Refactor telemetry acquisition and add dynamic climate module support

2026-05-26 10:22:57 +01:00
parent 6fefa2542e
commit 0a33f42502
10 changed files with 784 additions and 10 deletions
@@ -0,0 +1,538 @@
+# LitoralRegas Backend
+
+Spring Boot backend for the LitoralRegas agricultural monitoring and control platform.
+
+This backend communicates with agricultural controllers through Modbus, builds dynamic acquisition plans based on installed modules, collects live telemetry, stores historical data, and exposes APIs consumed by the frontend dashboard.
+
+---
+
+# Features
+
+* Modbus TCP communication
+* Dynamic controller capability discovery
+* Sensor definition import system
+* Live telemetry acquisition
+* Telemetry cache layer
+* Historical telemetry storage
+* Climate module API
+* Meteorology module API
+* Irrigation module foundation
+* Historical chart aggregation endpoints
+* SQLite persistence
+
+---
+
+# Technology Stack
+
+* Java 21
+* Spring Boot
+* Spring Web
+* Spring Data JPA
+* SQLite
+* Maven
+* Modbus TCP
+
+---
+
+# Architecture Overview
+
+```txt
+Sensor Definitions
+        ↓
+Controller Capabilities
+        ↓
+Acquisition Plan Builder
+        ↓
+Telemetry Acquisition Scheduler
+        ↓
+Telemetry Cache
+        ↓
+Historian + Module APIs
+        ↓
+Frontend Dashboard
+```
+
+---
+
+# Core Concepts
+
+## Sensor Definitions
+
+The `sensor_definition` table contains the full catalog of known sensors.
+
+A controller installation may not use every sensor present in the catalog.
+
+The acquisition plan decides which sensors are actually polled.
+
+---
+
+## Controller Capabilities
+
+Controller capabilities are read directly from Modbus registers.
+
+Current capability model:
+
+```txt
+Register 6 → irrigationControllerCount
+Register 7 → fertilizerChannelCount
+Register 8 → feature flags
+Register 9 → climateGreenhouseCount
+```
+
+Feature flags:
+
+```txt
+bit 0 → climateEnabled
+bit 1 → irrigationEnabled
+bit 2 → lightingEnabled
+```
+
+---
+
+## Acquisition Plan
+
+The acquisition plan dynamically selects sensors based on:
+
+* Installed controller modules
+* Greenhouse count
+* Irrigation controller count
+* Sensor category
+* Modbus address ranges
+
+This prevents polling sensors that do not exist in a specific installation.
+
+---
+
+## Telemetry Cache
+
+The telemetry cache stores the latest acquired value for each sensor.
+
+Module APIs read from the cache instead of directly querying Modbus.
+
+---
+
+## Historian
+
+The historian stores telemetry over time.
+
+It supports:
+
+* historical chart series
+* accumulated values
+* time range queries
+* future workspace/chart persistence
+
+---
+
+# Project Structure
+
+```txt
+src/main/java/com/litoralregas/backend
+├── acquisition
+├── historian
+├── modbus
+├── modules
+│   ├── climate
+│   ├── irrigation
+│   ├── meteo
+│   └── shared
+├── sensor
+├── telemetry
+└── config
+```
+
+---
+
+# Running the Backend
+
+## Requirements
+
+* Java 21+
+* Maven
+* SQLite
+* Reachable controller or Modbus simulator
+
+---
+
+## Start Backend
+
+```bash
+mvn spring-boot:run
+```
+
+Default backend URL:
+
+```txt
+http://localhost:18450
+```
+
+---
+
+# Configuration
+
+Main configuration file:
+
+```txt
+src/main/resources/application.yaml
+```
+
+Example acquisition scheduler configuration:
+
+```yaml
+litoralregas:
+  acquisition:
+    scheduler:
+      enabled: true
+      fixed-delay-millis: 10000
+```
+
+Important:
+
+YAML comments must be on a separate line.
+
+Correct:
+
+```yaml
+fixed-delay-millis: 10000
+# Longer delay between acquisition cycles
+```
+
+Wrong:
+
+```yaml
+fixed-delay-millis: 10000 // comment
+```
+
+---
+
+# Database
+
+Current database engine:
+
+```txt
+SQLite
+```
+
+Main table:
+
+```sql
+CREATE TABLE sensor_definition (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    name VARCHAR(255) NOT NULL,
+    modbus_address INTEGER NOT NULL,
+    bit_offset INTEGER,
+    value_type VARCHAR(50) NOT NULL,
+    unit VARCHAR(50),
+    decimal_places INTEGER NOT NULL DEFAULT 0,
+    category VARCHAR(100) NOT NULL,
+    source_type VARCHAR(50) NOT NULL,
+    polling_interval_seconds INTEGER NOT NULL DEFAULT 1,
+    enabled BOOLEAN NOT NULL DEFAULT TRUE,
+    created_at TIMESTAMP NOT NULL
+);
+```
+
+---
+
+# Sensor Definition Import
+
+Sensor definitions are imported from:
+
+```txt
+src/main/resources/config/sensor-map.txt
+```
+
+Run import manually:
+
+```bash
+curl -X POST http://localhost:18450/api/sensor-definition-import/run
+```
+
+The importer:
+
+* imports missing sensors
+* updates safe metadata
+* skips duplicates safely
+* supports clean reimports during development
+
+---
+
+# Clean Development Reimport
+
+To completely reset sensor definitions during development:
+
+```sql
+DELETE FROM sensor_definition;
+
+DELETE FROM sqlite_sequence
+WHERE name = 'sensor_definition';
+```
+
+Then re-run:
+
+```bash
+curl -X POST http://localhost:18450/api/sensor-definition-import/run
+```
+
+---
+
+# Main API Endpoints
+
+## Acquisition Plan
+
+```http
+GET /api/acquisition/plan
+```
+
+Returns:
+
+* module availability
+* greenhouse count
+* irrigation controller count
+* selected sensor IDs
+
+---
+
+## Latest Telemetry
+
+```http
+GET /api/telemetry/latest
+```
+
+Returns latest cached telemetry values.
+
+---
+
+## Meteo Module
+
+```http
+GET /api/modules/meteo
+```
+
+Provides:
+
+* exterior temperature
+* exterior humidity
+* wind speed
+* wind direction
+* radiation
+* rain sensors
+* CO2 overview
+
+---
+
+## Climate Module
+
+```http
+GET /api/modules/climate
+```
+
+Provides:
+
+* greenhouse temperature
+* greenhouse humidity
+* CO2
+* ventilation
+* extractors
+* screens
+* windows
+* lighting sectors
+* soil humidity
+* soil temperature
+
+---
+
+## Historical Accumulation
+
+```http
+GET /api/historian/accumulated
+```
+
+Example:
+
+```http
+GET /api/historian/accumulated?key=meteo.chuva.1&range=30d
+```
+
+Supported ranges:
+
+```txt
+7d
+30d
+month
+year
+```
+
+---
+
+# Dynamic Module Strategy
+
+Module availability is determined dynamically through:
+
+```txt
+ControllerCapabilities
+        ↓
+AcquisitionPlan
+        ↓
+TelemetryCache
+```
+
+This means:
+
+* climate sensors are only acquired if climate exists
+* irrigation sensors are only acquired if irrigation exists
+* lighting sensors are only acquired if lighting exists
+
+The frontend should eventually use acquisition plan data to decide which sections to render.
+
+---
+
+# Derived Sensors
+
+Some sensors are virtual/computed values.
+
+Examples:
+
+```txt
+DPV Estufa 1        → -121
+Hum. Absoluta 1     → -141
+```
+
+These are NOT real Modbus registers.
+
+Correct pipeline:
+
+```txt
+Raw temperature + humidity
+        ↓
+DerivedClimateService
+        ↓
+DPV / Absolute Humidity
+        ↓
+TelemetryCache + Historian
+```
+
+Negative Modbus addresses should never be polled directly.
+
+---
+
+# Current Development Notes
+
+Disconnected sensors may return unrealistic values.
+
+Examples:
+
+* invalid temperature values
+* unrealistic humidity values
+* disconnected soil probes
+
+This is expected in partially installed environments.
+
+The backend currently exposes all acquired sensors.
+
+The frontend chart builder will later allow users to choose only relevant variables.
+
+---
+
+# Planned Improvements
+
+## Chart Variables API
+
+Planned endpoint:
+
+```http
+GET /api/charts/variables?module=climate
+```
+
+Expected response:
+
+```json
+{
+  "sensorId": 13,
+  "name": "Temperatura estufa 1",
+  "key": "temperatura.estufa.1",
+  "historianKey": "climate.temperatura.estufa.1",
+  "module": "climate",
+  "unit": "C",
+  "category": "CLIMATE"
+}
+```
+
+---
+
+## Derived Climate Values
+
+Future derived telemetry:
+
+* DPV
+* absolute humidity
+* dew point
+* climate alarms
+* sensor health
+
+---
+
+## Workspace System
+
+Planned chart workspace support:
+
+* save layouts
+* detachable charts
+* multi-monitor support
+* reusable chart presets
+* draggable variables
+
+---
+
+# Useful Development Commands
+
+## Import Sensors
+
+```bash
+curl -X POST http://localhost:18450/api/sensor-definition-import/run
+```
+
+## Check Acquisition Plan
+
+```bash
+curl http://localhost:18450/api/acquisition/plan
+```
+
+## Check Meteo Module
+
+```bash
+curl http://localhost:18450/api/modules/meteo
+```
+
+## Check Climate Module
+
+```bash
+curl http://localhost:18450/api/modules/climate
+```
+
+## Check Latest Telemetry
+
+```bash
+curl http://localhost:18450/api/telemetry/latest
+```
+
+---
+
+# Current Status
+
+The backend foundation is currently stable for:
+
+* sensor catalog import
+* Modbus acquisition
+* capability-based acquisition planning
+* telemetry cache
+* meteorology module
+* climate module
+* historical accumulation
+* frontend integration
+
+Next major milestone:
+
+```txt
+Chart Variables API + Derived Climate Telemetry
+```
@@ -77,21 +77,55 @@ public class AcquisitionPlanBuilder {
    ) {
        Integer address = sensor.getModbusAddress();

-        if (address == null || address < 0) {
+        if (address == null) {
            return false;
        }

+        // Exterior/meteo climate block
        if (address >= 10 && address <= 22) {
            return true;
        }

-        if (address < 100 || address > 899) {
-            return false;
+        // Main greenhouse climate blocks:
+        // E1: 100-126
+        // E2: 140-166
+        // E3: 180-206
+        // ...
+        // stride = 40
+        if (address >= 100 && address <= 899) {
+            int greenhouseNumber = ((address - 100) / 40) + 1;
+            int offsetInBlock = (address - 100) % 40;
+
+            return greenhouseNumber >= 1
+                    && greenhouseNumber <= greenhouseCount
+                    && offsetInBlock >= 0
+                    && offsetInBlock <= 26;
        }

-        int greenhouseNumber = ((address - 100) / 40) + 1;
+        // Soil sensors:
+        // Humidade solo 1-36: 3200-3235
+        // Temperatura solo 1-36: 3236-3271
+        if (address >= 3200 && address <= 3271) {
+            return true;
+        }

-        return greenhouseNumber >= 1 && greenhouseNumber <= greenhouseCount;
+        // Computed climate values:
+        // DPV Estufa 1-20: -121 to -140
+        // Hum. Absoluta Estufa 1-20: -141 to -160
+        if (address >= -160 && address <= -121) {
+            int absoluteIndex = Math.abs(address);
+
+            int greenhouseNumber;
+            if (absoluteIndex >= 121 && absoluteIndex <= 140) {
+                greenhouseNumber = absoluteIndex - 120;
+            } else {
+                greenhouseNumber = absoluteIndex - 140;
+            }
+
+            return greenhouseNumber >= 1 && greenhouseNumber <= greenhouseCount;
+        }
+
+        return false;
    }

    private boolean belongsToEnabledIrrigationRange(
@@ -0,0 +1,21 @@
+package com.litoralregas.backend.modules.climate;
+
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+@RestController
+@RequestMapping("/api/modules/climate")
+public class ClimateModuleController {
+
+    private final ClimateModuleService climateModuleService;
+
+    public ClimateModuleController(ClimateModuleService climateModuleService) {
+        this.climateModuleService = climateModuleService;
+    }
+
+    @GetMapping
+    public ClimateModuleResponse getLatest() {
+        return climateModuleService.getLatest();
+    }
+}
@@ -0,0 +1,10 @@
+package com.litoralregas.backend.modules.climate;
+
+import com.litoralregas.backend.modules.shared.ModuleSensorResponse;
+import java.time.Instant;
+import java.util.List;
+
+public record ClimateModuleResponse(
+        Instant timestamp,
+        List<ModuleSensorResponse> sensors
+) {}
@@ -0,0 +1,118 @@
+package com.litoralregas.backend.modules.climate;
+
+import com.litoralregas.backend.acquisition.telemetry.TelemetryCache;
+import com.litoralregas.backend.acquisition.telemetry.TelemetrySnapshot;
+import com.litoralregas.backend.modules.shared.ModuleSensorResponse;
+import org.springframework.stereotype.Service;
+
+import java.time.Instant;
+import java.util.Comparator;
+import java.util.List;
+
+@Service
+public class ClimateModuleService {
+
+    private final TelemetryCache telemetryCache;
+
+    public ClimateModuleService(TelemetryCache telemetryCache) {
+        this.telemetryCache = telemetryCache;
+    }
+
+    public ClimateModuleResponse getLatest() {
+        List<ModuleSensorResponse> sensors = telemetryCache.getAll()
+                .stream()
+                .filter(this::isClimateSensor)
+                .sorted(Comparator.comparing(TelemetrySnapshot::sensorId))
+                .map(this::toResponse)
+                .toList();
+
+        return new ClimateModuleResponse(
+                Instant.now(),
+                sensors
+        );
+    }
+
+    private boolean isClimateSensor(TelemetrySnapshot snapshot) {
+        String name = normalize(snapshot.name());
+
+        boolean isIrrigationOrHydro =
+                name.contains(" ce")
+                        || name.contains("ph")
+                        || name.contains("bomba")
+                        || name.contains("rega")
+                        || name.contains("tanque")
+                        || name.contains("hidro")
+                        || name.contains("dren")
+                        || name.contains("bancada")
+                        || name.contains("pressao");
+
+        if (isIrrigationOrHydro) {
+            return false;
+        }
+
+        return name.contains("greenhouse")
+                || name.contains("exterior")
+                || name.contains("interior")
+                || name.contains("clima")
+                || name.contains("estufa")
+                || name.contains("zenital")
+                || name.contains("lateral")
+                || name.contains("topo")
+                || name.contains("ecra")
+                || name.contains("dpv")
+                || name.contains("absoluta")
+                || name.startsWith("il ")
+                || name.contains(" il ")
+                || name.contains("ventilacao")
+                || name.contains("ventilador")
+                || name.contains("extrator")
+                || name.contains("janela")
+                || name.contains("iluminacao")
+                || name.contains("luz")
+                || name.contains("sombra")
+                || name.contains("cortina")
+                || name.contains("co2")
+                || name.contains("humidade solo")
+                || name.contains("temperatura solo")
+                || name.contains("temperatura do solo")
+                || name.contains("humidade do solo");
+    }
+
+    private ModuleSensorResponse toResponse(TelemetrySnapshot snapshot) {
+        return new ModuleSensorResponse(
+                snapshot.sensorId(),
+                snapshot.name(),
+                buildKey(snapshot.name()),
+                snapshot.value(),
+                snapshot.unit(),
+                snapshot.modbusAddress(),
+                snapshot.bitOffset(),
+                snapshot.timestamp()
+        );
+    }
+
+    private String buildKey(String name) {
+        return normalize(name)
+                .replaceAll("[^a-z0-9]+", ".")
+                .replaceAll("^\\.|\\.$", "");
+    }
+
+    private String normalize(String value) {
+        if (value == null) {
+            return "";
+        }
+
+        return value
+                .toLowerCase()
+                .replace("ç", "c")
+                .replace("ã", "a")
+                .replace("á", "a")
+                .replace("à", "a")
+                .replace("é", "e")
+                .replace("ê", "e")
+                .replace("í", "i")
+                .replace("ó", "o")
+                .replace("õ", "o")
+                .replace("ú", "u");
+    }
+}
@@ -0,0 +1,4 @@
+package com.litoralregas.backend.modules.climate.websocket;
+
+public class ClimateModuleWebSocketPublisher {
+}
@@ -1,8 +1,11 @@
 package com.litoralregas.backend.sensor;

 import org.springframework.data.jpa.repository.JpaRepository;
+import org.springframework.data.jpa.repository.Query;
+import org.springframework.data.repository.query.Param;

 import java.util.List;
+import java.util.Optional;

 public interface SensorDefinitionRepository extends JpaRepository<SensorDefinition, Integer> {

@@ -11,4 +14,21 @@ public interface SensorDefinitionRepository extends JpaRepository<SensorDefiniti
    List<SensorDefinition> findByEnabledTrueOrderByNameAsc();

    List<SensorDefinition> findAllByOrderByNameAsc();
+
+    @Query("""
+        select s
+        from SensorDefinition s
+        where s.modbusAddress = :modbusAddress
+          and (
+              (:bitOffset is null and s.bitOffset is null)
+              or s.bitOffset = :bitOffset
+          )
+        order by s.id asc
+        """)
+    List<SensorDefinition> findAllByHardwareAddress(
+            @Param("modbusAddress") Integer modbusAddress,
+            @Param("bitOffset") Integer bitOffset
+    );
+
+    Optional<SensorDefinition> findByName(String name);
 }
@@ -43,9 +43,29 @@ public class SensorDefinitionImportService {
            }

            SensorDefinitionImportRow row = parser.parseLine(line)
-                    .orElseThrow(() -> new IllegalArgumentException("Invalid empty sensor row."));
+                    .orElseThrow(() -> new IllegalArgumentException(
+                            "Invalid sensor row: " + line
+                    ));

-            if (repository.existsByName(row.name())) {
+            List<SensorDefinition> existingSensors = repository.findAllByHardwareAddress(
+                    row.modbusAddress(),
+                    row.bitOffset()
+            );
+
+            if (!existingSensors.isEmpty()) {
+                for (SensorDefinition sensorDefinition : existingSensors) {
+                    sensorDefinition.setValueType(row.valueType());
+                    sensorDefinition.setUnit(row.unit());
+                    sensorDefinition.setDecimalPlaces(row.decimalPlaces());
+                    sensorDefinition.setCategory(row.category());
+                    sensorDefinition.setSourceType(row.sourceType());
+                }
+
+                skippedExisting += existingSensors.size();
+                continue;
+            }
+
+            if (repository.findByName(row.name()).isPresent()) {
                skippedExisting++;
                continue;
            }
@@ -33,7 +33,7 @@ litoralregas:
  acquisition:
    scheduler:
      enabled: true
-      fixed-delay-millis: 3000
+      fixed-delay-millis: 10000 # change here for longer wait between cycles

  weather:
    enabled: true
@@ -41,6 +41,15 @@ litoralregas:
    longitude: -8.7375
    location-name: Mira

+  modules:
+    climate:
+      enabled: true
+      exterior-enabled: true
+      enabled-sites:
+        - 1
+    irrigation:
+      enabled: true
+
 weather:
  api-key: 0aa355536b6c469eb4b82226262505
  base-url: https://api.weatherapi.com/v1
@@ -81,8 +81,8 @@ Humidade 5 estufa 4*231*0*%*c
 CO2 estufa 4*232*0*ppm*c
 Temperatura do solo Estufa 4*233*1*C*c
 Humidade do solo Estufa 4*234*0*%*c
-Ventiladores Estufa 5*235,0*0*SU*c
-Extratores Estufa 5*235,1*0*SU*c
+Ventiladores Estufa 4*235,0*0*SU*c
+Extratores Estufa 4*235,1*0*SU*c

 Temperatura estufa 5*260*1*C*c
 Humidade estufa 5*261*0*%*c