statistics webapp for smartvos
  • Python 63.6%
  • JavaScript 27.1%
  • CSS 6%
  • HTML 3.3%
Find a file
2026-10-08 14:01:34 +02:00
smartstats Fix pan action by includeing hammer 2026-10-08 14:01:34 +02:00
tests Improve ux 2026-10-07 15:39:31 +02:00
tools inial commit 2026-10-04 13:10:20 +02:00
.gitignore inial commit 2026-10-04 13:10:20 +02:00
app.py inial commit 2026-10-04 13:10:20 +02:00
README.md Fix pan action by includeing hammer 2026-10-08 14:01:34 +02:00
requirements.txt inial commit 2026-10-04 13:10:20 +02:00

Uvos Smart Statistics

Logs sensor readings from a smartvos primary instance into SQLite and serves a web UI for plotting them as interactive time-series graphs.

smartvos primary ──TCP 38940──▶ collector thread ──▶ SQLite ──▶ Flask API ──▶ Chart.js UI

Run

pip install -r requirements.txt
python3 app.py

Then open http://localhost:8080. The collector starts in a background thread inside the same process, so one command is all that is needed. It connects to 10.0.0.1:38940 by default and creates data/sensorlog.db on first start.

Configuration

All settings come from environment variables.

Variable Default Meaning
SMARTVOS_HOST 10.0.0.1 smartvos primary address
SMARTVOS_PORT 38940 TCP port of the primary
SMARTVOS_TRANSPORT tcp tcp or ws — see the note below
SMARTVOS_WS_URL ws://<host>:<port+1>/ override the WebSocket URL
SMARTVOS_DB sqlite:///data/sensorlog.db SQLAlchemy database URL
SMARTVOS_RETENTION_DAYS 0 drop readings older than N days (0 = keep forever)
SMARTVOS_DEDUPE_SECONDS 2.0 suppress an identical value seen this soon after the last one
SMARTVOS_LOG_HIDDEN 1 also log sensors flagged Hidden
SMARTVOS_SOCKET_TIMEOUT 15.0 per-read socket timeout in seconds
SMARTVOS_POLL_INTERVAL 10.0 snapshot interval, only used by the ws transport
WEB_HOST / WEB_PORT 0.0.0.0 / 8080 where the UI listens
WEB_MAX_POINTS 1500 target points per series before SQL downsampling

Transport: TCP vs WebSocket

TCP is the default. Historically it was the only endpoint that pushed live data: in SHinterface/src/mainobject.cpp the sensor store was wired up as

connect(&globalSensors, &SensorStore::sensorChangedState, tcpServer, &TcpServer::sensorEvent);

with no equivalent for webServer. The WebSocket server answered GetSensors with a snapshot but never received change notifications — verified against a live instance at the time: over TCP the collector received continuous SensorUpdate pushes, over WebSocket exactly one message in 15 seconds, the reply to its own request.

That gap has since been closed upstream: SHinterface commit e9d92de (2026-10-06, "server: websocket: fix clients not getting messages about sensor updates") also connects sensorChangedState to webServer, so current builds push over WebSocket as well.

TCP remains the default because deployed primaries often predate that fix, and the polling fallback costs nothing. SMARTVOS_TRANSPORT=ws still re-requests the snapshot every SMARTVOS_POLL_INTERVAL seconds, which keeps older primaries working at polling resolution; against a fixed build the pushes arrive on their own and the extra polling is harmless (dedupe absorbs the duplicate values).

Wire protocol

Reference: SHinterface/src/service/tcpserver.cpp, tcpclient.cpp, service.cpp. Framing is symmetric:

MSG JSON LEN <n>\n
<n raw bytes of UTF-8 JSON>

Interleaved with JSON frames, the plain-text command MSG GETVERSION\n is answered with MSG VERSION <version>\n.

The collector sends {"MessageType":"GetSensors","Data":[]} on connect and then persists every SensorUpdate (and AddSensor) it receives. SensorStore only emits on an actual value or metadata change, so the write rate tracks real activity instead of a polling interval.

A sensor object looks like:

{ "Type": "Sensor", "SensorType": 12, "Id": -4009434731138787459,
  "Field": 42.36, "Name": "Server Power", "GroupName": "Ground Floor",
  "Unit": "W", "Hidden": false, "LastSeen": "Tue Oct 6 09:31:57 2026" }

Two details worth knowing:

  • Identity is (SensorType, Id), matching Sensor::operator==. Id is a signed 64-bit value — MQTT-backed sensors use hashed ids that are often negative, so the column is BigInteger, not unsigned.
  • LastSeen is not usable. It is QDateTime::toString(), which is locale-dependent and unparseable. Readings are timestamped on arrival in UTC instead.

Data model

Table Columns
sensors pk, sensor_type, ext_id, name, group_name, unit, hidden, first_seen, last_seen, last_value
readings id, sensor_pk → sensors, ts (UTC), value

sensors has a unique constraint on (sensor_type, ext_id); readings has a composite index on (sensor_pk, ts) because every chart query is one sensor over one time window. SQLite runs in WAL mode so the collector can write while the web app reads.

HTTP API

Endpoint Purpose
GET / the plotting UI
GET /healthz liveness probe
GET /api/sensors all sensors with their latest value, plus the group list
GET /api/series?sensor=1&sensor=2&range=24h&agg=avg series data as {x: epoch_ms, y: value}
GET /api/status collector state, server version, reading counts

/api/series parameters:

  • sensor — repeatable, up to 128 ids
  • range — 1h, 6h, 24h, 7d, 30d, 90d, all; ignored if start is given
  • start / end — epoch millis or an ISO timestamp
  • agg — avg, min, max
  • max_points — cap per series, default WEB_MAX_POINTS

When a window holds more points than max_points, values are aggregated into equal-width buckets in SQL, so a 90-day query over dense data stays fast. Buckets align to the Unix epoch, which keeps the returned series sorted — required by the chart layer, which consumes {x,y} pairs without re-parsing.

UI notes

  • One chart per unit, so temperatures never share an axis with watts. Unit-less sensors are split by sensor type instead.
  • Line colours are assigned by position within a unit group, not by global sensor index, so two charts can never collide and neighbours in one chart get maximally different hues. The sidebar swatch matches the plotted line. More than 16 sensors in a single unit still repeat hues.
  • Doors, buttons and other binary types render as stepped lines.
  • Ctrl+scroll zooms — plain scroll keeps scrolling the page — and dragging pans the x axis; per-chart "Reset zoom" returns to the current time window. Legend clicks toggle a series.
  • Tooltips interpolate every series at the exact cursor time, so sensors with different reporting cadences all show a value instead of only the fastest one. Outside a series' own time range the value reads "–".
  • The sidebar groups sensors by room group or by sensor type (toggle next to the Clear button).
  • Auto-refresh updates charts in place: page scroll, the zoom/pan window and legend toggles survive it.
  • Selection, range, aggregation, grouping and refresh interval persist in localStorage.
  • Chart.js, its date adapter, the zoom plugin and hammerjs (required by the zoom plugin for drag-pan) are vendored under smartstats/static/vendor/, so the page needs no internet access.

Tests

python3 -m pytest tests/ -q

If collection crashes inside a plugin import, a stale entry point in the active environment is the cause; run with PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 to skip third-party plugin discovery.

tests/mock_server.py is a stand-in smartvos primary that speaks the real framing, so the end-to-end tests cover snapshot ingest, live pushes, reconnect with backoff, dedupe, hidden-sensor handling and the HTTP API without needing hardware.

tests/ui_logic_test.js exercises the browser-side helpers (unit grouping, value formatting, URL building) in Node with stubbed DOM:

PROJ=$PWD node tests/ui_logic_test.js

Visual check without a browser binary

tools/qt_shot.py renders the UI to PNG using Qt WebEngine (PySide6), which works where a standalone Chromium cannot. It drives the page with injected JS, prints what is on screen, and saves a screenshot:

pip install PySide6
python3 app.py &                     # serve first
QT_QPA_PLATFORM=offscreen python3 tools/qt_shot.py temps /tmp/shot.png

Scenarios: empty, all, temps, power, search. Override with BASE_URL, SHOT_W, SHOT_H.

Known limitations

  • Timestamps are recorded on arrival, not at the sensor. If the collector is disconnected, readings during the gap are lost — the primary does not backfill history.
  • Only sensors are logged. ItemUpdate, Message and MqttDevices frames are parsed and ignored.
  • The UI has no authentication; keep it on a trusted network.