Metrics: SQLite-persisted history by default, runtime-switchable backend (ADR-0013) - #28
Merged
Merged
Conversation
…e backend (ADR-0013) Metrics history no longer requires an external InfluxDB stack. Three backends now sit behind one runtime router: sqlite (new default) persists windowed history into watchtower.db next to the live ring, memory keeps the old live-only behaviour, and influxdb stays as the bring-your-own reader for existing observability stacks. Storage is two-tier and deliberately not a raw-sample TSDB: 1-minute averages of the 10s sampler ticks kept ~72h, 10-minute rollups kept Metrics:RetentionDays (default 30, clamped 1-365). Rollup and windowed deletion ride the sampler loop; historical queries bucket either tier into the requested step in SQL. Offline containers and unavailable host reads persist as gaps, never as zeros. The backend is switchable at runtime from Settings -> Metrics: the new metrics.getConfig/updateConfig handlers persist Watchtower:Metrics:* through the settings store (which layers over env config and re-binds IOptionsMonitor live), the router resolves the backend per call, and the sampler re-checks it per tick - idling entirely under influxdb so exactly one collector runs. A misconfigured influx setup now degrades to unavailable (influx-misconfigured) instead of faulting DI, and the reader logs a warning when rows carry memory but no container.cpu.utilization - the collector-version drift that silently renders every CPU graph as 0. The metrics-history client flag follows the switch (evaluated per session fetch); the Settings page reloads when a save flips it so the boot-frozen contribution registry rebuilds (upstream fix filed as Elarion#137). ADR-0013 supersedes ADR-0007's "no SQLite persistence" and "restart required" decisions; docs and stale comments swept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Metrics history is now on by default with zero dependencies, and the metrics backend is switchable at runtime from Settings → Metrics — no restart, no compose edits.
sqlite(new default)watchtower.db(default 30 days, configurable 1–365)memory(opt-in)influxdb(opt-in, BYO)Why
Two lessons from operating the ADR-0007 split (in-memory default, Influx opt-in):
docker_statsrenamed its CPU metric across collector releases; a bucket filled by the "wrong" version renders every CPU graph as a silent, plausible flat 0 while memory works (the incident that started this branch).ADR-0013 records the reversal of ADR-0007's "no SQLite persistence" rejection: that rejection assumed persisting the raw 10s firehose. This stores 1-minute averages kept ~72h + 10-minute rollups kept the retention window — a few hundred thousand small rows / tens of MB at default settings, trivially inside SQLite's comfort zone.
How
MetricsPersistenceService: accumulates sampler ticks in memory, flushes one row per series per minute; rollup + both deletion windows ride the sampler loop (no new job infra). Offline containers and unreadable host/procpersist as gaps, not zeros.SqliteMetricsSource: live window from the ring; historical ranges bucketed in SQL from whichever tier covers the range. Container identity in history is the container name (stable across recreation, same convention as the Influx reader).IMetricsSourceis nowMetricsSourceRouter, resolving the backend fromIOptionsMonitorper call.metrics.updateConfigpersistsWatchtower:Metrics:*through the settings store (the same live-rebind mechanism as the automation toggles). The sampler re-checks the backend per tick and idles entirely underinfluxdb, preserving the single-collector invariant behaviourally. The Influx reader is built lazily from the current options, rebuilt on change, and degrades toinflux-misconfiguredinstead of faulting DI.metrics-historyflag is evaluated per session fetch and follows the switch; a save that flips it reloads the page to rebuild the boot-frozen contribution registry (upstream reactive-capabilities enhancement filed as Elarion #137).container.cpu.utilizationvalues (the collector-version drift signature), and the docs gained a pinned-collector warning box with the explicitmetrics:enable snippet.Reviewer notes
Metrics:Backendstart persisting history on upgrade (deliberate; compose comments + README document it,memoryrestores the old behaviour). MigrationAddMetricsHistoryadds the two sample tables.MaxRollupBucketsPerPass) — a large backlog converges over successive hourly passes; the tests exercise this with two passes.metrics.updateConfigvalidates backend/retention/Influx connection before writing, so a bad save can't strand the dashboard; a null token keeps the stored one.Testing
memory→metrics-historyflag false on the next session fetch, nav gone, direct URL shows the "Enable in Settings" banner; switch back → restored. No restarts.