Aggregate metrics into per-minute rollups for fast, small stats

[dansimau]

Problem

• metrics table growing too large
• One raw row is stored per tick (tick_processing_time ≈ 11.9M rows/month), kept for 90 days.
• stats.go computed p99 by loading every matching row into Go and sorting it in memory (the 151s query in the report).

Fix

Store per-minute rollups and serve long-range queries from them, keeping raw points only for the high-resolution "last minute" view.

• store/histogram.go — a DDSketch-style log-scale histogram. Bucket boundaries are gamma^i, fixed for all time, so per-minute histograms sum exactly and a quantile over any window is a true quantile accurate to ~2% relative error (HistogramGamma = 1.0408). This is what makes the p99 correct rather than a proxy — MAX(per-minute p99) was rejected as it converges to "worst single minute," not a percentile.
• store/models.go — MetricRollup stores count, sum, histogram (per-minute, retained 90d, ~260k rows total).
• metrics/service.go — roll up completed minutes each minute; one-time Go backfill of historical raw (chunked by day, since histograms can't be built in SQL); prune raw to 24h and rollups to 90d, with raw pruning gated on backfill completion so history is never deleted before it's captured.
• cmd/hal/commands/stats.go — last minute from raw (exact); hour/day/month merge the window's rollup histograms and compute the p99 of the merged distribution.

Notes

• Reads now scale with rollup count, not raw count.
• First startup runs a one-time backfill over the existing ~7GB (backgrounded); a one-off VACUUM then reclaims the file.
• Accuracy is tunable via HistogramGamma.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: de75f7a090

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[dansimau]

Addressed the Codex findings in 254c1f7. Summary of how each is handled:

P1 — gap between backfill and incremental rollups: Replaced the separate backfill + fixed 5-minute lookback with a single watermark-based rollupPending: it rolls up from just after the latest existing rollup (or the earliest raw point) up to the current minute, in day chunks. The same mechanism does the initial backfill, fills gaps after downtime, and does steady-state rollups, so there's no window that escapes both paths. Added TestRollupPendingResumesFromWatermarkWithoutGaps.

P1 — backfill error permanently disabled raw pruning: Removed the one-shot backfillDone flag. A failed pass doesn't advance the watermark, so it's simply retried on the next tick. Raw pruning is now gated on the watermark reaching the raw cut-off (pruneOldData), so raw is never deleted before it's rolled up and pruning resumes automatically once caught up. Added TestPruneOldDataDefersRawUntilRolledUp.

P2 — stats report 0 before rollups exist: stats now falls back to raw when no rollups cover the window (rawTailStart returns the window start). Added TestSumMetricsFallsBackToRawWithoutRollups.

P2 — longer windows excluded the current minute: hour/day/month now combine rollups with the not-yet-rolled raw tail (everything after the latest rolled minute). The watermark guarantees the tail never overlaps a rollup, so no double-counting. Added TestCalculateP99IncludesRawTail.

Also added PRAGMA busy_timeout since the rollup loop now reads/writes concurrently with the async writer (and it helps the separate CLI processes too).

go build/go test ./.../go vet all pass.