Skip to content

fix(pg): alert_history retention never ran; resolve state lost on restart - #30

Draft
cursor[bot] wants to merge 1 commit into
masterfrom
cursor/critical-bug-management-65e4
Draft

fix(pg): alert_history retention never ran; resolve state lost on restart#30
cursor[bot] wants to merge 1 commit into
masterfrom
cursor/critical-bug-management-65e4

Conversation

@cursor

@cursor cursor Bot commented Sep 3, 2026

Copy link
Copy Markdown

Bug and impact

  1. Retention never deleted alert_history rowscleanupAlertHistory issued DELETE … WHERE ts / created_at, but bootstrap only creates fired_at. Errors were discarded, so Retention.AlertHistoryDays had no effect and the table grew without bound on PG deployments.
  2. Resolved alerts reappear as firing after restartresolveAlertRecord updated only the resolved_at column; loadRecentAlerts rebuilt memory from JSONB (status=firing, resolved_at=0). /api/v1/alerts/history?status=firing showed ghosts; MTTR/history reporting was wrong. Resolve also matched PG BIGSERIAL id against the in-memory alertSeq, which diverges after a failed insert.

Root cause

Stale cleanup SQL against the wrong columns; resolve/load asymmetry between the relational column and JSONB payload; async insert + id-based update.

Fix

  • Delete by fired_at < cut and log failures.
  • Resolve by key and update both resolved_at and JSONB status/resolved_at.
  • Load overlays the relational resolved_at onto JSONB records.
  • Persist alert history insert/resolve synchronously after releasing the store lock so resolve cannot race a missing row.
  • Drop orphan test refs (licenseResetForTest, classicAppJS) so ./cmd/server tests compile after license/Vue v2 removal.

Validation

  • go test -count=1 -run 'TestCleanupAlertHistoryUsesFiredAt|TestResolveAlertRecord|TestLoadRecentAlerts|TestStoreAlertHistory|TestMetricsEndpointAuthAndPayload|TestSweepSkipsInflight' ./cmd/server/
  • go build ./cmd/server ./cmd/agent
Open in Web View Automation 

cleanupAlertHistory deleted by nonexistent ts/created_at columns so
retention never ran. resolveAlertRecord updated only resolved_at while
loadRecentAlerts restored JSONB as still firing after restart.

Co-authored-by: Ethan <sreyun@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant