Skip to content

feat(alertd): add a blob correction rate check - #819

Closed
dannash100 wants to merge 1 commit into
mainfrom
feat/blob-correction-rate-check
Closed

feat(alertd): add a blob correction rate check#819
dannash100 wants to merge 1 commit into
mainfrom
feat/blob-correction-rate-check

Conversation

@dannash100

Copy link
Copy Markdown
Contributor

Tamanu's blob store can now repair a corrupt blob from parity stored beside it, bumping blobs.correction_count and stamping last_corrected_at. Nothing is lost when that happens, which is what makes it a different signal from blob_integrity: a rising repair rate means the disk under the store is failing, so the response is to plan replacing the media rather than to recover content. Nothing surfaced that.

Adds blob_correction_rate to the doctor registry, built from the Correction rate query in Tamanu's cookbook. Details carry the 24h/7d/all-time blob counts alongside the repair total, so an operator can read spread against repeat count.

Thresholds count distinct blobs, never repairs: repeated repair of one blob is a single bad region, the same count across separate blobs is the substrate going. WARN at 3 distinct blobs in 7 days, FAIL at 10 in 24h, or at 5 in 24h when that is over three times the week's daily average. One or two in a week passes, which is parity doing its job. alertd has no cross-sweep state, so "rising across scrub passes" comes from the two windows the query already carries, and the flat 24h threshold keeps a store that has plateaued at a high rate failing once the 7 day baseline catches up to it.

Two worth a look:

  • Runs on facility as well as central, unlike the central-only DB checks around it. Both hold durable blobs and the failing media is the server's own.
  • A missing blobs table or missing parity columns skips rather than reporting BROKEN, following sync_snapshot_tables. Error correction is version-gated and off by default, so the default path would break on every pre-R2 deployment.

Draft: blocked on Tamanu R2, which adds the columns.

@dannash100

Copy link
Copy Markdown
Contributor Author

🤖 Superseded by #820, which carries this commit alongside the blob_integrity and blob_antivirus checks. Landing the three together avoids the conflict in checks.rs and keeps the runbook set complete.

@dannash100 dannash100 closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant