feat(alertd): add a blob correction rate check - #819
Closed
dannash100 wants to merge 1 commit into
Closed
Conversation
Contributor
Author
|
🤖 Superseded by #820, which carries this commit alongside the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tamanu's blob store can now repair a corrupt blob from parity stored beside it, bumping
blobs.correction_countand stampinglast_corrected_at. Nothing is lost when that happens, which is what makes it a different signal fromblob_integrity: a rising repair rate means the disk under the store is failing, so the response is to plan replacing the media rather than to recover content. Nothing surfaced that.Adds
blob_correction_rateto the doctor registry, built from the Correction rate query in Tamanu's cookbook. Details carry the 24h/7d/all-time blob counts alongside the repair total, so an operator can read spread against repeat count.Thresholds count distinct blobs, never repairs: repeated repair of one blob is a single bad region, the same count across separate blobs is the substrate going. WARN at 3 distinct blobs in 7 days, FAIL at 10 in 24h, or at 5 in 24h when that is over three times the week's daily average. One or two in a week passes, which is parity doing its job. alertd has no cross-sweep state, so "rising across scrub passes" comes from the two windows the query already carries, and the flat 24h threshold keeps a store that has plateaued at a high rate failing once the 7 day baseline catches up to it.
Two worth a look:
blobstable or missing parity columns skips rather than reporting BROKEN, followingsync_snapshot_tables. Error correction is version-gated and off by default, so the default path would break on every pre-R2 deployment.Draft: blocked on Tamanu R2, which adds the columns.