manifest: cold checkpoint + hot append-only journal#16
Merged
Conversation
First half of removing the per-record full manifest rewrite. An entry carries the resulting state of the chain and file a record touched, so replay is "apply in order, last wins" and applying one twice changes nothing. That is not tidiness. Compaction writes the checkpoint and only then drops the journal, so a crash between the two replays lines onto a newer checkpoint. A delta-shaped entry would double-count there; a stated result cannot.
…d corrupt One append + fsync per line, so a partial line can only ever be the last one — a crash mid-write. That one is dropped. A malformed line anywhere else raises, because silently skipping it would drop an attestation, and dropping evidence quietly is the failure this library exists to prevent. _fsync_fd moves here from sinks/local_file.py so both writers share one definition rather than a copy.
The bump is required here, and the pubkeys change deliberately avoided one — the cases are asymmetric. pubkeys was additive: an old reader ignoring it still read correct heads. A v2 checkpoint's heads lag by up to a compaction interval, so an old reader would take them as authoritative and report MANIFEST_INTEGRITY — a FALSE integrity break on honest evidence. Refusing to load is the honest failure; a wrong verdict is not. So from_dict accepts v1.0 and v2.0 and only v2.0 is written. A v1 manifest has no journal and its checkpoint is the state, as written.
…nifest The sink rewrote and F_FULLFSYNCed the entire manifest on every record, and chains is never pruned. Measured under per-chain-per-run: ~90 KB/day against ~0.5 KB/day, ~179x faster growth, extrapolating to ~22.5 MB rewritten ~1800x a day after 250 busy days. Same fsync count, O(1) bytes. The manifest is an attestation — verify reports MANIFEST_INTEGRITY when it disagrees with the log — so durability is untouched; only the write size changes. Construction-time declare_pubkey still writes the checkpoint. It is not a per-record event. verify.py's manifest loader read manifest.json raw, so once the sink stopped rewriting it per record, every verify_tree cross-check saw a stale (empty) checkpoint instead of current chain/file state. It now replays manifest.journal on top, same as Manifest.load_or_create. Test fixtures that hand-edit manifest.json to simulate a lying manifest now collapse checkpoint+journal first, since a live journal restates a chain/file wholesale and would silently override the hand-edit otherwise.
Checkpoint first, fsync, then drop the journal. A crash between the two leaves both, and replay folds stale entries onto a newer checkpoint — a no-op, because entries state results rather than deltas. The reverse order would lose every attestation the journal still held. The line counter is seeded from the journal on construction: a fresh process (the Claude Code hook path is one per tool call) must not restart the count at zero and let the journal grow unbounded.
…id otherwise The module docstring claimed the manifest is "NOT load-bearing for chain integrity" and exists only so the hook handler can recover the chain head. verify disagrees in the same codebase: a manifest that contradicts the log is MANIFEST_INTEGRITY, "an integrity break, never a pass", and files[].sha256 is the anchor that catches a byte change. Both could not be true, and which one is decided this whole design: an attestation cannot have its durability batched away. v0.1-era prose the code outgrew, in the same shape as the NON-CLAIMS block and __version__. The equivalence test is the real deliverable: verify_tree returns identical verdicts against a journal-backed directory and a compacted one.
apply_journal_entry replayed JournalEntry.redaction_disabled (a bool, True only for DISABLED) through note_redaction_disabled, which latches: from UNKNOWN, note_redaction_disabled(False) reaches ENABLED. A directory where nobody ever attested a redaction state read as ENABLED after journal replay — the exact affirmative lie RedactionState exists to prevent, and a break from the full-rewrite path this branch replaced, which kept UNKNOWN. JournalEntry now carries redaction_state as the RedactionState value string itself. apply_journal_entry folds it in by severity (UNKNOWN < ENABLED < DISABLED), keeping whichever of the current and replayed state is more severe, instead of assigning or routing through the bool-taking latch. That gives both last-wins replay and reorder-safety: an older, less-severe entry replayed after a newer one can no longer regress the state. Also corrects the manifest module docstring's now-false claim that the sink updates manifest.json atomically on every write — this branch made the per-record path a journal append with manifest.json as a periodic checkpoint instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sink rewrote and
F_FULLFSYNCed the entiremanifest.jsonon every record, andmanifest.chainsis never pruned. Cheap at one chain per day; not cheap under per-chain-per-run, where bosun measured it: ~90 KB/day vs ~0.5 KB/day, ~179× faster growth, extrapolating to ~22.5 MB rewritten ~1800×/day after 250 busy days. A host cannot fix it — the rewrite is inside the sink, and pruningchainswould lie about which chains exist.Two findings reframed the fix
The manifest is an ATTESTATION, not a cache. The codebase said both.
sinks/local_file.pycalled it "NOT load-bearing for chain integrity";verify.pytreats disagreement with the log asMANIFEST_INTEGRITY— "an integrity break, never a pass" — andfiles[].sha256is "the robust anchor". The verifier is right; the docstring was v0.1-era prose the code outgrew. So this change reduces bytes written, never fsync count, never the guarantee.The head is not the only hot field.
files[<today>].sha256mutates per record exactly like the chain head. Moving only the head out would buy nothing. That is why the shape is a journal, not a per-chain head file: head and file-sha are keyed differently, so a sidecar-per-key needs two fsyncs per record and opens a head/sha inconsistency window — a falseMANIFEST_INTEGRITYunder attestation semantics. One journal line carries both, published together, one fsync.Design
manifest.json— cold checkpoint. Rewritten only on compaction and construction-time key declaration.manifest.journal— one append-only JSON line per record carrying the resulting state of the chain and file it touched. Resulting state, not deltas, so replay is "apply in order, last wins" — idempotent.close(): write the checkpoint, fsync, then drop the journal. A crash between is a no-op on replay — the reason entries state results.verify.pynow loads checkpoint + journal too, so the sink and verifier never disagree.Compatibility
Schema bumps
manifest.v1.0→manifest.v2.0, and here the bump is required (thepubkeyschange deliberately avoided one — the cases are asymmetric). A v2 checkpoint's heads lag by up to a compaction interval, so an old reader would take them as authoritative and report a falseMANIFEST_INTEGRITYon honest evidence. Refusing to load is the honest failure. Sofrom_dictaccepts v1.0 and v2.0 (a v1 manifest has no journal and its heads are authoritative as written), only v2.0 is written, and an unknown version still raises.Measured
The claim the item made, re-measured: 2000 records across 200 chains →
manifest.jsonwritten 4 times, not 2000.Reviewed
Six commits, each TDD with a task review; a whole-branch review on top. The load-bearing test is an equivalence property:
verify_treereturns identical verdicts on a journal-backed directory and a compacted one — it passed before any Task-6 change, confirming Tasks 1–5 preserved it. The final review caught one real gap — the journal first encoded the redaction tri-state as a bool, so replaying it latchedUNKNOWN → ENABLED, asserting "redaction enabled" where nothing was attested (the exact affirmative lie the tri-state exists to prevent). Fixed by carrying the tri-state and folding it monotonically; reorder-safety and the equivalence case are both pinned by non-vacuous tests.Deliberately not here
Cross-process append safety (deferred with PR #13); the pre-existing crash window between the JSONL append and the manifest write (neither fixed nor worsened — filed separately); signing the manifest; pruning
chains.560 tests green,
mypy --strictandruffclean.