fix: broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race - #603
Merged
Conversation
cache_dir() swept every *.uffs.tmp on every call while background compact-cache saves spend seconds writing such temps; std's FILE_SHARE_DELETE let the sweep delete the live temp, so the writer's atomic-write rename failed with os error 2 (the intermittent 'Background compact cache save failed' on all seven drives, and why drive S's cache was missing at the v0.6.35 start). Only temps at least an hour old are removed now, via a host-tested is_abandoned_temp helper; files with unreadable or future mtimes are conservatively kept.
Drive C's cache failed 'bloom k_hashes out of range' on every load from Aug 12 to Aug 22 — a corruption-class failure is permanent, but the loader left the file in place so every restart re-failed on the same bytes. LoadCacheError::is_corruption() classifies decrypt / decompress / parse / deserialise failures as permanent, and load_compact_cache now moves such a file aside as <name>.corrupt (one forensic sample kept, older sample replaced) so the next load reports Missing, the caller rebuilds, and the rebuild's save writes a fresh cache. Missing / stale / IO / key-unavailable are untouched.
Windows keeps a file object's completion-port association in the file object, one-shot for its lifetime. The broker registry's original volume handle lives as long as the daemon and every read adopts a DuplicateHandle copy, so the first IOCP read claims the association and every later one fails 0x80070057 at associate time; both existing fallback rungs open new handles, which needs exactly the elevation the broker exists to avoid — a drive whose cache was also unusable stayed Cold forever (drive C, 2026-08-22). New fallback rung 0 reads MFT chunks with OVERLAPPED-offset reads (read_handle_at) on the handle the reader already holds — no new opens, no IOCP, works on broker duplicates and plain handles alike. Chunk/FRS planning is the new pure MftExtentMap::read_plan, host- tested across contiguous, fragmented, sparse-hole, clamped, and sub-record-chunk layouts; parsing reuses the $MFT-file rung's parse_chunk_records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three production defects found while root-causing the 2026-08-22 status
anomalies on the 7-drive reference box (
LOG/Output+uffsd.log): drive Cstuck
Cold("encrypted cache only") across restarts under broker mode, andintermittent background cache-save failures. All three fixed end to end, each
with host-testable units and regression tests.
1. mft: broker-mode full MFT reads dead-ended after the first IOCP read
Windows stores a file object's completion-port association in the file object
(
CompletionContext), not the handle — and it is one-shot. The broker registrykeeps its original volume handle open for the daemon's lifetime and every read
adopts a
DuplicateHandlecopy, so all reads share one file object: the firstIOCP read after startup claims the association, and every later one fails at
associatetime withERROR_INVALID_PARAMETER(0x80070057). Both existingfallback rungs open new handles (
X:\$MFT, unbuffered volume), which needsexactly the elevation the broker exists to avoid:
Fix: a new fallback rung 0 —
OVERLAPPED-offset chunk reads on the volumehandle the reader already holds (
read_handle_at, per-read event, no IOCP,no new opens; works on broker duplicates and plain handles alike). Chunk/FRS
planning is a new pure
MftExtentMap::read_plan(compiled and unit-tested onevery platform: contiguous, fragmented, sparse-hole, clamped, sub-record-chunk
cases). Parsing reuses the
$MFT-file rung'sparse_chunk_records.2. core: corrupt compact caches were never healed
Drive C's cache failed
compact cache deserialise failed: bloom k_hashes out of rangeon every load from Aug 12 to Aug 22 — 39 occurrences. Acorruption-class failure is permanent (the same bytes can never load), but the
loader left the file in place, so every restart/promote re-failed identically;
combined with (1), C had no working path at all under broker mode.
Fix:
LoadCacheError::is_corruption()classifiesDecryptFailed/DecompressFailed/ParseError/Deserialize as permanent, and
load_compact_cachenow quarantines such a file as<name>.corrupt(keepingone forensic sample, replacing an older one) so the next load reports
Missing→ the caller rebuilds → the rebuild's background save writes a freshcache. Missing/stale/IO/KeyUnavailable are untouched. Tests cover the
classification split, the rename, sample replacement, the missing-file no-op,
and that the inner loader itself never touches the file.
3. mft:
.uffs.tmpsweep deleted in-flight background savescache_dir()swept every*.uffs.tmpon every call. Background compactsaves write multi-hundred-MB temps for seconds while journal loops call
cache_dir()concurrently; std opens files withFILE_SHARE_DELETE, so thesweep's delete succeeded (delete-pending), the writer's handle close made the
temp vanish, and the atomic-write rename failed with os error 2 — the
intermittent
Background compact cache save failedon every drive (S 5×,C/D/M 4×, F 3×). This is also why S's cache was missing at the v0.6.35 start
(45 s full rebuild) and why C's corrupt cache was never overwritten by the
rebuild saves that did succeed in reading.
Fix: the sweep is age-gated — only temps ≥ 1 h old (far beyond any live
write) are removed, via a pure
is_abandoned_temp(modified, now)helper withunit tests (fresh/boundary/old/future-mtime cases). Docs on
cache_dirupdatedto say what actually happens.
How the three compose
Corrupt cache (2) → rebuild required → broker mode can't read MFT (1) → stuck
Coldforever; and even elevated rebuilds couldn't heal the cache because there-save kept being killed by the temp sweep (3). Fixing any one helps; fixing
all three makes the loop converge: quarantine → offset-read rebuild → race-free
save.
Validation
just go(fmt, prod+test lints, nextest, audit gates) — green on macOSjust check-windows(cargo xwin cross-check incl. tests) — green