Skip to content

fix: broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race - #603

Merged
githubrobbi merged 3 commits into
mainfrom
fix/broker-read-cache-heal
Aug 22, 2026
Merged

fix: broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race#603
githubrobbi merged 3 commits into
mainfrom
fix/broker-read-cache-heal

Conversation

@githubrobbi

Copy link
Copy Markdown
Collaborator

Three production defects found while root-causing the 2026-08-22 status
anomalies on the 7-drive reference box (LOG/Output + uffsd.log): drive C
stuck Cold ("encrypted cache only") across restarts under broker mode, and
intermittent background cache-save failures. All three fixed end to end, each
with host-testable units and regression tests.

1. mft: broker-mode full MFT reads dead-ended after the first IOCP read

Windows stores a file object's completion-port association in the file object
(CompletionContext), not the handle — and it is one-shot. The broker registry
keeps its original volume handle open for the daemon's lifetime and every read
adopts a DuplicateHandle copy, so all reads share one file object: the first
IOCP read after startup claims the association, and every later one fails at
associate time with ERROR_INVALID_PARAMETER (0x80070057). Both existing
fallback rungs open new handles (X:\$MFT, unbuffered volume), which needs
exactly the elevation the broker exists to avoid:

13:53:31 IOCP inline read failed … 0x80070057                       volume=C
13:53:31 Fallback 1 ($MFT file) failed … Access is denied (os 5)    volume=C
13:53:31 WARN USN-refresh failed; falling back to bare compact-cache load

Fix: a new fallback rung 0 — OVERLAPPED-offset chunk reads on the volume
handle the reader already holds (read_handle_at, per-read event, no IOCP,
no new opens; works on broker duplicates and plain handles alike). Chunk/FRS
planning is a new pure MftExtentMap::read_plan (compiled and unit-tested on
every platform: contiguous, fragmented, sparse-hole, clamped, sub-record-chunk
cases). Parsing reuses the $MFT-file rung's parse_chunk_records.

2. core: corrupt compact caches were never healed

Drive C's cache failed compact cache deserialise failed: bloom k_hashes out of range on every load from Aug 12 to Aug 22 — 39 occurrences. A
corruption-class failure is permanent (the same bytes can never load), but the
loader left the file in place, so every restart/promote re-failed identically;
combined with (1), C had no working path at all under broker mode.

Fix: LoadCacheError::is_corruption() classifies
DecryptFailed/DecompressFailed/ParseError/Deserialize as permanent, and
load_compact_cache now quarantines such a file as <name>.corrupt (keeping
one forensic sample, replacing an older one) so the next load reports
Missing → the caller rebuilds → the rebuild's background save writes a fresh
cache. Missing/stale/IO/KeyUnavailable are untouched. Tests cover the
classification split, the rename, sample replacement, the missing-file no-op,
and that the inner loader itself never touches the file.

3. mft: .uffs.tmp sweep deleted in-flight background saves

cache_dir() swept every *.uffs.tmp on every call. Background compact
saves write multi-hundred-MB temps for seconds while journal loops call
cache_dir() concurrently; std opens files with FILE_SHARE_DELETE, so the
sweep's delete succeeded (delete-pending), the writer's handle close made the
temp vanish, and the atomic-write rename failed with os error 2 — the
intermittent Background compact cache save failed on every drive (S 5×,
C/D/M 4×, F 3×). This is also why S's cache was missing at the v0.6.35 start
(45 s full rebuild) and why C's corrupt cache was never overwritten by the
rebuild saves that did succeed in reading.

Fix: the sweep is age-gated — only temps ≥ 1 h old (far beyond any live
write) are removed, via a pure is_abandoned_temp(modified, now) helper with
unit tests (fresh/boundary/old/future-mtime cases). Docs on cache_dir updated
to say what actually happens.

How the three compose

Corrupt cache (2) → rebuild required → broker mode can't read MFT (1) → stuck
Cold forever; and even elevated rebuilds couldn't heal the cache because the
re-save kept being killed by the temp sweep (3). Fixing any one helps; fixing
all three makes the loop converge: quarantine → offset-read rebuild → race-free
save.

Validation

  • just go (fmt, prod+test lints, nextest, audit gates) — green on macOS
  • just check-windows (cargo xwin cross-check incl. tests) — green
  • New unit tests: 6 read-plan, 5 quarantine/classification, 5 temp-age

cache_dir() swept every *.uffs.tmp on every call while background
compact-cache saves spend seconds writing such temps; std's
FILE_SHARE_DELETE let the sweep delete the live temp, so the writer's
atomic-write rename failed with os error 2 (the intermittent
'Background compact cache save failed' on all seven drives, and why
drive S's cache was missing at the v0.6.35 start). Only temps at least
an hour old are removed now, via a host-tested is_abandoned_temp
helper; files with unreadable or future mtimes are conservatively
kept.
Drive C's cache failed 'bloom k_hashes out of range' on every load
from Aug 12 to Aug 22 — a corruption-class failure is permanent, but
the loader left the file in place so every restart re-failed on the
same bytes. LoadCacheError::is_corruption() classifies decrypt /
decompress / parse / deserialise failures as permanent, and
load_compact_cache now moves such a file aside as <name>.corrupt
(one forensic sample kept, older sample replaced) so the next load
reports Missing, the caller rebuilds, and the rebuild's save writes a
fresh cache. Missing / stale / IO / key-unavailable are untouched.
Windows keeps a file object's completion-port association in the file
object, one-shot for its lifetime. The broker registry's original
volume handle lives as long as the daemon and every read adopts a
DuplicateHandle copy, so the first IOCP read claims the association
and every later one fails 0x80070057 at associate time; both existing
fallback rungs open new handles, which needs exactly the elevation the
broker exists to avoid — a drive whose cache was also unusable stayed
Cold forever (drive C, 2026-08-22).

New fallback rung 0 reads MFT chunks with OVERLAPPED-offset reads
(read_handle_at) on the handle the reader already holds — no new
opens, no IOCP, works on broker duplicates and plain handles alike.
Chunk/FRS planning is the new pure MftExtentMap::read_plan, host-
tested across contiguous, fragmented, sparse-hole, clamped, and
sub-record-chunk layouts; parsing reuses the $MFT-file rung's
parse_chunk_records.
@githubrobbi githubrobbi changed the title Fix the 2026-08-22 daemon incidents: broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race fix(mft,core): broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race Aug 22, 2026
@githubrobbi githubrobbi changed the title fix(mft,core): broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race fix: broker IOCP dead-end, corrupt-cache poisoning, temp-sweep race Aug 22, 2026
@githubrobbi
githubrobbi added this pull request to the merge queue Aug 22, 2026
Merged via the queue into main with commit 19a9ac5 Aug 22, 2026
21 of 24 checks passed
@githubrobbi
githubrobbi deleted the fix/broker-read-cache-heal branch August 22, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant