Skip to content

fix(bin): give data/secondmates.md exactly one parsing owner - #1404

Open
JaredHuynhning wants to merge 7 commits into
kunchenguid:mainfrom
JaredHuynhning:fm/fm-registry-parser-owner
Open

fix(bin): give data/secondmates.md exactly one parsing owner#1404
JaredHuynhning wants to merge 7 commits into
kunchenguid:mainfrom
JaredHuynhning:fm/fm-registry-parser-owner

Conversation

@JaredHuynhning

Copy link
Copy Markdown

What and why

data/secondmates.md had five readers, each with its own regex for the structured (home: ...) field. Commit d939c49 (#686) fixed one of them because a registry summary can carry a parenthetical before that field, but the fix landed only in bin/fm-backlog-handoff.sh. The anchored ^[^(]*(home: ...) form survived in three siblings and bin/fm-update.sh carried a third set of rules.

This was live. A registered secondmate whose summary contained a parenthetical read as home-less, so spawn refused to launch it, and teardown deleted a parent home whose registered nested secondmate home it could no longer see.

bin/fm-registry-lib.sh is now the single owner of row selection and field extraction. The rule is written exactly once, as one awk program driving both the per-line helpers and a bulk single-pass stream. Every reader routes through it: home seeding, spawn, teardown, backlog handoff, update, the fast-forward library (and through it bootstrap), the public-followup parent binding, and the fleet snapshot.

Row selection is stated once: last-match-wins for by-id resolution with a literal id comparison, while whole-registry conflict scans deliberately visit every row, because "which row is authoritative" and "is any row in conflict" are different questions.

Why the owner is an awk program

The fleet snapshot parses a byte- and line-bounded window inside a 2 second timeout. A per-row helper loop was measured at 1.51s at the 256-row window cap, which would turn a healthy registry into a false "unreadable" snapshot. The single-pass stream does it in 0.024s. An earlier attempt kept a separate jq extractor there; independent review rejected it because it disagreed with the owner on two-fragment rows and on trailing whitespace. The snapshot's jq filter now only groups pre-parsed pairs and must never extract a field.

Defects found and fixed during validation

The pipeline caught four faults in this change that the original work had not:

  • Non-backtracking extraction. row_field committed to the last textual (home: and failed when that one was malformed, where the fix(bin): parse handoff homes after registry parentheticals #686 reference sed backtracks to the earlier valid field. Fixed to resolve the last valid occurrence.
  • Per-row lookup cost. fm_registry_row cost a process per row, making the update backstop O(ids x rows), measured at 1.17s for a 20-row registry. Now single-pass, measured at 0.003s.
  • registry_id_conflict_for_assignment turned fail-open. Narrowing it to a last-match-wins lookup silently disabled the guard, because validate_registry skips home-less rows before duplicate detection, so the invariant the code claimed did not hold. Reproduced end to end: with a home-less duplicate row present, seeding an already-registered id to a new home returned 0, fired no guard, and silently reassigned the id leaving the old home orphaned. Restored to an all-rows scan matching its sibling registry_home_conflict_for_assignment, with a regression test.
  • Silent deregistration failure. remove_secondmate_registry_entry returned success when it could not rewrite the registry, and it runs after the home has been deleted, so teardown would report success while leaving the registry naming a secondmate whose home is gone. Now fails loudly, with a regression test.

Row selection in teardown was also hardened on captain instruction: deregistration matched the whole id field exactly instead of interpolating the caller-supplied id into a regex (verified: deleting oss.triage previously removed an unrelated ossXtriage row), and the parent-binding selector now rejects indented rows the owner skips, keeping its exactly-one-match safety property.

Verification

  • Each broken reader was proven red independently against base f7d0d0a: home seeding, spawn, teardown and update each fail on their own. A single combined test was deliberately split, because a serial test fails at its first reader and never reaches the rest.
  • Backlog handoff is not a mutation proof and is not presented as one: it was the reference that already carried the correct parser. Its test asserts parity with the owner instead, and was verified to go red when the owner is mutated to disagree.
  • The snapshot assertion checks the selected home, not merely that a row was seen, and goes red on base for all three disagreements: first-row duplicate, preserved trailing whitespace, and first (home: fragment.
  • grep -rn '(home:' bin/ shows exactly one extraction site.

Full local suite: 102 scripts. Failures on that machine were each reproduced identically on base f7d0d0a (fm-backend-orca, fm-calm-pi-extension, fm-session-start, fm-busy-adapter-wiring, fm-pi-watch-extension, fm-turnend-guard, fm-backend-herdr-presentation-e2e), plus two load-sensitive timing flakes measured at 0/5 failures on this branch versus 1/5 on base under identical synthetic load.

Follow-ups, deliberately not patched here

The captain closed the fix loop after the round above, on the grounds that each successive round had introduced a new silently-passing path. These remain open and are recorded rather than fixed:

  • arch-teardown-lease-duplicate (docs/architecture.md:181, info, pre-existing): restates the teardown treehouse-lease fail-closed rule that .agents/skills/secondmate-provisioning/SKILL.md already owns, while the same section defers every other secondmate contract to that skill by pointer. Reduce it to a pointer. Not caused by this change.
  • fm-registry-ids-unused (bin/fm-registry-lib.sh, info): fm_registry_ids has no production caller since fm-update.sh uses fm_registry_id_homes. Either drop it with its contract test or give it a stated consumer.

Scope

Confined to registry parsing and row selection. No spawn or teardown lifecycle or cleanup ordering changed; fm-lifecycle-crash-safety owns that and will touch the same two files.

Note for maintainers: this is a fork PR, so CI needs maintainer approval before it will run.

Five readers of data/secondmates.md each carried their own regex for the
structured (home: ...) field, and three of them anchored on "no parenthesis
before the field". A registry summary or scope routinely carries its own
parenthetical, so those three read a registered secondmate as home-less:
spawn refused to launch it, and teardown removed a parent home whose
registered nested home it could no longer see. Commit d939c49 fixed one
reader; this makes the fix structural.

bin/fm-registry-lib.sh is now the single owner of row selection and field
extraction, and every shell reader sources it: home seeding, spawn, teardown,
update, bootstrap, backlog handoff, the fast-forward library, and the
public-followup fallback. Row selection is stated once and is last-match-wins
for by-id lookup, with a literal id comparison rather than a regex; whole
registry scans stay whole-registry scans.

The bounded jq reader in fm-fleet-snapshot.sh cannot source shell, because it
parses a byte- and line-bounded window inside a timed subshell. It now
mirrors the owner's rules with a comment saying so, and resolves a duplicated
id to the same row every other reader would.

Coverage: tests/fm-registry-lib.test.sh pins the library contract, and the
parenthetical fixture is extended from backlog handoff to home seeding, spawn,
teardown, update, the public-followup fallback, and the fleet snapshot. The
new cross-call-site tests and the strengthened teardown fixtures fail against
the pre-fix code.
…n root

tests/fm-gotmp.test.sh builds a synthetic bin/ holding exactly the siblings
fm-teardown.sh sources, so adding a new sourced library aborts teardown under
set -eu inside that fixture. Link bin/fm-registry-lib.sh in both fake roots,
the same way the other newly required siblings are already handled.
Independent review rejected the first attempt with four blocking findings.

B1. A second extractor survived in fm-fleet-snapshot.sh and disagreed with the
owner: given two "(home: ...)" fragments it took the first while the owner took
the last, and it preserved trailing whitespace the owner strips. That replaced
one wrong parser with two that disagree, which is harder to notice. The earlier
justification for keeping it - a bounded window under a hard timeout - was real
but the conclusion was not: measured, per-row helper calls cost 1.51s at the
256-row window cap against a 2s budget. The extraction rule is now written once
as a single awk program in fm-registry-lib.sh, driving both the per-line helpers
and a new single-pass stream; the snapshot parses that window in 0.024s and its
jq filter now only groups pre-parsed pairs. One extraction site remains in bin/.

B2. fm-update.sh walked rows and deduplicated by home, so it fast-forwarded a
superseded duplicate-id home. It now iterates registered ids and resolves each
through the owner, acting once at the authoritative last row.

B3. The bearings duplicate-id test could not detect the selection change it
existed to guard, passing identically before and after. Replaced with an
assertion on the selected home in the snapshot's registry records.

B4. The "every call site" test ran serially and proved only its first reader.
Split into independent per-call-site tests, plus executable coverage for the
ff-lib registry backfill, which had none.

Also fixes the reported follow-up: the advertised scope extractor rejected its
own fixture's "scope: triage (incl. security)". It now ends at the "; projects:"
delimiter and is asserted.

Red-proof against f7d0d0a, each in isolation: home-seed, spawn and teardown each
fail on their own; fm-update T8 fails; the snapshot record assertion fails on all
three disagreements; the public-followup parent binding fails. Backlog handoff
and the ff-lib backfill stay green against the old code because those readers
already carried the correct form - they are regression guards, not red proofs.
…ed proof

The recheck was right that backlog handoff cannot be mutation-proved against
f7d0d0a: that call site already carried the correct greedy parser at the base
commit. It was the file that HAD the kunchenguid#686 fix, which is how the four broken
siblings were found. Grouping it under "each call site proven red" overstated
the evidence.

Four of the five readers were broken - home seeding, spawn, teardown and
fm-update.sh - and each fails independently against the pre-fix code. Handoff
was the reference, and its test now says so and earns its place a different way:
it computes the expected home from fm-registry-lib.sh rather than hand-copying
it, and asserts the item lands in exactly the home the owner names.

That assertion can fail. Mutating only the owner's home rule back to the
anchored form, so the reference and the owner disagree, turns it red in
isolation; with the owner intact it passes. The reference and the owner can no
longer drift apart unnoticed, which is the failure this test exists to catch now
that it is not catching the original bug.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant