feat(bin): wake-outcome ledger for measured coordinator attention cost

[sbracewell64]

Intent

Add a wake-outcome ledger so firstmate's coordinator attention cost becomes measurable rather than estimated, and so the dormant model-promotion system can activate.

Why: the captain's scheduling policy (config/crew-dispatch.json _scheduling.telemetry.must_answer) requires telemetry answering why work was queued or promoted, which threshold fired, wait times, and whether thresholds are tuned right - none of which was collectable. The dormant model-promotion design (data/model-onboarding-policy/report.md sections 10.1/10.3/10.6) names this ledger plus a terminal per-task outcome line as its single named first instrument, and requires that it use no second evidence store and that a later task extend this file format rather than define a competing one. The captain's instrumentation doctrine also governs: the required instrumentation is whatever makes the quantity verifiable after deployment, and a signal used to detect change must be stable under no change.

This work followed a mandatory design review: the design was written to data/wake-outcome-ledger/design.md, reviewed and APPROVED by firstmate before any implementation, with three decisions explicitly ruled. Those three are deliberate and must not be flagged as mistakes:

1. The ledger lives at data/wake-ledger.tsv, NOT under state/, deviating from the referenced report's suggestion. Ruled because teardown deletes state/.* for every finished task and this evidence must outlive the tasks it describes; AGENTS.md section 2 defines data/ as durable and state/ as volatile.
2. TWO records per wake (a deterministic wake record written by the drain, plus a coordinator-written outcome record joined on the wake-queue sequence) rather than the report's single combined line. Ruled because one coordinator-written line would make measured attention cost fall whenever the coordinator skipped the recording step, so the metric would move without the underlying quantity moving. The split gives a denominator that is stable under no change and turns missing coverage into a reported number instead of a silent undercount. The extra join is the accepted cost.
3. Watcher-absorbed wakes are deliberately EXCLUDED. They never reach the coordinator, so they are not part of the quantity being measured, and recording them would mean touching seven call sites in the watcher's hot loop for a number no captain policy asks for. state/.watch-triage.log remains their disposable debug record.

Other deliberate choices a reviewer reading only the diff would not know:

• The ledger must never block, delay, alter, or fail a wake. Its only drain call site sits below the drain's authoritative print-and-delete boundary, with stdout discarded and failure ignored; the ledger takes no lock and writes one printf-appended line capped at 1024 characters, under the 4096-byte PIPE_BUF atomic-append bound. Two tests pin exactly this: an unwritable ledger leaves the drain's raw rows and exit status untouched, and a deliberately delayed ledger phase never blocks a concurrent wake append.
• FM_WAKE_LEDGER_TEST_DELAY is an intentional test-only latency seam, deliberately mirroring the existing FM_WAKE_ENRICH_TEST_DELAY seam in fm-wake-lib.sh rather than inventing a new pattern.
• Teardown writes the terminal record immediately before deleting the task metadata, because that is the last moment the task's harness, model, and effort are recoverable anywhere. That call is best effort on purpose: it warns on stderr but can never fail a teardown, since a telemetry write must never stand between the fleet and cleanup. A test pins that too.
• The wake count per task is deliberately NOT stored on the terminal record; it is computed from that task's wake records, per the report's no-duplication requirement.
• fm-wake-ledger.sh sources fm-pr-lib.sh solely to reuse fm_task_id_path_safe rather than restate that predicate, honoring the one-owner rule in the firstmate-coding-guidelines skill.
• The escalated field intentionally reads unknown until dispatch records it; the field exists and reads from task metadata so it fills in later with no format change. This limitation is stated in the approved design rather than hidden.
• No prune or rotation command in v1: growth is roughly 1-3 MB per year, and a prune that races an append can lose evidence.
• AGENTS.md deliberately gains only two lines (a layout entry and one wake-handling sentence) with all mechanics kept in the script header and --help, per the firstmate-coding-guidelines size-discipline and one-owner rules. No new skill was added because there is no situation-specific procedure to load. docs/supervision-protocols/* were deliberately left untouched because the recording step is harness-independent, so duplicating it into six harness files would violate the one-owner rule.

Verification already done locally: the new tests/fm-wake-ledger.test.sh passes all 10 cases; tests/fm-teardown.test.sh passes 35 cases including 3 new ledger cases; fm-backend old-vs-new teardown conformance, teardown-endpoint-safety, gotmp, test-run and lint suites all pass; bin/fm-lint.sh is clean at pinned ShellCheck 0.11.0; bin/fm-doc-audience-check.sh is clean; and the test coverage guard accepts the new test. Three watcher suites (fm-pi-watch-extension, fm-turnend-guard, fm-watcher-lock) fail or hang in this worktree, but that is a known pre-existing fleet defect: a Stop hook keeps re-arming a watcher from the worktree, and those suites are sensitive to an externally armed watcher. I proved this by reverting my drain change and re-running: they fail byte-identically at baseline, so they are unrelated to this change.

What Changed

• New bin/fm-wake-ledger.sh maintains a durable append-only ledger at data/wake-ledger.tsv (deliberately outside volatile state/) with two records per wake: a deterministic wake record appended by fm-wake-drain.sh below its print-and-delete boundary, and a coordinator-written outcome record joined on (seq, queued) so distinct wakes stay distinct across state resets — the join key was widened from bare seq after the pipeline review flagged silent collapse after a sequence reset. The ledger enforces a closed outcome vocabulary, sanitizes fields, caps lines under the PIPE_BUF atomic-append bound, and can never block, alter, or fail a wake.
• bin/fm-teardown.sh now writes a best-effort terminal task record (landed/abandoned, with --force mapping to abandoned) immediately before deleting task metadata — the last moment harness/model/effort are recoverable; an unwritable ledger warns on stderr but never fails teardown. A report command surfaces coverage, latency percentiles, outcome mix, and per-profile joins, turning missed coordinator recordings into a reported number instead of a silent undercount.
• Added tests/fm-wake-ledger.test.sh (11 cases, including unwritable-ledger isolation, delayed-ledger non-blocking, concurrent-append integrity, and reused-seq joins) plus 3 new teardown ledger cases; docs updated in AGENTS.md, docs/architecture.md, docs/configuration.md, and docs/scripts.md. Watcher-absorbed wakes are deliberately excluded — they never reach the coordinator.

Risk Assessment

✅ Low: The round-2 commit implements the requested (seq, queued) join fix exactly as instructed — write side, report dedup, coverage join, unknown-never-dedups rule, docs, and a targeted seq-reuse regression test — without touching the drain path or any captain-ruled decision, leaving no open findings.

Testing

Ran the two targeted suites (tests/fm-wake-ledger.test.sh, tests/fm-teardown.test.sh) — all 47 cases pass including the never-block and best-effort-teardown guarantees — then demonstrated the full workflow end-to-end with the real production scripts, capturing a CLI transcript showing wake records written by the drain, coordinator outcome records joined on (seq, queued), the terminal task record, the durable TSV at data/wake-ledger.tsv, and a report that answers the scheduling policy's telemetry questions (wake volume, coverage with missing outcomes reported not hidden, latency, outcome mix, per-profile model join). No findings.

Evidence: End-to-end ledger workflow transcript (enqueue → drain → outcomes → terminal record → report)

===== 2b. the drain wrote one deterministic wake record per deduped row ===== v1 wake 1785379594 seq=2 kind=signal key=task-demo1.status task=task-demo1 queued=1785379592 latency=2 v1 wake 1785379594 seq=3 kind=heartbeat key=heartbeat task=- queued=1785379592 latency=2 v1 wake 1785379594 seq=4 kind=check key=task-demo2.check.sh task=task-demo2 queued=1785379594 latency=0 ===== 3. out-of-vocabulary token ===== error: unknown outcome token: shrugged (absorbed inspected steered decided escalated repaired false-positive) -> refused, exit 2 ===== 6. fm-wake-ledger.sh report ===== wakes: 3 total, 2 with a recorded outcome, 1 unrecorded (67% coverage) outcomes: by token: absorbed 1 decided 1 tasks: 1 terminal (landed 1) per profile: pi/openai-codex/gpt-5.6-sol/medium tasks 1 wakes/task 1.0 coordinator response latency (queued -> drained), seconds: min 0 median 2 p90 2 max 2

===== 1. Watcher enqueues three wakes (production fm_wake_append) =====
  queued: 1785379592	1	signal	task-demo1.status	signal: task-demo1 needs a decision
  queued: 1785379592	2	signal	task-demo1.status	signal: task-demo1 duplicate (dedupes)
  queued: 1785379592	3	heartbeat	heartbeat	heartbeat
  queued: 1785379594	4	check	task-demo2.check.sh	check: PR poll for task-demo2

===== 2. Coordinator wake: fm-wake-drain.sh drains the queue (rows the coordinator reads) =====
1785379592	2	signal	task-demo1.status	signal: task-demo1 duplicate (dedupes)
1785379592	3	heartbeat	heartbeat	heartbeat
1785379594	4	check	task-demo2.check.sh	check: PR poll for task-demo2

===== 2b. ...and the drain wrote one deterministic wake record per deduped row =====
v1	wake	1785379594	seq=2	kind=signal	key=task-demo1.status	task=task-demo1	queued=1785379592	latency=2
v1	wake	1785379594	seq=3	kind=heartbeat	key=heartbeat	task=-	queued=1785379592	latency=2
v1	wake	1785379594	seq=4	kind=check	key=task-demo2.check.sh	task=task-demo2	queued=1785379594	latency=0

===== 3. Coordinator records outcomes per AGENTS.md (one wake deliberately left uncovered) =====
  drained seqs: 2 3 4
  (seq 4 intentionally not recorded - coverage must report it, not hide it)
  a token outside the closed vocabulary is refused:
error: unknown outcome token: shrugged (absorbed inspected steered decided escalated repaired false-positive)
  -> refused, exit 2

===== 4. Teardown writes the terminal task record (same call bin/fm-teardown.sh makes) =====

===== 5. The ledger file itself (durable evidence, data/wake-ledger.tsv) =====
v1	wake	1785379594	seq=2	kind=signal	key=task-demo1.status	task=task-demo1	queued=1785379592	latency=2
v1	wake	1785379594	seq=3	kind=heartbeat	key=heartbeat	task=-	queued=1785379592	latency=2
v1	wake	1785379594	seq=4	kind=check	key=task-demo2.check.sh	task=task-demo2	queued=1785379594	latency=0
v1	outcome	1785379594	seq=2	queued=1785379592	outcome=decided	task=task-demo1	after=0	note=answered task-demo1 merge gate
v1	outcome	1785379594	seq=3	queued=1785379592	outcome=absorbed	task=-	after=0
v1	task	1785379594	task=task-demo1	harness=pi	model=openai-codex/gpt-5.6-sol	effort=medium	mode=local-only	kind=crewmate	project=demo-proj	backend=tmux	outcome=landed	route=direct	escalated=unknown	findings=unknown	pr=https://github.com/demo/repo/pull/42

===== 6. fm-wake-ledger.sh report - the telemetry the captain's policy asks for =====
wake ledger: /tmp/fm-ledger-e2e.ipd3IO/data/wake-ledger.tsv
window: all records

wakes: 3 total, 2 with a recorded outcome, 1 unrecorded (67% coverage)
  by kind:  signal 1  check 1  heartbeat 1

outcomes: 2 recorded (1 not attributable to a task)
  by token:  absorbed 1  decided 1

tasks: 1 terminal (landed 1)

per profile (harness/model/effort):
  pi/openai-codex/gpt-5.6-sol/medium           tasks 1    wakes/task 1.0    steered 0    repaired 0    escalated 0    false-positive 0

busiest tasks by wake count:
  task-demo1                           1
  task-demo2                           1

coordinator response latency (queued -> drained), seconds: min 0  median 2  p90 2  max 2

Evidence: Reproducible demo script

#!/usr/bin/env bash
# End-to-end demonstration of the wake-outcome ledger, driving the real
# production scripts exactly as the fleet does:
#   1. watcher enqueues wakes (production fm_wake_append)
#   2. fm-wake-drain.sh hands them to the coordinator AND writes wake records
#   3. the coordinator records outcomes per AGENTS.md (fm-wake-ledger.sh outcome)
#   4. teardown-style terminal task record (fm-wake-ledger.sh task)
#   5. fm-wake-ledger.sh report answers the scheduling-telemetry questions
set -u
ROOT=$1
HOME_DIR=$(mktemp -d "${TMPDIR:-/tmp}/fm-ledger-e2e.XXXXXX")
mkdir -p "$HOME_DIR/state" "$HOME_DIR/data"
touch "$HOME_DIR/state/.last-watcher-beat"
export FM_ROOT_OVERRIDE="$HOME_DIR" FM_STATE_OVERRIDE="$HOME_DIR/state" FM_DATA_OVERRIDE="$HOME_DIR/data"
LEDGER_FILE="$HOME_DIR/data/wake-ledger.tsv"

# A task's metadata, so signal keys attribute to a real task id.
cat > "$HOME_DIR/state/task-demo1.meta" <<'META'
window=fm-task-demo1
harness=pi
model=openai-codex/gpt-5.6-sol
effort=medium
META

step() { printf '\n===== %s =====\n' "$1"; }

step "1. Watcher enqueues three wakes (production fm_wake_append)"
enqueue() {
  FM_STATE_OVERRIDE="$HOME_DIR/state" bash -c '
    . "$1"; fm_wake_append "$2" "$3" "$4"' _ "$ROOT/bin/fm-wake-lib.sh" "$2" "$3" "$4"
}
enqueue s signal  "task-demo1.status" "signal: task-demo1 needs a decision"
enqueue s signal  "task-demo1.status" "signal: task-demo1 duplicate (dedupes)"
enqueue s heartbeat heartbeat heartbeat
sleep 2   # real wait so latency is a measured, nonzero number
enqueue s check "task-demo2.check.sh" "check: PR poll for task-demo2"
sed 's/^/  queued: /' "$HOME_DIR/state/.wake-queue"

step "2. Coordinator wake: fm-wake-drain.sh drains the queue (rows the coordinator reads)"
"$ROOT/bin/fm-wake-drain.sh" 2>/dev/null
step "2b. ...and the drain wrote one deterministic wake record per deduped row"
grep -P '\twake\t' "$LEDGER_FILE"

step "3. Coordinator records outcomes per AGENTS.md (one wake deliberately left uncovered)"
seqs=$(grep -oP 'seq=\K[0-9]+' "$LEDGER_FILE" | sort -n)
set -- $seqs
echo "  drained seqs: $*"
"$ROOT/bin/fm-wake-ledger.sh" outcome decided "$1" --note "answered task-demo1 merge gate"
"$ROOT/bin/fm-wake-ledger.sh" outcome absorbed "$2"
echo "  (seq $3 intentionally not recorded - coverage must report it, not hide it)"
echo "  a token outside the closed vocabulary is refused:"
"$ROOT/bin/fm-wake-ledger.sh" outcome shrugged "$1" 2>&1 && echo UNEXPECTED-ACCEPT || echo "  -> refused, exit $?"

step "4. Teardown writes the terminal task record (same call bin/fm-teardown.sh makes)"
"$ROOT/bin/fm-wake-ledger.sh" task task-demo1 --outcome landed \
  --harness pi --model openai-codex/gpt-5.6-sol --effort medium \
  --mode local-only --kind crewmate --project demo-proj --backend tmux \
  --route direct --escalated unknown --pr https://github.com/demo/repo/pull/42

step "5. The ledger file itself (durable evidence, data/wake-ledger.tsv)"
cat "$LEDGER_FILE"

step "6. fm-wake-ledger.sh report - the telemetry the captain's policy asks for"
"$ROOT/bin/fm-wake-ledger.sh" report

rm -rf "$HOME_DIR"

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅

• ⚠️ bin/fm-wake-ledger.sh:482 - The durable ledger's identity/join key is the wake-queue sequence from state/.wake-queue.seq, but state/ is volatile while data/wake-ledger.tsv is durable evidence spanning weeks. After a state wipe or home rebuild, sequences restart and distinct wakes in the durable file share a seq. cmd_report dedups wake records by seq alone (wake_seen, line 482) and outcome records by seq alone (outcome_seen, line 493), so an all-records report silently collapses distinct wakes/outcomes across a seq reset — undercounting the very denominator the design requires to be stable under no change. The live outcome join is unaffected (the bounded tail read takes the newest matching wake), and wake records already carry queued= which could disambiguate report dedup (seq SUBSEP queued); outcome records carry no second key, so a complete fix would touch the approved seq-as-join-key ruling — hence ask-user rather than auto-fix.
• ℹ️ bin/fm-teardown.sh:1237 - The terminal outcome vocabulary accepts landed|failed|abandoned, but the only caller (fm-teardown.sh) maps --force to abandoned and everything else to landed, so 'failed' is currently unreachable vocabulary with no writer. This is coherent (teardown refuses un-landed work without --force), but worth confirming that reserving 'failed' for a future caller is intended rather than a missed mapping.

🔧 Fix: join wake outcomes on (seq, queued) across state resets
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-wake-ledger.test.sh — all 11 cases pass (record format, closed vocabulary, sanitization, task attribution, drain integration, unwritable-ledger never changes drain rows/exit, delayed ledger never blocks a concurrent wake append, concurrent-append integrity, report counts/coverage, reused-seq (seq,queued) join)
• bash tests/fm-teardown.test.sh — all 36 cases pass, including the 3 new ledger cases: terminal record with the meta's harness/model/effort written before metadata deletion, --force records outcome=abandoned, unwritable ledger warns on stderr but never fails or halts teardown
• Manual end-to-end demo (/tmp/no-mistakes-evidence/01KYRDQ3AZH2SPJ0X4CEQ869N5/e2e-demo.sh): enqueued wakes via production fm_wake_lib, drained with the real bin/fm-wake-drain.sh, recorded coordinator outcomes per the new AGENTS.md instruction, wrote the teardown-style terminal task record, verified an out-of-vocabulary token is refused with exit 2, and ran fm-wake-ledger.sh report showing coverage (67% with one deliberately unrecorded wake), latency percentiles, outcome mix, and the per-profile join
• git status --porcelain — worktree left clean, no transient test artifacts

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.