Loop: driver one-shot contract needs enforcement + tests — background orchestration produced phantom-success drivers and wedged the queue

[robercano-ghbot]

Incident (2026-07-10, drivers #91/#92)

Both drivers spawned their orchestrator in the BACKGROUND and ended their turn. In headless -p mode that ends the session; the background orchestrator was killed after the grace ceiling. Result: half-born local feat/issue-N-* worktree branches (no commits, no push, no PR), result=exit rc=0 in the ledger (phantom success), census read the local branches as in_flight, skipped both issues, and advanced #109 out of the owner's intended order.

Hotfixed on main (d37e951): the driver prompt now mandates foreground-only agent spawns, forbids ending the turn before the PR/feedback exists on GitHub, and requires deleting local branch/worktree debris on failure.

Second incident class (2026-07-14 → recovered 2026-07-15, issue #107): a driver was killed mid-flight by a service restart AFTER committing approved, gates-green work but BEFORE pushing/opening the PR. The original "clean the debris" design would have DELETED that finished work. It was recovered manually instead (re-ran gates, pushed, opened PR #117) — proving the wedge state can contain real work and that publishing it is purely mechanical. The 2026-07-15 overnight phantom (#106 driver, 01:18Z: empty out-file, no ledger completion, 4.8h tick silence) added a third data point.

Design: post-exit verification + debris CLASSIFIER (not a deleter)

After every advance driver exits, the daemon (or loop-event) verifies the claimed outcome instead of trusting rc:

1. Ledger honesty: check GitHub post-exit; record pr=<num> (or marker-posted) in the ledger line. rc=0 with no PR → re-ledger as result=phantom.
2. Classify the debris (deterministic, pure shell — states align with Loop: failure classification + crash-resume for dead workers (the recovery half of loop-health) #98's classifier):
   • A. Empty — branch has no commits ahead of main, worktree clean → delete branch + worktree, let census restart the issue. (The original Worktree hygiene: auto-remove worker worktrees (and branches) after their PR merges #91/Cockpit: group Live worker progress by task, link issue/PR, and make columns sortable #92 case; nothing is lost.)
   • B. Publishable — commits ahead of main, worktree clean, events.jsonl shows gates/reviews done → mechanical publish, no model needed: re-run the test gate (main may have moved), push, open the PR via bot-gh.sh. Exactly the manual Loop daemon: harden runtime-env resolution — bake resolved PATH into installed units, don't depend on nvm sourcing at runtime #107 → PR Loop daemon: bake resolved node/claude PATH into pr-loop.service unit #117 recovery, automated.
   • C. Half-done — commits ahead but work unfinished, or dirty worktree → NEVER delete. If dirty, preserve with a wip: commit; mark the issue resumable and hand off to Loop: failure classification + crash-resume for dead workers (the recovery half of loop-health) #98's continuation driver (fresh driver pointed at the existing branch with the recorded phase, or claude --resume <session-id> from the ledger). Bounded attempts, then needs-human.
3. Tests: loop-event.test.sh asserts the prompt contains the one-shot contract clauses; loop-daemon.test.sh scenarios for phantom-success (rc=0, no PR → classified + ledgered), case A (debris cleaned), case B (publish path invoked), case C (nothing deleted, resumable recorded).
4. Fail fast: consider CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 for driver spawns so backgrounded work dies loudly instead of half-completing.
5. Shrink case C's blast radius: implementer/orchestrator agent prompts mandate WIP commits at each milestone (scoped → implemented → tests green), so a crash loses minutes of work, not hours.

Scope boundary with #98

This issue = detection + classification + the mechanical paths (A delete, B publish) at the driver post-exit boundary. #98 = the model-driven continuation path for case C and stall detection for workers that are still nominally alive. The classifier states are shared so the two halves compose.

🤖 Generated with Claude Code