Loop: failure classification + crash-resume for dead workers (the recovery half of loop-health)

[robercano]

Context (from the orchestrator landscape comparison)

reCode's recovery story today is the spawn-lock TTL self-heal (900s, from #81) and in-flight detection. What it lacks is everything after that: a worker that dies mid-task orphans its partial work (worktree + branch sit there; the issue stays in_flight until a human intervenes), and every recovery is a cold restart that re-derives all context. Issue #85 (loop-health panel) covers seeing stalls; this issue is about recovering from them.

What the comparison offers:

• Looper has the most transferable design: boundary-aware failure classification — transient (network, rate limit) → bounded retry with exponential backoff; manual_intervention (dirty worktree, auth) → surfaced prominently, never auto-retried; deterministic failure → terminal with an explicit operator-retry command. Plus reconcile-stale to repair queue state after laptop sleep/wake, and native session resume — a re-spawned agent continues the same CLI session instead of cold-starting (runner-retry spec).
• Composio AO: conservative liveness — a failed probe is an observation, not proof of death; termination requires multiple independent conditions; never force-delete a dirty worktree (architecture doc).
• Gas Town: agents-watching-agents (Witness per repo, Deacon patrol, Boot watchdog-of-the-watchdog) and gt seance — a successor agent interrogates its dead predecessor's session for context instead of starting cold. Heavy machinery, but the seance idea maps cheaply onto Claude Code's --resume.

Proposal

1. Detect: census learns a stalled=N state — issue in_flight but no events.jsonl activity for its task in T minutes AND no open PR. (Depends on / extends Cockpit: loop-health panel — last tick, cadence, verdict history, stall detection #85's last-activity tracking.)
2. Classify (deterministic, in shell): worktree exists + dirty → manual_intervention (label needs-human, never delete — AO's rule); worktree exists + clean or branch pushed → resumable; nothing but the branch → orphaned.
3. Recover: for resumable, the next ADVANCE for that issue re-spawns an implementer pointed at the EXISTING worktree/branch with instructions to inspect git log/git status and continue (cheap seance: the git state is the memory; optionally attempt Claude Code session resume if a session id was logged in events.jsonl). Bounded: 2 resume attempts, then needs-human.
4. Log every transition to events.jsonl so Cockpit: loop-health panel — last tick, cadence, verdict history, stall detection #85's panel can render verdict/recovery history.

Acceptance criteria

• Kill an implementer mid-task in the smoke harness (self/smoke-fanout.sh fixture): next tick detects the stall, classifies it, and either resumes to completion or escalates — no human unwedging, no lost dirty work, ever.
• Classification + attempt counters visible in cockpit; covered by loop-tick.test.sh cases.

Decision points for you

• Stall threshold T (suggest 30 min default, adapter-overridable)?
• Is Claude Code session-resume worth wiring (needs session-id capture at spawn), or is git-state-as-memory enough for v1? (Recommend git-state for v1.)

🤖 Generated with Claude Code

https://claude.ai/code/session_014Gu82LWNkdzhrDFSQ7EZmB

---

Update 2026-07-15 — split with #111, and field evidence

Issue #111 has been redesigned as the driver post-exit half of this recovery story: after every advance driver exits, it verifies the claimed outcome against GitHub (phantom detection / ledger honesty) and runs a deterministic debris classifier whose states mirror the Proposal's step 2:

• Empty (no commits, clean) → delete + restart — handled entirely in Loop: driver one-shot contract needs enforcement + tests — background orchestration produced phantom-success drivers and wedged the queue #111.
• Publishable (commits ahead, clean, gates/reviews recorded done) → mechanical publish (re-run gate, push, open PR via bot) — handled entirely in Loop: driver one-shot contract needs enforcement + tests — background orchestration produced phantom-success drivers and wedged the queue #111. Field-proven 2026-07-15: issue Loop daemon: harden runtime-env resolution — bake resolved PATH into installed units, don't depend on nvm sourcing at runtime #107's killed driver left exactly this state; manual recovery (→ PR Loop daemon: bake resolved node/claude PATH into pr-loop.service unit #117) confirmed no model is needed.
• Half-done / dirty → never deleted; wip:-committed if dirty and marked resumable — then handed off to THIS issue's continuation driver (step 3 of the Proposal: fresh implementer pointed at the existing worktree/branch, git state as memory, optional session-resume from the ledgered session id, bounded attempts → needs-human).

So this issue's remaining scope = stall detection for nominally-alive workers (steps 1, 4) + the case-C continuation/resume machinery (step 3). Classification for dead drivers moves to #111; keep the state names shared so the halves compose.

Supporting change (filed under #111, benefits this issue directly): agent prompts mandate WIP commits at each milestone, so the half-done case loses minutes of work, not hours.

🤖 Generated with Claude Code

[robercano]

Stall threshold T (suggest 30 min default, adapter-overridable)? Yes do this

Let's do git-state-as-memory for v1