Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn

[rysweet]

Summary

An engineer instance completed cleanly (status=completed, exit 0) but its engineer_claim was never released on completion, so the claim leaked and the worktree sat idle 7009s until the overseer stale-engineer reaper picked it up. The completion→claim-release path is compounded by an effect-dispatch race: require_goal_repository raises permanent("goal disappeared before effect dispatch") (src/ooda_actions/advance_goal/typed_goal_session.rs:460), surfaced as CycleErrorCode::DownstreamFailed (src/typed_ooda/executor.rs), which then drives repeated reason=no-worktree immediate-reclaims while the worktree is transiently absent/re-created.

This is a completion → claim-release / effect-dispatch race (finished-unreported leak). It is related to but distinct from:

• Simard OODA no-progress breaker reaps+re-dispatches engineers that already completed work and opened a mergeable PR (termination/done-detection defect → duplicate PRs) #4441 (no-progress breaker reaps+re-dispatches an engineer that already opened a mergeable PR — PR done-detection → duplicate PRs). Here there is no PR; completion is a sessions.jsonl status=completed that never propagated to the ledger.
• Stale-assignment sweep is non-idempotent: same dead engineer id re-cleared every OODA cycle (clear not durably persisted) #4462 (stale-assignment sweep clear not durably persisted — sweep_stale_assignments). Here the churn is the claim_reaper NoWorktree path racing the goal-session effect dispatch, not the per-cycle assignment-clear.

Concrete occurrence (durable evidence)

• Goal: continuously-research-and-improve-your-own-cogn-70ab8541 (standing research goal, Make Simard's research goal NEVER idle: every cycle it must seek a NEW source or design a NEW experiment to improve metacognition (dedup vs recent directions); an idle cycle is a fault, not normal. Pr #4399 never-idle machinery)
• Claim: rysweet/Simard:continuously-research-and-improve-your-own-cogn-70ab8541
• Engineer instance: pid 3246134, session session-18c4a54eb48ffa36 (non-nested, parent_session_id=null)
• Archive: ~/.simard/reaped-engineers/rysweet_Simard_continuously-research-and-improve-your-own-cogn-70ab8541-1784744035/

Triangulated across three independent sources in the archive:

1. sessions.jsonl tail (evidence.txt L2 → L9): initial status=active registration, terminal update {"session_id":"session-18c4a54eb48ffa36","status":"completed","end_time":1784737025.79} → completed. All nested --version sessions also completed (L6-8).
2. manifest.json: idle_age_secs=7009, archived_unix_ts=1784744035. 1784744035 − 7009 = 1784737026 ≈ end_time 1784737025.79 → the newest-file idle age is exactly the elapsed time since the session completed (16:17:05 UTC). The worktree went quiet because the engineer finished, not because it hung.
3. journal.txt: L27 terminal heartbeat COMMAND_EXIT_CODE="0" / Script done ... exit; evidence.txt L424 cargo ... command_complete exitCode 0 (305 passed; 0 failed). Then L28 & L34 claim-reaper: reclaimed ... reason=no-worktree, verdict=no-investigation; L30 typed goal-session effect incomplete (DownstreamFailed): goal disappeared before effect dispatch; L35 worktree re-created (native git hooks enrolled in engineer worktree).

Actual system state at investigation: pid 3246134 not alive; worktree still registered (git worktree list → branch engineer/...-3285bd); no hard-fault signature (0 matches for panic/OOM/E2BIG/exit-126/SIGKILL across journal+evidence).

Root cause (candidate paths)

1. No completion→claim-release hook. When an engineer session transitions to status=completed, nothing releases its engineer_claim through the release_engineer_claim chokepoint; the claim survives until the idle-staleness reaper investigates it ~2h later.
2. Effect-dispatch/goal-lifecycle race. require_goal_repository (typed_goal_session.rs:452-461) reads active_goals.active at dispatch time; if the goal is momentarily absent (re-orient / roll_to_new_cycle for this standing goal), it returns permanent("goal disappeared before effect dispatch") → DownstreamFailed, and the reaper's NoWorktree path immediately reclaims the claim whose worktree is transiently gone, then a new worktree is re-created — churn.

Impact

• Leaked engineer-claim consumes a cap-limited admission slot for up to the staleness window after the work already finished.
• no-worktree reclaim ↔ worktree re-create churn on a standing goal; noise that masks genuine liveness/reap reasoning.

Suggested fix direction

1. Release the engineer_claim on session completion (drive status=completed through release_engineer_claim) so a finished engineer never idles as a leaked claim.
2. Make the completion→effect-dispatch path race-free: treat "goal absent from active_goals.active at dispatch" for a standing/re-orienting goal as a benign no-op (or re-resolve the goal) rather than a permanent DownstreamFailed, so it cannot trigger a spurious NoWorktree reclaim + worktree re-create.
3. Regression test: a session that reaches status=completed releases its claim within one tick; a standing goal mid re-orient does not raise goal disappeared before effect dispatch.

Dedup

Dedup key: simard-claim-not-released-on-completion:effect-dispatch-race-downstreamfailed:continuously-research-and-improve-your-own-cogn-70ab8541. Searched open+closed issues for claim-release/finished-unreported/DownstreamFailed/effect-dispatch/no-worktree — closest are #4441 and #4462 (related, distinct as noted above); no issue tracks the completion→claim-release/effect-dispatch race. Filing as a new systemic defect.

Filed by the stale-engineer investigator (Overseer investigate-before-reap, prompt_assets/simard/overseer/investigate_stale_engineer.md). Verdict: dead / finished-unreported. Durable evidence archive preserved.

[rysweet]

Consolidation from newest stale-engineer investigation (archive -1784744035)

A follow-up investigate-before-reap pass over the same durable archive
(~/.simard/reaped-engineers/rysweet_Simard_continuously-research-and-improve-your-own-cogn-70ab8541-1784744035/)
independently reproduced this root cause and confirms the verdict dead / finished-unreported. No duplicate issue filed — this is the canonical tracker.

Re-verified system state (now, ~2.5h after completion):

• Engineer session session-18c4a54eb48ffa36, pid 3246134 → not alive (finished).
• Newest session status=completed, end_time=1784737025.79; manifest.idle_age_secs=7009; 1784744035 − 7009 = 1784737026 ≈ end_time → idle clock started exactly at clean completion (COMMAND_EXIT_CODE=0, 305 passed; 0 failed).
• Worktree still git-registered (engineer/…-3285bd @ 8f81e125), clean git status, no live process holding it → leaked claim, not work-in-flight.
• No panic/OOM/E2BIG/exit-126/SIGKILL anywhere in journal+evidence; the only failure signature is DownstreamFailed: goal disappeared before effect dispatch (typed_goal_session.rs:460) driving reason=no-worktree, verdict=no-investigation reclaim↔re-create churn — exactly as this issue describes.

Cross-link: related to #4437 (claim-reaper reaps healthy standing/perpetual-goal engineers as false positives). This goal is the standing cognition goal, so the completion→claim-release race here compounds the perpetual-goal handling gap in #4437; a robust fix should cover both the completion-release hook and the standing-goal re-orient no-op path.

Action taken: engineer is genuinely dead + unrecoverable and its evidence is durably archived, so the Overseer reap proceeds (release claim + remove worktree). Labeling workflow:default to dispatch the systemic fix.

[rysweet]

Corroborating evidence for the no-worktree churn (goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c, archive rysweet_Simard_...-f29bb15c-1784746010).

The single journal for this goal shows the reaper interleaving two paths on the same claim_key across sweeps, never converging (10 sibling archives, 04:00→18:46 UTC):

L4/L5/L11  verdict=pending        (HeartbeatStale: evidence archived, claim+evidence preserved)
L52/L62    reason=no-worktree, verdict=no-investigation  → RECLAIMED (unconditional, #4400 gate BYPASSED)
L83/L84/L95   verdict=pending     (re-archived — worktree back)
L106/L107/L118 verdict=pending    (re-archived again)

State desync: at investigation time the manifest worktree advance-...-f29bb15c-1784733114-bba449 is present on disk right now (mtime Jul 22 15:11), yet earlier sweeps logged reason=no-worktree reclaims for the same claim. The NoWorktree branch (claim_reaper.rs probe L532-538 → router L332-333, verdict=None) reclaims immediately with no investigation and no evidence gate — so a transient/racy worktree-name mismatch (leaked claim outliving one worktree incarnation, before the next epoch -1784733114- is created) reclaims the slot, the engineer is re-launched under a new worktree epoch, re-claimed, and re-archived. Net: perpetual re-archival, admission-slot thrash, and the #4400 investigate-before-reap protection is skipped precisely for the leaked-claim case this issue describes.

Related non-convergence facet tracked in #4462 (verdict/clear not durably persisted between sweeps → same engineer re-processed every cycle). The stale-engineer path lacks the persisted-dedupe/atomic-write-back invariants (I2/I3/I7) the no-progress re-investigation loop uses.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[rysweet]

Corroborating evidence — new archive -1784755469, a distinct trigger: daemon handover, not a completion-race

Investigate-before-reap pass (#4400) on the same goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c. Archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784755469/. Verdict: STILL-ALIVE false positive — claim kept, worktree NOT removed.

The 20:35:42 reason=no-worktree reclaim came from a second daemon incarnation, not the one that ran the investigations

journal.txt shows a daemon restart/handover, not simultaneous daemons:

PID	first line	last line	lines
2802920	15:29:15	20:04:40	89
1664591	20:05:21	21:22:50	46

• PID 2802920 logged 7× NOT reaping … (verdict=pending, claim + evidence preserved) between 16:05→20:04, idle climbing 3196 → 8452 → 12894 → 17564s.
• PID 1664591 (the new incarnation, no in-memory record of those 7 Pending investigations) hit a transient NoWorktree and fired the verdict-free fast-path:

  20:35:42 simard[1664591]: [simard] claim-reaper: reclaimed …f29bb15c
           (reason=no-worktree, age=n/a, verdict=no-investigation)
  

This is the NoWorktree fast-path at src/overseer/claim_reaper.rs:308-311,332-333 → verdict = None → unconditional reclaim (:374-402). Its guarding comment (:330-331 "NOTHING to preserve or investigate") is false during daemon handover / worktree churn: absence is transient.

Ground truth OVERRIDES the stale log signal (state > transcript)

• Worktree exists on disk, mtime 2026-07-22 20:45:05 — ~9 min AFTER the 20:35:42 reclaim → it was re-created, the engineer is alive.
• .simard-engineer-claim sentinel present, written 20:45, contents pid=1664591 (the daemon PID again — see BUG: engineer claims record daemon PID (defeats is_pid_alive); stale-claim cleanup not automatic #1713).
• 19× no-progress breaker: standing/perpetual goal idled this cycle (normal, not a fault) → idle 2363s is expected, not death (missing reaper exemption = claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437).
• All 8 evidence.txt "newest files" are checked-in fixtures (tests/**/fixtures, src/**/fixtures, prompt_assets, package.json) — no engineer transcript (claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449).

Distinct root cause vs. the completion-race already tracked here

This issue frames the no-worktree churn as claim-not-released-on-completion + effect-dispatch race. This occurrence adds an orthogonal trigger: there is no cross-incarnation single-writer/lease on the reaper, so a fresh daemon has zero memory of the prior incarnation's in-flight Pending investigations and reclaims on a transient NoWorktree. That memory/lease gap is #1713 (claim records daemon PID; no durable single-writer). Combined with #4467 (the recipe verdict is never written back, so the investigation loops Pending forever and re-archives — this is the 12th archive of this goal), the NoWorktree fast-path becomes the escape hatch that regresses to pre-#4400 verdict-free reaping.

Suggested systemic fix (captured here to avoid a duplicate issue)

Guard the NoWorktree fast-path (claim_reaper.rs:332-333) before verdict-free reclaim with:

1. a cross-incarnation lease (durable owner token, not daemon PID) so only one incarnation reaps a claim_key/cycle — BUG: engineer claims record daemon PID (defeats is_pid_alive); stale-claim cleanup not automatic #1713 / Live self-migration spike — KEYSTONE: exactly-one-primary via a fenced single-writer role lease #2725 PrimaryLease;
2. a transient-absence / stability check (e.g., re-probe or require absence to persist across N sweeps) so worktree churn/handover can't trip it;
3. the is_perpetual() exemption the reaper still lacks — claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437.

No new issue filed (deduplicated into #4464/#1713/#4437/#4467/#4449). No claim released, no worktree removed — verdict is still-alive.

[rysweet]

Recurrence — same defect, more severe manifestation: no-worktree path reclaimed a genuinely-LIVE claim (not a finished/leaked one)

New durable evidence archive of the same goal / same claim, from a stale-engineer investigation run by the Overseer investigate-before-reap path:

• Archive: ~/.simard/reaped-engineers/rysweet_Simard_continuously-research-and-improve-your-own-cogn-70ab8541-1784803462/
• Goal / claim: continuously-research-and-improve-your-own-cogn-70ab8541 / rysweet/Simard:continuously-research-and-improve-your-own-cogn-70ab8541
• manifest: idle_age_secs=6090, worktree ...-1784797372-d6f5c6
• Investigator verdict: still-alive / cause=null (fail-closed — the engineer is alive, NOT finished-unreported this time)

Why this strengthens #4464

The original occurrence was a finished-unreported leak (session status=completed, exit 0). This occurrence has no completion event at all — the engineer is still alive, yet the same NoWorktree path reclaimed its claim verdict=no-investigation three times while the OODA loop was actively cycling and the per-goal reasoner was spawn-ing new work:

• journal.txt L10 05:30:26 reclaimed ... reason=no-worktree, verdict=no-investigation
• journal.txt L12 05:46:36 typed goal-session effect incomplete (DownstreamFailed): goal disappeared before effect dispatch
• journal.txt L13 05:47:26 reclaimed ... reason=no-worktree, verdict=no-investigation
• journal.txt L50 08:50:12 reclaimed ... reason=no-worktree, verdict=no-investigation

…interleaved with active liveness in the same journal window:

• OODA cycle counter advancing monotonically 2463 → 2480 (04:57 → 10:42, ~5.75h)
• per-goal reasoner spawn decisions at L48/L54/L57/L63/L68/L73 ("standing research goal must not sit idle … No stale/fault signals, so investigate is unwarranted")
• claim-reaper correctly NOT reaping (verdict=pending, claim + evidence preserved) on the HeartbeatStale path (L6/L27/L34/L42/L60/L65/L70)
• zero crash signatures (0 matches for panic / OOM / E2BIG / SIGKILL / segfault / non-zero-exit across journal + evidence)

Live cross-check at investigation time: overseer pid 3159928 alive (simard ooda run, the same pid as journal tail cycle 2480) and worktree ...-70ab8541-1784797372-d6f5c6 still present on disk. So the no-worktree reclaims fired in the transient worktree-gap between spawn bursts of a live standing goal.

Root-cause anchor (confirms candidate path #2 in this issue)

src/overseer/claim_reaper.rs:332-333 — the NoWorktree arm sets verdict = None, an unconditional reap that bypasses investigate-before-reap (// A NoWorktree claim has NOTHING to preserve or investigate — it is reclaimed directly). Combined with the require_goal_repository → permanent("goal disappeared before effect dispatch") → DownstreamFailed race (typed_goal_session.rs:452-461), a standing/re-orienting goal that is momentarily worktree-less mid-spawn is reclaimed as if dead — a false-positive reclaim of live work, strictly worse than the leaked-completed-claim framing.

Take-away for the fix

The suggested-fix directions in the issue body still hold, but this occurrence shows the NoWorktree immediate-reclaim needs to be guarded against a live/standing goal mid-effect-dispatch, not only made race-free for the completion path — e.g. treat "goal present + in-flight/spawning + worktree transiently absent" as not an unconditional reap. The evidence-hardening (#4449) note also applies: this archive's evidence.txt is entirely checked-in fixtures (newest-mtime of tracked tests/fixtures/**, src/coin_gym/fixtures/**, package.json), i.e. no real engineer transcript — a fail-closed "no transcript" signal, not death evidence.

Filed by the stale-engineer investigator (Overseer investigate-before-reap, prompt_assets/simard/overseer/investigate_stale_engineer.md). Verdict: still-alive / cause=null. No reclaim performed; claim + worktree preserved. Dedup key: simard-claim-not-released-on-completion:effect-dispatch-race-downstreamfailed:continuously-research-and-improve-your-own-cogn-70ab8541 — same as this issue.

[rysweet]

Recurrence evidence — stale-engineer investigation f29bb15c (finished-unreported / leaked claim on completion)

Investigated a would-be-stale engineer before reap, per prompt_assets/simard/overseer/investigate_stale_engineer.md. This is a fresh, independently-grounded datapoint for this bug.

Verdict: dead / cause finished-unreported. The engineer completed cleanly but never released its claim → leaked claim + idle worktree the reaper then flags.

Durable evidence (survives worktree cleanup):
~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784908047/

Grounded facts from evidence.txt / sessions.jsonl:

• Session session-18c53ddfca2a1494 (PID 196805) recorded {"status":"completed","end_time":1784903860} — it finished, it did not crash.
• Its verification test tests/gadugi/kgpacks-rs-hybrid-ranking.sh completed exitCode:0 ("Test session completed"). No panic / OOM / SIGKILL anywhere in evidence.
• PID 196805 is provably gone (no /proc/196805, no ps entry).
• Newest-file idle age 4187s is measured from the status:completed write itself → the worktree is idle because the work finished, not because it wedged.
• The claim was never released on completion (this bug) and no commit/branch/PR carries the result — the only artifacts are redundant uncommitted edits, because the parity feature was already merged as feat(kgpacks-rs): hybrid retrieval ranking (KGP-Q10) — full agent-kgpacks parity #4503 (MERGED 2026-07-23, KGP-Q10 full parity).

Systemic driver (recurrence): 38 archives for goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c. Two compounding defects:

1. This issue (Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn #4464): claim not released on session completion → leaked claim.
2. claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437: claim_reaper uses newest-file-mtime only and lacks the is_perpetual() exemption no_progress.rs already applies, so it keeps relaunching engineers for an already-satisfied standing goal (feat(kgpacks-rs): hybrid retrieval ranking (KGP-Q10) — full agent-kgpacks parity #4503 merged). Each finishes redundantly and leaks its claim → this loop.

claim_reaper.rs confirmed to derive staleness purely from worktree presence + newest-file mtime; it never reads sessions.jsonl / status:completed, so a cleanly-finished session is indistinguishable from a wedged one at the mechanical layer.

Reap disposition: reap is permitted (process already gone, evidence archived). Deferring the mechanical reclaim to the live daemon's claim_reaper rather than mutating the claim ledger by hand — a second concurrent writer would be exactly the race in #4477.

Filed as a deduplicated comment (no new issue) — self-improvement signal, not a silent reclaim.

[rysweet]

Recurrence data point — finished-unreported / leaked claim (stale-engineer investigation)

Another confirmed occurrence of this defect, caught by the #4104 stale-claim reaper safety net and investigated via investigate_stale_engineer.md.

Claim: rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c
Durable archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784920781/

Evidence (triangulated, all from the durable archive):

• Clean completion, not a crash — sessions.jsonl tail: session session-18c53ddfca2a1494 went status:"active" (PID 196805, start 1784900858) → status:"completed", end_time:1784903860. No later active record; PID 196805 gone.
• Idle-age is measured from that clean completion — archived_ts 1784920781 − idle_age 16921 = 1784903860 = exact end_time. The engineer simply finished and stopped writing.
• No crash signature — zero non-zero exits ("exitCode":[1-9] → none). panic/panicked/e2big appear ONLY inside JSON keyword lists ("panicked","false"), never as events. No OOM/SIGKILL/segfault/deadlock. Killing process is only normal agentic-test teardown after an exitCode:0 sub-run (14:18:23), which is intermediate — the terminus is the 14:37 session completion.
• Leaked claim — the engineer_claims row was still present on the ledger (outcome_id=019f945e-20e8-7480-9009-313cb5504b24, request_id=cycle-2537-...-spawn) with no open PR and no reported result (daemon journal repeatedly logs no open PR, worker present but quiet, goal not-started). The engineer completed but its result never reached the ledger and the claim was never released — the primary release-on-termination path (fix(ooda): make engineer_claims a releasable liveness lease (#4094) #4095, release_engineer_claim at the cleanup_engineer_worktree_for_goal chokepoint) did not fire.

Compounding (see #4467): the reaper re-archived evidence and dispatched the agentic investigation at 15:47, 16:54, and 18:17, each returning verdict=pending / NOT reaping — the reconcile_inflight_investigations write-back gap prevents convergence, so this same completed engineer was re-investigated repeatedly (unbounded re-archival).

Manual reap performed (automated path blocked by #4467): released the claim via the shared chokepoint (DELETE FROM engineer_claims WHERE claim_key=?) and removed the worktree (git worktree remove); durable archive left intact.

Filed as a recurrence here rather than a new issue to keep the finished-unreported/leaked-claim signature deduplicated. Root fix remains the release-on-completion gap tracked by this issue; the never-converging investigation is #4467.