Skip to content

fix: key stall termination on read watermark, not prompt lock (0.14.3) - #86

Merged
william0wang merged 1 commit into
mainfrom
fix/subagent-stall-kill
Sep 1, 2026
Merged

fix: key stall termination on read watermark, not prompt lock (0.14.3)#86
william0wang merged 1 commit into
mainfrom
fix/subagent-stall-kill

Conversation

@william0wang

Copy link
Copy Markdown
Owner

Problem

Since #85 (0.14.2), silently-running turns — most visibly sub-agent tasks — were killed after 120s of stream silence. Zed.log captured two live hits:

10:08:56 [stall] no-progress deadline reached; prompt lock=released
10:08:56 [stop] v4/command stop sent for sess_31dbe177

Root cause

#85's deadline check probed the prompt lock via session/goal show and killed the turn whenever the probe did not report the 1308 lock-busy error. Raw-backend probes (.zcode/scratch/probe-goal-lock.mjs) against the Aug-28 app-server proved the prompt lock is not a liveness signal at all:

  • session/goal show succeeds mid-turn (11/11 probes across a 185s live turn, projection running the whole time) — it never reports 1308;
  • a probe session/send is ACCEPTED while the turn runs (queued as steer input) — the lock is only held during turn finalisation;
  • the read projection's watermark (contextUsed/turnCount) keeps advancing the whole time a sub-agent works behind a silent stream (msgs 2→20, contextUsed 0→147k over 5 minutes with zero session/events).

So "lock released" is the normal state of a healthy mid-turn backend, and killing on it murdered live sub-agent turns after 2 quiet minutes.

Fix

Replace the lock probe with a watermark heartbeat taken from the existing 15s stall-reconcile session/read (zero extra RPCs):

  • watermark advancing → backend demonstrably alive → defer the terminal decision indefinitely;
  • watermark frozen < 10 min (STALE_FREEZE_MS) → still defer (long CoT / quiet tools legitimately freeze it for 60s+);
  • watermark frozen ≥ 10 min → stale projection confirmed: fetch the reply and end gently (end_turn, stallRecovered), bounded stop only when nothing was ever delivered.

This keeps #85's convergence goal (a projection stuck at running no longer hangs forever) while never killing a live turn. probePromptLock is deleted; tests assert the goal channel is never consulted again.

Verification

  • tests/stale-running-recovery.test.ts rewritten: watermark-advancing turn survives 120s and 10 min of silence and ends normally on turn.completed; frozen watermark converges to max_turn_requests after the 10-min budget (with stop); frozen + delivered output ends gently without stop.
  • Full suite 838/838 green; typecheck/lint/build/smoke clean; smoke reports 0.14.3.
  • Docs: docs/ARCHITECTURE.md turn-state diagram updated; new AGENTS.md gotcha "Prompt lock ≠ turn liveness" with the probe evidence.

@william0wang
william0wang merged commit ca7a739 into main Sep 1, 2026
1 check passed
@william0wang
william0wang deleted the fix/subagent-stall-kill branch September 1, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant