fix: key stall termination on read watermark, not prompt lock (0.14.3) - #86
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #85 (0.14.2), silently-running turns — most visibly sub-agent tasks — were killed after 120s of stream silence. Zed.log captured two live hits:
Root cause
#85's deadline check probed the prompt lock via
session/goal showand killed the turn whenever the probe did not report the 1308 lock-busy error. Raw-backend probes (.zcode/scratch/probe-goal-lock.mjs) against the Aug-28 app-server proved the prompt lock is not a liveness signal at all:session/goal showsucceeds mid-turn (11/11 probes across a 185s live turn, projectionrunningthe whole time) — it never reports 1308;session/sendis ACCEPTED while the turn runs (queued as steer input) — the lock is only held during turn finalisation;contextUsed/turnCount) keeps advancing the whole time a sub-agent works behind a silent stream (msgs 2→20, contextUsed 0→147k over 5 minutes with zerosession/events).So "lock released" is the normal state of a healthy mid-turn backend, and killing on it murdered live sub-agent turns after 2 quiet minutes.
Fix
Replace the lock probe with a watermark heartbeat taken from the existing 15s stall-reconcile
session/read(zero extra RPCs):STALE_FREEZE_MS) → still defer (long CoT / quiet tools legitimately freeze it for 60s+);end_turn,stallRecovered), bounded stop only when nothing was ever delivered.This keeps #85's convergence goal (a projection stuck at
runningno longer hangs forever) while never killing a live turn.probePromptLockis deleted; tests assert the goal channel is never consulted again.Verification
tests/stale-running-recovery.test.tsrewritten: watermark-advancing turn survives 120s and 10 min of silence and ends normally onturn.completed; frozen watermark converges tomax_turn_requestsafter the 10-min budget (with stop); frozen + delivered output ends gently without stop.0.14.3.docs/ARCHITECTURE.mdturn-state diagram updated; new AGENTS.md gotcha "Prompt lock ≠ turn liveness" with the probe evidence.