fix(cockpit): derive accurate live task state — no more phantom in-flight workers - #126
Merged
Merged
Conversation
…ight workers The live-progress panel keyed worker identity on the RAW task string, so the same worker logging "81" then "issue-81" (or "issue-70-worker-inspector") became several identities, and a "done" logged under one variant never overwrote the "reviewing" logged under another. Combined with workers that crash or simply never emit their own done, the cockpit showed tasks 70/81 as reviewing/implementing a week after their PRs merged. Three derivation rules fix it: - Worker identity is (role, normalized task, lens): id variants collapse via the existing taskGroupKey() normalizer, so a later done actually overwrites the earlier phase. Lens stays in the key — reviewers of the same task under different lenses are distinct workers. - The orchestrator owns the task lifecycle: a group whose orchestrator's latest event is done — with no worker activity logged after it (file order) — renders as a single done-badged header, not a table of phantom in-flight rows. No orchestrator events falls back to every-worker-done. Activity logged AFTER an orchestrator done (task re-scoped) keeps the group active. - Unfinished groups silent for >COCKPIT_STALE_AFTER_SECONDS (default 2h, same override style as COCKPIT_NOW) are badged "stale · no events for Nh/Nd" with muted rows — a wedged worker should be visible as wedged, never as active work. Verified against the real event log: 70/81 (and 8 other finished tasks) now render done, three genuinely-abandoned groups render stale, and the one task the loop is actually driving (#95, scoped 25 min ago) still renders active. Adds 10 checks to cockpit.test.sh (fixture mirrors the exact production log shape that exposed the bug); all 80 existing checks pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
robercano
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The cockpit showed tasks (e.g. #70 "reviewing", #81 "implementing") as in-flight a week after they finished and their PRs merged.
Two compounding causes in the live-progress panel:
81,issue-81,issue-70-worker-inspector), so adonelogged under one variant never overwrote thereviewinglogged under another — the stale identity rendered as active forever.donefor the task, but sub-workers that crashed or never emitted their owndone(e.g. Harden PR-loop tick: deterministic loop-tick.sh with single verdict, in-flight detection, spawn lock #81's implementer) kept their last phase indefinitely, with no staleness cutoff.Fix — three derivation rules
taskGroupKey()normalizer, so later phases actually overwrite earlier ones. Lens stays in the key (same-role reviewers under different lenses are distinct workers).donefinishes the task: a group whose orchestrator's latest event isdone, with no worker activity after it (file order), renders as one done-badged header — no phantom rows. Falls back to every-worker-done when there are no orchestrator events; activity logged after an orchestrator done (re-scoped task) keeps the group active.COCKPIT_STALE_AFTER_SECONDS(default 2h; same override style asCOCKPIT_NOW) get astale · no events for Nh/Ndbadge with muted rows — a wedged worker shows as wedged, never as active work.Verified on the real event log
#70, #81 and 8 other finished tasks now render done; 3 genuinely-abandoned groups (#90, #106, #107) render stale; the one task the loop is actually driving right now (#95, scoped 25 min before the check) still renders active with its warn badge.
Tests
+10 checks in
cockpit.test.sh(fixture mirrors the exact production log shape that exposed the bug: implementer that never logs done, reviewer whose done lands under an id variant, orchestrator close, post-done re-scope, 49h-silent group, threshold override). All 80 existing checks pass unchanged. Gates:build✅lint✅test✅🤖 Generated with Claude Code