Skip to content

feat(loop): stall detection + bounded resume for hung in_flight issues (issue #98) #146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robercano-ghbot merged 2 commits into from
Jul 16, 2026
Merged

feat(loop): stall detection + bounded resume for hung in_flight issues (issue #98) #146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b24e54f
feat(loop): stall detection + bounded resume for hung in_flight issue…
robercano Jul 16, 2026
6fef3e7
fix(loop): make stall/resume actually reachable, stop charging #95 bu…
robercano Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 83 additions & 1 deletion .claude/scripts/loop-census.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@
# hasn't reached PR stage. A tick uses this to
# avoid double-spawning an orchestrator for an
# issue that already has a worktree in progress.
# stalled=<n> age_min=<m> one line PER in_flight issue whose most recent
# events.jsonl activity is older than the stall
# threshold (issue #98) — see STALL DETECTION
# below. A tick uses this to RESUME a hung/dead
# in_flight issue instead of refusing it forever.
# blocked=<n> by=<N> one line per candidate that would otherwise be
# advance_ready but is skipped because its body
# says "Blocked by #N" and issue N is still OPEN
Expand Down Expand Up @@ -57,6 +62,30 @@
# one candidate was otherwise eligible; with zero eligible candidates,
# advance_ready stays "none" exactly as before this feature.
#
# --- STALL DETECTION (issue #98) --------------------------------------------
# An `in_flight` issue (feat/issue-N-* branch exists, no open PR yet) can sit
# forever if the driver that created it hung or died without ever reaching
# loop-daemon.sh's post-exit verification (issue #111) — e.g. the daemon
# process itself was restarted/killed mid-drive. This census can't see driver
# health directly, but it CAN see whether anything has logged progress for
# that issue recently: log-event.sh (issue #52) appends one JSONL line per
# phase transition to events.jsonl, with a `task` field that's been observed
# in BOTH a bare issue number ("42") and an "issue-42" form across this
# project's real history — an in_flight issue is STALLED when the newest
# event naming it (by either form) is older than `budget.stall_minutes`
# (adapter-configurable, default 30; see gates.json).
#
# CONSERVATIVE FALSE-POSITIVE RULE: an issue with ZERO events at all is NEVER
# reported stalled — a branch/worktree just created by a driver that hasn't
# logged its first event yet looks identical to a permanently-abandoned one
# from events.jsonl's point of view alone; treating "no data yet" as "stalled"
# would kill fresh work. Only a issue with AT LEAST ONE event, whose newest is
# past the threshold, is reported.
#
# Events file: defaults to <root>/.claude/state/events.jsonl; override with
# $CLAUDE_EVENTS_FILE (same env var log-event.sh itself honors) for
# testability without touching the real, gitignored state dir.
#
# Repo derived from the git remote; override with $1. Bot login via $BOT_LOGIN.
# Invoke as `bash .claude/scripts/loop-census.sh` (pre-approve that exact
# command). Read-only: advances no cursor, mutates nothing — safe to re-run.
Expand All @@ -76,6 +105,46 @@ case "$gates_rel" in /*) gates="$gates_rel" ;; *) gates="$root/$gates_rel" ;; es
base=$(node -e 'const g=require(process.argv[1]); console.log((g.merge&&g.merge.baseBranch)||"main")' "$gates")
module_labels=$(node -e 'const g=require(process.argv[1]); console.log(g.modules.map(m=>"module:"+m.name).join("\n"))' "$gates")

# Stall threshold (issue #98), adapter-overridable via budget.stall_minutes;
# same node -e / require(gates) pattern as base/module_labels above.
stall_minutes=$(node -e '
const g = require(process.argv[1]);
const v = g.budget && g.budget.stall_minutes;
console.log((Number.isFinite(v) && v > 0) ? v : 30);
' "$gates" 2>/dev/null)
case "$stall_minutes" in ''|*[!0-9]*) stall_minutes=30 ;; esac

events_file="${CLAUDE_EVENTS_FILE:-$root/.claude/state/events.jsonl}"

# --- stall detection helper (issue #98) --------------------------------------
# $1 = issue number. Prints the age in whole minutes of the NEWEST
# events.jsonl line whose `task` field is either "$1" or "issue-$1" (both
# forms occur in this project's real log), or nothing when there is no such
# event at all — callers must treat empty as "do not report stalled" (see the
# conservative false-positive rule in the header comment above), never as 0.
last_event_age_minutes() {
CLAUDE_STALL_EVENTS_FILE="$events_file" CLAUDE_STALL_ISSUE="$1" node -e '
const fs = require("fs");
const file = process.env.CLAUDE_STALL_EVENTS_FILE;
const num = process.env.CLAUDE_STALL_ISSUE;
let latest = null;
try {
const text = fs.readFileSync(file, "utf8");
for (const line of text.split("\n")) {
if (!line.trim()) continue;
let o;
try { o = JSON.parse(line); } catch (e) { continue; }
if (o.task !== num && o.task !== ("issue-" + num)) continue;
const t = Date.parse(o.ts);
if (!Number.isFinite(t)) continue;
if (latest === null || t > latest) latest = t;
}
} catch (e) { /* no file / unreadable -> latest stays null */ }
if (latest === null) process.exit(0);
console.log(Math.floor((Date.now() - latest) / 60000));
' 2>/dev/null
}

# --- driver-unit guard (issue #119): never advance an issue whose transient
# driver unit (pr-loop-driver-issue<N>, spawned by loop-daemon.sh's run_driver)
# is currently active — e.g. the driver hasn't reached `git checkout -b` yet,
Expand Down Expand Up @@ -135,6 +204,7 @@ advance_ready="none"
fallback_ready="none"
detail=""
in_flight=""
stalled_lines=""
blocked_lines=""
while IFS=$'\t' read -r num labels title; do
[ -z "${num:-}" ] && continue
Expand Down Expand Up @@ -198,7 +268,18 @@ while IFS=$'\t' read -r num labels title; do
"$b"|*"/$b") has_open_pr=1; break ;;
esac
done <<< "$open_pr_branches"
[ "$has_open_pr" -eq 1 ] || in_flight+="in_flight=$num"$'\n'
if [ "$has_open_pr" -eq 1 ]; then
: # already has an open PR -- never in_flight, never stalled
else
in_flight+="in_flight=$num"$'\n'
# --- stall detection (issue #98): only for genuinely in_flight issues,
# and only when at least one event exists (see the conservative
# false-positive rule in the header comment above) ---
age_min="$(last_event_age_minutes "$num")"
if [ -n "$age_min" ] && [ "$age_min" -ge "$stall_minutes" ]; then
stalled_lines+="stalled=$num age_min=$age_min"$'\n'
fi
fi
fi
done <<< "$planned"

Expand All @@ -212,6 +293,7 @@ fi
echo "planned_issues=$planned_count"
[ -n "$detail" ] && printf '%s' "$detail"
[ -n "$in_flight" ] && printf '%s' "$in_flight"
[ -n "$stalled_lines" ] && printf '%s' "$stalled_lines"
[ -n "$blocked_lines" ] && printf '%s' "$blocked_lines"
echo "advance_ready=$advance_ready"

Expand Down
91 changes: 91 additions & 0 deletions .claude/scripts/loop-census.test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,97 @@ check "(d) task-list ref alone does not gate — advance_ready=40" bash -c \
check "(d) no blocked= line emitted for a task-list-only reference" bash -c \
'! printf "%s\n" "$1" | grep -q "^blocked="' _ "$outTasklist"

# ---------------------------------------------------------------------------
# Stall detection (issue #98): loop-census.sh must emit `stalled=N age_min=M`
# for an in_flight issue whose newest events.jsonl activity (task field either
# "N" or "issue-N" -- both forms occur in real logs) is older than
# budget.stall_minutes, while leaving fresh/zero-event/has-a-PR issues alone.
#
# Fixture: four planned+module:test issues, each with its own
# feat/issue-N-* branch and NO open PR (issue 90 is the one exception, with
# an open PR, to prove "stalled branch + open PR -> NOT stalled"):
# 80 stale events (task="80", well past the threshold) -> stalled
# 81 fresh events (task="issue-81", well within the threshold) -> NOT stalled
# 82 zero events at all for this task -> NOT stalled
# (conservative false-positive rule: never kill a just-created branch)
# 90 stale events (task="90") but an OPEN PR already exists -> NOT stalled
# (never even in_flight, so never even considered for staleness)
# stall_minutes is set to 2 in this fixture's own gates.json so the test
# doesn't need to wait a real 30 minutes -- timestamps below are computed
# relative to the ACTUAL wall clock at test run time via `date -u -d`.
# ---------------------------------------------------------------------------
dirStall="$work/stall"
scriptsStall="$dirStall/.claude/scripts"
mkdir -p "$scriptsStall"
cp "$census_src" "$scriptsStall/loop-census.sh"
cp "$resolve_roots_src" "$scriptsStall/resolve-roots.sh"
cat > "$dirStall/.claude/gates.json" <<'EOF'
{
"modules": [{ "name": "test", "path": ".", "description": "", "owner": "" }],
"merge": { "baseBranch": "main" },
"budget": { "stall_minutes": 2 }
}
EOF
cat > "$scriptsStall/pr-feedback.sh" <<'EOF'
#!/usr/bin/env bash
exit 0
EOF
cat > "$scriptsStall/bot-gh.sh" <<'EOF'
#!/usr/bin/env bash
case "$1" in
repo) echo "acme/repo" ;;
pr)
if printf '%s\n' "$*" | grep -q 'headRefName'; then
printf '%s\n' "feat/issue-90-x"
else
echo 1
fi
;;
issue)
printf '80\tplanned,module:test\tStale issue eighty\n'
printf '81\tplanned,module:test\tFresh issue eighty one\n'
printf '82\tplanned,module:test\tNo-events issue eighty two\n'
printf '90\tplanned,module:test\tStale-but-has-PR issue ninety\n'
;;
*) echo "fake-bot-gh.sh: unhandled args: $*" >&2; exit 1 ;;
esac
EOF
chmod +x "$scriptsStall"/*.sh
git -C "$dirStall" init -q -b main
git -C "$dirStall" -c user.email=t@e.st -c user.name=t commit -q --allow-empty -m init
git -C "$dirStall" branch feat/issue-80-a main >/dev/null
git -C "$dirStall" branch feat/issue-81-a main >/dev/null
git -C "$dirStall" branch feat/issue-82-a main >/dev/null
git -C "$dirStall" branch feat/issue-90-x main >/dev/null

eventsStall="$work/stall-events.jsonl"
stale_ts="$(date -u -d '-45 minutes' +%Y-%m-%dT%H:%M:%SZ)"
fresh_ts="$(date -u -d '-1 minutes' +%Y-%m-%dT%H:%M:%SZ)"
{
printf '{"ts":"%s","role":"implementer","model":"sonnet","task":"80","phase":"implementing","lens":"","detail":""}\n' "$stale_ts"
printf '{"ts":"%s","role":"implementer","model":"sonnet","task":"issue-81","phase":"implementing","lens":"","detail":""}\n' "$fresh_ts"
printf '{"ts":"%s","role":"implementer","model":"sonnet","task":"90","phase":"implementing","lens":"","detail":""}\n' "$stale_ts"
} > "$eventsStall"

outStall="$(env -u GATES_FILE CLAUDE_EVENTS_FILE="$eventsStall" bash "$scriptsStall/loop-census.sh" "acme/repo")"

check "stall: stale in_flight issue 80 (task=\"80\" form) IS reported stalled" bash -c \
'printf "%s\n" "$1" | grep -q "^stalled=80 age_min="' _ "$outStall"
check "stall: fresh in_flight issue 81 (task=\"issue-81\" form) is NOT stalled" bash -c \
'! printf "%s\n" "$1" | grep -q "^stalled=81 "' _ "$outStall"
check "stall: issue 82 has zero events -> NOT stalled (conservative false-positive rule)" bash -c \
'! printf "%s\n" "$1" | grep -q "^stalled=82 "' _ "$outStall"
check "stall: issue 90 has stale events but an OPEN PR -> NOT stalled" bash -c \
'! printf "%s\n" "$1" | grep -q "^stalled=90 "' _ "$outStall"
check "stall: issue 90 with an open PR is also NOT in_flight" bash -c \
'! printf "%s\n" "$1" | grep -qx "in_flight=90"' _ "$outStall"
check "stall: exactly one stalled= line total" bash -c \
'[ "$(printf "%s\n" "$1" | grep -c "^stalled=")" -eq 1 ]' _ "$outStall"
check "stall: age_min on the stalled line is at least the 2-minute threshold" bash -c '
age="$(printf "%s\n" "$1" | sed -n "s/^stalled=80 age_min=\([0-9]*\)/\1/p")"
[ -n "$age" ] && [ "$age" -ge 2 ]
' _ "$outStall"

echo ""
if [ "$fail" -eq 0 ]; then
echo "loop-census.test.sh: PASS ($ok checks)"
Expand Down
Loading
Loading