From 16404ee68103f19407eb8dd29eefe05384733e0c Mon Sep 17 00:00:00 2001 From: Konner Moshier <39997550+Moshik21@users.noreply.github.com> Date: Fri, 31 Jul 2026 10:58:49 -0700 Subject: [PATCH 1/4] fix(bin): detect and repair panes that survive a restart unable to work A session-provider restart can bring a crewmate pane back looking perfectly healthy while it is silently unable to work, and endpoint liveness reports it ALIVE. This happened twice on 2026-07-31 and stopped the fleet both times. Mechanism, verified against herdr 0.7.5 and Claude Code 2.1.220: herdr persists only each pane's CREATION cwd, and its default `[session] resume_agents_on_restore` relaunches the agent from the integration-reported session ref (`claude --resume `). That command is herdr's own, so it carries none of the launch shape fm-spawn resolved. The pane returns without its autonomy flag - stalling forever on its first tool call - and in the project checkout, because the task worktree was only ever entered by a `treehouse get` subshell the persisted creation cwd never followed. Detection: fm-crew-fitness.sh reports whether a recorded task's pane still carries the autonomy flags its own recorded launch command specified and is still in its recorded worktree. The session-start digest runs it for every live endpoint - the crash that causes this takes the watcher down too, so a session start is where it surfaces. The flag set is intersected with the task's recorded launch rather than mapped per harness, so a template change can never leave the check asserting a stale flag. Exact argv-element matching is required, not a substring scan: a brief travels in argv and can quote the very flag being looked for. Repair: fm-spawn.sh now records the fully resolved launch command as `launch=` in the task's metadata, and fm-crew-relaunch.sh replays it in the recorded worktree. It refuses to relaunch over a live agent, refuses when the recorded worktree is unaccounted for, never allocates a second worktree, and never touches worktree contents. Both fail closed toward "unknown": a false "fit" is the exact silent failure being fixed. The endpoint is proven present through the recovery-grade state contract first, because tmux silently resolves an absent target to the caller's own window - which would otherwise condemn a vanished task by measuring firstmate's own pane. Also corrects the recorded reason a keystroke cannot repair this: bypassPermissions is a launch-time grant, so a process started without it does not offer that mode at all, independent of key transmission. --- .agents/skills/harness-adapters/SKILL.md | 6 + .../skills/stuck-crewmate-recovery/SKILL.md | 24 +- AGENTS.md | 6 +- bin/backends/herdr.sh | 27 ++ bin/backends/tmux.sh | 32 ++ bin/fm-backend.sh | 62 ++++ bin/fm-crew-fitness.sh | 268 ++++++++++++++ bin/fm-crew-relaunch.sh | 146 ++++++++ bin/fm-session-start.sh | 8 + bin/fm-spawn.sh | 31 +- bin/fm-test-run.sh | 1 + docs/herdr-backend.md | 7 + tests/fm-crew-fitness.test.sh | 329 ++++++++++++++++++ 13 files changed, 930 insertions(+), 17 deletions(-) create mode 100755 bin/fm-crew-fitness.sh create mode 100755 bin/fm-crew-relaunch.sh create mode 100755 tests/fm-crew-fitness.test.sh diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index afe9eb4440..d66a97767e 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -183,6 +183,12 @@ First launch in a fresh worktree, or first ever on a machine, may show a trust o After every spawn, peek the pane within about 20 seconds. If such a dialog is showing, accept it from an active firstmate session using `FM_HOME= bin/fm-send.sh --key Enter`, or the choice the dialog requires, unless `FM_HOME` is already set to the active firstmate home; verify the brief started processing. +**Autonomy is a launch-time grant and cannot be restored from the keyboard (verified 2026-07-31, Claude Code 2.1.220).** +`bypassPermissions` is only available to a session that was started with `--dangerously-skip-permissions`, or with `--allow-dangerously-skip-permissions`, whose own help states it "enable[s] bypassing all permission checks as an option, without it being enabled by default". +A process started with neither does not offer that mode at all, so no interrupt, keystroke, or mode-cycle can put it back into bypass; `shift+tab` cycles only the modes the running process already has. +This is the durable reason a restored pane cannot be repaired remotely with a keystroke, independent of whether any given session provider transmits that key. +The only repair is to exit the agent and relaunch it with the original launch command, which is what `bin/fm-crew-relaunch.sh` replays; `stuck-crewmate-recovery` owns the procedure. + Claude renders a predicted-next-prompt suggestion as dim/faint text inside an otherwise-empty composer after a turn completes. A plain `tmux capture-pane` cannot tell that ghost text apart from typed text. Firstmate launches every claude crewmate and secondmate with `CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false`, scoped to firstmate-launched agents through `bin/fm-spawn.sh`, so it never touches the captain's global config. diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index d97ebee302..622841fee6 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -2,7 +2,7 @@ name: stuck-crewmate-recovery description: >- Agent-only playbook for stuck or missing ordinary Firstmate direct reports. - Use when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. + Use when the session-start digest reports an ordinary direct report's endpoint dead, reports a live endpoint as unfit, or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. Reconciles recorded work before escalating from targeted inspection through safe relaunch or failure. user-invocable: false metadata: @@ -12,10 +12,32 @@ metadata: # stuck-crewmate-recovery Use this playbook when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or when a direct report is stale, looping, repeatedly confused, asking a question its brief already answers, unresponsive, or when a steer failed to land. +Also use it when the digest reports a live endpoint as `fitness: unfit`. Load `harness-adapters` before sending an interrupt, exit command, resume command, or harness-specific skill invocation. The target window's harness is recorded as `harness=` in `state/.meta`. +## Alive but unable to work + +A live endpoint does not prove the worker can still act. +After a session-provider restart, a pane can come back with full conversation context, no error, and a live endpoint, yet have lost the autonomy flag it was launched with and be sitting outside its task worktree. +Such a worker stalls forever on its first tool call, waiting for an approval nobody is watching for. +`bin/fm-crew-fitness.sh ` is the owner of that verdict, and the session-start digest runs it for every live endpoint. +`bin/fm-crew-fitness.sh --all` sweeps this home when a restart is suspected mid-session. + +Repair an `unfit` worker in this order. + +1. Exit the agent with its adapter's exit command from `harness-adapters`. + The pane must fall back to its shell first, because the repair refuses to relaunch over a live agent. +2. Run `bin/fm-crew-relaunch.sh `, which replays the exact launch command recorded at spawn time in the task's own worktree. + Do not hand-assemble the relaunch: the recorded command already carries the resolved autonomy flag, model, effort, env prefixes, and encoded brief. +3. Confirm with `bin/fm-crew-fitness.sh ` that the pane reads `fit` again. + +A refusal from that script is a stop-and-investigate result. +It refuses when an agent still owns the task, when the recorded worktree is gone or is no longer its own checkout, or when no launch command was recorded, and it never allocates a replacement worktree. +`unknown` is not `unfit`: it means the check could not establish an answer, so inspect the pane before acting rather than relaunching on it. +Interrupting the harness cannot restore autonomy that was never granted at launch, so never try to fix an `unfit` pane with a keystroke; `harness-adapters` owns why. + ## Session-start reconciliation for a dead ordinary direct report This procedure covers ordinary `kind=ship` and `kind=scout` direct reports. diff --git a/AGENTS.md b/AGENTS.md index 199d914454..0a720cddf1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,7 +90,7 @@ state/ volatile runtime signals; gitignored .turn-ended touched by turn-end hooks .grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown .kimi-turnend-token firstmate-owned Kimi hook registry token for the task; removed by teardown - .meta written by fm-spawn: window=, endpoint_task_id=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=; kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, records one canonical pr= and the forge's pr_head= when available (GitHub pull requests and GitLab merge requests; docs/gitlab-merge-watch.md); fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) + .meta written by fm-spawn: window=, endpoint_task_id=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, launch=; kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, records one canonical pr= and the forge's pr_head= when available (GitHub pull requests and GitLab merge requests; docs/gitlab-merge-watch.md); fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) .herdr-presentation quarantinable attempt and restart-binding journal for Herdr's optional visual projection; never task or endpoint authority; see docs/herdr-backend.md "Optional presentation spaces" .check.sh authenticated slow poll; the watcher dispatches validated PR data and the byte-identified X shim through trusted repository scripts, runs registered custom checks from hash-validated private snapshots, and rejects every other state check without execution .check-trust private content binding created by fm-check-register.sh for an intentional custom check @@ -191,7 +191,7 @@ Honor lock-refused read-only mode exactly as section 3 requires. Treat digest status tails as wake-event history and use targeted current-state reconciliation when the live state matters. Reconcile only this home's recorded direct reports and their recorded backend inventory; never sweep a shared endpoint namespace for matching names or claim another home's work. -For an ordinary direct report whose endpoint is dead or metadata has no window, load `stuck-crewmate-recovery` and preserve the recorded worktree and unlanded work while reconciling ownership. +For an ordinary direct report whose endpoint is dead, whose live endpoint reports `fitness: unfit`, or whose metadata has no window, load `stuck-crewmate-recovery` and preserve the recorded worktree and unlanded work while reconciling ownership. For a dead secondmate direct report, load `secondmate-provisioning` and reconcile only that secondmate, never its whole child tree from the main home. Each secondmate reconciles work already in its own home and then idles; recovery never authorizes it to invent work. @@ -494,7 +494,7 @@ These skills are not captain-invocable; load them only at their precise triggers - `firstmate-orca` - load before switching to Orca, spawning or supervising Orca-backed work, smoke-testing Orca backend behavior, debugging Orca task state, or reconciling Orca-backed task metadata. - `project-management` - load before adding, creating, removing, or initializing a project. Cloning or registering a project is add intake and uses the same trigger. -- `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. +- `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead, reports a live endpoint as `fitness: unfit`, or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. - `secondmate-provisioning` - load before creating, seeding, validating, launching, handing backlog to, recovering, pushing inherited local material into, or retiring a secondmate home, and before editing `data/secondmates.md`. - `decision-hold-lifecycle` - load before treating an investigation or visual review as complete, before ending a visual review that exposed a decision, and when recording or routing the captain's answer. - `fmx-respond` - load on an `x-mention ` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the X-mode configuration blocker, on a `public-followup ...` `check:` wake or a startup-surfaced public commitment, and on any milestone or terminal wake for an X-mode-linked task before posting its completion follow-up; relevant only when X mode is on. diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 30f420d07b..061f2449c6 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -1890,6 +1890,33 @@ fm_backend_herdr_current_path() { # | jq -r '.result.pane.foreground_cwd // empty' 2>/dev/null } +# fm_backend_herdr_pane_argv_has: 0 if any argv element of the pane's FOREGROUND +# process group is EXACTLY , 1 if it is confidently absent, 2 if the +# answer cannot be established. See fm-backend.sh's fm_backend_pane_argv_has for +# the shared contract. +# +# `pane process-info` reports `.result.process_info.foreground_processes[].argv` +# as a real JSON array, so element boundaries survive intact and jq can compare +# whole elements. An absent or empty foreground_processes list is an +# undeterminable read (2), never proof of absence: a pane whose agent has not +# been attributed yet must not be reported as having lost its autonomy flag. +fm_backend_herdr_pane_argv_has() { # + local target=$1 want=$2 info n + fm_backend_herdr_target_ready "$target" || return 2 + info=$(fm_backend_herdr_cli "$FM_BACKEND_HERDR_SESSION" pane process-info \ + --pane "$FM_BACKEND_HERDR_PANE" 2>/dev/null) || return 2 + [ -n "$info" ] || return 2 + n=$(printf '%s' "$info" \ + | jq -r '[.result.process_info.foreground_processes[]? | select(.argv != null)] | length' 2>/dev/null) + case "$n" in ''|*[!0-9]*) return 2 ;; esac + [ "$n" -gt 0 ] || return 2 + if printf '%s' "$info" | jq -e --arg want "$want" \ + 'any(.result.process_info.foreground_processes[]?.argv[]?; . == $want)' >/dev/null 2>&1; then + return 0 + fi + return 1 +} + # fm_backend_herdr_send_text_line: send one line of TEXT then submit, # ATOMICALLY - mirrors tmux's `send-keys -t T text Enter`. Used for the fixed # spawn-time commands (treehouse get, the GOTMPDIR export). `pane run` types diff --git a/bin/backends/tmux.sh b/bin/backends/tmux.sh index f8da21bf0d..ad88ef099e 100644 --- a/bin/backends/tmux.sh +++ b/bin/backends/tmux.sh @@ -101,6 +101,38 @@ fm_backend_tmux_current_path() { # tmux display-message -p -t "$1" '#{pane_current_path}' 2>/dev/null } +# fm_backend_tmux_pane_argv_has: 0 if any argv element of the pane's FOREGROUND +# process group is EXACTLY , 1 if it is confidently absent, 2 if the +# answer cannot be established. See fm-backend.sh's fm_backend_pane_argv_has for +# the shared contract and why exact-element (never substring) matching is +# mandatory. +# +# tmux exposes no argv formatter, so this resolves the pane's controlling tty, +# reads that tty's foreground process-group id, and reads the NUL-separated +# /proc//cmdline of every process in that group. NUL separation is what +# makes exact-element matching possible; a space-joined `ps -o args=` fallback +# could not tell a real flag apart from a brief that merely mentions it, so a +# host without a readable /proc returns 2 (undeterminable) rather than guessing. +fm_backend_tmux_pane_argv_has() { # + local target=$1 want=$2 tty tpgid pid found=1 + [ -r /proc/self/cmdline ] || return 2 + tty=$(tmux display-message -p -t "$target" '#{pane_tty}' 2>/dev/null) || return 2 + [ -n "$tty" ] || return 2 + tpgid=$(ps -t "$tty" -o tpgid= 2>/dev/null | tr -d ' ' | grep -E '^[0-9]+$' | head -1) + case "$tpgid" in ''|*[!0-9]*) return 2 ;; esac + for pid in $(ps -t "$tty" -o pgid=,pid= 2>/dev/null | awk -v g="$tpgid" '$1 == g { print $2 }'); do + [ -r "/proc/$pid/cmdline" ] || continue + found=0 + if tr '\0' '\n' < "/proc/$pid/cmdline" 2>/dev/null | grep -qxF -- "$want"; then + return 0 + fi + done + # found stays 1 when no foreground process was readable at all: that is an + # undeterminable read, not proof the element is absent. + [ "$found" -eq 0 ] || return 2 + return 1 +} + # fm_backend_tmux_send_text_line: send one line of TEXT then Enter, with no # composer verification - used for the fixed spawn-time commands # (`treehouse get`, the GOTMPDIR export) that already ran this exact sequence diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index e505b99f75..51097544a9 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -721,6 +721,26 @@ fm_backend_send_key() { # [expected-label] esac } +# fm_backend_send_text_line: send one line of TEXT and submit it, with no +# composer verification. This is the SHELL-command send - used for fixed +# commands typed at a pane's shell prompt (`treehouse get`, the GOTMPDIR export, +# a recovery `cd` plus relaunch), never for talking to a running agent. Use +# fm_backend_send_text_submit for anything that must prove it landed in an +# agent's composer. +fm_backend_send_text_line() { # [session] + local backend=$1 + shift + fm_backend_source "$backend" || return 1 + case "$backend" in + tmux) fm_backend_tmux_send_text_line "$1" "$2" ;; + herdr) fm_backend_herdr_send_text_line "$1" "$2" ;; + zellij) fm_backend_zellij_send_text_line "$1" "$2" "${3:-}" ;; + orca) fm_backend_orca_send_text_line "$1" "$2" ;; + cmux) fm_backend_cmux_send_text_line "$1" "$2" "${3:-}" ;; + *) echo "error: no send-text-line implementation for backend '$backend'" >&2; return 1 ;; + esac +} + # fm_backend_send_text_submit: type text once, then submit and verify, # retrying only the submission (never retyping). Echoes the backend's # proof-carrying verdict; callers require exact empty for confirmed delivery. @@ -869,6 +889,48 @@ fm_backend_target_exists() { # [expected-label] esac } +# fm_backend_current_path: the live FOREGROUND process's working directory for +# , or empty when the backend cannot report one. The per-backend readers +# deliberately track the foreground process rather than the pane's creation-time +# cwd, because a firstmate crewmate reaches its worktree through a `treehouse +# get` subshell that the creation-time value never follows. +fm_backend_current_path() { # [session] + local backend=$1 target=$2 session=${3:-} + fm_backend_source "$backend" || return 0 + case "$backend" in + tmux) fm_backend_tmux_current_path "$target" ;; + herdr) fm_backend_herdr_current_path "$target" ;; + zellij) fm_backend_zellij_current_path "$target" "$session" ;; + cmux) fm_backend_cmux_current_path "$target" "$session" ;; + *) return 0 ;; + esac +} + +# fm_backend_pane_argv_has: the single three-state contract for "is an +# argv element of the process actually running in right now". +# 0 - present as an EXACT whole argv element. +# 1 - confidently absent. +# 2 - undeterminable (unsupported backend, failed read, no attributable +# foreground process). +# +# Exact-element matching is a correctness requirement, not a style choice. A +# crewmate's argv carries its whole brief as one element, and a brief can quote +# the very flag being looked for - this task's own brief does. Substring +# matching over a flattened command line would report a stripped autonomy flag +# as still present, which is precisely the silent failure this check exists to +# catch, so any reader that cannot preserve element boundaries must return 2. +# +# Callers must treat 2 as "unknown", never as either verdict. +fm_backend_pane_argv_has() { # + local backend=$1 target=$2 element=$3 + fm_backend_source "$backend" || return 2 + case "$backend" in + tmux) fm_backend_tmux_pane_argv_has "$target" "$element" ;; + herdr) fm_backend_herdr_pane_argv_has "$target" "$element" ;; + *) return 2 ;; + esac +} + # fm_backend_agent_state: the single recovery-grade agent/endpoint state # contract. It is deliberately richer than fm_backend_target_exists's cheap # pane-presence read and prints exactly one of: diff --git a/bin/fm-crew-fitness.sh b/bin/fm-crew-fitness.sh new file mode 100755 index 0000000000..60386ed800 --- /dev/null +++ b/bin/fm-crew-fitness.sh @@ -0,0 +1,268 @@ +#!/usr/bin/env bash +# fm-crew-fitness.sh - can this task's recorded pane actually still do the work? +# +# Endpoint liveness answers "is the pane there". It cannot answer "is the thing +# in the pane still able to act", and those two came apart in production: a +# session-provider restart can leave a pane that is alive, has full conversation +# context, shows no error, and is nonetheless permanently unable to work. +# +# The observed mechanism (verified 2026-07-31, herdr 0.7.5, Claude Code 2.1.220): +# herdr persists only each pane's CREATION cwd in ~/.config/herdr/session.json, +# and its `[session] resume_agents_on_restore` default relaunches the agent from +# the integration-reported session ref - `claude --resume `. That resume +# command is herdr's, not firstmate's, so it carries NONE of the launch shape +# fm-spawn resolved: no autonomy flag, no model/effort, no env prefixes. The +# restored pane therefore comes back (a) without permission bypass, so it stalls +# forever on its first tool call waiting for an approval nobody is watching for, +# and (b) in the project checkout instead of the task worktree, because the +# worktree was only ever entered by a `treehouse get` subshell that the persisted +# creation cwd never followed. See docs/herdr-backend.md "Restart and liveness +# behavior" for the backend-side statement of the same facts. +# +# So this check reads the two properties that failure destroys: +# +# autonomy - every autonomy-granting flag that THIS task's recorded launch +# command specified is still an exact argv element of the process +# running in the pane right now. +# cwd - the pane's live foreground cwd is still the task's recorded +# worktree. +# +# The autonomy flag set is never inferred per harness. It is intersected with the +# task's own recorded `launch=`, so whatever fm-spawn.sh's launch_template +# actually used for this task is what gets enforced, and changing a template can +# never leave this check silently asserting a stale flag. +# +# Output is one stable line: +# +# fitness: (autonomy= cwd=)[ - ] +# +# Verdict rules, in order: +# 1. Either axis definitely bad (autonomy=lost or cwd=wrong) -> unfit. +# 2. Otherwise either axis unknown -> unknown. +# 3. Otherwise -> fit. +# +# It FAILS CLOSED toward unknown and never toward fit: every unreadable target, +# unsupported backend, missing record, or unattributable process is `unknown`, +# because a false `fit` is exactly the silent failure this script exists to end. +# `n/a` is not a failure - it means the recorded launch command granted autonomy +# by some means other than a recognized flag (pi needs none; opencode uses an env +# prefix), so there is no flag whose loss could be detected. +# +# Read-only and side-effect free. Exit status: 0 fit, 1 unfit, 3 unknown, +# 2 usage error. The exit status lets a caller branch without parsing the line. +set -u + +usage() { + cat <<'EOF' +usage: fm-crew-fitness.sh + fm-crew-fitness.sh --all + +Reports whether a recorded task's pane is still able to work: its harness still +carries the autonomy flags its recorded launch command specified, and it is +still in the task's recorded worktree. + + report on one task + --all report on every state/*.meta in this home, one line per task, + prefixed ": "; exit 1 if any task is unfit, else 3 if any is + unknown, else 0 + +Exit: 0 fit, 1 unfit, 3 unknown, 2 usage error. +Repair an unfit pane with bin/fm-crew-relaunch.sh after exiting its agent. +EOF +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" + +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" + +# Recognized autonomy-granting flags across every verified adapter's launch +# template in bin/fm-spawn.sh. This list only has to CONTAIN the flags; it never +# maps harness -> flag, because the flag actually in force is read from the +# task's own recorded launch command. A harness whose autonomy rides an env +# prefix (opencode's OPENCODE_CONFIG_CONTENT) or needs no grant at all (pi) +# simply matches nothing here and reports n/a. +AUTONOMY_FLAGS='--dangerously-skip-permissions +--dangerously-bypass-approvals-and-sandbox +--always-approve +--auto' + +# real_path: physical path, or the input unchanged when it cannot be resolved. +# Worktree comparison must be physical: a symlinked project or worktree prefix +# would otherwise read as a mismatch and manufacture a false `wrong`. +real_path() { # + [ -n "$1" ] || return 0 + ( cd "$1" 2>/dev/null && pwd -P ) || printf '%s' "$1" +} + +# launch_autonomy_flags: the recognized autonomy flags present as whitespace- +# delimited words in . Word matching (not substring) keeps a longer flag +# from being reported because a shorter one is its prefix. +launch_autonomy_flags() { # + local launch=$1 flag word + while IFS= read -r flag; do + [ -n "$flag" ] || continue + for word in $launch; do + if [ "$word" = "$flag" ]; then + printf '%s\n' "$flag" + break + fi + done + done < [detail] + local verdict=$1 autonomy=$2 cwd=$3 detail=${4:-} + if [ -n "$detail" ]; then + printf 'fitness: %s (autonomy=%s cwd=%s) - %s\n' "$verdict" "$autonomy" "$cwd" "$detail" + else + printf 'fitness: %s (autonomy=%s cwd=%s)\n' "$verdict" "$autonomy" "$cwd" + fi + case "$verdict" in + fit) return 0 ;; + unfit) return 1 ;; + *) return 3 ;; + esac +} + +check_one() { # + local id=$1 meta backend target window worktree launch + local autonomy=unknown cwd=unknown detail='' flag rc live live_real want_real + local -a lost=() + + meta="$STATE/$id.meta" + [ -f "$meta" ] || { report unknown unknown unknown "no durable record for task '$id'"; return; } + + window=$(fm_meta_get "$meta" window) + worktree=$(fm_meta_get "$meta" worktree) + launch=$(fm_meta_get "$meta" launch) + backend=$(fm_backend_of_meta "$meta") + target=$(fm_backend_target_of_meta "$meta") + target=${target:-$window} + + [ -n "$target" ] || { report unknown unknown unknown "no endpoint recorded"; return; } + + # The exact recorded endpoint must be proven present BEFORE either axis is + # read, and only the recovery-grade state contract can prove it. Tmux + # silently resolves an absent target to the CALLER'S OWN active window, and + # the cheap fm_backend_target_exists probe inherits that fallback - so a task + # whose window is gone would be measured against firstmate's own pane and + # condemned as having lost its flag and left its worktree. That false + # accusation is worse than no answer: it could send a relaunch after a task + # that no longer has an endpoint at all. fm_backend_agent_state is the + # primitive that checks the exact window against a real inventory. + # + # Only `missing` and `unreadable` short-circuit. `dead` deliberately does + # NOT: a pane that fell back to a bare shell still exists, still reports + # alive to the digest's liveness probe, and is genuinely unfit - that is the + # third shape of this same failure. + case "$(fm_backend_agent_state "$backend" "$target")" in + missing) + report unknown unknown unknown "recorded endpoint $target is gone; its liveness, not its fitness, is the question" + return + ;; + unreadable) + report unknown unknown unknown "could not read the recorded endpoint $target on backend=$backend" + return + ;; + esac + + # --- autonomy axis -------------------------------------------------------- + if [ -z "$launch" ]; then + # Spawned before launch= was recorded. The flags in force cannot be known, + # and guessing them per harness is exactly the drift this design avoids. + detail="no launch command recorded (spawned before this was tracked); deterministic relaunch is unavailable for this task" + else + local -a want=() + while IFS= read -r flag; do + [ -n "$flag" ] || continue + want+=("$flag") + done < <(launch_autonomy_flags "$launch") + if [ "${#want[@]}" -eq 0 ]; then + # The recorded launch grants autonomy by some means other than a + # recognized flag, so no flag exists whose loss could be detected. + autonomy=n/a + else + autonomy=ok + for flag in "${want[@]}"; do + fm_backend_pane_argv_has "$backend" "$target" "$flag" + rc=$? + case "$rc" in + 0) ;; + 1) lost+=("$flag"); autonomy=lost ;; + # An undeterminable read cannot clear an ALREADY-PROVEN loss: one + # confidently absent flag is enough to condemn the pane. + *) [ "$autonomy" = lost ] || autonomy=unknown; break ;; + esac + done + fi + fi + + # --- worktree axis -------------------------------------------------------- + if [ -z "$worktree" ]; then + cwd=unknown + else + live=$(fm_backend_current_path "$backend" "$target" "$window" 2>/dev/null || true) + if [ -z "$live" ]; then + cwd=unknown + else + live_real=$(real_path "$live") + want_real=$(real_path "$worktree") + if [ "$live_real" = "$want_real" ]; then + cwd=ok + else + cwd=wrong + fi + fi + fi + + # --- verdict -------------------------------------------------------------- + if [ "$autonomy" = lost ] || [ "$cwd" = wrong ]; then + [ "$autonomy" != lost ] || detail="autonomy flag no longer in force: ${lost[*]:-}" + if [ "$cwd" = wrong ]; then + if [ -n "$detail" ]; then + detail="$detail; pane is in $live, not the recorded worktree $worktree" + else + detail="pane is in $live, not the recorded worktree $worktree" + fi + fi + report unfit "$autonomy" "$cwd" "$detail" + return + fi + if [ "$autonomy" = unknown ] || [ "$cwd" = unknown ]; then + [ -n "$detail" ] || detail="could not read the pane's process or working directory on backend=$backend" + report unknown "$autonomy" "$cwd" "$detail" + return + fi + report fit "$autonomy" "$cwd" +} + +case "${1:-}" in + ''|-h|--help) usage; [ -n "${1:-}" ] && exit 0; exit 2 ;; +esac + +if [ "$1" = --all ]; then + worst=0 + found=0 + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] || continue + found=1 + id=$(basename "$meta" .meta) + line=$(check_one "$id") + rc=$? + printf '%s: %s\n' "$id" "$line" + case "$rc" in + 1) worst=1 ;; + 3) [ "$worst" -eq 1 ] || worst=3 ;; + esac + done + [ "$found" -eq 1 ] || printf '(no recorded tasks)\n' + exit "$worst" +fi + +check_one "$1" diff --git a/bin/fm-crew-relaunch.sh b/bin/fm-crew-relaunch.sh new file mode 100755 index 0000000000..116069f551 --- /dev/null +++ b/bin/fm-crew-relaunch.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# fm-crew-relaunch.sh - relaunch a recorded task in its own worktree, exactly as +# it was originally launched. +# +# This exists because the correct relaunch used to be assembled from memory +# during an incident: the right env prefixes, the autonomy flag, the recorded +# model and effort, the encoded brief, and a `cd` into the right worktree, all +# reconstructed by hand while a crashed fleet sat idle. Every one of those is +# already decided at spawn time, so fm-spawn.sh records the resolved command as +# `launch=` in state/.meta and this script replays it. Nothing is re-derived. +# +# Typical use: a session-provider restart brought a pane back alive but unable to +# work (see bin/fm-crew-fitness.sh for the mechanism and detection). Exit that +# pane's agent with the adapter's exit command from the harness-adapters skill, +# then run this. +# +# SAFETY - this script refuses rather than risking the task: +# * It NEVER relaunches over a live agent. Only the recovery-grade `dead` and +# `missing` endpoint states license a relaunch; `alive`, `ambiguous`, +# `unreadable`, and `unverified` all refuse, so two agents can never end up +# owning one task. Exit the agent first - do not force past this. +# * It NEVER allocates a worktree. It relaunches into the recorded worktree or +# it refuses, because allocating a second worktree for a task whose first one +# is unaccounted for is how one task becomes two diverging copies. +# * It NEVER touches the worktree's contents: no checkout, no clean, no stash, +# no reset. Uncommitted work is preserved by not being touched at all. +# * It appends a recovery note to the brief BEFORE relaunching, so the agent +# builds on the work already in the worktree instead of starting over. +# +# Read `bin/fm-crew-fitness.sh --help` for the detection side of the same +# failure. Exit status: 0 relaunched, 1 refused (with the reason on stderr), +# 2 usage error. +set -u + +usage() { + cat <<'EOF' +usage: fm-crew-relaunch.sh [--note ] [--dry-run] + +Replays the task's recorded launch command in its recorded worktree, after +proving no live agent still owns it. + + --note recovery note appended to data//brief.md before relaunch + (default: a generic interrupted-and-resumed note) + --dry-run print what would be sent and exit without sending or writing + +Refuses unless the recorded endpoint is dead or missing: exit the agent with its +adapter's exit command first (see the harness-adapters skill). +EOF +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" + +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" + +die() { printf 'error: %s\n' "$1" >&2; exit 1; } + +ID= +NOTE= +DRY_RUN=0 +while [ "$#" -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --note) [ "$#" -ge 2 ] || { usage >&2; exit 2; }; NOTE=$2; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -*) usage >&2; exit 2 ;; + *) [ -z "$ID" ] || { usage >&2; exit 2; }; ID=$1; shift ;; + esac +done +[ -n "$ID" ] || { usage >&2; exit 2; } + +META="$STATE/$ID.meta" +[ -f "$META" ] || die "no durable record for task '$ID' ($META)" + +LAUNCH=$(fm_meta_get "$META" launch) +WORKTREE=$(fm_meta_get "$META" worktree) +WINDOW=$(fm_meta_get "$META" window) +BACKEND=$(fm_backend_of_meta "$META") +TARGET=$(fm_backend_target_of_meta "$META") +TARGET=${TARGET:-$WINDOW} + +[ -n "$LAUNCH" ] || die "task '$ID' has no recorded launch command; it was spawned before launch= was tracked, so a deterministic relaunch is not possible - recover it by hand through the stuck-crewmate-recovery playbook" +[ -n "$TARGET" ] || die "task '$ID' has no recorded endpoint to relaunch into" +[ -n "$WORKTREE" ] || die "task '$ID' has no recorded worktree; refusing to relaunch somewhere else" + +# --- worktree must still be the same accounted-for worktree ------------------ +# Never allocate, never substitute. A missing or moved worktree is a +# stop-and-investigate result, because the task's uncommitted work lives there. +[ -d "$WORKTREE" ] || die "recorded worktree $WORKTREE no longer exists; refusing to allocate another one for task '$ID' - investigate where its work went before relaunching" +WT_TOP=$(git -C "$WORKTREE" rev-parse --show-toplevel 2>/dev/null) || \ + die "recorded worktree $WORKTREE is not a git worktree any more; refusing to relaunch task '$ID' into it" +WT_REAL=$( cd "$WORKTREE" && pwd -P ) +TOP_REAL=$( cd "$WT_TOP" && pwd -P ) +[ "$WT_REAL" = "$TOP_REAL" ] || \ + die "recorded worktree $WORKTREE is now inside a different checkout ($WT_TOP); refusing to relaunch task '$ID'" + +# --- no live agent may still own this task ----------------------------------- +AGENT_STATE=$(fm_backend_agent_state "$BACKEND" "$TARGET") +case "$AGENT_STATE" in + dead|missing) ;; + alive) + die "an agent is still running in task '$ID''s pane; exit it with its adapter's exit command first (harness-adapters skill), then re-run - relaunching over a live agent would put two agents on one task" + ;; + *) + die "cannot confirm task '$ID''s pane is free (endpoint state: $AGENT_STATE); refusing to relaunch until it reads dead or missing" + ;; +esac + +BRIEF="$DATA/$ID/brief.md" +[ -n "$NOTE" ] || NOTE="This session was interrupted and relaunched in place. Your worktree, its commits, and its uncommitted changes are exactly as you left them. Inspect the worktree first and continue from that state - do not restart the task from the beginning." + +if [ "$DRY_RUN" -eq 1 ]; then + printf 'would append recovery note to: %s\n' "$BRIEF" + printf 'would cd pane %s (backend=%s) to: %s\n' "$TARGET" "$BACKEND" "$WORKTREE" + printf 'would send launch: %s\n' "$LAUNCH" + exit 0 +fi + +# Append the recovery note BEFORE relaunching: the launch command re-reads the +# brief from disk, so a note written afterwards would be invisible to the agent +# it is meant for. +if [ -f "$BRIEF" ]; then + { + printf '\n## Recovery note\n\n' + printf '%s\n' "$NOTE" + } >> "$BRIEF" || die "could not append the recovery note to $BRIEF" +else + printf 'warning: no brief at %s; relaunching without a recovery note\n' "$BRIEF" >&2 +fi + +# cd first, then launch. The recorded launch command is worktree-relative in +# effect - it is the same string fm-spawn.sh sent once the pane was already +# inside the worktree - so sending it from the wrong directory is what produced +# the failure this script repairs. +fm_backend_send_text_line "$BACKEND" "$TARGET" "cd $WORKTREE" "$WINDOW" \ + || die "could not send the directory change to task '$ID''s pane" +sleep 0.3 +fm_backend_send_text_line "$BACKEND" "$TARGET" "$LAUNCH" "$WINDOW" \ + || die "could not send the launch command to task '$ID''s pane" + +printf 'relaunched %s harness=%s backend=%s worktree=%s\n' \ + "$ID" "$(fm_meta_get "$META" harness)" "$BACKEND" "$WORKTREE" diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index e4f21d0358..2cba895704 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -361,6 +361,14 @@ for meta in "$STATE"/*.meta; do backend=$(fm_backend_of_meta "$meta") if fm_backend_target_exists "$backend" "${target:-$window}" "fm-$id"; then printf 'endpoint: alive (backend=%s window=%s)\n' "$backend" "$window" + # A live endpoint is not proof the pane can still work. After a + # session-provider restart a pane can come back alive, with full context + # and no error, yet stripped of its autonomy flag and sitting outside its + # worktree - permanently stalled on an approval nobody is watching for. + # That failure is only ever discovered at a session start like this one, + # because the crash that causes it takes the watcher down too. Only alive + # endpoints are checked: a dead one already routes to recovery. + "$SCRIPT_DIR/fm-crew-fitness.sh" "$id" 2>/dev/null || true else printf 'endpoint: dead (backend=%s window=%s)\n' "$backend" "$window" fi diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index ec485fd241..505cbb3900 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -111,6 +111,11 @@ # __PITURNEND__ absolute path to .pi/extensions/fm-primary-turnend-guard.ts in a pi secondmate home # __PIWATCH__ absolute path to .pi/extensions/fm-primary-pi-watch.ts in a pi secondmate home # __OPINPUT__ absolute path to the canonical operational-input encoder +# The fully resolved launch command is recorded as launch= in state/.meta. +# It is the one durable record of how a task was actually started, so recovery +# can replay it byte-for-byte instead of re-deriving flags from memory: +# bin/fm-crew-relaunch.sh replays it, and bin/fm-crew-fitness.sh reads it to +# learn which autonomy flag that task is supposed to be running with. # Verified per-harness turn-end hooks are installed automatically where enabled; some live outside the worktree. # Kimi uses one surgically installed Firstmate region in $HOME/.kimi-code/config.toml, # a firstmate-owned global hook and registry, and a gitignored per-task pointer. @@ -1208,21 +1213,10 @@ esac # worktree-detection steps below must never reference an unbound WT_TARGET under set -u. : "${WT_TARGET:=$T}" spawn_send_text_line() { # - case "$BACKEND" in - tmux) fm_backend_tmux_send_text_line "$1" "$2" ;; - herdr) fm_backend_herdr_send_text_line "$1" "$2" ;; - zellij) fm_backend_zellij_send_text_line "$1" "$2" "$W" ;; - orca) fm_backend_orca_send_text_line "$1" "$2" ;; - cmux) fm_backend_cmux_send_text_line "$1" "$2" "$W" ;; - esac + fm_backend_send_text_line "$BACKEND" "$1" "$2" "$W" } spawn_current_path() { # - case "$BACKEND" in - tmux) fm_backend_tmux_current_path "$1" ;; - herdr) fm_backend_herdr_current_path "$1" ;; - zellij) fm_backend_zellij_current_path "$1" "$W" ;; - cmux) fm_backend_cmux_current_path "$1" "$W" ;; - esac + fm_backend_current_path "$BACKEND" "$1" "$W" } spawn_send_literal() { # case "$BACKEND" in @@ -1679,6 +1673,17 @@ if [ "$KIND" = secondmate ]; then sq_primary_home=$(shell_quote "$FM_HOME") LAUNCH="FM_ROOT_OVERRIDE= FM_STATE_OVERRIDE= FM_DATA_OVERRIDE= FM_PROJECTS_OVERRIDE= FM_CONFIG_OVERRIDE= FM_PUBLIC_FOLLOWUP_PRIMARY_HOME=$sq_primary_home FM_HOME=$sq_home $LAUNCH" fi +# Record the EXACT resolved launch command, after every placeholder, env prefix, +# and harness/model/effort decision above has been applied. Recovery must be able +# to relaunch a task byte-identically without re-deriving any of that from memory +# under pressure; bin/fm-crew-relaunch.sh is its only consumer, and +# bin/fm-crew-fitness.sh reads it to learn which autonomy flag THIS task was +# actually launched with. Appended rather than folded into the block above +# because the launch string is not fully resolved until here, matching how +# fm-pr-check and fm-x-link already extend a task's metadata after the fact. +# One line, no newlines: every launch template is a single command line. +printf 'launch=%s\n' "$LAUNCH" >> "$STATE/$ID.meta" + # Export GOTMPDIR into the crewmate's pane shell so the agent and every child # process (go build, go test, ...) inherit it. Sent before the launch command so # the env is set when the agent starts; the brief sleep lets the export land. diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 4765427a3b..5f1672a76b 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -163,6 +163,7 @@ family_for_basename() { printf '%s\n' live-harness-optin ;; fm-backend-herdr.test.sh|fm-backend-tmux-smoke.test.sh|fm-backend.test.sh|\ + fm-crew-fitness.test.sh|\ fm-herdr-session-cleanup.test.sh|fm-send-strict.test.sh|fm-spawn-batch.test.sh|\ fm-spawn-dispatch-profile.test.sh|fm-spawn-worktree-settle.test.sh|\ fm-teardown-endpoint-safety.test.sh) diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index 4bd10f6dfb..55bafbb016 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -216,6 +216,13 @@ No Herdr-specific copy of that protocol exists. ## Restart and liveness behavior +Herdr restores a stopped server's layout from its own persisted session state, which records each pane's CREATION cwd and no command. +Its `[session] resume_agents_on_restore` setting, on by default, then relaunches supported agents from the integration-reported session ref rather than from the command that originally started them. +For a Firstmate crewmate both halves are lossy: the resume command is Herdr's own, so it carries none of the autonomy flag, model, effort, or env prefixes `bin/fm-spawn.sh` resolved, and the restored pane opens in the project checkout because the task worktree was only ever entered by a `treehouse get` subshell that the persisted creation cwd never followed. +The result is a pane that is alive, has full conversation context, shows no error, and is permanently unable to work. +`bin/fm-crew-fitness.sh` detects exactly that pair of losses and the session-start digest runs it for every live endpoint; `bin/fm-crew-relaunch.sh` repairs it by replaying the recorded launch command. +A stale session ref instead leaves a bare shell where the resume command failed, which the husk classifier below already covers. + Stopping and restarting a named Herdr server preserves workspace, tab, pane, and label ids, but the underlying harness processes and live agent registrations do not survive. A restored same-labeled tab with a missing pane or no registered agent is a husk. Create replaces only a confidently dead or no-agent husk, creates the replacement before closing the old tab, and refuses live or unknown states. diff --git a/tests/fm-crew-fitness.test.sh b/tests/fm-crew-fitness.test.sh new file mode 100755 index 0000000000..44600077df --- /dev/null +++ b/tests/fm-crew-fitness.test.sh @@ -0,0 +1,329 @@ +#!/usr/bin/env bash +# tests/fm-crew-fitness.test.sh - the restart-recovery contract, end to end: +# bin/fm-crew-fitness.sh detects a pane that is alive but unable to work, and +# bin/fm-crew-relaunch.sh repairs it. +# +# The failure these guard against is a session-provider restart that brings a +# crewmate pane back looking perfectly healthy while it is silently useless: the +# harness relaunched without its autonomy flag (so it stalls on the first tool +# call waiting for an approval nobody is watching for) and outside its task +# worktree. Endpoint liveness reports that pane ALIVE, which is what let it cost +# real time twice in one day. +# +# Detection and repair share one expensive fixture - a real tmux server on a +# private socket plus a real git worktree - so they are exercised in one script +# rather than duplicating it. The tmux socket is private (`-L`) and reached +# through a PATH shim, exactly as tests/fm-backend-tmux-smoke.test.sh does, so +# this never touches the host's real sessions. +# +# The agent is a copy of bash named `claude`: that gives tmux's +# pane_current_command the value the tmux agent-state classifier needs to report +# `alive`, while /proc//cmdline carries whatever argv the case under test +# wants. No real harness is launched. +set -u + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +FAILED=0 +fail() { printf 'not ok - %s\n' "$1" >&2; FAILED=1; } +pass() { printf 'ok - %s\n' "$1"; } + +assert_contains() { #