diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index afe9eb4440..d66a97767e 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -183,6 +183,12 @@ First launch in a fresh worktree, or first ever on a machine, may show a trust o After every spawn, peek the pane within about 20 seconds. If such a dialog is showing, accept it from an active firstmate session using `FM_HOME= bin/fm-send.sh --key Enter`, or the choice the dialog requires, unless `FM_HOME` is already set to the active firstmate home; verify the brief started processing. +**Autonomy is a launch-time grant and cannot be restored from the keyboard (verified 2026-07-31, Claude Code 2.1.220).** +`bypassPermissions` is only available to a session that was started with `--dangerously-skip-permissions`, or with `--allow-dangerously-skip-permissions`, whose own help states it "enable[s] bypassing all permission checks as an option, without it being enabled by default". +A process started with neither does not offer that mode at all, so no interrupt, keystroke, or mode-cycle can put it back into bypass; `shift+tab` cycles only the modes the running process already has. +This is the durable reason a restored pane cannot be repaired remotely with a keystroke, independent of whether any given session provider transmits that key. +The only repair is to exit the agent and relaunch it with the original launch command, which is what `bin/fm-crew-relaunch.sh` replays; `stuck-crewmate-recovery` owns the procedure. + Claude renders a predicted-next-prompt suggestion as dim/faint text inside an otherwise-empty composer after a turn completes. A plain `tmux capture-pane` cannot tell that ghost text apart from typed text. Firstmate launches every claude crewmate and secondmate with `CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false`, scoped to firstmate-launched agents through `bin/fm-spawn.sh`, so it never touches the captain's global config. diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index d97ebee302..fa34260529 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -2,7 +2,7 @@ name: stuck-crewmate-recovery description: >- Agent-only playbook for stuck or missing ordinary Firstmate direct reports. - Use when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. + Use when the session-start digest reports an ordinary direct report's endpoint dead, reports a live endpoint as unfit, or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. Reconciles recorded work before escalating from targeted inspection through safe relaunch or failure. user-invocable: false metadata: @@ -12,10 +12,33 @@ metadata: # stuck-crewmate-recovery Use this playbook when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or when a direct report is stale, looping, repeatedly confused, asking a question its brief already answers, unresponsive, or when a steer failed to land. +Also use it when the digest reports a live endpoint as `fitness: unfit`. Load `harness-adapters` before sending an interrupt, exit command, resume command, or harness-specific skill invocation. The target window's harness is recorded as `harness=` in `state/.meta`. +## Alive but unable to work + +A live endpoint does not prove the worker can still act. +After a session-provider restart, a pane can come back with full conversation context, no error, and a live endpoint, yet have lost the autonomy flag it was launched with and be sitting outside its task worktree. +Such a worker stalls forever on its first tool call, waiting for an approval nobody is watching for. +`bin/fm-crew-fitness.sh ` is the owner of that verdict, and the session-start digest runs it for every live endpoint. +`bin/fm-crew-fitness.sh --all` sweeps this home when a restart is suspected mid-session. + +Repair an `unfit` worker in this order. + +1. Exit the agent with its adapter's exit command from `harness-adapters`. + The pane must fall back to its shell first, because the repair refuses to relaunch over a live agent. +2. Run `bin/fm-crew-relaunch.sh `, which replays the exact launch command recorded at spawn time in the task's own worktree. + Do not hand-assemble the relaunch: the recorded command already carries the resolved autonomy flag, model, effort, env prefixes, and encoded brief. +3. Confirm with `bin/fm-crew-fitness.sh ` that the pane reads `fit` again. + +A refusal from that script is a stop-and-investigate result. +It refuses when an agent still owns the task, when the recorded worktree is gone or is no longer its own checkout, and when no launch command was recorded, and it never allocates a replacement worktree. +It also refuses when the recorded endpoint is gone entirely rather than merely idle, because there is then no pane to relaunch into: that case is a respawn, which the dead-direct-report procedure below owns, and it refuses before writing anything so a refusal never leaves a half-applied repair behind. +`unknown` is not `unfit`: it means the check could not establish an answer, so inspect the pane before acting rather than relaunching on it. +Interrupting the harness cannot restore autonomy that was never granted at launch, so never try to fix an `unfit` pane with a keystroke; `harness-adapters` owns why. + ## Session-start reconciliation for a dead ordinary direct report This procedure covers ordinary `kind=ship` and `kind=scout` direct reports. @@ -43,6 +66,7 @@ Escalate in order: 3. If the crewmate is confused or looping, interrupt with the adapter's interrupt key, then redirect with one corrective line. For example, for a single-Escape adapter: `FM_HOME= bin/fm-send.sh --key Escape`. 4. If the crewmate is genuinely wedged after redirection, exit the agent with the adapter's exit command and relaunch with the same brief plus a `progress so far` note appended to it. + Use `bin/fm-crew-relaunch.sh --note ''` for that relaunch whenever the task recorded a `launch=` command: it replays the resolved launch in the recorded worktree and appends the note itself, so nothing has to be reassembled by hand. Genuine wedging means looping, unresponsive, repeating the same obstacle, or truly dead. A low context reading is not wedging; modern harnesses auto-compact and keep going. The worktree and commits persist, so relaunch is cheap. diff --git a/AGENTS.md b/AGENTS.md index 199d914454..a9d8025260 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,7 +90,7 @@ state/ volatile runtime signals; gitignored .turn-ended touched by turn-end hooks .grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown .kimi-turnend-token firstmate-owned Kimi hook registry token for the task; removed by teardown - .meta written by fm-spawn: window=, endpoint_task_id=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=; kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, records one canonical pr= and the forge's pr_head= when available (GitHub pull requests and GitLab merge requests; docs/gitlab-merge-watch.md); fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) + .meta written by fm-spawn: window=, endpoint_task_id=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, launch=; kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, records one canonical pr= and the forge's pr_head= when available (GitHub pull requests and GitLab merge requests; docs/gitlab-merge-watch.md); fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) .herdr-presentation quarantinable attempt and restart-binding journal for Herdr's optional visual projection; never task or endpoint authority; see docs/herdr-backend.md "Optional presentation spaces" .check.sh authenticated slow poll; the watcher dispatches validated PR data and the byte-identified X shim through trusted repository scripts, runs registered custom checks from hash-validated private snapshots, and rejects every other state check without execution .check-trust private content binding created by fm-check-register.sh for an intentional custom check @@ -147,6 +147,8 @@ A lock-refused session must not spawn, steer, merge, drain the wake queue, repai A file that does not exist prints an explicit `ABSENT` marker, never confused with an empty-but-present file: absence is meaningful (`captain.md` absent means use the firstmate repo's built-in defaults, `projects.md` absent means rebuild it from the clones under `projects/`, etc.). 5. **Fleet-state digest** - the compact backlog listing owned by `bin/fm-session-start.sh`; every `state/.meta`; a bounded tail of each task's `state/.status` (labeled as wake-EVENT history, not current state, with the full log path printed for a deeper read); the `state/.afk` flag; and one cheap alive/dead read of each task's recorded backend endpoint. That liveness line is a fast presence check only, not a full state read - when you need a crew's actual current state (a run-step, not just "is the pane there"), read it with `bin/fm-crew-state.sh ` as before; the digest deliberately skips that deeper, slower read for every task so it stays fast and bounded. + Every alive endpoint additionally prints one `fitness:` line from `bin/fm-crew-fitness.sh`, because a pane that came back from a session-provider restart without its autonomy grant or outside its worktree still reads as alive while being unable to work. + `unfit` routes to `stuck-crewmate-recovery` (section 5), while `unknown` only means the check could not establish an answer, so inspect that pane rather than repairing it. 6. **Supervision operating instructions and next step** - after the wake queue and before context, the digest emits exactly one operating block for the detected primary harness. The closing reminder points back to that emitted block and preserves only the lock, afk, X-mode, and read-once reminders. The script itself never starts supervision; the emitted harness protocol owns the exact wait or wake mechanism. @@ -191,7 +193,7 @@ Honor lock-refused read-only mode exactly as section 3 requires. Treat digest status tails as wake-event history and use targeted current-state reconciliation when the live state matters. Reconcile only this home's recorded direct reports and their recorded backend inventory; never sweep a shared endpoint namespace for matching names or claim another home's work. -For an ordinary direct report whose endpoint is dead or metadata has no window, load `stuck-crewmate-recovery` and preserve the recorded worktree and unlanded work while reconciling ownership. +For an ordinary direct report whose endpoint is dead, whose live endpoint reports `fitness: unfit`, or whose metadata has no window, load `stuck-crewmate-recovery` and preserve the recorded worktree and unlanded work while reconciling ownership. For a dead secondmate direct report, load `secondmate-provisioning` and reconcile only that secondmate, never its whole child tree from the main home. Each secondmate reconciles work already in its own home and then idles; recovery never authorizes it to invent work. @@ -494,7 +496,7 @@ These skills are not captain-invocable; load them only at their precise triggers - `firstmate-orca` - load before switching to Orca, spawning or supervising Orca-backed work, smoke-testing Orca backend behavior, debugging Orca task state, or reconciling Orca-backed task metadata. - `project-management` - load before adding, creating, removing, or initializing a project. Cloning or registering a project is add intake and uses the same trigger. -- `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. +- `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead, reports a live endpoint as `fitness: unfit`, or its metadata has no window, or after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer. - `secondmate-provisioning` - load before creating, seeding, validating, launching, handing backlog to, recovering, pushing inherited local material into, or retiring a secondmate home, and before editing `data/secondmates.md`. - `decision-hold-lifecycle` - load before treating an investigation or visual review as complete, before ending a visual review that exposed a decision, and when recording or routing the captain's answer. - `fmx-respond` - load on an `x-mention ` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the X-mode configuration blocker, on a `public-followup ...` `check:` wake or a startup-surfaced public commitment, and on any milestone or terminal wake for an X-mode-linked task before posting its completion follow-up; relevant only when X mode is on. diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 30f420d07b..061f2449c6 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -1890,6 +1890,33 @@ fm_backend_herdr_current_path() { # | jq -r '.result.pane.foreground_cwd // empty' 2>/dev/null } +# fm_backend_herdr_pane_argv_has: 0 if any argv element of the pane's FOREGROUND +# process group is EXACTLY , 1 if it is confidently absent, 2 if the +# answer cannot be established. See fm-backend.sh's fm_backend_pane_argv_has for +# the shared contract. +# +# `pane process-info` reports `.result.process_info.foreground_processes[].argv` +# as a real JSON array, so element boundaries survive intact and jq can compare +# whole elements. An absent or empty foreground_processes list is an +# undeterminable read (2), never proof of absence: a pane whose agent has not +# been attributed yet must not be reported as having lost its autonomy flag. +fm_backend_herdr_pane_argv_has() { # + local target=$1 want=$2 info n + fm_backend_herdr_target_ready "$target" || return 2 + info=$(fm_backend_herdr_cli "$FM_BACKEND_HERDR_SESSION" pane process-info \ + --pane "$FM_BACKEND_HERDR_PANE" 2>/dev/null) || return 2 + [ -n "$info" ] || return 2 + n=$(printf '%s' "$info" \ + | jq -r '[.result.process_info.foreground_processes[]? | select(.argv != null)] | length' 2>/dev/null) + case "$n" in ''|*[!0-9]*) return 2 ;; esac + [ "$n" -gt 0 ] || return 2 + if printf '%s' "$info" | jq -e --arg want "$want" \ + 'any(.result.process_info.foreground_processes[]?.argv[]?; . == $want)' >/dev/null 2>&1; then + return 0 + fi + return 1 +} + # fm_backend_herdr_send_text_line: send one line of TEXT then submit, # ATOMICALLY - mirrors tmux's `send-keys -t T text Enter`. Used for the fixed # spawn-time commands (treehouse get, the GOTMPDIR export). `pane run` types diff --git a/bin/backends/tmux.sh b/bin/backends/tmux.sh index f8da21bf0d..1a6c3d0e86 100644 --- a/bin/backends/tmux.sh +++ b/bin/backends/tmux.sh @@ -101,6 +101,84 @@ fm_backend_tmux_current_path() { # tmux display-message -p -t "$1" '#{pane_current_path}' 2>/dev/null } +# fm_backend_tmux_foreground_pids: the pids of 's FOREGROUND process +# group, one per line, or nonzero when that group cannot be resolved at all. +# tmux exposes no process formatter beyond the pane's controlling tty, so the +# tty's foreground process-group id is the only handle on "what is actually +# running in this pane right now". Shared by the argv and environment readers, +# which ask different questions about exactly the same processes. +fm_backend_tmux_foreground_pids() { # + local target=$1 tty tpgid pids + tty=$(tmux display-message -p -t "$target" '#{pane_tty}' 2>/dev/null) || return 1 + [ -n "$tty" ] || return 1 + tpgid=$(ps -t "$tty" -o tpgid= 2>/dev/null | tr -d ' ' | grep -E '^[0-9]+$' | head -1) + case "$tpgid" in ''|*[!0-9]*) return 1 ;; esac + pids=$(ps -t "$tty" -o pgid=,pid= 2>/dev/null | awk -v g="$tpgid" '$1 == g { print $2 }') + [ -n "$pids" ] || return 1 + printf '%s\n' "$pids" +} + +# fm_backend_tmux_pane_argv_has: 0 if any argv element of the pane's FOREGROUND +# process group is EXACTLY , 1 if it is confidently absent, 2 if the +# answer cannot be established. See fm-backend.sh's fm_backend_pane_argv_has for +# the shared contract and why exact-element (never substring) matching is +# mandatory. +# +# The element boundaries come from the NUL-separated /proc//cmdline of +# every process in the pane's foreground group. NUL separation is what makes +# exact-element matching possible; a space-joined `ps -o args=` fallback could +# not tell a real flag apart from a brief that merely mentions it, so a host +# without a readable /proc returns 2 (undeterminable) rather than guessing. +fm_backend_tmux_pane_argv_has() { # + local want=$2 pid found=1 pids + [ -r /proc/self/cmdline ] || return 2 + pids=$(fm_backend_tmux_foreground_pids "$1") || return 2 + for pid in $pids; do + [ -r "/proc/$pid/cmdline" ] || continue + found=0 + if tr '\0' '\n' < "/proc/$pid/cmdline" 2>/dev/null | grep -qxF -- "$want"; then + return 0 + fi + done + # found stays 1 when no foreground process was readable at all: that is an + # undeterminable read, not proof the element is absent. + [ "$found" -eq 0 ] || return 2 + return 1 +} + +# fm_backend_tmux_pane_env_has: 0 if any environment entry of the pane's +# FOREGROUND process group is EXACTLY (a whole NAME=value string), 1 if +# it is confidently absent, 2 if the answer cannot be established. See +# fm-backend.sh's fm_backend_pane_env_has for the shared contract. +# +# /proc//environ is NUL-separated exactly like cmdline, so whole entries +# survive and can be compared without any parsing of the value. It holds the +# environment the process was EXECed with, which is precisely the question: +# whether the env prefix its recorded launch command specified is still in force +# or was dropped by a restart that rebuilt the command. +# +# Verified readable for a same-uid process on Linux 6.18 (WSL2) with +# kernel.yama.ptrace_scope=1 and no hidepid mount option, against a live agent +# pane the reader did not start (docs/verification/runtime-backends.md, +# "Foreground argv and environment"). Readability is NOT universal: a host with +# no /proc at all (macOS), a hidepid=2 mount, and a process owned by another +# user each make the read fail, and every one of them returns 2 rather than +# reporting a grant absent that was merely unreadable. +fm_backend_tmux_pane_env_has() { # + local want=$2 pid found=1 pids + [ -r /proc/self/environ ] || return 2 + pids=$(fm_backend_tmux_foreground_pids "$1") || return 2 + for pid in $pids; do + [ -r "/proc/$pid/environ" ] || continue + found=0 + if tr '\0' '\n' < "/proc/$pid/environ" 2>/dev/null | grep -qxF -- "$want"; then + return 0 + fi + done + [ "$found" -eq 0 ] || return 2 + return 1 +} + # fm_backend_tmux_send_text_line: send one line of TEXT then Enter, with no # composer verification - used for the fixed spawn-time commands # (`treehouse get`, the GOTMPDIR export) that already ran this exact sequence diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index e505b99f75..a7e357b660 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -721,6 +721,31 @@ fm_backend_send_key() { # [expected-label] esac } +# fm_backend_send_text_line: send one line of TEXT and submit it, with no +# composer verification. This is the SHELL-command send - used for fixed +# commands typed at a pane's shell prompt (`treehouse get`, the GOTMPDIR export, +# a recovery `cd` plus relaunch), never for talking to a running agent. Use +# fm_backend_send_text_submit for anything that must prove it landed in an +# agent's composer. +# +# The 4th argument is the task's LABEL (`fm-`), never a session name or a +# recorded target string: zellij and cmux verify the tab/workspace title against +# it before sending, so passing anything else makes every send on those two +# backends fail. +fm_backend_send_text_line() { # [expected-label] + local backend=$1 + shift + fm_backend_source "$backend" || return 1 + case "$backend" in + tmux) fm_backend_tmux_send_text_line "$1" "$2" ;; + herdr) fm_backend_herdr_send_text_line "$1" "$2" ;; + zellij) fm_backend_zellij_send_text_line "$1" "$2" "${3:-}" ;; + orca) fm_backend_orca_send_text_line "$1" "$2" ;; + cmux) fm_backend_cmux_send_text_line "$1" "$2" "${3:-}" ;; + *) echo "error: no send-text-line implementation for backend '$backend'" >&2; return 1 ;; + esac +} + # fm_backend_send_text_submit: type text once, then submit and verify, # retrying only the submission (never retyping). Echoes the backend's # proof-carrying verdict; callers require exact empty for confirmed delivery. @@ -869,6 +894,117 @@ fm_backend_target_exists() { # [expected-label] esac } +# fm_backend_current_path: the live FOREGROUND process's working directory for +# , or empty when the backend cannot report one. The per-backend readers +# deliberately track the foreground process rather than the pane's creation-time +# cwd, because a firstmate crewmate reaches its worktree through a `treehouse +# get` subshell that the creation-time value never follows. +# +# NOT side-effect free on every backend. zellij and cmux expose no live-process +# cwd field at all, so their readers ACTIVELY PROBE: they type a marked `pwd` +# into the pane and read it back. That is only acceptable at a pane's shell +# prompt before any harness is launched, which is why this op is scoped to +# fm-spawn.sh's pre-launch worktree poll. Anything reading the cwd of a pane +# that may already be running an agent must call fm_backend_current_path_passive +# instead - typing into a live agent's composer would submit a bogus prompt. +# +# The 4th argument is the task's LABEL (`fm-`), which zellij and cmux verify +# the tab/workspace title against before probing. +fm_backend_current_path() { # [expected-label] + local backend=$1 target=$2 label=${3:-} + fm_backend_source "$backend" || return 0 + case "$backend" in + tmux) fm_backend_tmux_current_path "$target" ;; + herdr) fm_backend_herdr_current_path "$target" ;; + zellij) fm_backend_zellij_current_path "$target" "$label" ;; + cmux) fm_backend_cmux_current_path "$target" "$label" ;; + *) return 0 ;; + esac +} + +# fm_backend_current_path_passive: the same live foreground working directory, +# but ONLY from backends that can report it without sending anything into the +# pane. Empty means "no passive reader here", never "the pane has no cwd", so a +# caller must treat empty as unknown rather than as a mismatch. +# +# This is the reader for any pane that may already be running an agent. It +# supports exactly the backends that expose a live-process cwd as data - tmux's +# `#{pane_current_path}` and herdr's `.result.pane.foreground_cwd` - which is +# the same split as fm_backend_pane_argv_has, for the same reason: an answer +# that costs a keystroke in a live composer is not an answer worth having. +fm_backend_current_path_passive() { # + local backend=$1 target=$2 + fm_backend_source "$backend" || return 0 + case "$backend" in + tmux) fm_backend_tmux_current_path "$target" ;; + herdr) fm_backend_herdr_current_path "$target" ;; + *) return 0 ;; + esac +} + +# fm_backend_pane_argv_has: the single three-state contract for "is an +# argv element of the process actually running in right now". +# 0 - present as an EXACT whole argv element. +# 1 - confidently absent. +# 2 - undeterminable (unsupported backend, failed read, no attributable +# foreground process). +# +# Exact-element matching is a correctness requirement, not a style choice. A +# crewmate's argv carries its whole brief as one element, and a brief can quote +# the very flag being looked for - this task's own brief does. Substring +# matching over a flattened command line would report a stripped autonomy flag +# as still present, which is precisely the silent failure this check exists to +# catch, so any reader that cannot preserve element boundaries must return 2. +# +# Callers must treat 2 as "unknown", never as either verdict. +fm_backend_pane_argv_has() { # + local backend=$1 target=$2 element=$3 + fm_backend_source "$backend" || return 2 + case "$backend" in + tmux) fm_backend_tmux_pane_argv_has "$target" "$element" ;; + herdr) fm_backend_herdr_pane_argv_has "$target" "$element" ;; + *) return 2 ;; + esac +} + +# fm_backend_pane_env_has: the same three-state contract for "is , a +# whole NAME=value environment entry, part of the environment of the process +# actually running in right now". +# 0 - present as an EXACT whole entry. +# 1 - confidently absent. +# 2 - undeterminable (unsupported backend, failed read, no attributable +# foreground process). +# +# This exists because not every grant a launch command carries rides in argv: +# opencode's permission grant and a secondmate's home are env prefixes, and a +# restart that rebuilds the launch command drops those exactly as it drops a +# flag. Whole-entry matching is required for the same reason as argv - a value +# can contain anything, including another entry's name. +# +# Backend support is narrower than argv on purpose. tmux resolves the pane's own +# foreground pids and reads /proc//environ. herdr's `pane process-info` +# reports argv, argv0, cmdline, cwd, name, and pid, and NO environment at all, +# so herdr answers 2 - honestly undeterminable rather than falsely absent. +# +# Callers must treat 2 as "unknown", never as either verdict. +fm_backend_pane_env_has() { # + local backend=$1 target=$2 entry=$3 + # An entry must be one whole NAME=value line. A malformed entry has nothing to + # match against, and a newline inside one cannot survive the line-oriented + # comparison every reader uses, so both are left unanswered rather than + # answered wrongly. + case "$entry" in + *$'\n'*) return 2 ;; + [A-Za-z_]*=*) ;; + *) return 2 ;; + esac + fm_backend_source "$backend" || return 2 + case "$backend" in + tmux) fm_backend_tmux_pane_env_has "$target" "$entry" ;; + *) return 2 ;; + esac +} + # fm_backend_agent_state: the single recovery-grade agent/endpoint state # contract. It is deliberately richer than fm_backend_target_exists's cheap # pane-presence read and prints exactly one of: diff --git a/bin/fm-crew-fitness.sh b/bin/fm-crew-fitness.sh new file mode 100755 index 0000000000..c63ee0579f --- /dev/null +++ b/bin/fm-crew-fitness.sh @@ -0,0 +1,380 @@ +#!/usr/bin/env bash +# fm-crew-fitness.sh - can this task's recorded pane actually still do the work? +# +# Endpoint liveness answers "is the pane there". It cannot answer "is the thing +# in the pane still able to act", and those two came apart in production: a +# session-provider restart can leave a pane that is alive, has full conversation +# context, shows no error, and is nonetheless permanently unable to work. +# +# The observed mechanism (verified 2026-07-31, herdr 0.7.5, Claude Code 2.1.220): +# herdr persists only each pane's CREATION cwd in ~/.config/herdr/session.json, +# and its `[session] resume_agents_on_restore` default relaunches the agent from +# the integration-reported session ref - `claude --resume `. That resume +# command is herdr's, not firstmate's, so it carries NONE of the launch shape +# fm-spawn resolved: no autonomy flag, no model/effort, no env prefixes. The +# restored pane therefore comes back (a) without permission bypass, so it stalls +# forever on its first tool call waiting for an approval nobody is watching for, +# and (b) in the project checkout instead of the task worktree, because the +# worktree was only ever entered by a `treehouse get` subshell that the persisted +# creation cwd never followed. See docs/herdr-backend.md "Restart and liveness +# behavior" for the backend-side statement of the same facts. +# +# So this check reads the two properties that failure destroys: +# +# autonomy - every autonomy-granting flag that THIS task's recorded launch +# command specified is still an exact argv element of the process +# running in the pane right now, and every autonomy-granting env +# prefix it specified is still an exact entry in that process's +# environment. +# cwd - the pane's live foreground cwd is still the task's recorded +# worktree. +# +# The grant set is never inferred per harness. It is intersected with the +# task's own recorded `launch=`, so whatever fm-spawn.sh's launch_template +# actually used for this task is what gets enforced, and changing a template can +# never leave this check silently asserting a stale flag. +# +# Output is one stable line: +# +# fitness: (autonomy= cwd=)[ - ] +# +# Verdict rules, in order: +# 1. Either axis definitely bad (autonomy=lost or cwd=wrong) -> unfit. +# 2. Otherwise either axis unknown -> unknown. +# 3. Otherwise -> fit. +# +# A grant that rides an ENV PREFIX rather than a flag counts on the same axis, +# because a restart that rebuilds the launch command drops an env prefix exactly +# as it drops a flag. It is read where it can be read and reported `unknown` +# where it cannot; it is never assumed satisfied. +# +# It FAILS CLOSED toward unknown and never toward fit: every unreadable target, +# unsupported backend, missing record, or unattributable process is `unknown`, +# because a false `fit` is exactly the silent failure this script exists to end. +# `n/a` is not a failure - it means the recorded launch command carried no grant +# of either kind (pi), so there is nothing whose loss could be detected. +# +# Read-only and side-effect free. Exit status: 0 fit, 1 unfit, 3 unknown, +# 2 usage error. The exit status lets a caller branch without parsing the line. +set -u + +usage() { + cat <<'EOF' +usage: fm-crew-fitness.sh + fm-crew-fitness.sh --all + +Reports whether a recorded task's pane is still able to work: its harness still +carries the autonomy grants its recorded launch command specified, as flags and +as env prefixes, and it is still in the task's recorded worktree. + + report on one task + --all report on every state/*.meta in this home, one line per task, + prefixed ": "; exit 1 if any task is unfit, else 3 if any is + unknown, else 0 + +Exit: 0 fit, 1 unfit, 3 unknown, 2 usage error. +Repair an unfit pane with bin/fm-crew-relaunch.sh after exiting its agent. +EOF +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" + +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" + +# Recognized autonomy-granting flags across every verified adapter's launch +# template in bin/fm-spawn.sh. This list only has to CONTAIN the flags; it never +# maps harness -> flag, because the flag actually in force is read from the +# task's own recorded launch command. A harness that needs no grant at all (pi) +# simply matches nothing here. +AUTONOMY_FLAGS='--dangerously-skip-permissions +--dangerously-bypass-approvals-and-sandbox +--always-approve +--auto' + +# Env-prefix names in a recorded launch command that carry a grant the worker +# cannot work without. These do not appear in argv at all, so they are read +# through fm_backend_pane_env_has, which answers from the running process's own +# environment where the backend can reach it (tmux) and `undeterminable` where +# it cannot (herdr's process-info exposes no environment; a host without /proc). +# An undeterminable grant is `unknown`, never satisfied. +# +# Membership is deliberately narrow - only prefixes whose loss leaves the worker +# unable to do its job. opencode's OPENCODE_CONFIG_CONTENT IS its permission +# grant, and FM_HOME is what makes a secondmate address its own home rather than +# its parent's. Prefixes that merely tune a working agent (claude's +# CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION, pi's FM_PI_HARNESS) are not listed, +# because a task is not unfit for having lost one. +AUTONOMY_ENV_GRANTS='OPENCODE_CONFIG_CONTENT +FM_HOME' + +# real_path: physical path, or the input unchanged when it cannot be resolved. +# Worktree comparison must be physical: a symlinked project or worktree prefix +# would otherwise read as a mismatch and manufacture a false `wrong`. +real_path() { # + [ -n "$1" ] || return 0 + ( cd "$1" 2>/dev/null && pwd -P ) || printf '%s' "$1" +} + +# launch_words: the recorded launch command split on whitespace, with NO +# pathname expansion. `read -ra` rather than an unquoted `for word in $launch`, +# because a launch command legitimately contains glob characters - opencode's +# grant is literally OPENCODE_CONFIG_CONTENT='{"permission":{"*":"allow"}}' - +# and letting the shell expand one against the cwd would silently rewrite the +# very evidence being examined. +launch_words() { # + local -a words=() + read -ra words <<<"$1" + [ "${#words[@]}" -eq 0 ] || printf '%s\n' "${words[@]}" +} + +# launch_autonomy_flags: the recognized autonomy flags present as whitespace- +# delimited words in . Word matching (not substring) keeps a longer flag +# from being reported because a shorter one is its prefix. +launch_autonomy_flags() { # + local flag word + local -a words=() + while IFS= read -r word; do words+=("$word"); done < <(launch_words "$1") + while IFS= read -r flag; do + [ -n "$flag" ] || continue + for word in ${words[@]+"${words[@]}"}; do + if [ "$word" = "$flag" ]; then + printf '%s\n' "$flag" + break + fi + done + done <, whole, +# as they were recorded. Only the LEADING assignment words are considered, +# because only those are env prefixes of the command itself; a `NAME=value` +# appearing later is an argument to the harness, not part of its environment. +launch_env_grants() { # + local word name grant + local -a words=() + while IFS= read -r word; do words+=("$word"); done < <(launch_words "$1") + for word in ${words[@]+"${words[@]}"}; do + case "$word" in + [A-Za-z_]*=*) name=${word%%=*} ;; + *) break ;; + esac + while IFS= read -r grant; do + [ -n "$grant" ] || continue + [ "$name" = "$grant" ] && printf '%s\n' "$word" + done < + local v=$1 tmp + case "$v" in + \'*\') + [ "${#v}" -ge 2 ] || return 1 + v=${v#\'} + v=${v%\'} + tmp=${v//\'\\\'\'/$'\x01'} + case "$tmp" in *\'*) return 1 ;; esac + printf '%s' "${tmp//$'\x01'/\'}" + ;; + *[\'\"\\\$\`]*) return 1 ;; + *) printf '%s' "$v" ;; + esac +} + +report() { # [detail] + local verdict=$1 autonomy=$2 cwd=$3 detail=${4:-} + if [ -n "$detail" ]; then + printf 'fitness: %s (autonomy=%s cwd=%s) - %s\n' "$verdict" "$autonomy" "$cwd" "$detail" + else + printf 'fitness: %s (autonomy=%s cwd=%s)\n' "$verdict" "$autonomy" "$cwd" + fi + case "$verdict" in + fit) return 0 ;; + unfit) return 1 ;; + *) return 3 ;; + esac +} + +check_one() { # + local id=$1 meta backend target window worktree launch + local autonomy=unknown cwd=unknown detail='' flag rc live live_real want_real + local -a lost=() + + meta="$STATE/$id.meta" + [ -f "$meta" ] || { report unknown unknown unknown "no durable record for task '$id'"; return; } + + window=$(fm_meta_get "$meta" window) + worktree=$(fm_meta_get "$meta" worktree) + launch=$(fm_meta_get "$meta" launch) + backend=$(fm_backend_of_meta "$meta") + target=$(fm_backend_target_of_meta "$meta") + target=${target:-$window} + + [ -n "$target" ] || { report unknown unknown unknown "no endpoint recorded"; return; } + + # The exact recorded endpoint must be proven present BEFORE either axis is + # read, and only the recovery-grade state contract can prove it. Tmux + # silently resolves an absent target to the CALLER'S OWN active window, and + # the cheap fm_backend_target_exists probe inherits that fallback - so a task + # whose window is gone would be measured against firstmate's own pane and + # condemned as having lost its flag and left its worktree. That false + # accusation is worse than no answer: it could send a relaunch after a task + # that no longer has an endpoint at all. fm_backend_agent_state is the + # primitive that checks the exact window against a real inventory. + # + # Only `missing` and `unreadable` short-circuit. `dead` deliberately does + # NOT: a pane that fell back to a bare shell still exists, still reports + # alive to the digest's liveness probe, and is genuinely unfit - that is the + # third shape of this same failure. + case "$(fm_backend_agent_state "$backend" "$target")" in + missing) + report unknown unknown unknown "recorded endpoint $target is gone; its liveness, not its fitness, is the question" + return + ;; + unreadable) + report unknown unknown unknown "could not read the recorded endpoint $target on backend=$backend" + return + ;; + esac + + # --- autonomy axis -------------------------------------------------------- + if [ -z "$launch" ]; then + # Spawned before launch= was recorded. The flags in force cannot be known, + # and guessing them per harness is exactly the drift this design avoids. + detail="no launch command recorded (spawned before this was tracked); deterministic relaunch is unavailable for this task" + else + local -a want=() env_grants=() + local grant name value + while IFS= read -r flag; do + [ -n "$flag" ] || continue + want+=("$flag") + done < <(launch_autonomy_flags "$launch") + while IFS= read -r grant; do + [ -n "$grant" ] || continue + env_grants+=("$grant") + done < <(launch_env_grants "$launch") + if [ "${#want[@]}" -eq 0 ] && [ "${#env_grants[@]}" -eq 0 ]; then + # The recorded launch carries no grant whose loss could be detected. + autonomy=n/a + else + autonomy=ok + for flag in "${want[@]+"${want[@]}"}"; do + fm_backend_pane_argv_has "$backend" "$target" "$flag" + rc=$? + case "$rc" in + 0) ;; + 1) lost+=("$flag"); autonomy=lost ;; + # An undeterminable read cannot clear an ALREADY-PROVEN loss: one + # confidently absent grant is enough to condemn the pane. + *) [ "$autonomy" = lost ] || autonomy=unknown; break ;; + esac + done + # Env-prefix grants are read from the live process's own environment. Only + # a grant actually READ as present counts as satisfied: an undeterminable + # read is unknown, never assumed in force, because assuming it is the + # false `fit` this whole script exists to prevent. + for grant in "${env_grants[@]+"${env_grants[@]}"}"; do + name=${grant%%=*} + if ! value=$(shell_unquote "${grant#*=}"); then + [ "$autonomy" = lost ] || autonomy=unknown + [ -n "$detail" ] || detail="could not decode the recorded env-prefix grant $name= to compare it against the running process" + continue + fi + fm_backend_pane_env_has "$backend" "$target" "$name=$value" + rc=$? + case "$rc" in + 0) ;; + 1) lost+=("$name="); autonomy=lost ;; + *) + [ "$autonomy" = lost ] || autonomy=unknown + [ -n "$detail" ] || detail="could not read the running process's environment on backend=$backend, so the env-prefix grant $name= is unproven; verify it by hand or relaunch" + ;; + esac + done + fi + fi + + # --- worktree axis -------------------------------------------------------- + # PASSIVE read only. This runs against a pane that is expected to be running + # an agent, so the cwd must come from data the backend already holds. The + # general fm_backend_current_path is not usable here: on zellij and cmux it + # types a marked `pwd` into the pane, which in a live agent's composer would + # submit a bogus prompt - and this script promises to be side-effect free. + # Those backends therefore report cwd=unknown, never a guess. + if [ -z "$worktree" ]; then + cwd=unknown + else + live=$(fm_backend_current_path_passive "$backend" "$target" 2>/dev/null || true) + if [ -z "$live" ]; then + cwd=unknown + else + live_real=$(real_path "$live") + want_real=$(real_path "$worktree") + if [ "$live_real" = "$want_real" ]; then + cwd=ok + else + cwd=wrong + fi + fi + fi + + # --- verdict -------------------------------------------------------------- + if [ "$autonomy" = lost ] || [ "$cwd" = wrong ]; then + [ "$autonomy" != lost ] || detail="autonomy grant no longer in force: ${lost[*]:-}" + if [ "$cwd" = wrong ]; then + if [ -n "$detail" ]; then + detail="$detail; pane is in $live, not the recorded worktree $worktree" + else + detail="pane is in $live, not the recorded worktree $worktree" + fi + fi + report unfit "$autonomy" "$cwd" "$detail" + return + fi + if [ "$autonomy" = unknown ] || [ "$cwd" = unknown ]; then + [ -n "$detail" ] || detail="could not read the pane's process or working directory on backend=$backend" + report unknown "$autonomy" "$cwd" "$detail" + return + fi + report fit "$autonomy" "$cwd" +} + +case "${1:-}" in + ''|-h|--help) usage; [ -n "${1:-}" ] && exit 0; exit 2 ;; +esac + +if [ "$1" = --all ]; then + worst=0 + found=0 + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] || continue + found=1 + id=$(basename "$meta" .meta) + line=$(check_one "$id") + rc=$? + printf '%s: %s\n' "$id" "$line" + case "$rc" in + 1) worst=1 ;; + 3) [ "$worst" -eq 1 ] || worst=3 ;; + esac + done + [ "$found" -eq 1 ] || printf '(no recorded tasks)\n' + exit "$worst" +fi + +check_one "$1" diff --git a/bin/fm-crew-relaunch.sh b/bin/fm-crew-relaunch.sh new file mode 100755 index 0000000000..6502ebe8eb --- /dev/null +++ b/bin/fm-crew-relaunch.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +# fm-crew-relaunch.sh - relaunch a recorded task in its own worktree, exactly as +# it was originally launched. +# +# This exists because the correct relaunch used to be assembled from memory +# during an incident: the right env prefixes, the autonomy flag, the recorded +# model and effort, the encoded brief, and a `cd` into the right worktree, all +# reconstructed by hand while a crashed fleet sat idle. Every one of those is +# already decided at spawn time, so fm-spawn.sh records the resolved command as +# `launch=` in state/.meta and this script replays it. Nothing is re-derived. +# +# Typical use: a session-provider restart brought a pane back alive but unable to +# work (see bin/fm-crew-fitness.sh for the mechanism and detection). Exit that +# pane's agent with the adapter's exit command from the harness-adapters skill, +# then run this. +# +# SAFETY - this script refuses rather than risking the task: +# * It NEVER relaunches over a live agent. Only the recovery-grade `dead` +# endpoint state licenses a relaunch - a pane that still exists but has +# fallen back to its shell. `alive`, `ambiguous`, `unreadable`, and +# `unverified` all refuse, so two agents can never end up owning one task. +# Exit the agent first - do not force past this. +# * It NEVER pretends to repair a vanished endpoint. `missing` means the +# recorded pane is authoritatively gone, so there is nothing to relaunch +# INTO: that is a respawn, which owns worktree and endpoint allocation and +# is not this script's job. It refuses before touching anything, so a +# refusal never leaves a half-applied repair behind. +# * It NEVER allocates a worktree. It relaunches into the recorded worktree or +# it refuses, because allocating a second worktree for a task whose first one +# is unaccounted for is how one task becomes two diverging copies. +# * It NEVER touches the worktree's contents: no checkout, no clean, no stash, +# no reset. Uncommitted work is preserved by not being touched at all. +# * It appends a recovery note to the brief BEFORE relaunching, so the agent +# builds on the work already in the worktree instead of starting over. +# +# Read `bin/fm-crew-fitness.sh --help` for the detection side of the same +# failure. Exit status: 0 relaunched, 1 refused (with the reason on stderr), +# 2 usage error. +set -u + +usage() { + cat <<'EOF' +usage: fm-crew-relaunch.sh [--note ] [--dry-run] + +Replays the task's recorded launch command in its recorded worktree, after +proving no live agent still owns it. + + --note recovery note appended to data//brief.md before relaunch + (default: a generic interrupted-and-resumed note) + --dry-run print what would be sent and exit without sending or writing + +Refuses unless the recorded endpoint is dead - present, but with no agent left +running in it. Exit the agent with its adapter's exit command first (see the +harness-adapters skill). An endpoint that is gone entirely needs a respawn, not +a relaunch. +EOF +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" + +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" + +die() { printf 'error: %s\n' "$1" >&2; exit 1; } + +ID= +NOTE= +DRY_RUN=0 +while [ "$#" -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --note) [ "$#" -ge 2 ] || { usage >&2; exit 2; }; NOTE=$2; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -*) usage >&2; exit 2 ;; + *) [ -z "$ID" ] || { usage >&2; exit 2; }; ID=$1; shift ;; + esac +done +[ -n "$ID" ] || { usage >&2; exit 2; } + +META="$STATE/$ID.meta" +[ -f "$META" ] || die "no durable record for task '$ID' ($META)" + +LAUNCH=$(fm_meta_get "$META" launch) +WORKTREE=$(fm_meta_get "$META" worktree) +WINDOW=$(fm_meta_get "$META" window) +BACKEND=$(fm_backend_of_meta "$META") +TARGET=$(fm_backend_target_of_meta "$META") +TARGET=${TARGET:-$WINDOW} + +[ -n "$LAUNCH" ] || die "task '$ID' has no recorded launch command; it was spawned before launch= was tracked, so a deterministic relaunch is not possible - recover it by hand through the stuck-crewmate-recovery playbook" +[ -n "$TARGET" ] || die "task '$ID' has no recorded endpoint to relaunch into" +[ -n "$WORKTREE" ] || die "task '$ID' has no recorded worktree; refusing to relaunch somewhere else" + +# --- worktree must still be the same accounted-for worktree ------------------ +# Never allocate, never substitute. A missing or moved worktree is a +# stop-and-investigate result, because the task's uncommitted work lives there. +[ -d "$WORKTREE" ] || die "recorded worktree $WORKTREE no longer exists; refusing to allocate another one for task '$ID' - investigate where its work went before relaunching" +WT_TOP=$(git -C "$WORKTREE" rev-parse --show-toplevel 2>/dev/null) || \ + die "recorded worktree $WORKTREE is not a git worktree any more; refusing to relaunch task '$ID' into it" +WT_REAL=$( cd "$WORKTREE" && pwd -P ) +TOP_REAL=$( cd "$WT_TOP" && pwd -P ) +[ "$WT_REAL" = "$TOP_REAL" ] || \ + die "recorded worktree $WORKTREE is now inside a different checkout ($WT_TOP); refusing to relaunch task '$ID'" + +# --- no live agent may still own this task ----------------------------------- +AGENT_STATE=$(fm_backend_agent_state "$BACKEND" "$TARGET") +case "$AGENT_STATE" in + dead) ;; + alive) + die "an agent is still running in task '$ID''s pane; exit it with its adapter's exit command first (harness-adapters skill), then re-run - relaunching over a live agent would put two agents on one task" + ;; + missing) + die "task '$ID''s recorded endpoint $TARGET no longer exists, so there is nothing to relaunch into; this is a respawn, not a relaunch - its worktree at $WORKTREE is untouched, so recover it through the stuck-crewmate-recovery playbook" + ;; + *) + die "cannot confirm task '$ID''s pane is free (endpoint state: $AGENT_STATE); refusing to relaunch until it reads dead" + ;; +esac + +BRIEF="$DATA/$ID/brief.md" +[ -n "$NOTE" ] || NOTE="This session was interrupted and relaunched in place. Your worktree, its commits, and its uncommitted changes are exactly as you left them. Inspect the worktree first and continue from that state - do not restart the task from the beginning." + +# Everything sent below is typed into a shell, so every recorded path must be +# shell-quoted. A worktree path holding a space or a glob character would +# otherwise `cd` somewhere else entirely and the launch command would land in +# the wrong directory while this script reported success - reproducing the exact +# failure it exists to repair. +printf -v CD_CMD 'cd %q' "$WORKTREE" + +# The pane shell is fresh after a restart, so the env fm-spawn.sh established +# around the original launch is gone with it. GOTMPDIR is the one piece the +# agent and every child process it starts inherit, and the task's own tmp root +# is already recorded, so replay it exactly as fm-spawn.sh does. +# +# The directory has to be recreated, not just named. Go does not create GOTMPDIR +# (fm-spawn.sh makes the same point where it first sets it), and the tmp root is +# /tmp/fm-, which the very reboot this script recovers from is what clears - +# so exporting the path alone would hand the agent a GOTMPDIR that does not +# exist and break its first build instead of repairing it. mkdir -p creates only +# that path and touches nothing in the recorded worktree. If it cannot be +# created the export is dropped entirely, because Go's own default temp is a +# working fallback and a dangling GOTMPDIR is not. +TASKTMP=$(fm_meta_get "$META" tasktmp) +GOTMP_CMD= +GOTMP_DIR= +if [ -n "$TASKTMP" ]; then + GOTMP_DIR="$TASKTMP/gotmp" + printf -v GOTMP_CMD 'export GOTMPDIR=%q' "$GOTMP_DIR" +fi + +# The task LABEL, not the recorded target string: zellij and cmux verify the +# tab/workspace title against this before sending. It is the same value +# fm-spawn.sh sends every spawn-time line with. +LABEL="fm-$ID" + +if [ "$DRY_RUN" -eq 1 ]; then + printf 'would append recovery note to: %s\n' "$BRIEF" + printf 'would cd pane %s (backend=%s) to: %s\n' "$TARGET" "$BACKEND" "$WORKTREE" + if [ -n "$GOTMP_CMD" ]; then + printf 'would create: %s\n' "$GOTMP_DIR" + printf 'would send env: %s\n' "$GOTMP_CMD" + else + printf 'would send no GOTMPDIR export: task has no recorded tasktmp\n' + fi + printf 'would send launch: %s\n' "$LAUNCH" + exit 0 +fi + +if [ -n "$GOTMP_CMD" ]; then + if ! mkdir -p "$GOTMP_DIR" 2>/dev/null; then + printf 'warning: could not create %s; relaunching without the GOTMPDIR export so the agent falls back to the default temp\n' "$GOTMP_DIR" >&2 + GOTMP_CMD= + fi +else + printf 'warning: task %s has no recorded tasktmp; relaunching without the GOTMPDIR export fm-spawn.sh normally sets\n' "$ID" >&2 +fi + +# Append the recovery note BEFORE relaunching: the launch command re-reads the +# brief from disk, so a note written afterwards would be invisible to the agent +# it is meant for. +if [ -f "$BRIEF" ]; then + { + printf '\n## Recovery note\n\n' + printf '%s\n' "$NOTE" + } >> "$BRIEF" || die "could not append the recovery note to $BRIEF" +else + printf 'warning: no brief at %s; relaunching without a recovery note\n' "$BRIEF" >&2 +fi + +# cd first, then launch. The recorded launch command is worktree-relative in +# effect - it is the same string fm-spawn.sh sent once the pane was already +# inside the worktree - so sending it from the wrong directory is what produced +# the failure this script repairs. +fm_backend_send_text_line "$BACKEND" "$TARGET" "$CD_CMD" "$LABEL" \ + || die "could not send the directory change to task '$ID''s pane" +sleep 0.3 +if [ -n "$GOTMP_CMD" ]; then + fm_backend_send_text_line "$BACKEND" "$TARGET" "$GOTMP_CMD" "$LABEL" \ + || die "could not send the GOTMPDIR export to task '$ID''s pane" + sleep 0.3 +fi +fm_backend_send_text_line "$BACKEND" "$TARGET" "$LAUNCH" "$LABEL" \ + || die "could not send the launch command to task '$ID''s pane" + +printf 'relaunched %s harness=%s backend=%s worktree=%s\n' \ + "$ID" "$(fm_meta_get "$META" harness)" "$BACKEND" "$WORKTREE" diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index e4f21d0358..38a9c17d7f 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -40,8 +40,10 @@ # always safe, always runs. # 5. fleet digest - a compact data/backlog.md identity/metadata listing, # every state/*.meta, a bounded state/*.status tail, -# state/.afk, and a cheap per-task endpoint-liveness read: -# read-only, always runs. +# state/.afk, a cheap per-task endpoint-liveness read, +# and, for each endpoint that read as alive, one +# fm-crew-fitness.sh line (that script's header owns the +# verdict): read-only, always runs. # 6. closing reminder - prints the context-specific watcher next step; this # script points back to the emitted harness supervision # block and deliberately never arms the watcher itself. @@ -361,6 +363,14 @@ for meta in "$STATE"/*.meta; do backend=$(fm_backend_of_meta "$meta") if fm_backend_target_exists "$backend" "${target:-$window}" "fm-$id"; then printf 'endpoint: alive (backend=%s window=%s)\n' "$backend" "$window" + # A live endpoint is not proof the pane can still work. After a + # session-provider restart a pane can come back alive, with full context + # and no error, yet stripped of its autonomy flag and sitting outside its + # worktree - permanently stalled on an approval nobody is watching for. + # That failure is only ever discovered at a session start like this one, + # because the crash that causes it takes the watcher down too. Only alive + # endpoints are checked: a dead one already routes to recovery. + "$SCRIPT_DIR/fm-crew-fitness.sh" "$id" 2>/dev/null || true else printf 'endpoint: dead (backend=%s window=%s)\n' "$backend" "$window" fi diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index ec485fd241..0c135f2b37 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -111,6 +111,11 @@ # __PITURNEND__ absolute path to .pi/extensions/fm-primary-turnend-guard.ts in a pi secondmate home # __PIWATCH__ absolute path to .pi/extensions/fm-primary-pi-watch.ts in a pi secondmate home # __OPINPUT__ absolute path to the canonical operational-input encoder +# The fully resolved launch command is recorded as launch= in state/.meta. +# It is the one durable record of how a task was actually started, so recovery +# can replay it byte-for-byte instead of re-deriving flags from memory: +# bin/fm-crew-relaunch.sh replays it, and bin/fm-crew-fitness.sh reads it to +# learn which autonomy flag that task is supposed to be running with. # Verified per-harness turn-end hooks are installed automatically where enabled; some live outside the worktree. # Kimi uses one surgically installed Firstmate region in $HOME/.kimi-code/config.toml, # a firstmate-owned global hook and registry, and a gitignored per-task pointer. @@ -495,6 +500,14 @@ launch_template() { case "$ARG3" in *' '*) # raw launch command (unverified-adapter escape hatch) + # A launch command is one command line. A newline in this caller-supplied + # string could neither be typed into the pane as one command nor recorded as + # one launch= line: it would write extra lines into state/.meta, and + # fm_meta_get's tail -1 would then prefer an injected trailing worktree= or + # window= over the genuine one for every consumer, teardown included. + case "$ARG3" in + *$'\n'*) echo "error: the raw launch command contains a newline; a launch command must be a single command line" >&2; exit 1 ;; + esac LAUNCH=$ARG3 HARNESS="" for word in $LAUNCH; do @@ -1208,21 +1221,10 @@ esac # worktree-detection steps below must never reference an unbound WT_TARGET under set -u. : "${WT_TARGET:=$T}" spawn_send_text_line() { # - case "$BACKEND" in - tmux) fm_backend_tmux_send_text_line "$1" "$2" ;; - herdr) fm_backend_herdr_send_text_line "$1" "$2" ;; - zellij) fm_backend_zellij_send_text_line "$1" "$2" "$W" ;; - orca) fm_backend_orca_send_text_line "$1" "$2" ;; - cmux) fm_backend_cmux_send_text_line "$1" "$2" "$W" ;; - esac + fm_backend_send_text_line "$BACKEND" "$1" "$2" "$W" } spawn_current_path() { # - case "$BACKEND" in - tmux) fm_backend_tmux_current_path "$1" ;; - herdr) fm_backend_herdr_current_path "$1" ;; - zellij) fm_backend_zellij_current_path "$1" "$W" ;; - cmux) fm_backend_cmux_current_path "$1" "$W" ;; - esac + fm_backend_current_path "$BACKEND" "$1" "$W" } spawn_send_literal() { # case "$BACKEND" in @@ -1679,6 +1681,26 @@ if [ "$KIND" = secondmate ]; then sq_primary_home=$(shell_quote "$FM_HOME") LAUNCH="FM_ROOT_OVERRIDE= FM_STATE_OVERRIDE= FM_DATA_OVERRIDE= FM_PROJECTS_OVERRIDE= FM_CONFIG_OVERRIDE= FM_PUBLIC_FOLLOWUP_PRIMARY_HOME=$sq_primary_home FM_HOME=$sq_home $LAUNCH" fi +# Record the EXACT resolved launch command, after every placeholder, env prefix, +# and harness/model/effort decision above has been applied. Recovery must be able +# to relaunch a task byte-identically without re-deriving any of that from memory +# under pressure; bin/fm-crew-relaunch.sh is its only consumer, and +# bin/fm-crew-fitness.sh reads it to learn which autonomy flag THIS task was +# actually launched with. Appended rather than folded into the block above +# because the launch string is not fully resolved until here, matching how +# fm-pr-check and fm-x-link already extend a task's metadata after the fact. +# Exactly one line. A launch string carrying a newline - a raw launch command is +# rejected for it above, but a substituted path could still hold one - would +# write extra lines here, and fm_meta_get's tail -1 would then prefer an +# injected trailing worktree= or window= over the genuine one for every +# consumer, teardown included. Corrupting the record is worse than not having +# it: recording nothing leaves fitness reading unknown and relaunch refusing, +# which is where both already fail closed. +case "$LAUNCH" in + *$'\n'*) echo "warning: the resolved launch command contains a newline; not recording launch= for $ID (deterministic relaunch will be unavailable for this task)" >&2 ;; + *) printf 'launch=%s\n' "$LAUNCH" >> "$STATE/$ID.meta" ;; +esac + # Export GOTMPDIR into the crewmate's pane shell so the agent and every child # process (go build, go test, ...) inherit it. Sent before the launch command so # the env is set when the agent starts; the brief sleep lets the export land. diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 4765427a3b..5f1672a76b 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -163,6 +163,7 @@ family_for_basename() { printf '%s\n' live-harness-optin ;; fm-backend-herdr.test.sh|fm-backend-tmux-smoke.test.sh|fm-backend.test.sh|\ + fm-crew-fitness.test.sh|\ fm-herdr-session-cleanup.test.sh|fm-send-strict.test.sh|fm-spawn-batch.test.sh|\ fm-spawn-dispatch-profile.test.sh|fm-spawn-worktree-settle.test.sh|\ fm-teardown-endpoint-safety.test.sh) diff --git a/docs/configuration.md b/docs/configuration.md index e035870970..e03ff8c017 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -62,6 +62,7 @@ Any value other than `tmux`, `herdr`, `zellij`, `orca`, or `cmux` is rejected un The session-start secondmate liveness sweep uses the recovery-grade `fm_backend_agent_state` classifier where verified. The comment above that function in `bin/fm-backend.sh` is the single owner of its detailed state contract and recovery authorization. The compatibility helper `fm_backend_agent_alive` continues to collapse those detailed results to `alive`, `dead`, or `unknown` for older callers. +The session-start digest additionally runs `bin/fm-crew-fitness.sh` for every live endpoint, which needs a live pane's argv, environment, or foreground working directory; only tmux and herdr can report any of those, so a task on zellij, Orca, or cmux always reads `unknown` there, and the tmux and herdr guides' limits own their remaining blind spots. A herdr spawn additionally version-gates against the installed `herdr` binary's protocol and requires `jq`, refusing loudly on an incompatible or missing installation. A zellij spawn additionally version-gates against the installed `zellij` binary's version and requires `jq`, refusing loudly when either is missing or the version is older than 0.44. A cmux spawn additionally version-gates against the installed `cmux` binary's version, requires `jq`, and requires the control socket to be reachable and accessible (see [`docs/cmux-backend.md`](cmux-backend.md) "Setup" for the one-time socket-access configuration this needs; Automation mode is the recommended socket control mode, with Password mode supported via `config/cmux-socket-password`), refusing loudly and non-retryably on a `cmuxOnly`/unauthenticated socket. diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index 4bd10f6dfb..ed672b8ea0 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -216,6 +216,13 @@ No Herdr-specific copy of that protocol exists. ## Restart and liveness behavior +Herdr restores a stopped server's layout from its own persisted session state, which records each pane's CREATION cwd and no command. +Its `[session] resume_agents_on_restore` setting, on by default, then relaunches supported agents from the integration-reported session ref rather than from the command that originally started them. +For a Firstmate crewmate both halves are lossy: the resume command is Herdr's own, so it carries none of the autonomy flag, model, effort, or env prefixes `bin/fm-spawn.sh` resolved, and the restored pane opens in the project checkout because the task worktree was only ever entered by a `treehouse get` subshell that the persisted creation cwd never followed. +The result is a pane that is alive, has full conversation context, shows no error, and is permanently unable to work. +`bin/fm-crew-fitness.sh` detects exactly that pair of losses and the session-start digest runs it for every live endpoint; `bin/fm-crew-relaunch.sh` repairs it by replaying the recorded launch command. +A stale session ref instead leaves a bare shell where the resume command failed, which the husk classifier below already covers. + Stopping and restarting a named Herdr server preserves workspace, tab, pane, and label ids, but the underlying harness processes and live agent registrations do not survive. A restored same-labeled tab with a missing pane or no registered agent is a husk. Create replaces only a confidently dead or no-agent husk, creates the replacement before closing the old tab, and refuses live or unknown states. @@ -278,6 +285,7 @@ Tests use thin compatibility wrappers in `tests/herdr-test-safety.sh` and never - A Firstmate outside Herdr cannot resolve a launcher workspace, so a colliding home label refuses new spawns until the collision is cleared. - Ghost and placeholder recognition depends on ANSI de-emphasis and fails safely to pending when unavailable. - Mid-session secondmate liveness is not implemented. +- `pane process-info` exposes argv but no environment, so `bin/fm-crew-fitness.sh` can check a flag grant on this backend but reports its autonomy axis `unknown` for a task whose grant rides an env prefix instead (OpenCode's permission grant, and a secondmate's own home). - OpenCode 1.18.4 can accept Enter while busy without clearing the composer. The tmux backend has a busy-queue fallback, but Herdr still reports this case as submit pending and needs a separate adapter fix. - Only tmux and Herdr can host the away-mode supervisor terminal. diff --git a/docs/scripts.md b/docs/scripts.md index 767980b75a..32d3fc75ef 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -64,6 +64,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-supervisor-target-lib.sh` | Resolve the shared supervisor target and backend for the daemon and launcher | | `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery | | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | +| `fm-crew-fitness.sh` | Report whether a live pane still carries its recorded launch command's autonomy grants and worktree | +| `fm-crew-relaunch.sh` | Replay a task's recorded launch command in its recorded worktree, only into a pane whose agent has exited | | `fm-tangle-lib.sh` | Shared default-branch resolution and primary-checkout tangle classification | | `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | | `fm-ff-lib.sh` | Shared guarded fast-forward helper for origin pulls and local secondmate syncs | diff --git a/docs/tmux-backend.md b/docs/tmux-backend.md index 936e0e7b47..32c9c42f24 100644 --- a/docs/tmux-backend.md +++ b/docs/tmux-backend.md @@ -81,6 +81,7 @@ Ambiguous pending text never receives the busy-queue conversion. - tmux is the reference path and supports secondmate homes. - The OpenCode busy-queue exception is tmux-specific; Herdr retains its separately documented gap. +- The crewmate-fitness argv and environment reads resolve a pane's foreground processes through `/proc`, so on a host without a readable one - macOS, or a `hidepid=2` mount - `bin/fm-crew-fitness.sh` reports its autonomy axis `unknown` rather than guessing, and only its recorded-worktree axis can still condemn a pane there; [`verification/runtime-backends.md`](verification/runtime-backends.md#foreground-argv-and-environment) owns the evidence and boundary. ```sh tests/fm-backend-tmux-smoke.test.sh diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 40b8aac86a..10ed6ecdb7 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -117,6 +117,39 @@ Valid cleanup removed only the exact task-bound target and left the control wind The metadata-only validation covers tmux, Herdr, Zellij, Orca, and cmux before backend dispatch. Claude, Codex, OpenCode, Pi, pi-signed, Grok, and Kimi share that backend cleanup boundary; their harness-specific hook files and token cleanup run only after it, so no harness needs a separate endpoint parser. +### Foreground argv and environment + +`fm_backend_tmux_pane_argv_has` and `fm_backend_tmux_pane_env_has` answer whether an exact argv element or an exact `NAME=value` environment entry is still in force in a live pane, which is what `bin/fm-crew-fitness.sh` reads to detect a worker that came back from a restart without the autonomy grant it was launched with. +tmux exposes no argv or environment formatter, so both resolve the pane's controlling tty, take that tty's foreground process-group id, and read the NUL-separated `/proc//cmdline` and `/proc//environ` of the processes in that group. + +The environment half was verified on 2026-07-31 on Linux 6.18.33.1 (WSL2, x86_64): + +```sh +grep ' /proc ' /proc/mounts +cat /proc/sys/kernel/yama/ptrace_scope +stat -c '%U' /proc/ +tr '\0' '\n' < /proc//environ | wc -l +MY_TEST_GRANT='{"permission":{"*":"allow"}}' bash -c 'sleep 3 & tr "\0" "\n" < /proc/$!/environ | grep -c "^MY_TEST_GRANT="' +``` + +```text +proc /proc proc rw,nosuid,nodev,noexec,noatime 0 0 +1 +konne +31 +1 +``` + +A same-uid process the reader did not start was readable with `kernel.yama.ptrace_scope=1` and no `hidepid` mount option, because that gate covers ptrace ATTACH rather than this read. +An env prefix of the `NAME=value command` form appears verbatim as a whole NUL-separated entry in the child's environment, including a JSON value carrying glob characters, so whole-entry matching needs no parsing of the value. + +Readability is deliberately NOT claimed to be universal. +A host with no `/proc` at all, which is every macOS host and therefore the captain's own primary machine, a `hidepid=2` mount, and a process owned by another user each make the read fail, and none of those three was exercised here. +Every one of them returns the shared contract's `2` (undeterminable), which `bin/fm-crew-fitness.sh` reports as `unknown`, so an unreadable grant is never reported as a grant that was lost and never counted as one still in force. +The same `2` is what the herdr adapter returns for every environment question, because its `pane process-info` exposes argv, argv0, cmdline, cwd, name, and pid and no environment at all. + +The three-state contract for both readers is exercised through the fitness and relaunch behavior in `tests/fm-crew-fitness.test.sh`, and the herdr argv reader additionally in `tests/fm-backend-herdr.test.sh`. + ## Herdr The compatibility floor is protocol 14. @@ -150,10 +183,36 @@ The CLI matrix was checked directly: | Native state | `herdr agent get ` | Working and done transitions were visible; native `busy` remains positive activity evidence, while native `idle` cannot close a turn and the adapter's semantic lifecycle decides worker state. | | Restart | guarded named-session stop then start | Workspace, tab, pane, and labels persisted; the agent process and registration did not. | | Close | `herdr pane close --session ` | The exact one-pane task tab closed; closing a final tab could remove the workspace. | +| Foreground argv | `herdr pane process-info --pane --session ` | Reported the pane's live foreground processes with `argv` as a real JSON array, so whole-element matching is possible; see "Foreground argv" below. | All destructive verification used `bin/fm-herdr-lab.sh` with a non-default `fm-lab-` name and a byte-identical default-session tripwire. No ambient `herdr server stop` command is a supported test operation. +### Foreground argv + +`fm_backend_herdr_pane_argv_has` answers whether an exact argv element is still in force in a live pane, which is what `bin/fm-crew-fitness.sh` reads to detect a worker that came back from a restart without its autonomy flag. +The claim it rests on is that `pane process-info` reports argv as a real array, because substring matching over a flattened command line could not tell a real flag apart from a brief that merely quotes one. + +Verified read-only against herdr 0.7.5, protocol 17, on 2026-07-31, with the schema and two live panes of the captain's own default session: + +```sh +herdr api schema --json | jq -c '.schemas.success_response["$defs"].PaneProcessInfoProcess' +herdr pane process-info --pane +``` + +```text +{"properties":{"argv":{"items":{"type":"string"},"type":["array","null"]},"argv0":{"type":["string","null"]},"cmdline":{"type":["string","null"]},"cwd":{"type":["string","null"]},"name":{"type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"}},"required":["pid","name"],"type":"object"} +{"id":"cli:pane:process_info","result":{"process_info":{"foreground_process_group_id":5221,"foreground_processes":[{"argv":["claude","--resume","21637ebd-8ca7-41bc-9c3b-1b8079c08ed0"],"cmdline":"claude --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0","cwd":"...","name":"claude","pid":5221}],"pane_id":"wK:p1","shell_pid":5138},"type":"pane_process_info"}} +``` + +The response envelope is `.result.process_info`, `foreground_processes[].argv` is an array of strings or null, and `argv` is nullable while `cmdline` is a flattened string, which is why the adapter reads the array and never the string. +The same schema is the reason herdr can answer nothing about a process's ENVIRONMENT: its per-process fields are exactly argv, argv0, cmdline, cwd, name, and pid, so `fm_backend_pane_env_has` returns undeterminable for herdr rather than reporting an env-prefix grant absent that it simply cannot see. +The second live pane was a firstmate crewmate launched with the full autonomy shape, whose argv carried `--dangerously-skip-permissions` as its own element alongside the whole encoded brief as a single further element. +Running the adapter's own predicate `any(.result.process_info.foreground_processes[]?.argv[]?; . == $want)` against that pair returned true for the autonomous pane and false for the `claude --resume` pane, which is exactly the restart failure shape the check exists to catch. + +The three-state contract itself is exercised deterministically against this schema in `tests/fm-backend-herdr.test.sh`. +The adapter function has not been driven end to end against a live herdr pane inside a lab session, so the evidence above is a read-only schema and output verification rather than a full E2E run. + ### Prune and respawn The real label-collision reproduction is owned by: diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index 314bc8605c..adbb707983 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -1829,6 +1829,117 @@ test_current_path_reads_cwd() { pass "fm_backend_herdr_current_path: reads pane foreground_cwd (the live running process), not the frozen creation-time cwd" } +# --- pane_argv_has (three-state exact argv-element read) --------------------- +# +# The canned responses below reproduce the shape `herdr pane process-info` +# actually returns, verified read-only against herdr 0.7.5 protocol 17 on +# 2026-07-31 and recorded in docs/verification/runtime-backends.md "Foreground +# argv": the envelope is `.result.process_info`, `foreground_processes[].argv` +# is an array of strings or null, and `cmdline` is a separate flattened string +# the adapter must never fall back to. +# +# What these cases prove is the adapter's own contract - present / confidently +# absent / undeterminable - against that shape. They cannot prove herdr still +# returns it; that is what the recorded verification evidence is for. + +# herdr_process_info: one canned pane process-info response. Each argument is a +# whole argv element of a single foreground process. +herdr_process_info() { # ... + local first=1 e + printf '{"id":"cli:pane:process_info","result":{"type":"pane_process_info","process_info":{"pane_id":"w1:p2","shell_pid":5138,"foreground_process_group_id":5221,"foreground_processes":[{"pid":5221,"name":"claude","cmdline":"' + for e in "$@"; do + [ "$first" -eq 1 ] || printf ' ' + first=0 + printf '%s' "$e" + done + printf '","argv":' + printf '%s\n' "$@" | jq -R . | jq -sc . + printf '}]}}}\n' +} + +test_pane_argv_has_finds_an_exact_element() { + local dir log resp fb status + dir="$TMP_ROOT/argv-present"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + herdr_process_info claude --dangerously-skip-permissions --model opus > "$resp/1.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 0 "$status" "an argv element that is present must report 0" + assert_contains "$(cat "$log")" $'\x1f''pane'$'\x1f''process-info'$'\x1f''--pane'$'\x1f''w1:p2' \ + "pane_argv_has did not read the requested pane's process info" + pass "fm_backend_herdr_pane_argv_has: a flag present as its own argv element reports present" +} + +test_pane_argv_has_reports_a_stripped_flag_absent() { + local dir log resp fb status + dir="$TMP_ROOT/argv-absent"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + # The exact shape a herdr restart leaves behind: the agent relaunched from the + # integration-reported session ref, carrying none of firstmate's launch shape. + herdr_process_info claude --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0 > "$resp/1.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 1 "$status" "a stripped autonomy flag must report confidently absent" + pass "fm_backend_herdr_pane_argv_has: an autonomy flag stripped by a resume reports confidently absent" +} + +test_pane_argv_has_does_not_match_inside_an_element() { + local dir log resp fb status + dir="$TMP_ROOT/argv-quoted"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + # A crewmate's whole brief travels as one argv element, and a brief can quote + # the very flag being looked for. Only whole-element equality is correct. + herdr_process_info claude --resume abc123 \ + 'the restored process no longer carried --dangerously-skip-permissions, so it stalls' > "$resp/1.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 1 "$status" "a brief that merely quotes the flag must not count as the flag" + pass "fm_backend_herdr_pane_argv_has: a brief quoting the flag is not mistaken for the flag itself" +} + +test_pane_argv_has_is_undeterminable_without_an_attributed_process() { + local dir log resp fb status + dir="$TMP_ROOT/argv-noproc"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + # No foreground process attributed yet. Reporting "absent" here would condemn + # a healthy pane, so the contract requires 2 (undeterminable). + printf '{"id":"cli:pane:process_info","result":{"type":"pane_process_info","process_info":{"pane_id":"w1:p2","foreground_processes":[]}}}\n' > "$resp/1.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 2 "$status" "an empty foreground process list must be undeterminable, never proof of absence" + pass "fm_backend_herdr_pane_argv_has: no attributable foreground process is undeterminable, not absent" +} + +test_pane_argv_has_is_undeterminable_on_a_null_argv() { + local dir log resp fb status + dir="$TMP_ROOT/argv-null"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + # herdr's schema declares argv nullable. A null argv carries no element + # boundaries at all, and cmdline is not a substitute, so it cannot answer. + printf '{"id":"cli:pane:process_info","result":{"type":"pane_process_info","process_info":{"pane_id":"w1:p2","foreground_processes":[{"pid":5221,"name":"claude","cmdline":"claude --dangerously-skip-permissions","argv":null}]}}}\n' > "$resp/1.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 2 "$status" "a null argv must be undeterminable and must never fall back to the flattened cmdline" + pass "fm_backend_herdr_pane_argv_has: a null argv is undeterminable and never falls back to cmdline" +} + +test_pane_argv_has_is_undeterminable_when_the_read_fails() { + local dir log resp fb status + dir="$TMP_ROOT/argv-fail"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + printf '1\n' > "$resp/1.exit" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_argv_has default:w1:p2 --dangerously-skip-permissions' "$ROOT" + status=$? + expect_code 2 "$status" "a failed process-info read must be undeterminable" + pass "fm_backend_herdr_pane_argv_has: a failed read is undeterminable, never a verdict" +} + # --- busy_state (semantic agent state) --------------------------------------- test_busy_state_working_maps_to_busy() { @@ -3164,6 +3275,12 @@ test_capture_preserves_pane_read_failure test_send_key_normalizes_and_targets_pane test_kill_is_best_effort test_current_path_reads_cwd +test_pane_argv_has_finds_an_exact_element +test_pane_argv_has_reports_a_stripped_flag_absent +test_pane_argv_has_does_not_match_inside_an_element +test_pane_argv_has_is_undeterminable_without_an_attributed_process +test_pane_argv_has_is_undeterminable_on_a_null_argv +test_pane_argv_has_is_undeterminable_when_the_read_fails test_busy_state_working_maps_to_busy test_busy_state_done_and_blocked_map_to_idle test_busy_state_unknown_on_no_agent diff --git a/tests/fm-crew-fitness.test.sh b/tests/fm-crew-fitness.test.sh new file mode 100755 index 0000000000..83e97e5642 --- /dev/null +++ b/tests/fm-crew-fitness.test.sh @@ -0,0 +1,464 @@ +#!/usr/bin/env bash +# tests/fm-crew-fitness.test.sh - the restart-recovery contract, end to end: +# bin/fm-crew-fitness.sh detects a pane that is alive but unable to work, and +# bin/fm-crew-relaunch.sh repairs it. +# +# The failure these guard against is a session-provider restart that brings a +# crewmate pane back looking perfectly healthy while it is silently useless: the +# harness relaunched without its autonomy flag (so it stalls on the first tool +# call waiting for an approval nobody is watching for) and outside its task +# worktree. Endpoint liveness reports that pane ALIVE, which is what let it cost +# real time twice in one day. +# +# Detection and repair share one expensive fixture - a real tmux server on a +# private socket plus a real git worktree - so they are exercised in one script +# rather than duplicating it. The tmux socket is private (`-L`) and reached +# through a PATH shim, exactly as tests/fm-backend-tmux-smoke.test.sh does, so +# this never touches the host's real sessions. +# +# The agent is a copy of bash named `claude`: that gives tmux's +# pane_current_command the value the tmux agent-state classifier needs to report +# `alive`, while /proc//cmdline carries whatever argv the case under test +# wants. No real harness is launched. +set -u + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +FAILED=0 +fail() { printf 'not ok - %s\n' "$1" >&2; FAILED=1; } +pass() { printf 'ok - %s\n' "$1"; } + +assert_contains() { #