You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Three separate loop outages share one root cause family: the systemd spawn environment differs from an interactive shell, and nothing checks it until a real driver burns a real attempt.
Each fix was correct; each bug was invisible to unit tests (they stub the environment) and surfaced only in production, days after merge, because daemon code goes live only at the next restart/re-arm.
Proposal
After every daemon start (arm, re-arm, restart, crash-bounce), before the first real ADVANCE, spawn a canary driver through the exact same run_driver path (same systemd-run/setsid branching, same env threading) with a trivial probe prompt and a short ceiling (e.g. RuntimeMaxSec=3m):
print pwd (must equal the repo root)
report the session's effective permission mode (must be bypassPermissions)
run bot-gh.sh auth status (must succeed)
Ledger it as verdict=canary result=pass|fail. On fail: do NOT advance anything — surface loudly (needs-human comment on a pinned ops issue, or cockpit banner) and keep ticking in WATCH. On pass: proceed normally.
This converts the whole "systemd env drift" bug class from burns an issue's attempt budget over hours to caught in under a minute, zero attempts spent.
Notes
The probe prompt must NOT be able to mutate repo state (read-only checks only).
Canary result should be cached per daemon start, not re-run every tick.
Problem
Three separate loop outages share one root cause family: the systemd spawn environment differs from an interactive shell, and nothing checks it until a real driver burns a real attempt.
systemd-runtransient units default WorkingDirectory to $HOME — drivers never loaded the repo's bypass-permissions profile, exitedphantom, and burned all 5 of issue Loop: dependency-aware issue selection — census must consult the blocking graph it already parses #97's attempts on 2026-07-16 (fix(loop): pin transient driver units to the repo root via --working-directory #131)Each fix was correct; each bug was invisible to unit tests (they stub the environment) and surfaced only in production, days after merge, because daemon code goes live only at the next restart/re-arm.
Proposal
After every daemon start (arm, re-arm, restart, crash-bounce), before the first real ADVANCE, spawn a canary driver through the exact same
run_driverpath (same systemd-run/setsid branching, same env threading) with a trivial probe prompt and a short ceiling (e.g.RuntimeMaxSec=3m):pwd(must equal the repo root)bypassPermissions)bot-gh.sh auth status(must succeed)Ledger it as
verdict=canary result=pass|fail. On fail: do NOT advance anything — surface loudly (needs-human comment on a pinned ops issue, or cockpit banner) and keep ticking in WATCH. On pass: proceed normally.This converts the whole "systemd env drift" bug class from burns an issue's attempt budget over hours to caught in under a minute, zero attempts spent.
Notes
needs-humanlabeling silently no-oped until 2026-07-16 because the label didn't exist in the repo (created by hand today) — the canary's failure path should fail loudly if its own escalation plumbing is missing.🤖 Generated with Claude Code