Skip to content

Loop: post-arm canary driver — verify the spawn environment before advancing real issues #132

Description

@robercano-ghbot

Problem

Three separate loop outages share one root cause family: the systemd spawn environment differs from an interactive shell, and nothing checks it until a real driver burns a real attempt.

  1. systemd's minimal PATH lacked nvm node/claude — daemon dead July 11–13 (Loop daemon: harden runtime-env resolution — bake resolved PATH into installed units, don't depend on nvm sourcing at runtime #107, fixed 4bf7dbb/f9dcfcb)
  2. claude binary not resolvable in the RC unit (fixed e4f35f6, baked at arm time)
  3. systemd-run transient units default WorkingDirectory to $HOME — drivers never loaded the repo's bypass-permissions profile, exited phantom, and burned all 5 of issue Loop: dependency-aware issue selection — census must consult the blocking graph it already parses #97's attempts on 2026-07-16 (fix(loop): pin transient driver units to the repo root via --working-directory #131)

Each fix was correct; each bug was invisible to unit tests (they stub the environment) and surfaced only in production, days after merge, because daemon code goes live only at the next restart/re-arm.

Proposal

After every daemon start (arm, re-arm, restart, crash-bounce), before the first real ADVANCE, spawn a canary driver through the exact same run_driver path (same systemd-run/setsid branching, same env threading) with a trivial probe prompt and a short ceiling (e.g. RuntimeMaxSec=3m):

  • print pwd (must equal the repo root)
  • report the session's effective permission mode (must be bypassPermissions)
  • run bot-gh.sh auth status (must succeed)

Ledger it as verdict=canary result=pass|fail. On fail: do NOT advance anything — surface loudly (needs-human comment on a pinned ops issue, or cockpit banner) and keep ticking in WATCH. On pass: proceed normally.

This converts the whole "systemd env drift" bug class from burns an issue's attempt budget over hours to caught in under a minute, zero attempts spent.

Notes

  • The probe prompt must NOT be able to mutate repo state (read-only checks only).
  • Canary result should be cached per daemon start, not re-run every tick.
  • Related: the Loop: spend ceilings — per-tick caps, daily budget with auto-halt, stop-after self-disarm #95 escalation's needs-human labeling silently no-oped until 2026-07-16 because the label didn't exist in the repo (created by hand today) — the canary's failure path should fail loudly if its own escalation plumbing is missing.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmodule:harnessOrchestrator machinery under .claude

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions