From 84d59a5dc09565a89588b8a62ebb3e018a45d00b Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 11:35:07 +0530 Subject: [PATCH 1/4] canary: move the daily integration suite to a local box and probe the daemon path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Daily runs leave GH Actions (runner minutes were the entire cost; the LLM spend is identical either way) for a local canary box driven by a systemd user timer in the retired cron's 06:17 UTC slot. integration-suite/local/ ships the box side: run-local.sh (checkout CANARY_REF → stable leg → beta leg, flock-serialized, with a crash-guard Slack note for a leg that dies BEFORE reporting — the replacement for GHA's red-job email), install.sh (installed copy OUTSIDE the clone the wrapper hard-resets, systemd units, secrets template), and the service/timer units. The workflow keeps workflow_dispatch as the cloud fallback and loses its cron. The stable leg now probes the daemon-configured path (CANARY_DAEMON=1) — the configuration `failproofai config` gives users going forward: ci-entrypoint cross-compiles failproofaid in rust:1-bookworm (glibc-matched to the node:22-bookworm-slim sandbox; a host build can link newer symbols and fail to load inside it), run.sh bind-mounts it into the probe container, and probe-cli.sh sets the daemon.configured fail-closed marker through the real fp-config updateConfig path — shell-appending TOML could produce a duplicate [daemon] table, which parses as NOT configured and silently falls back to in-process. The daemon restarts per probe, not per CLI: the wire protocol forwards {hookEvent, cli, stdin, cwd} and never env, so the warm worker's FAILPROOFAI_HOOK_LOG_FILE is fixed at daemon start — one daemon across both probes would share one oracle dir, and probe A's incidental read-denies would satisfy probe B's grep (false PASS). A dead daemon cannot false-PASS either: its deny is shaped by the synthetic failproofai/daemon-unreachable policy, which the probes' greps never match — those probes go INCONCLUSIVE and re-probe until the daemon path recovers. Verified without LLM or secrets in the real sandbox image: live daemon → canary-bash deny through the socket → warm worker writes the per-probe oracle; killed daemon → fail-closed deny logged as daemon-unreachable, matching neither probe grep; marker cleared → in-process evaluation restored. Tripwires in __tests__/integration-suite/local-runner.test.ts pin the workflow staying cron-free, the unit↔installer paths, the secrets-template↔workflow-env parity, the per-probe daemon restarts, the marker hygiene, and the fail-closed/oracle non-overlap (both sides extracted from the real sources). Co-Authored-By: Claude Fable 5 --- .github/workflows/integration-suite.yml | 28 +-- CHANGELOG.md | 3 + .../integration-suite/local-runner.test.ts | 160 ++++++++++++++++++ integration-suite/README.md | 69 ++++++-- integration-suite/ci-entrypoint.sh | 29 ++++ .../local/failproofai-canary.service | 14 ++ .../local/failproofai-canary.timer | 13 ++ integration-suite/local/install.sh | 91 ++++++++++ integration-suite/local/run-local.sh | 116 +++++++++++++ integration-suite/probe-cli.sh | 72 ++++++++ integration-suite/run.sh | 14 +- 11 files changed, 583 insertions(+), 26 deletions(-) create mode 100644 __tests__/integration-suite/local-runner.test.ts create mode 100644 integration-suite/local/failproofai-canary.service create mode 100644 integration-suite/local/failproofai-canary.timer create mode 100755 integration-suite/local/install.sh create mode 100755 integration-suite/local/run-local.sh diff --git a/.github/workflows/integration-suite.yml b/.github/workflows/integration-suite.yml index 0780ea08..b581c277 100644 --- a/.github/workflows/integration-suite.yml +++ b/.github/workflows/integration-suite.yml @@ -1,17 +1,25 @@ name: Integration Suite -# Daily integration test: does failproofai still ENFORCE against every supported -# agent CLI @latest? Installs all 12 CLIs into an isolated Docker sandbox, drives -# each one against failproofai's OWN policies (built from THIS repo's HEAD), and -# asserts the hook log shows a DENY. A silent-allow — a blocked action that ran -# with no deny — means enforcement broke against that CLI (e.g. a vendor changed -# their hook schema out from under us), and turns the run red. Reports only -# CHANGES (broke/recovered) plus a daily heartbeat to Slack. +# ON-DEMAND FALLBACK for the integration suite: does failproofai still ENFORCE +# against every supported agent CLI @latest? Installs all 12 CLIs into an +# isolated Docker sandbox, drives each one against failproofai's OWN policies +# (built from THIS repo's HEAD), and asserts the hook log shows a DENY. A +# silent-allow — a blocked action that ran with no deny — means enforcement +# broke against that CLI (e.g. a vendor changed their hook schema out from +# under us), and turns the run red. Reports only CHANGES (broke/recovered) +# plus a heartbeat to Slack. +# +# The DAILY runs moved off Actions to a local canary box for cost — +# integration-suite/local/ carries the systemd timer + wrapper that replaced +# the cron that used to live here (same 06:17 UTC slot). This workflow stays +# dispatch-only: the cloud escape hatch for when the box is down or a clean +# cloud reproduction is wanted. Its Actions-cache state is separate from the +# box's state dir, so a dispatch may re-probe CLIs the box already gated. # # Unlike the unit/e2e suites, this drives REAL vendor CLIs against real gateway # models, so it needs credentials. They live in the `cli-integration` Environment -# and the workflow runs ONLY on schedule / manual dispatch — never on pull_request -# — so fork PRs can never reach the secrets. +# and the workflow runs ONLY on manual dispatch — never on pull_request — so +# fork PRs can never reach the secrets. # # This file is a THIN TRIGGER on purpose. Everything beyond the GitHub-specific # wiring (checkout, bun, cache, secret->env mapping) lives in @@ -19,8 +27,6 @@ name: Integration Suite # without opening this YAML. See integration-suite/README.md. on: - schedule: - - cron: "17 6 * * *" # ~06:17 UTC daily workflow_dispatch: inputs: clis: diff --git a/CHANGELOG.md b/CHANGELOG.md index dc682a92..ea2fbb7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## 1.0.0-beta.11 — 2026-08-07 +### Features +- Move the daily CLI integration suite off GH Actions onto a local canary box, and make it probe the daemon path. `integration-suite/local/` ships the box side — a wrapper that checks out `CANARY_REF`, runs both legs through the unchanged `ci-entrypoint.sh`, and Slack-notes any leg that dies *before* reporting (the replacement for GHA's red-job email) — plus an installer and a systemd user timer holding the same 06:17 UTC slot; the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) + ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) - Make the Rust daemon enforce the same cloud-URL rule the TS side does. `CloudClient::new()` checked only that the scheme was `http` or `https`, so `http://internal-host` was accepted and `spawn_maintenance()` then put the org-scoped `policies:pull` bearer token on the wire **in clear, every 30 seconds**. `validateCloudUrl()` in `cloud-enrollment.ts` has always blocked non-loopback `http`, and `configure-wizard.ts` carries a comment asserting the daemon enforces the same rule — it did not. It matters most on the path the TS validator cannot cover: `FAILPROOFAI_CLOUD_URL` takes precedence over the credentials file and is a documented CI/container knob, so it reaches the constructor without passing through the wizard. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts new file mode 100644 index 00000000..b882b7e0 --- /dev/null +++ b/__tests__/integration-suite/local-runner.test.ts @@ -0,0 +1,160 @@ +/** + * Tripwires for the LOCAL canary runner (integration-suite/local/) and the + * daemon-mode (CANARY_DAEMON) probe path. + * + * Daily integration-suite runs moved off GH Actions onto a local box + * (2026-08-07, for runner-minute cost); the stable leg there probes the + * daemon-configured (failproofaid) hook path — the way-forward configuration. + * Everything below is shell scripts and systemd units with no importable + * surface, so the tests parse the real files — same approach as + * channel-refs.test.ts, and for the same reason: the alternative is a second + * copy of each contract to drift against. + */ +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; + +const ROOT = path.join(__dirname, "../.."); +const SUITE = path.join(ROOT, "integration-suite"); +const LOCAL = path.join(SUITE, "local"); +const runLocal = readFileSync(path.join(LOCAL, "run-local.sh"), "utf8"); +const installSh = readFileSync(path.join(LOCAL, "install.sh"), "utf8"); +const service = readFileSync(path.join(LOCAL, "failproofai-canary.service"), "utf8"); +const timer = readFileSync(path.join(LOCAL, "failproofai-canary.timer"), "utf8"); +const workflow = readFileSync( + path.join(ROOT, ".github/workflows/integration-suite.yml"), + "utf8", +); +const probeSh = readFileSync(path.join(SUITE, "probe-cli.sh"), "utf8"); +const runSh = readFileSync(path.join(SUITE, "run.sh"), "utf8"); +const entrypoint = readFileSync(path.join(SUITE, "ci-entrypoint.sh"), "utf8"); +const handlerTs = readFileSync(path.join(ROOT, "src/hooks/handler.ts"), "utf8"); + +describe("GHA workflow is dispatch-only", () => { + it("has no schedule trigger (daily runs live on the canary box)", () => { + // Dropping the daily cron WAS the cost decision. A schedule reappearing + // here must be deliberate — this makes it a conversation, not a re-spend. + expect(workflow).not.toMatch(/^\s*schedule:/m); + expect(workflow).not.toMatch(/\bcron:/); + }); + + it("keeps workflow_dispatch as the cloud fallback", () => { + expect(workflow).toMatch(/^\s*workflow_dispatch:/m); + }); +}); + +describe("local runner wiring", () => { + it("service ExecStart points at the exact path install.sh installs to", () => { + // run-local.sh hard-resets the runner clone, so the unit must exec the + // INSTALLED copy — a unit pointing into the clone would run whatever the + // checked-out ref happens to carry, mid-reset. + const m = /^ExecStart=%h\/(\S+)$/m.exec(service); + expect(m).not.toBeNull(); + expect(m![1]).toBe(".config/failproofai-canary/bin/run-local.sh"); + expect(installSh).toContain("CANARY_CONF_DIR:-$HOME/.config/failproofai-canary"); + expect(installSh).toMatch( + /install -m 755 "\$HERE\/run-local\.sh" "\$CONF_DIR\/bin\/run-local\.sh"/, + ); + }); + + it("install.sh installs both systemd units", () => { + expect(installSh).toContain("failproofai-canary.service"); + expect(installSh).toContain("failproofai-canary.timer"); + }); + + it("timer keeps the retired GHA cron slot and catches up after downtime", () => { + expect(timer).toMatch(/OnCalendar=.*06:17.*UTC/); + expect(timer).toMatch(/^Persistent=true$/m); + }); + + it("run-local.sh drives the same front door CI does", () => { + expect(runLocal).toContain("integration-suite/ci-entrypoint.sh"); + }); + + it("refuses to run without an explicit CANARY_REF", () => { + // A baked-in default ref would silently keep probing a stale branch after + // the daemon branch merges to main — every box states what it tests. + expect(runLocal).toMatch(/\$\{CANARY_REF:\?/); + }); + + it("stable leg defaults to the daemon path, beta to in-process", () => { + expect(runLocal).toContain("${CANARY_DAEMON_STABLE:-1}"); + expect(runLocal).toContain("${CANARY_DAEMON_BETA:-0}"); + }); + + it("secrets template offers every secret-fed env var the workflow maps", () => { + // The box's secrets.env and the GHA Environment must stay interchangeable. + // A secret added to the workflow but not the template means the box runs + // without it and that CLI quietly reports ERROR forever. + const envNames = [...workflow.matchAll(/^\s+([A-Z0-9_]+):\s+\$\{\{\s*secrets\./gm)].map( + (m) => m[1], + ); + expect(envNames.length).toBeGreaterThanOrEqual(10); + for (const name of envNames) { + expect(installSh, `secrets.env template is missing ${name}`).toContain(name); + } + }); +}); + +describe("daemon-mode probe path", () => { + it("ci-entrypoint builds the daemon in a bookworm container and hands it to run.sh", () => { + // rust:1-bookworm ⇔ node:22-bookworm-slim sandbox: same glibc line. A + // host build can link a newer glibc and fail to load inside the sandbox. + expect(entrypoint).toMatch(/rust:1-bookworm cargo build --release --locked -p failproofaid/); + expect(entrypoint).toMatch(/^CANARY_DAEMON=/m); + expect(entrypoint).toMatch(/^CANARY_DAEMON_BIN=/m); + }); + + it("run.sh mounts the binary exactly where probe-cli.sh executes it", () => { + const mount = /-v "\$DBIN:(\S+):ro"/.exec(runSh); + expect(mount).not.toBeNull(); + expect(probeSh).toContain(mount![1]); + }); + + it("probe-cli.sh restarts the daemon per probe (oracle isolation)", () => { + // The wire protocol forwards {hookEvent, cli, stdin, cwd} — never env — so + // the worker's FAILPROOFAI_HOOK_LOG_FILE is fixed at daemon start. One + // daemon across both probes would mean one shared log dir, and probe A's + // incidental read-denies (an agent exploring before it touches the marker) + // would satisfy probe B's grep: a false PASS. + expect(probeSh).toContain('daemon_cycle "$LOGA"'); + expect(probeSh).toContain('daemon_cycle "$LOGB"'); + }); + + it("sets the fail-closed marker in daemon mode and clears it otherwise", () => { + // The HOME volume persists across daily runs: a marker left behind by a + // daemon-mode run would make an in-process run fail closed on every hook + // event with no daemon anywhere. + expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:true\}\}\)/); + expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:false\}\}\)/); + }); + + it("a dead daemon cannot false-PASS either probe", () => { + // bin/failproofai.mjs shapes an unreachable-daemon deny through a + // synthetic policy (see handler.ts's forceDecision branch). Its oracle + // line must NEVER satisfy the probes' deny greps — otherwise a crashed + // daemon reads as healthy enforcement, the exact inversion this suite + // exists to catch. Both sides are extracted from the real sources so a + // rename on either side trips this test. + const idMatch = /registerPolicy\(\s*"(failproofai\/[a-z-]+)",\s*"Fail-closed/.exec(handlerTs); + expect(idMatch).not.toBeNull(); + // handler.ts:463 — `result=${decision} policy=${policyName} duration=…` + const failClosedLine = `result=deny policy=${idMatch![1]} duration=3ms`; + + const deniedPat = /denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); + const readDeniedPat = /read_denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); + expect(deniedPat).not.toBeNull(); + expect(readDeniedPat).not.toBeNull(); + + const deniedRe = new RegExp(deniedPat![1].replace("$1", "canary-bash")); + const readDeniedRe = new RegExp(readDeniedPat![1]); + // Sanity both ways: the real canary/builtin lines must still match… + expect(deniedRe.test("result=deny policy=custom/canary-bash duration=2ms")).toBe(true); + expect( + readDeniedRe.test("result=deny policy=failproofai/block-read-outside-cwd duration=2ms"), + ).toBe(true); + // …and the fail-closed line must match neither. + expect(deniedRe.test(failClosedLine)).toBe(false); + expect(readDeniedRe.test(failClosedLine)).toBe(false); + }); +}); diff --git a/integration-suite/README.md b/integration-suite/README.md index b98ea94d..f549a22b 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -4,27 +4,65 @@ A daily **live-enforcement integration test** for failproofai. It answers one question the unit/e2e suites can't: *does failproofai still enforce against every supported agent CLI, at the versions users actually install today?* -Every day (`.github/workflows/integration-suite.yml`) it installs all 12 agent -CLIs **@latest** into an isolated Docker sandbox, drives each one against -failproofai's own policies (built from this repo's HEAD), and confirms the hook -log shows a **DENY**. A *silent-allow* — a blocked action that ran with no deny — -means enforcement broke against that CLI (e.g. a vendor changed their hook schema -out from under us). The test asserts the deny **positively**, so drift surfaces -as a red run + a Slack alert instead of going unnoticed until a user hits it. +Every day (a systemd user timer on the canary box — see **Local runner** below; +`.github/workflows/integration-suite.yml` is the on-demand cloud fallback) it +installs all 12 agent CLIs **@latest** into an isolated Docker sandbox, drives +each one against failproofai's own policies (built from the ref under test), and +confirms the hook log shows a **DENY**. A *silent-allow* — a blocked action that +ran with no deny — means enforcement broke against that CLI (e.g. a vendor +changed their hook schema out from under us). The test asserts the deny +**positively**, so drift surfaces as a red run + a Slack alert instead of going +unnoticed until a user hits it. + +On the box, the stable leg also runs **daemon-configured** (`CANARY_DAEMON=1`): +hooks route CLI → `failproofaid` (Rust supervisor) → warm bun worker, fail-closed +— the configuration `failproofai config` gives users — so the canary tests the +transport users actually run, not just the in-process path. See the +`CANARY_DAEMON` block in `probe-cli.sh` for the mechanics (per-probe daemon +restarts, the `daemon.configured` marker, and why a dead daemon scores +INCONCLUSIVE rather than a false PASS). ## Why it's separate from `__tests__/` It drives **real vendor CLIs against real gateway models** — it needs network, Docker, credentials, and ~7-10 min, none of which belong in the fast in-process -vitest suites. So it's a scheduled workflow, not a PR gate. +vitest suites. So it's a scheduled run, not a PR gate. + +## Local runner (the daily driver) + +Daily runs live on a **local canary box**, not GH Actions — runner minutes were +the entire cost of the old daily cron; the LLM spend is identical either way. +`local/` holds everything box-side: + +``` +local/run-local.sh the cron replacement: checkout CANARY_REF → + stable leg (daemon) → beta leg → crash-guard +local/install.sh box setup: installed copy + units + secrets template +local/failproofai-canary.service systemd user unit (oneshot, 4h ceiling) +local/failproofai-canary.timer daily 06:17 UTC, Persistent=true +``` + +Box setup: clone the repo anywhere once, `bash integration-suite/local/install.sh`, +fill `~/.config/failproofai-canary/secrets.env` (same variables the GHA +Environment supplied; token tarballs still come from `capture-tokens.sh` on a +logged-in machine), `loginctl enable-linger`, enable the timer. The wrapper runs +from an **installed copy** because it hard-resets the runner clone every run — +nothing that must survive a run may live inside the clone. + +State (`integration-suite-state[-beta].json`) sits in +`~/.local/state/failproofai-canary/` instead of the Actions cache; the +version-gate logic is unchanged. Verdict reports POST to Slack exactly as +before; a leg that dies *before* reporting gets a distinct crash-note (that's +the replacement for GHA's red-job email). ## How a run works -The workflow is a thin trigger; `ci-entrypoint.sh` is the front door and does -everything below except the Actions cache restore/save. +The trigger (box: `local/run-local.sh`; cloud: the workflow) is thin; +`ci-entrypoint.sh` is the front door and does everything below except state +restore/save. -1. Restore `integration-suite-state.json` from Actions cache (version-gate + - broke/recovered diff) — *workflow*. +1. Point `CANARY_STATE` at `integration-suite-state.json` (version-gate + + broke/recovered diff) — box state dir, or Actions cache on a dispatch. 2. Build failproofai under test (`dist/index.js` + `dist/cli.mjs`) from this repo. 3. Decode the OAuth token secrets, build the sandbox image, create the per-run HOME volume, install all 12 CLIs (`install-clis.sh`), inject the credential @@ -96,8 +134,10 @@ able to overwrite the stable leg's gating record. Because this repo is public, all credentials live in a scoped **GitHub Environment** (`cli-integration`) — only this workflow's job can read them — and -the workflow triggers on `schedule`/`workflow_dispatch` **only**, so fork PRs can -never reach them. +the workflow triggers on `workflow_dispatch` **only**, so fork PRs can never +reach them. (The canary box keeps its own copy of the same variables in +`~/.config/failproofai-canary/secrets.env`, chmod 600 — updating one does not +update the other.) | Auth | CLIs | Secret(s) | |------|------|-----------| @@ -126,4 +166,5 @@ canary-policies.mjs benign-marker custom policies the probe trips run.sh orchestrator (gate → probe → report → Slack) report.js build the Slack report + diff state (broke/recovered) capture-tokens.sh (run on a logged-in machine) refresh the OAuth token secrets +local/ the daily driver: box wrapper + systemd units (see above) ``` diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index 0a62f616..12647526 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -38,6 +38,11 @@ # CANARY_VERSION_GATED "all" (default) | comma-sep list | "none" to force-probe # CANARY_CLIS space-separated CLI subset (empty = all 12) # CANARY_SKIP_BUILD set to 1 to reuse an existing dist/ (local iteration) +# CANARY_DAEMON set to 1 to probe the daemon-configured (failproofaid) +# hook path: builds the Rust daemon and routes every +# probe's hooks through it, fail-closed (see probe-cli.sh) +# CANARY_CARGO_CACHE cargo home+target cache dir for the daemon build +# (default ~/.cache/failproofai-canary/cargo) # ───────────────────────────────────────────────────────────────────────────── set -u @@ -101,6 +106,28 @@ if [ ! -s "$REPO/dist/index.js" ] || [ ! -s "$REPO/dist/cli.mjs" ]; then exit 1 fi +# ── 1b. build failproofaid under test (daemon mode only) ──────────────────── +# Built in a rust:1-bookworm container, NOT on the host: the sandbox image is +# node:22-bookworm-slim (glibc 2.36), and a binary linked against a newer host +# glibc would fail to load inside it. The repo mounts read-only — cargo writes +# only to the mounted cache (registry + target), so the checkout stays clean. +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + step "building failproofaid (daemon) under test" + if [ ! -f "$REPO/crates/failproofaid/Cargo.toml" ]; then + echo "✗ CANARY_DAEMON=1 but $REPO has no crates/failproofaid — this ref predates the daemon; unset CANARY_DAEMON or pick a ref that carries it" >&2 + exit 1 + fi + CARGO_CACHE="${CANARY_CARGO_CACHE:-$HOME/.cache/failproofai-canary/cargo}" + mkdir -p "$CARGO_CACHE/home" "$CARGO_CACHE/target" + docker run --rm -u "$(id -u):$(id -g)" \ + -e HOME=/cargo/home -e CARGO_HOME=/cargo/home -e CARGO_TARGET_DIR=/cargo/target \ + -v "$CARGO_CACHE:/cargo" -v "$REPO:/src:ro" -w /src \ + rust:1-bookworm cargo build --release --locked -p failproofaid \ + || { echo "✗ failproofaid build failed" >&2; exit 1; } + export CANARY_DAEMON_BIN="$CARGO_CACHE/target/release/failproofaid" + [ -x "$CANARY_DAEMON_BIN" ] || { echo "✗ built failproofaid missing at $CANARY_DAEMON_BIN" >&2; exit 1; } +fi + # ── 2. decode OAuth token secrets ─────────────────────────────────────────── # Each is a base64 gzip-tar rooted at $HOME. A missing secret is NOT fatal: that # CLI simply reports ERROR (can't auth) rather than taking the whole run down. @@ -176,4 +203,6 @@ CANARY_STATE="$STATE" \ CANARY_ENVFILE="$ENVFILE" \ CANARY_CHANNEL="$CHANNEL" \ CANARY_PEER_STATE="$PEER_STATE" \ +CANARY_DAEMON="${CANARY_DAEMON:-0}" \ +CANARY_DAEMON_BIN="${CANARY_DAEMON_BIN:-}" \ bash "$HERE/run.sh" ${CANARY_CLIS:-} diff --git a/integration-suite/local/failproofai-canary.service b/integration-suite/local/failproofai-canary.service new file mode 100644 index 00000000..364d1a0a --- /dev/null +++ b/integration-suite/local/failproofai-canary.service @@ -0,0 +1,14 @@ +[Unit] +Description=failproofai CLI integration suite (canary) — one full run, both legs +# No network-online dependency: user-manager network targets are unreliable. +# run-local.sh's own `git fetch` fails loudly (and Slack-notes) if offline. + +[Service] +Type=oneshot +# The INSTALLED copy (install.sh) — never a path inside the runner clone, +# which run-local.sh hard-resets on every run. +ExecStart=%h/.config/failproofai-canary/bin/run-local.sh +# Two legs at up to 90 min each (run-local.sh's per-leg timeout) plus CLI +# installs and the daemon build; anything past this is a wedged run. +TimeoutStartSec=4h +Nice=10 diff --git a/integration-suite/local/failproofai-canary.timer b/integration-suite/local/failproofai-canary.timer new file mode 100644 index 00000000..1b6309de --- /dev/null +++ b/integration-suite/local/failproofai-canary.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Daily failproofai integration suite (canary) + +[Timer] +# Same slot the retired GHA cron used (06:17 UTC) — off-peak for the gateway. +OnCalendar=*-*-* 06:17:00 UTC +# A box that was off (or asleep) at 06:17 runs the canary on the next boot +# instead of silently skipping the day. +Persistent=true +RandomizedDelaySec=5m + +[Install] +WantedBy=timers.target diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh new file mode 100755 index 00000000..31f28c19 --- /dev/null +++ b/integration-suite/local/install.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# One-time (and safely re-runnable) setup for the canary BOX. +# +# Copies run-local.sh OUTSIDE the runner clone (the clone is hard-reset on +# every run, so nothing that survives a run may live inside it), installs the +# systemd user units, prepares the state dirs, and writes a secrets template. +# Re-running refreshes the installed copies but NEVER touches an existing +# secrets.env. +# ───────────────────────────────────────────────────────────────────────────── +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +CONF_DIR="${CANARY_CONF_DIR:-$HOME/.config/failproofai-canary}" +STATE_DIR="${CANARY_STATE_DIR:-$HOME/.local/state/failproofai-canary}" +UNIT_DIR="$HOME/.config/systemd/user" +ME="${USER:-$(id -un)}" + +echo "── checking box requirements ──" +missing=0 +for bin in docker git bun node curl flock; do + command -v "$bin" >/dev/null 2>&1 || { echo " ✗ $bin not found on PATH"; missing=1; } +done +if command -v docker >/dev/null 2>&1 && ! docker info >/dev/null 2>&1; then + echo " ✗ docker daemon not reachable as $ME (docker group membership?)"; missing=1 +fi +[ "$missing" = 0 ] && echo " ✓ docker git bun node curl flock all present" + +echo "── installing ──" +mkdir -p "$CONF_DIR/bin" "$STATE_DIR/logs" "$UNIT_DIR" +install -m 755 "$HERE/run-local.sh" "$CONF_DIR/bin/run-local.sh" +install -m 644 "$HERE/failproofai-canary.service" "$HERE/failproofai-canary.timer" "$UNIT_DIR/" +echo " ✓ $CONF_DIR/bin/run-local.sh" +echo " ✓ $UNIT_DIR/failproofai-canary.{service,timer}" +systemctl --user daemon-reload 2>/dev/null \ + || echo " ⚠ systemctl --user unavailable in this shell — run 'systemctl --user daemon-reload' from a login session" + +if [ ! -f "$CONF_DIR/secrets.env" ]; then + ( + umask 177 + cat > "$CONF_DIR/secrets.env" <<'EOF' +# failproofai canary — box configuration. chmod 600; sourced by run-local.sh. +# Same variables the GHA `cli-integration` Environment supplied — see +# integration-suite/ci-entrypoint.sh's header for what each one does. + +# ── gateway + PAT credentials ──────────────────────────────────────────────── +CANARY_LLM_API_KEY= +#CANARY_LLM_BASE_URL=https://models.aikin.club +#CANARY_LLM_MODEL=deepseek-v4-pro +#CANARY_CLAUDE_MODEL=claude-haiku-4-5 +#CANARY_PI_MODEL=claude-haiku-4-5 +#CANARY_CODEX_MODEL=gpt-5.1-codex-mini +COPILOT_GITHUB_TOKEN= + +# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── +# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh +# and paste the output here; the box itself never needs vendor logins. An empty +# value just makes that CLI report ERROR (can't auth), not a failed run. +CURSOR_TOKEN_TGZ_B64= +DEVIN_TOKEN_TGZ_B64= +ANTIGRAVITY_TOKEN_TGZ_B64= + +# ── reporting ──────────────────────────────────────────────────────────────── +CANARY_SLACK_WEBHOOK= + +# ── what to test ───────────────────────────────────────────────────────────── +# REQUIRED. Deliberately explicit (no baked-in default): flip to origin/main +# once the failproofaid branch (#632) merges. +CANARY_REF=origin/failproofaid +# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays +# in-process. Flip these to move the daemon dimension between legs. +#CANARY_DAEMON_STABLE=1 +#CANARY_DAEMON_BETA=0 +#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git +#CANARY_CLONE=$HOME/canary/failproofai +EOF + ) + echo " ✓ wrote template $CONF_DIR/secrets.env (fill it in)" +else + echo " ✓ kept existing $CONF_DIR/secrets.env" +fi + +cat <&2; exit 1; } +perms="$(stat -c %a "$SECRETS" 2>/dev/null || stat -f %Lp "$SECRETS" 2>/dev/null)" +[ "$perms" = 600 ] || { echo "✗ $SECRETS must be chmod 600 (is $perms) — it holds credentials" >&2; exit 1; } +set -a; . "$SECRETS"; set +a + +# Required, no default ON PURPOSE: a baked-in default ref would silently keep +# probing a stale branch after the daemon branch merges to main. Every box +# states what it tests. +: "${CANARY_REF:?CANARY_REF unset — set it in $SECRETS (origin/failproofaid until #632 merges, then origin/main)}" +CLONE="${CANARY_CLONE:-$HOME/canary/failproofai}" +GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" +LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout + +# One run at a time — the local stand-in for GHA's `concurrency` group. A +# still-running yesterday (hung vendor CLI) must not race today's volume. +exec 9>"$STATE_DIR/.lock" +flock -n 9 || { echo "another canary run holds $STATE_DIR/.lock — exiting" >&2; exit 0; } + +TS="$(date -u +%Y%m%dT%H%M%SZ)" + +slack_note() { # $1 = text; best-effort, never fails the run + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +# ── checkout the ref under test ────────────────────────────────────────────── +if [ ! -d "$CLONE/.git" ]; then + git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } +fi +git -C "$CLONE" fetch --prune origin \ + || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } +{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ + || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } +FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +echo "── canary run $TS: $CANARY_REF @ $FP_SHA ──" + +# ── legs (the same two the GHA matrix ran; sequential on one Docker host) ──── +run_leg() { # $1 = channel + local channel="$1" leg_log="$STATE_DIR/logs/leg-$1-$TS.log" rc daemon state peer + if [ "$channel" = stable ]; then + # Stable probes the daemon-configured (failproofaid) path — the way-forward + # configuration users get from `failproofai config`. Beta stays in-process: + # it answers "is the vendor about to break us", which is independent of our + # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. + daemon="${CANARY_DAEMON_STABLE:-1}" + state="$STATE_DIR/integration-suite-state.json" + peer="" + else + daemon="${CANARY_DAEMON_BETA:-0}" + state="$STATE_DIR/integration-suite-state-$channel.json" + peer="$STATE_DIR/integration-suite-state.json" + fi + echo "── leg: $channel (daemon=$daemon) ──" + GITHUB_WORKSPACE="$CLONE" \ + CANARY_CHANNEL="$channel" \ + CANARY_STATE="$state" \ + CANARY_PEER_STATE="$peer" \ + CANARY_FP_SHA="$FP_SHA" \ + CANARY_DAEMON="$daemon" \ + CANARY_ENVFILE="$STATE_DIR/tmp/canary-$channel.env" \ + CANARY_TOKENS_DIR="$STATE_DIR/tmp/tokens-$channel" \ + timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" + rc=${PIPESTATUS[0]} + # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack + # already carries the story); non-zero WITHOUT one means the harness died + # before reporting, which on GHA surfaced as a red-job email and here would + # otherwise be silence. + if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then + slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — $CANARY_REF @ $FP_SHA +\`\`\` +$(tail -12 "$leg_log") +\`\`\`" + fi + return "$rc" +} + +rc_stable=0; rc_beta=0 +run_leg stable || rc_stable=$? +run_leg beta || rc_beta=$? + +find "$STATE_DIR/logs" -name '*.log' -mtime +14 -delete 2>/dev/null || true + +echo "── done: stable rc=$rc_stable, beta rc=$rc_beta ──" +[ "$rc_stable" -eq 0 ] && [ "$rc_beta" -eq 0 ] diff --git a/integration-suite/probe-cli.sh b/integration-suite/probe-cli.sh index b517601b..a60f326c 100644 --- a/integration-suite/probe-cli.sh +++ b/integration-suite/probe-cli.sh @@ -63,6 +63,76 @@ printf '#!/bin/sh\nexec bun /repo/bin/failproofai.mjs "$@"\n' > "$HOME/bin/failp chmod +x "$HOME/bin/failproofai" export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" +# ── Daemon mode (CANARY_DAEMON=1) ──────────────────────────────────────────── +# Probes the configuration users get after `failproofai config`: hooks route +# CLI → failproofaid (Rust supervisor) → warm bun worker over Unix sockets, +# fail-CLOSED when the daemon is unreachable. The binary is cross-compiled on +# the host by ci-entrypoint.sh (rust:1-bookworm, so its glibc matches this +# sandbox) and bind-mounted at /opt/failproofaid/failproofaid by run.sh. +# +# The daemon is started PER PROBE, not once per CLI. The worker inherits the +# DAEMON's environment — the wire protocol carries only {hookEvent, cli, +# stdin, cwd}, never the hook process's env — so FAILPROOFAI_HOOK_LOG_FILE +# only reaches the oracle if the daemon itself is (re)started pointing at that +# probe's log dir. Sharing one log dir across both probes instead would let +# probe A's incidental denies (an agent exploring with reads trips +# block-read-outside-cwd) satisfy probe B's grep — a false PASS. +# +# A DEAD daemon cannot false-PASS: the client's fail-closed deny is shaped by +# a synthetic `failproofai/daemon-unreachable` policy (bin/failproofai.mjs), +# which denied()/read_denied() below can never match — those probes go +# INCONCLUSIVE and re-probe until the daemon path recovers. +DAEMON_PID="" +daemon_stop() { + [ -n "$DAEMON_PID" ] || return 0 + kill "$DAEMON_PID" 2>/dev/null + wait "$DAEMON_PID" 2>/dev/null + DAEMON_PID="" +} +daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) + [ "${CANARY_DAEMON:-0}" = 1 ] || return 0 + daemon_stop + rm -f "$FAILPROOFAI_DAEMON_SOCKET" + # Env is the worker's too (worker.rs spawns `sh -c "$FAILPROOFAI_WORKER_CMD"` + # inheriting it): the writable FP_DIST for the custom-policy loader's shim, + # and this probe's oracle dir. The worker entry only sets DIST when unset. + FAILPROOFAI_HOOK_LOG_FILE="$1" \ + FAILPROOFAI_WORKER_CMD="bun /repo/bin/failproofai-worker.mjs" \ + /opt/failproofaid/failproofaid >> "$BASE/daemon.log" 2>&1 & + DAEMON_PID=$! + for _ in $(seq 1 100); do # ≤10s; readiness = the socket ACCEPTS, not exists + if node -e 'const s=require("net").createConnection(process.argv[1]);s.on("connect",()=>process.exit(0));s.on("error",()=>process.exit(1));' \ + "$FAILPROOFAI_DAEMON_SOCKET" 2>/dev/null; then return 0; fi + kill -0 "$DAEMON_PID" 2>/dev/null || break + sleep 0.1 + done + echo "✗ failproofaid did not come up — daemon.log tail:" >&2 + tail -5 "$BASE/daemon.log" >&2 + exit 1 +} +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + [ -x /opt/failproofaid/failproofaid ] \ + || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + # Socket under /tmp: container-local, so a stale socket file in the PERSISTENT + # volume can never shadow a live daemon across daily runs. The override + # relocates the whole run dir — lock and worker.sock land beside it — and the + # dir is NOT pre-created here: failproofaid creates it 0700 itself and refuses + # one it didn't create with other perms (paths.rs ensure_run_dir). + export FAILPROOFAI_DAEMON_SOCKET="/tmp/fpai-canary/failproofaid.sock" + # The fail-closed marker, written via the REAL code path (fp-config's + # updateConfig) rather than shell-appending TOML — the volume's config.toml + # survives with its other tables intact, and a duplicate [daemon] table + # (invalid TOML) would silently read back as NOT configured. + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ + || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } + trap daemon_stop EXIT +else + # The HOME volume persists across runs: a marker left behind by a daemon-mode + # run would make this in-process run fail closed on every hook event with no + # daemon anywhere. Clear it unconditionally. + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true +fi + BASE="$HOME/probe-$CLI" # DEFINITE probes: BENIGN actions (echo/touch a token, read a plain file) the # model never refuses → a tool call is guaranteed, so no INCONCLUSIVE from @@ -206,6 +276,7 @@ ATTEMPTS=3 # retry up to N times to absorb LLM nondeterminism (flaky tool-call # despite no deny); ERROR=vendor quota/auth failure; INCONCLUSIVE=no tool call. LOGA="$BASE/log-bash" BASH_PROMPT='Using your shell/bash tool, run exactly this command: touch CANARY_PROBE_ran' +daemon_cycle "$LOGA" # daemon mode only: worker must inherit THIS probe's oracle dir OUTA="" for _ in $(seq 1 $ATTEMPTS); do export FAILPROOFAI_HOOK_LOG_FILE="$LOGA"; rm -f "$BASE/CANARY_PROBE_ran" @@ -226,6 +297,7 @@ else VA=INCONCLUSIVE; fi # ── Probe B: benign file read (canary-read) ────────────────────────────────── LOGB="$BASE/log-read" READ_PROMPT='Read the file named CANARY_MARKER.txt in the current directory and print its exact contents.' +daemon_cycle "$LOGB" # fresh daemon env — probe A's log dir must not receive B's denies OUTB="" for _ in $(seq 1 $ATTEMPTS); do export FAILPROOFAI_HOOK_LOG_FILE="$LOGB" diff --git a/integration-suite/run.sh b/integration-suite/run.sh index fab41fcb..b4df7c5d 100644 --- a/integration-suite/run.sh +++ b/integration-suite/run.sh @@ -61,8 +61,20 @@ FP_SHA="${CANARY_FP_SHA:-$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null || # Installed versions from the install step, keyed by cli. VERSIONS_JSON="$(docker run --rm -v "$VOL:/home/canary" "$IMAGE" cat /home/canary/canary-tier0.json 2>/dev/null || echo '[]')" +# Daemon mode: bind the host-built failproofaid binary into the probe container +# and tell probe-cli.sh to route hooks through it (see its CANARY_DAEMON block). +# A file→file bind mount, read-only — executing from an ro mount is fine. +DAEMON_FLAGS=() +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" + [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } + # docker reads a relative -v source as a NAMED VOLUME — absolutize first. + DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" + DAEMON_FLAGS=(-e CANARY_DAEMON=1 -v "$DBIN:/opt/failproofaid/failproofaid:ro") +fi + run_probe() { - docker run --rm --env-file "$ENVFILE" \ + docker run --rm --env-file "$ENVFILE" "${DAEMON_FLAGS[@]}" \ -v "$REPO:/repo:ro" -v "$SANDBOX:/opt/canary:ro" -v "$VOL:/home/canary" \ "$IMAGE" bash /opt/canary/probe-cli.sh "$1" 2>&1 } From 9e0533dc4c0e06d752cbafed5aaf3491f58bbee1 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 11:47:09 +0530 Subject: [PATCH 2/4] =?UTF-8?q?canary:=20one=20container,=20one=20cron=20l?= =?UTF-8?q?ine=20=E2=80=94=20repackage=20the=20box=20runner=20for=20zero-t?= =?UTF-8?q?ouch=20hosts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The box story shrinks to Docker + one cron line + one env file: the systemd units, install.sh and host-toolchain requirements are gone. A self-contained runner image (local/Dockerfile.runner — node+bun+git+docker CLIENT) drives the HOST's Docker through the mounted socket, so the sandbox image, the per-channel volumes and every probe container are exactly the ones CI runs, as siblings. Two decisions carry the design: - Path parity. The one work dir is mounted at an IDENTICAL path inside and out (-v "$HOME/fp-canary:$HOME/fp-canary") because paths under it serve both as in-container file paths and as sibling-container -v sources, which the host daemon resolves against the host filesystem. The entrypoint auto-detects the parity mount from its own container's mount table and names the exact flag to add when it is missing. runner-daily.sh pins the daemon build's cargo cache under the work dir — the only harness default rooted outside it ($HOME), where the rust sibling's mount would silently create an empty root-owned host dir and cache nothing. - A thin baked entrypoint, everything else from the checkout. The image carries only runner-entrypoint.sh (preflight, work-dir detection, host-side flock so overlapping cron fires share one lock across containers, clone/ fetch/checkout of $CANARY_REF, Slack crash-note for the checkout phase); it then execs integration-suite/local/runner-daily.sh FROM THE CHECKOUT. Harness changes reach the box through git — nobody rebuilds the boss's image for a leg tweak. runner-daily.sh keeps the leg contract from the systemd iteration verbatim: stable leg daemon-configured (CANARY_DAEMON=1) then beta in-process, per-leg 90-min timeout, crash-guard keyed on the absence of run.sh's own posted-to-Slack line, 14-day log prune. secrets.env.example documents every variable the GHA Environment supplied, in docker --env-file's literal KEY=value format. Tripwires updated in local-runner.test.ts: the image must never bake the daily driver, the crash-guard grep must match run.sh's actual wording, the example must offer every secret-fed env var the workflow maps and must contain no shell expansion on value lines. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- .../integration-suite/local-runner.test.ts | 132 ++++++++++++------ integration-suite/README.md | 60 +++++--- integration-suite/local/Dockerfile.runner | 41 ++++++ .../local/failproofai-canary.service | 14 -- .../local/failproofai-canary.timer | 13 -- integration-suite/local/install.sh | 91 ------------ integration-suite/local/run-local.sh | 116 --------------- integration-suite/local/runner-daily.sh | 88 ++++++++++++ integration-suite/local/runner-entrypoint.sh | 67 +++++++++ integration-suite/local/secrets.env.example | 43 ++++++ 11 files changed, 372 insertions(+), 295 deletions(-) create mode 100644 integration-suite/local/Dockerfile.runner delete mode 100644 integration-suite/local/failproofai-canary.service delete mode 100644 integration-suite/local/failproofai-canary.timer delete mode 100755 integration-suite/local/install.sh delete mode 100755 integration-suite/local/run-local.sh create mode 100755 integration-suite/local/runner-daily.sh create mode 100755 integration-suite/local/runner-entrypoint.sh create mode 100644 integration-suite/local/secrets.env.example diff --git a/CHANGELOG.md b/CHANGELOG.md index ea2fbb7c..71e8024e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box, and make it probe the daemon path. `integration-suite/local/` ships the box side — a wrapper that checks out `CANARY_REF`, runs both legs through the unchanged `ci-entrypoint.sh`, and Slack-notes any leg that dies *before* reporting (the replacement for GHA's red-job email) — plus an installer and a systemd user timer holding the same 06:17 UTC slot; the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index b882b7e0..9688a7b7 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -2,25 +2,30 @@ * Tripwires for the LOCAL canary runner (integration-suite/local/) and the * daemon-mode (CANARY_DAEMON) probe path. * - * Daily integration-suite runs moved off GH Actions onto a local box - * (2026-08-07, for runner-minute cost); the stable leg there probes the - * daemon-configured (failproofaid) hook path — the way-forward configuration. - * Everything below is shell scripts and systemd units with no importable - * surface, so the tests parse the real files — same approach as + * Daily integration-suite runs moved off GH Actions (2026-08-07, for + * runner-minute cost) onto a box whose entire contract is: Docker + one cron + * line + one env file. A self-contained runner image drives the HOST's Docker + * through the mounted socket; its baked entrypoint checks out CANARY_REF and + * hands off to runner-daily.sh FROM THE CHECKOUT, so harness changes reach the + * box through git with no image rebuild. The stable leg probes the + * daemon-configured (failproofaid) path — the way-forward configuration. + * + * Everything below is shell scripts, a Dockerfile and an env template with no + * importable surface, so the tests parse the real files — same approach as * channel-refs.test.ts, and for the same reason: the alternative is a second * copy of each contract to drift against. */ -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import path from "node:path"; import { describe, expect, it } from "vitest"; const ROOT = path.join(__dirname, "../.."); const SUITE = path.join(ROOT, "integration-suite"); const LOCAL = path.join(SUITE, "local"); -const runLocal = readFileSync(path.join(LOCAL, "run-local.sh"), "utf8"); -const installSh = readFileSync(path.join(LOCAL, "install.sh"), "utf8"); -const service = readFileSync(path.join(LOCAL, "failproofai-canary.service"), "utf8"); -const timer = readFileSync(path.join(LOCAL, "failproofai-canary.timer"), "utf8"); +const dockerfile = readFileSync(path.join(LOCAL, "Dockerfile.runner"), "utf8"); +const entrypointSh = readFileSync(path.join(LOCAL, "runner-entrypoint.sh"), "utf8"); +const dailySh = readFileSync(path.join(LOCAL, "runner-daily.sh"), "utf8"); +const secretsExample = readFileSync(path.join(LOCAL, "secrets.env.example"), "utf8"); const workflow = readFileSync( path.join(ROOT, ".github/workflows/integration-suite.yml"), "utf8", @@ -43,55 +48,100 @@ describe("GHA workflow is dispatch-only", () => { }); }); -describe("local runner wiring", () => { - it("service ExecStart points at the exact path install.sh installs to", () => { - // run-local.sh hard-resets the runner clone, so the unit must exec the - // INSTALLED copy — a unit pointing into the clone would run whatever the - // checked-out ref happens to carry, mid-reset. - const m = /^ExecStart=%h\/(\S+)$/m.exec(service); - expect(m).not.toBeNull(); - expect(m![1]).toBe(".config/failproofai-canary/bin/run-local.sh"); - expect(installSh).toContain("CANARY_CONF_DIR:-$HOME/.config/failproofai-canary"); - expect(installSh).toMatch( - /install -m 755 "\$HERE\/run-local\.sh" "\$CONF_DIR\/bin\/run-local\.sh"/, - ); +describe("runner image (the boss's one container)", () => { + it("bakes the thin entrypoint and nothing else of the harness", () => { + // The image must stay rebuild-free across harness changes: it may carry + // runner-entrypoint.sh (thin, stable) but must NOT bake runner-daily.sh + // or any other harness file — those are executed from the checkout. + expect(dockerfile).toMatch(/^COPY runner-entrypoint\.sh /m); + expect(dockerfile).toMatch(/^ENTRYPOINT \["\/usr\/local\/bin\/runner-entrypoint\.sh"\]$/m); + // (comments may mention the daily driver; COPY lines must not) + expect(dockerfile).not.toMatch(/^COPY .*runner-daily/m); }); - it("install.sh installs both systemd units", () => { - expect(installSh).toContain("failproofai-canary.service"); - expect(installSh).toContain("failproofai-canary.timer"); + it("ships the docker CLIENT for the mounted host socket", () => { + expect(dockerfile).toMatch(/download\.docker\.com\/linux\/static/); }); - it("timer keeps the retired GHA cron slot and catches up after downtime", () => { - expect(timer).toMatch(/OnCalendar=.*06:17.*UTC/); - expect(timer).toMatch(/^Persistent=true$/m); + it("entrypoint refuses to run without the socket and without CANARY_REF", () => { + // A baked-in default ref would silently keep probing a stale branch after + // the daemon branch merges to main — the env file states what it tests. + expect(entrypointSh).toContain("/var/run/docker.sock"); + expect(entrypointSh).toMatch(/\$\{CANARY_REF:\?/); }); - it("run-local.sh drives the same front door CI does", () => { - expect(runLocal).toContain("integration-suite/ci-entrypoint.sh"); + it("entrypoint serializes runs and hands off to the in-repo daily driver", () => { + // The lock file lives on the host work dir so overlapping cron fires + // share one lock across separate containers. + expect(entrypointSh).toMatch(/flock -n/); + expect(entrypointSh).toMatch(/exec bash "\$CLONE\/integration-suite\/local\/runner-daily\.sh"/); + expect(existsSync(path.join(LOCAL, "runner-daily.sh"))).toBe(true); }); - it("refuses to run without an explicit CANARY_REF", () => { - // A baked-in default ref would silently keep probing a stale branch after - // the daemon branch merges to main — every box states what it tests. - expect(runLocal).toMatch(/\$\{CANARY_REF:\?/); + it("entrypoint explains the identical-path work-dir mount when it is missing", () => { + // Path parity is the load-bearing trick of the whole design: paths under + // the work dir serve as sibling-container -v sources, resolved by the + // HOST daemon. The failure message must teach the fix. + expect(entrypointSh).toMatch(/-v \\"\\\$HOME\/fp-canary:\\\$HOME\/fp-canary\\"/); + }); +}); + +describe("daily driver (in-repo, evolves with the harness)", () => { + it("drives the same front door CI does, one leg per channel", () => { + expect(dailySh).toContain("integration-suite/ci-entrypoint.sh"); + expect(dailySh).toMatch(/\$\{CANARY_LEGS:-stable beta\}/); }); it("stable leg defaults to the daemon path, beta to in-process", () => { - expect(runLocal).toContain("${CANARY_DAEMON_STABLE:-1}"); - expect(runLocal).toContain("${CANARY_DAEMON_BETA:-0}"); + expect(dailySh).toContain("${CANARY_DAEMON_STABLE:-1}"); + expect(dailySh).toContain("${CANARY_DAEMON_BETA:-0}"); + }); + + it("pins the cargo cache under the work dir (path parity for the sibling build)", () => { + // ci-entrypoint's default cargo cache is under $HOME — inside the runner + // container that path does not exist on the host, so the rust sibling + // container's -v mount would silently create a root-owned host dir and + // cache nothing. The only harness default rooted outside $WORK. + expect(dailySh).toMatch(/CANARY_CARGO_CACHE="\$\{CANARY_CARGO_CACHE:-\$WORK\/cargo\}"/); }); - it("secrets template offers every secret-fed env var the workflow maps", () => { - // The box's secrets.env and the GHA Environment must stay interchangeable. - // A secret added to the workflow but not the template means the box runs + it("crash-guard greps the exact success line run.sh prints", () => { + // "leg died WITHOUT reporting" is detected by the absence of run.sh's own + // posted-to-Slack line — if that wording changes in run.sh, the crash + // guard goes blind and every FAIL verdict would double-post a crash note. + const m = /grep -q "([^"]+)" "\$leg_log"/.exec(dailySh); + expect(m).not.toBeNull(); + expect(runSh).toContain(m![1]); + }); +}); + +describe("secrets.env.example (the one file the boss edits)", () => { + it("offers every secret-fed env var the workflow maps", () => { + // The box's env file and the GHA Environment must stay interchangeable. + // A secret added to the workflow but not the example means the box runs // without it and that CLI quietly reports ERROR forever. const envNames = [...workflow.matchAll(/^\s+([A-Z0-9_]+):\s+\$\{\{\s*secrets\./gm)].map( (m) => m[1], ); expect(envNames.length).toBeGreaterThanOrEqual(10); for (const name of envNames) { - expect(installSh, `secrets.env template is missing ${name}`).toContain(name); + expect(secretsExample, `secrets.env.example is missing ${name}`).toContain(name); + } + }); + + it("states CANARY_REF uncommented (the runner refuses to start without it)", () => { + expect(secretsExample).toMatch(/^CANARY_REF=\S+$/m); + }); + + it("is valid docker --env-file material: no shell expansion on value lines", () => { + // docker --env-file is literal KEY=value — a $HOME in a value would reach + // the container as the four characters "$HOM"+"E". Comments may mention + // $HOME freely; value lines must not. + const valueLines = secretsExample + .split("\n") + .filter((l) => l.trim() && !l.trim().startsWith("#")); + for (const line of valueLines) { + expect(line, `value line must not rely on shell expansion: ${line}`).not.toContain("$"); } }); }); @@ -138,7 +188,7 @@ describe("daemon-mode probe path", () => { // rename on either side trips this test. const idMatch = /registerPolicy\(\s*"(failproofai\/[a-z-]+)",\s*"Fail-closed/.exec(handlerTs); expect(idMatch).not.toBeNull(); - // handler.ts:463 — `result=${decision} policy=${policyName} duration=…` + // handler.ts — `result=${decision} policy=${policyName} duration=…` const failClosedLine = `result=deny policy=${idMatch![1]} duration=3ms`; const deniedPat = /denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); diff --git a/integration-suite/README.md b/integration-suite/README.md index f549a22b..d6101877 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -32,28 +32,50 @@ vitest suites. So it's a scheduled run, not a PR gate. Daily runs live on a **local canary box**, not GH Actions — runner minutes were the entire cost of the old daily cron; the LLM spend is identical either way. -`local/` holds everything box-side: +The box needs exactly **Docker + one cron line + one env file**; there is no +host toolchain, no installed scripts, no systemd. Everything else happens +inside a self-contained runner image that drives the host's Docker through the +mounted socket (sibling containers — the sandbox image, volumes and probe +containers are the exact ones CI runs). -``` -local/run-local.sh the cron replacement: checkout CANARY_REF → - stable leg (daemon) → beta leg → crash-guard -local/install.sh box setup: installed copy + units + secrets template -local/failproofai-canary.service systemd user unit (oneshot, 4h ceiling) -local/failproofai-canary.timer daily 06:17 UTC, Persistent=true +Box setup, in full: + +```bash +# 1. one-time: build the runner image (from a clone, or straight from GitHub) +docker build -t failproofai-canary-runner \ + -f integration-suite/local/Dockerfile.runner integration-suite/local/ + +# 2. one-time: work dir + secrets +mkdir -p ~/fp-canary +cp integration-suite/local/secrets.env.example ~/fp-canary/secrets.env +chmod 600 ~/fp-canary/secrets.env # then fill it in + +# 3. cron (pick any quiet hour; overlapping fires share a lock and no-op) +17 6 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$HOME/fp-canary:$HOME/fp-canary" --env-file "$HOME/fp-canary/secrets.env" failproofai-canary-runner >/dev/null 2>&1 ``` -Box setup: clone the repo anywhere once, `bash integration-suite/local/install.sh`, -fill `~/.config/failproofai-canary/secrets.env` (same variables the GHA -Environment supplied; token tarballs still come from `capture-tokens.sh` on a -logged-in machine), `loginctl enable-linger`, enable the timer. The wrapper runs -from an **installed copy** because it hard-resets the runner clone every run — -nothing that must survive a run may live inside the clone. - -State (`integration-suite-state[-beta].json`) sits in -`~/.local/state/failproofai-canary/` instead of the Actions cache; the -version-gate logic is unchanged. Verdict reports POST to Slack exactly as -before; a leg that dies *before* reporting gets a distinct crash-note (that's -the replacement for GHA's red-job email). +The work dir is mounted at an **identical path** inside and out — that is +load-bearing, not style: paths under it are used both for in-container file +ops and as sibling-container `-v` sources, which the host daemon resolves +against the host filesystem. The entrypoint auto-detects it (and says exactly +what to mount if it can't). + +At each run the image's baked entrypoint (`runner-entrypoint.sh` — thin on +purpose) locks, clones/fetches `CANARY_REF` into `~/fp-canary/clone`, and +hands off to `runner-daily.sh` **from that checkout** — so harness changes +reach the box through git, and the image only needs a rebuild when the +entrypoint itself changes. The daily driver runs the stable leg +(daemon-configured) then the beta leg (in-process), exactly like the old GHA +matrix. + +Everything lands under the work dir: version-gate state in `state/` (instead +of the Actions cache — the gate logic is unchanged), run + per-leg logs in +`logs/` (pruned after 14 days), the clone, and the daemon build's cargo cache. +Verdict reports POST to Slack exactly as before; a leg that dies *before* +reporting gets a distinct crash-note with the log tail (that's the replacement +for GHA's red-job email — cron's own output can go to `/dev/null`). Token +tarballs still come from `capture-tokens.sh` on a logged-in machine; the first +run probes all 12 CLIs (~1h, empty gate) and steady-state runs are short. ## How a run works diff --git a/integration-suite/local/Dockerfile.runner b/integration-suite/local/Dockerfile.runner new file mode 100644 index 00000000..7dd8e366 --- /dev/null +++ b/integration-suite/local/Dockerfile.runner @@ -0,0 +1,41 @@ +# failproofai canary — the self-contained daily RUNNER image. +# +# The whole box story is: build this once, add one cron line, done. +# +# docker build -t failproofai-canary-runner -f Dockerfile.runner . +# 17 6 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ +# -v "$HOME/fp-canary:$HOME/fp-canary" --env-file "$HOME/fp-canary/secrets.env" \ +# failproofai-canary-runner >/dev/null 2>&1 +# +# At each run the baked entrypoint clones/fetches the repo at $CANARY_REF into +# the work dir and hands off to integration-suite/local/runner-daily.sh FROM +# THAT CHECKOUT — so harness changes reach the box through git, and this image +# only needs a rebuild when the entrypoint or this file change. +# +# It drives the HOST's Docker through the mounted socket (sibling containers, +# not docker-in-docker): the sandbox image, per-channel HOME volumes and probe +# containers are the exact ones CI runs. That is why the work dir must be +# mounted at an IDENTICAL path inside and out — paths under it are used both +# for in-container file ops and as `-v` sources that the HOST daemon resolves. + +FROM node:22-bookworm-slim + +# bun from the official image — same trick as the sandbox Dockerfile. +COPY --from=oven/bun:latest /usr/local/bin/bun /usr/local/bin/bun + +RUN apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends git ca-certificates curl \ + && rm -rf /var/lib/apt/lists/* + +# docker CLIENT only — the daemon is the host's, reached through the socket. +# Static binary, arch-mapped (x86_64 / aarch64 — uname -m matches the URL). +ARG DOCKER_VERSION=27.5.1 +RUN arch="$(uname -m)" \ + && curl -fsSL "https://download.docker.com/linux/static/stable/${arch}/docker-${DOCKER_VERSION}.tgz" \ + | tar -xz --strip-components=1 -C /usr/local/bin docker/docker + +COPY runner-entrypoint.sh /usr/local/bin/runner-entrypoint.sh +RUN chmod +x /usr/local/bin/runner-entrypoint.sh \ + && bun --version && node --version && git --version && docker --version + +ENTRYPOINT ["/usr/local/bin/runner-entrypoint.sh"] diff --git a/integration-suite/local/failproofai-canary.service b/integration-suite/local/failproofai-canary.service deleted file mode 100644 index 364d1a0a..00000000 --- a/integration-suite/local/failproofai-canary.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=failproofai CLI integration suite (canary) — one full run, both legs -# No network-online dependency: user-manager network targets are unreliable. -# run-local.sh's own `git fetch` fails loudly (and Slack-notes) if offline. - -[Service] -Type=oneshot -# The INSTALLED copy (install.sh) — never a path inside the runner clone, -# which run-local.sh hard-resets on every run. -ExecStart=%h/.config/failproofai-canary/bin/run-local.sh -# Two legs at up to 90 min each (run-local.sh's per-leg timeout) plus CLI -# installs and the daemon build; anything past this is a wedged run. -TimeoutStartSec=4h -Nice=10 diff --git a/integration-suite/local/failproofai-canary.timer b/integration-suite/local/failproofai-canary.timer deleted file mode 100644 index 1b6309de..00000000 --- a/integration-suite/local/failproofai-canary.timer +++ /dev/null @@ -1,13 +0,0 @@ -[Unit] -Description=Daily failproofai integration suite (canary) - -[Timer] -# Same slot the retired GHA cron used (06:17 UTC) — off-peak for the gateway. -OnCalendar=*-*-* 06:17:00 UTC -# A box that was off (or asleep) at 06:17 runs the canary on the next boot -# instead of silently skipping the day. -Persistent=true -RandomizedDelaySec=5m - -[Install] -WantedBy=timers.target diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh deleted file mode 100755 index 31f28c19..00000000 --- a/integration-suite/local/install.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env bash -# ───────────────────────────────────────────────────────────────────────────── -# One-time (and safely re-runnable) setup for the canary BOX. -# -# Copies run-local.sh OUTSIDE the runner clone (the clone is hard-reset on -# every run, so nothing that survives a run may live inside it), installs the -# systemd user units, prepares the state dirs, and writes a secrets template. -# Re-running refreshes the installed copies but NEVER touches an existing -# secrets.env. -# ───────────────────────────────────────────────────────────────────────────── -set -u -HERE="$(cd "$(dirname "$0")" && pwd)" -CONF_DIR="${CANARY_CONF_DIR:-$HOME/.config/failproofai-canary}" -STATE_DIR="${CANARY_STATE_DIR:-$HOME/.local/state/failproofai-canary}" -UNIT_DIR="$HOME/.config/systemd/user" -ME="${USER:-$(id -un)}" - -echo "── checking box requirements ──" -missing=0 -for bin in docker git bun node curl flock; do - command -v "$bin" >/dev/null 2>&1 || { echo " ✗ $bin not found on PATH"; missing=1; } -done -if command -v docker >/dev/null 2>&1 && ! docker info >/dev/null 2>&1; then - echo " ✗ docker daemon not reachable as $ME (docker group membership?)"; missing=1 -fi -[ "$missing" = 0 ] && echo " ✓ docker git bun node curl flock all present" - -echo "── installing ──" -mkdir -p "$CONF_DIR/bin" "$STATE_DIR/logs" "$UNIT_DIR" -install -m 755 "$HERE/run-local.sh" "$CONF_DIR/bin/run-local.sh" -install -m 644 "$HERE/failproofai-canary.service" "$HERE/failproofai-canary.timer" "$UNIT_DIR/" -echo " ✓ $CONF_DIR/bin/run-local.sh" -echo " ✓ $UNIT_DIR/failproofai-canary.{service,timer}" -systemctl --user daemon-reload 2>/dev/null \ - || echo " ⚠ systemctl --user unavailable in this shell — run 'systemctl --user daemon-reload' from a login session" - -if [ ! -f "$CONF_DIR/secrets.env" ]; then - ( - umask 177 - cat > "$CONF_DIR/secrets.env" <<'EOF' -# failproofai canary — box configuration. chmod 600; sourced by run-local.sh. -# Same variables the GHA `cli-integration` Environment supplied — see -# integration-suite/ci-entrypoint.sh's header for what each one does. - -# ── gateway + PAT credentials ──────────────────────────────────────────────── -CANARY_LLM_API_KEY= -#CANARY_LLM_BASE_URL=https://models.aikin.club -#CANARY_LLM_MODEL=deepseek-v4-pro -#CANARY_CLAUDE_MODEL=claude-haiku-4-5 -#CANARY_PI_MODEL=claude-haiku-4-5 -#CANARY_CODEX_MODEL=gpt-5.1-codex-mini -COPILOT_GITHUB_TOKEN= - -# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── -# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh -# and paste the output here; the box itself never needs vendor logins. An empty -# value just makes that CLI report ERROR (can't auth), not a failed run. -CURSOR_TOKEN_TGZ_B64= -DEVIN_TOKEN_TGZ_B64= -ANTIGRAVITY_TOKEN_TGZ_B64= - -# ── reporting ──────────────────────────────────────────────────────────────── -CANARY_SLACK_WEBHOOK= - -# ── what to test ───────────────────────────────────────────────────────────── -# REQUIRED. Deliberately explicit (no baked-in default): flip to origin/main -# once the failproofaid branch (#632) merges. -CANARY_REF=origin/failproofaid -# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays -# in-process. Flip these to move the daemon dimension between legs. -#CANARY_DAEMON_STABLE=1 -#CANARY_DAEMON_BETA=0 -#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git -#CANARY_CLONE=$HOME/canary/failproofai -EOF - ) - echo " ✓ wrote template $CONF_DIR/secrets.env (fill it in)" -else - echo " ✓ kept existing $CONF_DIR/secrets.env" -fi - -cat <&2; exit 1; } -perms="$(stat -c %a "$SECRETS" 2>/dev/null || stat -f %Lp "$SECRETS" 2>/dev/null)" -[ "$perms" = 600 ] || { echo "✗ $SECRETS must be chmod 600 (is $perms) — it holds credentials" >&2; exit 1; } -set -a; . "$SECRETS"; set +a - -# Required, no default ON PURPOSE: a baked-in default ref would silently keep -# probing a stale branch after the daemon branch merges to main. Every box -# states what it tests. -: "${CANARY_REF:?CANARY_REF unset — set it in $SECRETS (origin/failproofaid until #632 merges, then origin/main)}" -CLONE="${CANARY_CLONE:-$HOME/canary/failproofai}" -GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" -LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout - -# One run at a time — the local stand-in for GHA's `concurrency` group. A -# still-running yesterday (hung vendor CLI) must not race today's volume. -exec 9>"$STATE_DIR/.lock" -flock -n 9 || { echo "another canary run holds $STATE_DIR/.lock — exiting" >&2; exit 0; } - -TS="$(date -u +%Y%m%dT%H%M%SZ)" - -slack_note() { # $1 = text; best-effort, never fails the run - [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 - local payload - payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" - curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ - -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true -} - -# ── checkout the ref under test ────────────────────────────────────────────── -if [ ! -d "$CLONE/.git" ]; then - git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } -fi -git -C "$CLONE" fetch --prune origin \ - || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } -{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ - || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } -FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" -echo "── canary run $TS: $CANARY_REF @ $FP_SHA ──" - -# ── legs (the same two the GHA matrix ran; sequential on one Docker host) ──── -run_leg() { # $1 = channel - local channel="$1" leg_log="$STATE_DIR/logs/leg-$1-$TS.log" rc daemon state peer - if [ "$channel" = stable ]; then - # Stable probes the daemon-configured (failproofaid) path — the way-forward - # configuration users get from `failproofai config`. Beta stays in-process: - # it answers "is the vendor about to break us", which is independent of our - # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. - daemon="${CANARY_DAEMON_STABLE:-1}" - state="$STATE_DIR/integration-suite-state.json" - peer="" - else - daemon="${CANARY_DAEMON_BETA:-0}" - state="$STATE_DIR/integration-suite-state-$channel.json" - peer="$STATE_DIR/integration-suite-state.json" - fi - echo "── leg: $channel (daemon=$daemon) ──" - GITHUB_WORKSPACE="$CLONE" \ - CANARY_CHANNEL="$channel" \ - CANARY_STATE="$state" \ - CANARY_PEER_STATE="$peer" \ - CANARY_FP_SHA="$FP_SHA" \ - CANARY_DAEMON="$daemon" \ - CANARY_ENVFILE="$STATE_DIR/tmp/canary-$channel.env" \ - CANARY_TOKENS_DIR="$STATE_DIR/tmp/tokens-$channel" \ - timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" - rc=${PIPESTATUS[0]} - # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack - # already carries the story); non-zero WITHOUT one means the harness died - # before reporting, which on GHA surfaced as a red-job email and here would - # otherwise be silence. - if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then - slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — $CANARY_REF @ $FP_SHA -\`\`\` -$(tail -12 "$leg_log") -\`\`\`" - fi - return "$rc" -} - -rc_stable=0; rc_beta=0 -run_leg stable || rc_stable=$? -run_leg beta || rc_beta=$? - -find "$STATE_DIR/logs" -name '*.log' -mtime +14 -delete 2>/dev/null || true - -echo "── done: stable rc=$rc_stable, beta rc=$rc_beta ──" -[ "$rc_stable" -eq 0 ] && [ "$rc_beta" -eq 0 ] diff --git a/integration-suite/local/runner-daily.sh b/integration-suite/local/runner-daily.sh new file mode 100755 index 00000000..de36cbb7 --- /dev/null +++ b/integration-suite/local/runner-daily.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# The daily driver, invoked by the runner image's baked entrypoint AFTER it has +# locked, cloned and checked out $CANARY_REF into $CANARY_WORK/clone. It plays +# the role the GHA workflow YAML played — env → state paths → leg fan-out — +# then hands each leg to ci-entrypoint.sh, exactly as CI does. +# +# It lives IN THE REPO (not baked into the image) on purpose: the leg logic +# evolves with the harness, and the box picks changes up through the checkout — +# nobody rebuilds the boss's image for a harness tweak. +# +# Report delivery is unchanged (run.sh POSTs verdicts to CANARY_SLACK_WEBHOOK). +# What GHA gave for free — a notification when the JOB ITSELF died — is the +# crash-guard below: a leg that exits non-zero WITHOUT having posted its report +# gets a short Slack note carrying the log tail. +# ───────────────────────────────────────────────────────────────────────────── +set -u + +WORK="${CANARY_WORK:?CANARY_WORK missing — runner-entrypoint.sh sets it}" +CLONE="${CANARY_CLONE:-$WORK/clone}" +STATE_DIR="$WORK/state" +LOGS="$WORK/logs" +mkdir -p "$STATE_DIR" "$LOGS" +LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout + +# Everything a SIBLING container mounts must live under $WORK — the one dir +# shared with the host at an identical path. The daemon build's cargo cache is +# the only harness default rooted elsewhere ($HOME), so pin it here. +export CANARY_CARGO_CACHE="${CANARY_CARGO_CACHE:-$WORK/cargo}" + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +echo "── canary run $TS: ${CANARY_REF:-?} @ $FP_SHA ──" + +slack_note() { # $1 = text; best-effort, never fails the run + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +run_leg() { # $1 = channel + local channel="$1" leg_log="$LOGS/leg-$1-$TS.log" rc daemon state peer + if [ "$channel" = stable ]; then + # Stable probes the daemon-configured (failproofaid) path — the way-forward + # configuration users get from `failproofai config`. Beta stays in-process: + # it answers "is the vendor about to break us", which is independent of our + # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. + daemon="${CANARY_DAEMON_STABLE:-1}" + state="$STATE_DIR/integration-suite-state.json" + peer="" + else + daemon="${CANARY_DAEMON_BETA:-0}" + state="$STATE_DIR/integration-suite-state-$channel.json" + peer="$STATE_DIR/integration-suite-state.json" + fi + echo "── leg: $channel (daemon=$daemon) ──" + GITHUB_WORKSPACE="$CLONE" \ + CANARY_CHANNEL="$channel" \ + CANARY_STATE="$state" \ + CANARY_PEER_STATE="$peer" \ + CANARY_FP_SHA="$FP_SHA" \ + CANARY_DAEMON="$daemon" \ + timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" + rc=${PIPESTATUS[0]} + # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack + # already carries the story); non-zero WITHOUT one means the harness died + # before reporting, which on GHA surfaced as a red-job email and here would + # otherwise be silence. + if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then + slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — ${CANARY_REF:-?} @ $FP_SHA +\`\`\` +$(tail -12 "$leg_log") +\`\`\`" + fi + return "$rc" +} + +overall=0 +for channel in ${CANARY_LEGS:-stable beta}; do + run_leg "$channel" || overall=1 +done + +find "$LOGS" -name '*.log' -mtime +14 -delete 2>/dev/null || true + +echo "── done (overall rc=$overall) ──" +exit "$overall" diff --git a/integration-suite/local/runner-entrypoint.sh b/integration-suite/local/runner-entrypoint.sh new file mode 100755 index 00000000..933b8adc --- /dev/null +++ b/integration-suite/local/runner-entrypoint.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# Baked into the runner image (Dockerfile.runner). Keep this THIN and stable: +# preflight → work-dir detection → lock → checkout $CANARY_REF → hand off to +# integration-suite/local/runner-daily.sh FROM THE CHECKOUT. Everything that +# evolves with the harness lives in the repo side of that split, so changes +# reach the box through git without anyone rebuilding this image. +# ───────────────────────────────────────────────────────────────────────────── +set -u + +SOCK=/var/run/docker.sock +[ -S "$SOCK" ] || { echo "✗ docker socket not mounted — add: -v /var/run/docker.sock:/var/run/docker.sock" >&2; exit 1; } +docker info >/dev/null 2>&1 || { echo "✗ cannot talk to the host docker daemon through $SOCK" >&2; exit 1; } + +# The ONE host work dir, mounted at an IDENTICAL path inside and out +# (-v "$HOME/fp-canary:$HOME/fp-canary"). Identical is load-bearing: paths +# under it are used both for in-container file ops AND as sibling-container +# `-v` sources, which the HOST daemon resolves against the host filesystem. +# Auto-detected from this container's own mounts; CANARY_WORK settles it if +# more than one identical-path mount is present. +if [ -z "${CANARY_WORK:-}" ]; then + parity="$(docker inspect "$(cat /etc/hostname)" \ + --format '{{range .Mounts}}{{if eq .Source .Destination}}{{.Destination}}{{"\n"}}{{end}}{{end}}' 2>/dev/null \ + | grep -v '^/var/run/docker.sock$' | grep -v '^$' || true)" + case "$(printf '%s\n' "$parity" | grep -c .)" in + 1) CANARY_WORK="$parity" ;; + 0) echo "✗ no work dir found — mount one at an identical path: -v \"\$HOME/fp-canary:\$HOME/fp-canary\"" >&2; exit 1 ;; + *) echo "✗ several identical-path mounts found — set CANARY_WORK to the one to use:" >&2 + printf '%s\n' "$parity" >&2; exit 1 ;; + esac +fi +export CANARY_WORK +mkdir -p "$CANARY_WORK/logs" + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +exec > >(tee -a "$CANARY_WORK/logs/run-$TS.log") 2>&1 +echo "── canary runner $TS (work dir: $CANARY_WORK) ──" + +# One run at a time. The lock file lives on the host work dir, so overlapping +# cron fires — yesterday's run wedged on a vendor CLI — share one lock even +# though each is its own container. +exec 9>"$CANARY_WORK/.lock" +flock -n 9 || { echo "another canary run holds $CANARY_WORK/.lock — exiting"; exit 0; } + +slack_note() { # $1 = text; best-effort — the checkout phase's own crash-guard + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +# Required, no default ON PURPOSE: a baked-in ref would silently keep probing a +# stale branch after the daemon branch merges to main. The env file states it. +: "${CANARY_REF:?CANARY_REF missing from --env-file (origin/failproofaid until #632 merges, then origin/main)}" +CLONE="$CANARY_WORK/clone" +GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" + +if [ ! -d "$CLONE/.git" ]; then + git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } +fi +git -C "$CLONE" fetch --prune origin \ + || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } +{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ + || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } + +exec bash "$CLONE/integration-suite/local/runner-daily.sh" diff --git a/integration-suite/local/secrets.env.example b/integration-suite/local/secrets.env.example new file mode 100644 index 00000000..82aec522 --- /dev/null +++ b/integration-suite/local/secrets.env.example @@ -0,0 +1,43 @@ +# failproofai canary — box configuration. Copy to ~/fp-canary/secrets.env, +# fill in, chmod 600 (it holds credentials). +# +# This is a `docker --env-file` file, NOT a shell script: KEY=value lines +# only — no quotes, no $expansion, no spaces around `=`. `#` starts a comment. +# Same variables the GHA `cli-integration` Environment supplies — see +# integration-suite/ci-entrypoint.sh's header for what each one does. + +# ── REQUIRED: what to test ─────────────────────────────────────────────────── +# Deliberately explicit (the runner refuses to start without it): flip to +# origin/main once the failproofaid branch (#632) merges. +CANARY_REF=origin/failproofaid + +# ── gateway + PAT credentials ──────────────────────────────────────────────── +CANARY_LLM_API_KEY= +COPILOT_GITHUB_TOKEN= +#CANARY_LLM_BASE_URL=https://models.aikin.club +#CANARY_LLM_MODEL=deepseek-v4-pro +#CANARY_CLAUDE_MODEL=claude-haiku-4-5 +#CANARY_PI_MODEL=claude-haiku-4-5 +#CANARY_CODEX_MODEL=gpt-5.1-codex-mini + +# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── +# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh +# and paste the output here; the box itself never needs vendor logins. An empty +# value just makes that CLI report ERROR (can't auth), not a failed run. +CURSOR_TOKEN_TGZ_B64= +DEVIN_TOKEN_TGZ_B64= +ANTIGRAVITY_TOKEN_TGZ_B64= + +# ── reporting ──────────────────────────────────────────────────────────────── +CANARY_SLACK_WEBHOOK= + +# ── knobs (defaults shown) ─────────────────────────────────────────────────── +# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays +# in-process. Flip these to move the daemon dimension between legs. +#CANARY_DAEMON_STABLE=1 +#CANARY_DAEMON_BETA=0 +# Which legs to run — handy for support ("run just stable"). +#CANARY_LEGS=stable beta +# Force a full re-probe of all 12 CLIs (one-offs only; not in cron). +#CANARY_VERSION_GATED=none +#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git From 42139482e679516339bc22008c72085d86ed6eca Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 13:24:31 +0530 Subject: [PATCH 3/4] canary: port the fail-closed leg and its live-test lessons from the daemon test session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports the daemon-leg findings from the parallel host-run session that drove all three legs against 10 real, locally-installed CLIs (2026-08-07): the daemon does not regress enforcement on any CLI, denies land in 2-3ms warm versus 7-8ms cold — and the fail-closed pass surfaced an availability defect (factory fired 202 denied hook calls and antigravity 1,002, retrying a deny that can never succeed until the harness killed them at ten minutes) that only a fail-closed leg keeps visible. CANARY_DAEMON_DEAD=1 is that leg: configure the machine for the daemon exactly as CANARY_DAEMON=1 does, then never start it. Every CLI must DENY; the benign probe command executing anyway means the machine believed it was fail-closed and was not. The deny is scored through the existing daemon-unreachable detector, which also now breaks the probe retry loops early in live-daemon mode (a dead daemon denies everything — further LLM attempts can only reproduce the same deny) and prints a triage note so a mid-probe daemon death reads as DAEMON FAILED CLOSED instead of a quiet INCONCLUSIVE. Two hazards closed on the way in: - The DEAD leg gets its own state lane ($STATE.dead). Its PASS means "denied while dead" — recorded in the enforcement gate it would skip the next REAL probe of the same (CLI, failproofai) pair as already-green. - The daemon.configured marker is now cleared before wire() in EVERY mode and set only after it. wire() runs vendor CLIs whose hooks route through the marker (openclaw onboard), and a marker with no daemon up yet — set too early today, or surviving from yesterday in the persistent volume — would fail-close the wiring itself. Also carried from that session's debugging: the SUN_LEN (108-byte) Unix socket path cap is documented on the socket-path choice. All of it pinned in __tests__/integration-suite/local-runner.test.ts (49 tests). Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- .../integration-suite/local-runner.test.ts | 54 ++++++++++++ integration-suite/ci-entrypoint.sh | 8 +- integration-suite/local/secrets.env.example | 3 + integration-suite/probe-cli.sh | 85 ++++++++++++++----- integration-suite/run.sh | 19 +++-- 6 files changed, 144 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71e8024e..8b2e6f36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index 9688a7b7..5cf715d0 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -179,6 +179,60 @@ describe("daemon-mode probe path", () => { expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:false\}\}\)/); }); + it("CANARY_DAEMON_DEAD implies daemon mode in every layer", () => { + // The fail-closed leg (daemon-configured, daemon never started — every CLI + // must deny; live-verified 2026-08-07 against 10 real CLIs). Each layer + // normalizes independently because each can be invoked directly. + for (const [name, src] of [ + ["probe-cli.sh", probeSh], + ["run.sh", runSh], + ["ci-entrypoint.sh", entrypoint], + ] as const) { + expect(src, `${name} missing the DEAD→DAEMON normalization`).toMatch( + /CANARY_DAEMON_DEAD[^\n]*&& CANARY_DAEMON=1|CANARY_DAEMON_DEAD[^\n]*\]; then CANARY_DAEMON=1/, + ); + } + }); + + it("the DEAD leg never starts the daemon, needs no binary, and skips the build", () => { + expect(probeSh).toMatch(/CANARY_DAEMON_DEAD[^\n]*mkdir -p "\$1"; return 0/); + // run.sh: the binary requirement sits inside the not-DEAD guard + expect(runSh).toMatch(/CANARY_DAEMON_DEAD[^\n]*!= 1[\s\S]{0,200}CANARY_DAEMON_BIN/); + // ci-entrypoint: the cargo build is skipped on the DEAD leg + expect(entrypoint).toMatch(/CANARY_DAEMON[^\n]*= 1[^\n]*&&[^\n]*CANARY_DAEMON_DEAD[^\n]*!= 1/); + }); + + it("the DEAD leg scores the fail-closed deny as PASS on both probes", () => { + const scored = probeSh.match( + /\[ "\$\{CANARY_DAEMON_DEAD:-0\}" = 1 \] && daemon_failed_closed "\$LOG[AB]\/hooks\.log"; then V[AB]=PASS/g, + ); + expect(scored?.length).toBe(2); + // and the detector keys on the synthetic fail-closed policy name + expect(probeSh).toMatch(/daemon_failed_closed\(\) \{ grep -q "daemon-unreachable"/); + }); + + it("DEAD-leg results live in their own state lane", () => { + // A PASS on the DEAD leg means "denied while dead". Written into the + // enforcement gate it would skip the next REAL probe of the same + // (CLI, failproofai) pair as already-green. + expect(runSh).toMatch(/STATE="\$STATE\.dead"/); + }); + + it("the marker is cleared BEFORE wire and set AFTER it (wire fires vendor hooks)", () => { + // A marker with no daemon up yet would fail-close the vendor CLI calls + // wire() itself makes (openclaw onboard fires plugin hooks), breaking the + // wiring before any probe runs. That marker can come from TODAY (set too + // early) or YESTERDAY (persistent volume) — so the clear must run in every + // mode before wire, and daemon mode re-sets only after wire. + const wireCall = probeSh.indexOf("\nwire\n"); + const markerClear = probeSh.indexOf("m.updateConfig({daemon:{configured:false}})"); + const markerSet = probeSh.indexOf("m.updateConfig({daemon:{configured:true}})"); + expect(wireCall).toBeGreaterThan(0); + expect(markerClear).toBeGreaterThan(0); + expect(markerClear).toBeLessThan(wireCall); + expect(markerSet).toBeGreaterThan(wireCall); + }); + it("a dead daemon cannot false-PASS either probe", () => { // bin/failproofai.mjs shapes an unreachable-daemon deny through a // synthetic policy (see handler.ts's forceDecision branch). Its oracle diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index 12647526..38942152 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -41,6 +41,9 @@ # CANARY_DAEMON set to 1 to probe the daemon-configured (failproofaid) # hook path: builds the Rust daemon and routes every # probe's hooks through it, fail-closed (see probe-cli.sh) +# CANARY_DAEMON_DEAD set to 1 for the fail-closed leg: daemon-configured +# but the daemon is never started — every CLI must +# DENY. Implies CANARY_DAEMON=1; skips the Rust build. # CANARY_CARGO_CACHE cargo home+target cache dir for the daemon build # (default ~/.cache/failproofai-canary/cargo) # ───────────────────────────────────────────────────────────────────────────── @@ -111,7 +114,9 @@ fi # node:22-bookworm-slim (glibc 2.36), and a binary linked against a newer host # glibc would fail to load inside it. The repo mounts read-only — cargo writes # only to the mounted cache (registry + target), so the checkout stays clean. -if [ "${CANARY_DAEMON:-0}" = 1 ]; then +# The DEAD (fail-closed) leg needs no binary — the daemon is never started. +[ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && CANARY_DAEMON=1 +if [ "${CANARY_DAEMON:-0}" = 1 ] && [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then step "building failproofaid (daemon) under test" if [ ! -f "$REPO/crates/failproofaid/Cargo.toml" ]; then echo "✗ CANARY_DAEMON=1 but $REPO has no crates/failproofaid — this ref predates the daemon; unset CANARY_DAEMON or pick a ref that carries it" >&2 @@ -204,5 +209,6 @@ CANARY_ENVFILE="$ENVFILE" \ CANARY_CHANNEL="$CHANNEL" \ CANARY_PEER_STATE="$PEER_STATE" \ CANARY_DAEMON="${CANARY_DAEMON:-0}" \ +CANARY_DAEMON_DEAD="${CANARY_DAEMON_DEAD:-0}" \ CANARY_DAEMON_BIN="${CANARY_DAEMON_BIN:-}" \ bash "$HERE/run.sh" ${CANARY_CLIS:-} diff --git a/integration-suite/local/secrets.env.example b/integration-suite/local/secrets.env.example index 82aec522..2c44fa99 100644 --- a/integration-suite/local/secrets.env.example +++ b/integration-suite/local/secrets.env.example @@ -38,6 +38,9 @@ CANARY_SLACK_WEBHOOK= #CANARY_DAEMON_BETA=0 # Which legs to run — handy for support ("run just stable"). #CANARY_LEGS=stable beta +# One-off fail-closed audit: daemon-configured but never started — every CLI +# must DENY. Results go to a separate state lane; not part of the daily legs. +#CANARY_DAEMON_DEAD=1 # Force a full re-probe of all 12 CLIs (one-offs only; not in cron). #CANARY_VERSION_GATED=none #CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git diff --git a/integration-suite/probe-cli.sh b/integration-suite/probe-cli.sh index a60f326c..d8110f9b 100644 --- a/integration-suite/probe-cli.sh +++ b/integration-suite/probe-cli.sh @@ -70,6 +70,14 @@ export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" # the host by ci-entrypoint.sh (rust:1-bookworm, so its glibc matches this # sandbox) and bind-mounted at /opt/failproofaid/failproofaid by run.sh. # +# CANARY_DAEMON_DEAD=1 is the fail-closed probe: configure the machine for the +# daemon exactly as CANARY_DAEMON=1 does, then never start it. On a +# daemon-configured machine an unreachable daemon must DENY every hook event; +# if the benign probe command runs anyway, the machine believed it was +# fail-closed and was not. (Live-verified 2026-08-07 against 10 real CLIs: all +# denied — and factory/antigravity retry-stormed the deny for 10 minutes, an +# availability finding this leg exists to keep visible.) +# # The daemon is started PER PROBE, not once per CLI. The worker inherits the # DAEMON's environment — the wire protocol carries only {hookEvent, cli, # stdin, cwd}, never the hook process's env — so FAILPROOFAI_HOOK_LOG_FILE @@ -77,11 +85,7 @@ export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" # probe's log dir. Sharing one log dir across both probes instead would let # probe A's incidental denies (an agent exploring with reads trips # block-read-outside-cwd) satisfy probe B's grep — a false PASS. -# -# A DEAD daemon cannot false-PASS: the client's fail-closed deny is shaped by -# a synthetic `failproofai/daemon-unreachable` policy (bin/failproofai.mjs), -# which denied()/read_denied() below can never match — those probes go -# INCONCLUSIVE and re-probe until the daemon path recovers. +[ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && CANARY_DAEMON=1 DAEMON_PID="" daemon_stop() { [ -n "$DAEMON_PID" ] || return 0 @@ -91,6 +95,10 @@ daemon_stop() { } daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) [ "${CANARY_DAEMON:-0}" = 1 ] || return 0 + # Fail-closed probe: the daemon is deliberately never started. The client's + # forced deny is evaluated in-process, so its oracle lands in the CLI hook + # process's own env — the log dir still needs to exist. + if [ "${CANARY_DAEMON_DEAD:-0}" = 1 ]; then mkdir -p "$1"; return 0; fi daemon_stop rm -f "$FAILPROOFAI_DAEMON_SOCKET" # Env is the worker's too (worker.rs spawns `sh -c "$FAILPROOFAI_WORKER_CMD"` @@ -111,27 +119,25 @@ daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) exit 1 } if [ "${CANARY_DAEMON:-0}" = 1 ]; then - [ -x /opt/failproofaid/failproofaid ] \ - || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + if [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + [ -x /opt/failproofaid/failproofaid ] \ + || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + fi # Socket under /tmp: container-local, so a stale socket file in the PERSISTENT # volume can never shadow a live daemon across daily runs. The override # relocates the whole run dir — lock and worker.sock land beside it — and the # dir is NOT pre-created here: failproofaid creates it 0700 itself and refuses - # one it didn't create with other perms (paths.rs ensure_run_dir). + # one it didn't create with other perms (paths.rs ensure_run_dir). Keep the + # path SHORT and FLAT: a Unix socket path is capped at SUN_LEN (108 bytes on + # Linux) and the daemon dies before its first accept when the cap is blown. export FAILPROOFAI_DAEMON_SOCKET="/tmp/fpai-canary/failproofaid.sock" - # The fail-closed marker, written via the REAL code path (fp-config's - # updateConfig) rather than shell-appending TOML — the volume's config.toml - # survives with its other tables intact, and a duplicate [daemon] table - # (invalid TOML) would silently read back as NOT configured. - bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ - || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } trap daemon_stop EXIT -else - # The HOME volume persists across runs: a marker left behind by a daemon-mode - # run would make this in-process run fail closed on every hook event with no - # daemon anywhere. Clear it unconditionally. - bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true fi +# The HOME volume persists across runs, so YESTERDAY's marker survives into +# today. Clear it EARLY in every mode — before install/wire — because wire() +# runs vendor CLIs (openclaw onboard fires plugin hooks) that would fail closed +# against a marker with no daemon up yet. Daemon mode re-sets it after wire. +bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true BASE="$HOME/probe-$CLI" # DEFINITE probes: BENIGN actions (echo/touch a token, read a plain file) the @@ -241,7 +247,28 @@ printf '%s\n' "$MARKER_CONTENT" > "$BASE/CANARY_MARKER.txt" install_hooks wire +# The fail-closed marker is set AFTER install/wire, not before: wire() runs +# vendor CLIs (openclaw onboard fires its plugin hooks), and a marker with no +# daemon up yet would fail-close those calls and break the wiring itself. The +# installer never routes through the daemon either way (only `--hook` does). +# Written via the REAL code path (fp-config's updateConfig) rather than +# shell-appending TOML — the volume's config.toml survives with its other +# tables intact, and a duplicate [daemon] table (invalid TOML) would silently +# read back as NOT configured. +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ + || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } + echo " daemon: socket=$FAILPROOFAI_DAEMON_SOCKET configured=true dead=${CANARY_DAEMON_DEAD:-0}" +fi + denied() { grep -qE "result=deny policy=(failproofai/|custom/)?$1" "$2" 2>/dev/null; } +# A fail-closed deny (synthetic policy `failproofai/daemon-unreachable`, shaped +# by bin/failproofai.mjs) means the daemon was unreachable. It denies EVERY +# event, so probe A's marker never appears and probe B never leaks — silently +# reading as INCONCLUSIVE. It can never match denied()/read_denied(), so it +# can never forge a PASS; detect it so a dead daemon is loud, and so the +# CANARY_DAEMON_DEAD leg can score the deny as its expected outcome. +daemon_failed_closed() { grep -q "daemon-unreachable" "$1" 2>/dev/null; } # The read probe accepts EITHER the benign canary-read deny OR block-read-outside-cwd: # both are failproofai denies on the CLI's read payload, and BOTH require a populated # tool_input.file_path (so a real normalization-drift still leaks the sentinel → FAIL, @@ -283,14 +310,20 @@ for _ in $(seq 1 $ATTEMPTS); do OUTA="$(drive "$BASH_PROMPT" 2>&1)" denied canary-bash "$LOGA/hooks.log" && break [ -f "$BASE/CANARY_PROBE_ran" ] && break + # A dead daemon denies everything — further LLM attempts can only reproduce + # the same fail-closed deny, so stop burning them. + daemon_failed_closed "$LOGA/hooks.log" && break done # Ground truth FIRST. A CLI that logs our deny and runs the command anyway is # the silent-allow this suite exists to catch (copilot 1.0.70 did exactly that) # — and checking our own hooks.log first scored it PASS, because our log says # "denied" whether or not the CLI honoured it. The marker file is the only -# evidence of what the CLI actually did, so it decides. +# evidence of what the CLI actually did, so it decides. On the DEAD leg the +# marker existing is the catastrophic finding: the machine believed it was +# fail-closed and executed anyway. if [ -f "$BASE/CANARY_PROBE_ran" ]; then VA=FAIL elif denied canary-bash "$LOGA/hooks.log"; then VA=PASS +elif [ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && daemon_failed_closed "$LOGA/hooks.log"; then VA=PASS elif is_error "$OUTA"; then VA=ERROR else VA=INCONCLUSIVE; fi @@ -304,11 +337,13 @@ for _ in $(seq 1 $ATTEMPTS); do OUTB="$(drive "$READ_PROMPT" 2>&1)" read_denied "$LOGB/hooks.log" && break printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT" && break + daemon_failed_closed "$LOGB/hooks.log" && break done # Same ordering rule as probe A: the sentinel leaking into the transcript proves # the read happened, which outranks our own log claiming we denied it. if printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT"; then VB=FAIL elif read_denied "$LOGB/hooks.log"; then VB=PASS +elif [ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && daemon_failed_closed "$LOGB/hooks.log"; then VB=PASS elif is_error "$OUTB"; then VB=ERROR else VB=INCONCLUSIVE; fi @@ -317,4 +352,14 @@ echo " Probe A (touch token → canary-bash) : $VA" echo " Probe B (read marker → canary-read) : $VB" echo "--- deny evidence in oracle ---" grep -E "result=deny" "$LOGA/hooks.log" "$LOGB/hooks.log" 2>/dev/null | sed 's#.*/hooks.log:# #' | head -4 +# Triage note for the LIVE daemon leg: fail-closed denies mid-probe mean these +# verdicts measured the fail-closed path, not per-CLI enforcement — say so +# rather than leaving a quiet INCONCLUSIVE to be misread as "model didn't try". +if [ "${CANARY_DAEMON:-0}" = 1 ] && [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + if daemon_failed_closed "$LOGA/hooks.log" || daemon_failed_closed "$LOGB/hooks.log"; then + echo " ⚠️ DAEMON FAILED CLOSED mid-probe — verdicts reflect the fail-closed path, NOT per-CLI enforcement; see $BASE/daemon.log" + else + echo " daemon: routed, no fail-closed denies (verdicts reflect real daemon evaluation)" + fi +fi printf 'VERDICT_JSON {"cli":"%s","probes":{"bash":"%s","read":"%s"}}\n' "$CLI" "$VA" "$VB" diff --git a/integration-suite/run.sh b/integration-suite/run.sh index b4df7c5d..26dd615f 100644 --- a/integration-suite/run.sh +++ b/integration-suite/run.sh @@ -64,13 +64,22 @@ VERSIONS_JSON="$(docker run --rm -v "$VOL:/home/canary" "$IMAGE" cat /home/canar # Daemon mode: bind the host-built failproofaid binary into the probe container # and tell probe-cli.sh to route hooks through it (see its CANARY_DAEMON block). # A file→file bind mount, read-only — executing from an ro mount is fine. +# CANARY_DAEMON_DEAD=1 (fail-closed leg) implies daemon mode but needs no +# binary: the daemon is deliberately never started, only the marker is set. +# It also gets its OWN state lane: a PASS here means "denied while dead", and +# recording that as green in the enforcement gate would skip the next REAL +# probe of the same (CLI, failproofai) pair as already-verified. +if [ "${CANARY_DAEMON_DEAD:-0}" = 1 ]; then CANARY_DAEMON=1; STATE="$STATE.dead"; fi DAEMON_FLAGS=() if [ "${CANARY_DAEMON:-0}" = 1 ]; then - DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" - [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } - # docker reads a relative -v source as a NAMED VOLUME — absolutize first. - DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" - DAEMON_FLAGS=(-e CANARY_DAEMON=1 -v "$DBIN:/opt/failproofaid/failproofaid:ro") + DAEMON_FLAGS=(-e CANARY_DAEMON=1 -e "CANARY_DAEMON_DEAD=${CANARY_DAEMON_DEAD:-0}") + if [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" + [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } + # docker reads a relative -v source as a NAMED VOLUME — absolutize first. + DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" + DAEMON_FLAGS+=(-v "$DBIN:/opt/failproofaid/failproofaid:ro") + fi fi run_probe() { From 79ee17a548658e62fcba8103068c39b236395d62 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 13:36:34 +0530 Subject: [PATCH 4/4] chore: fill in the canary PR number in the changelog (#656) Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b2e6f36..77bd6519 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#656) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR)