From f36d9acdfedeca27125a5ec5b42bef8a820a6c28 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 11:35:07 +0530 Subject: [PATCH 1/7] canary: move the daily integration suite to a local box and probe the daemon path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Daily runs leave GH Actions (runner minutes were the entire cost; the LLM spend is identical either way) for a local canary box driven by a systemd user timer in the retired cron's 06:17 UTC slot. integration-suite/local/ ships the box side: run-local.sh (checkout CANARY_REF → stable leg → beta leg, flock-serialized, with a crash-guard Slack note for a leg that dies BEFORE reporting — the replacement for GHA's red-job email), install.sh (installed copy OUTSIDE the clone the wrapper hard-resets, systemd units, secrets template), and the service/timer units. The workflow keeps workflow_dispatch as the cloud fallback and loses its cron. The stable leg now probes the daemon-configured path (CANARY_DAEMON=1) — the configuration `failproofai config` gives users going forward: ci-entrypoint cross-compiles failproofaid in rust:1-bookworm (glibc-matched to the node:22-bookworm-slim sandbox; a host build can link newer symbols and fail to load inside it), run.sh bind-mounts it into the probe container, and probe-cli.sh sets the daemon.configured fail-closed marker through the real fp-config updateConfig path — shell-appending TOML could produce a duplicate [daemon] table, which parses as NOT configured and silently falls back to in-process. The daemon restarts per probe, not per CLI: the wire protocol forwards {hookEvent, cli, stdin, cwd} and never env, so the warm worker's FAILPROOFAI_HOOK_LOG_FILE is fixed at daemon start — one daemon across both probes would share one oracle dir, and probe A's incidental read-denies would satisfy probe B's grep (false PASS). A dead daemon cannot false-PASS either: its deny is shaped by the synthetic failproofai/daemon-unreachable policy, which the probes' greps never match — those probes go INCONCLUSIVE and re-probe until the daemon path recovers. Verified without LLM or secrets in the real sandbox image: live daemon → canary-bash deny through the socket → warm worker writes the per-probe oracle; killed daemon → fail-closed deny logged as daemon-unreachable, matching neither probe grep; marker cleared → in-process evaluation restored. Tripwires in __tests__/integration-suite/local-runner.test.ts pin the workflow staying cron-free, the unit↔installer paths, the secrets-template↔workflow-env parity, the per-probe daemon restarts, the marker hygiene, and the fail-closed/oracle non-overlap (both sides extracted from the real sources). Co-Authored-By: Claude Fable 5 --- .github/workflows/integration-suite.yml | 28 +-- CHANGELOG.md | 3 + .../integration-suite/local-runner.test.ts | 160 ++++++++++++++++++ integration-suite/README.md | 69 ++++++-- integration-suite/ci-entrypoint.sh | 29 ++++ .../local/failproofai-canary.service | 14 ++ .../local/failproofai-canary.timer | 13 ++ integration-suite/local/install.sh | 91 ++++++++++ integration-suite/local/run-local.sh | 116 +++++++++++++ integration-suite/probe-cli.sh | 72 ++++++++ integration-suite/run.sh | 14 +- 11 files changed, 583 insertions(+), 26 deletions(-) create mode 100644 __tests__/integration-suite/local-runner.test.ts create mode 100644 integration-suite/local/failproofai-canary.service create mode 100644 integration-suite/local/failproofai-canary.timer create mode 100755 integration-suite/local/install.sh create mode 100755 integration-suite/local/run-local.sh diff --git a/.github/workflows/integration-suite.yml b/.github/workflows/integration-suite.yml index 0780ea08..b581c277 100644 --- a/.github/workflows/integration-suite.yml +++ b/.github/workflows/integration-suite.yml @@ -1,17 +1,25 @@ name: Integration Suite -# Daily integration test: does failproofai still ENFORCE against every supported -# agent CLI @latest? Installs all 12 CLIs into an isolated Docker sandbox, drives -# each one against failproofai's OWN policies (built from THIS repo's HEAD), and -# asserts the hook log shows a DENY. A silent-allow — a blocked action that ran -# with no deny — means enforcement broke against that CLI (e.g. a vendor changed -# their hook schema out from under us), and turns the run red. Reports only -# CHANGES (broke/recovered) plus a daily heartbeat to Slack. +# ON-DEMAND FALLBACK for the integration suite: does failproofai still ENFORCE +# against every supported agent CLI @latest? Installs all 12 CLIs into an +# isolated Docker sandbox, drives each one against failproofai's OWN policies +# (built from THIS repo's HEAD), and asserts the hook log shows a DENY. A +# silent-allow — a blocked action that ran with no deny — means enforcement +# broke against that CLI (e.g. a vendor changed their hook schema out from +# under us), and turns the run red. Reports only CHANGES (broke/recovered) +# plus a heartbeat to Slack. +# +# The DAILY runs moved off Actions to a local canary box for cost — +# integration-suite/local/ carries the systemd timer + wrapper that replaced +# the cron that used to live here (same 06:17 UTC slot). This workflow stays +# dispatch-only: the cloud escape hatch for when the box is down or a clean +# cloud reproduction is wanted. Its Actions-cache state is separate from the +# box's state dir, so a dispatch may re-probe CLIs the box already gated. # # Unlike the unit/e2e suites, this drives REAL vendor CLIs against real gateway # models, so it needs credentials. They live in the `cli-integration` Environment -# and the workflow runs ONLY on schedule / manual dispatch — never on pull_request -# — so fork PRs can never reach the secrets. +# and the workflow runs ONLY on manual dispatch — never on pull_request — so +# fork PRs can never reach the secrets. # # This file is a THIN TRIGGER on purpose. Everything beyond the GitHub-specific # wiring (checkout, bun, cache, secret->env mapping) lives in @@ -19,8 +27,6 @@ name: Integration Suite # without opening this YAML. See integration-suite/README.md. on: - schedule: - - cron: "17 6 * * *" # ~06:17 UTC daily workflow_dispatch: inputs: clis: diff --git a/CHANGELOG.md b/CHANGELOG.md index a9e93431..0c2b0cd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -212,6 +212,9 @@ never "blocked". ## 1.0.0-beta.11 — 2026-08-07 +### Features +- Move the daily CLI integration suite off GH Actions onto a local canary box, and make it probe the daemon path. `integration-suite/local/` ships the box side — a wrapper that checks out `CANARY_REF`, runs both legs through the unchanged `ci-entrypoint.sh`, and Slack-notes any leg that dies *before* reporting (the replacement for GHA's red-job email) — plus an installer and a systemd user timer holding the same 06:17 UTC slot; the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) + ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) - Make the Rust daemon enforce the same cloud-URL rule the TS side does. `CloudClient::new()` checked only that the scheme was `http` or `https`, so `http://internal-host` was accepted and `spawn_maintenance()` then put the org-scoped `policies:pull` bearer token on the wire **in clear, every 30 seconds**. `validateCloudUrl()` in `cloud-enrollment.ts` has always blocked non-loopback `http`, and `configure-wizard.ts` carries a comment asserting the daemon enforces the same rule — it did not. It matters most on the path the TS validator cannot cover: `FAILPROOFAI_CLOUD_URL` takes precedence over the credentials file and is a documented CI/container knob, so it reaches the constructor without passing through the wizard. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts new file mode 100644 index 00000000..b882b7e0 --- /dev/null +++ b/__tests__/integration-suite/local-runner.test.ts @@ -0,0 +1,160 @@ +/** + * Tripwires for the LOCAL canary runner (integration-suite/local/) and the + * daemon-mode (CANARY_DAEMON) probe path. + * + * Daily integration-suite runs moved off GH Actions onto a local box + * (2026-08-07, for runner-minute cost); the stable leg there probes the + * daemon-configured (failproofaid) hook path — the way-forward configuration. + * Everything below is shell scripts and systemd units with no importable + * surface, so the tests parse the real files — same approach as + * channel-refs.test.ts, and for the same reason: the alternative is a second + * copy of each contract to drift against. + */ +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; + +const ROOT = path.join(__dirname, "../.."); +const SUITE = path.join(ROOT, "integration-suite"); +const LOCAL = path.join(SUITE, "local"); +const runLocal = readFileSync(path.join(LOCAL, "run-local.sh"), "utf8"); +const installSh = readFileSync(path.join(LOCAL, "install.sh"), "utf8"); +const service = readFileSync(path.join(LOCAL, "failproofai-canary.service"), "utf8"); +const timer = readFileSync(path.join(LOCAL, "failproofai-canary.timer"), "utf8"); +const workflow = readFileSync( + path.join(ROOT, ".github/workflows/integration-suite.yml"), + "utf8", +); +const probeSh = readFileSync(path.join(SUITE, "probe-cli.sh"), "utf8"); +const runSh = readFileSync(path.join(SUITE, "run.sh"), "utf8"); +const entrypoint = readFileSync(path.join(SUITE, "ci-entrypoint.sh"), "utf8"); +const handlerTs = readFileSync(path.join(ROOT, "src/hooks/handler.ts"), "utf8"); + +describe("GHA workflow is dispatch-only", () => { + it("has no schedule trigger (daily runs live on the canary box)", () => { + // Dropping the daily cron WAS the cost decision. A schedule reappearing + // here must be deliberate — this makes it a conversation, not a re-spend. + expect(workflow).not.toMatch(/^\s*schedule:/m); + expect(workflow).not.toMatch(/\bcron:/); + }); + + it("keeps workflow_dispatch as the cloud fallback", () => { + expect(workflow).toMatch(/^\s*workflow_dispatch:/m); + }); +}); + +describe("local runner wiring", () => { + it("service ExecStart points at the exact path install.sh installs to", () => { + // run-local.sh hard-resets the runner clone, so the unit must exec the + // INSTALLED copy — a unit pointing into the clone would run whatever the + // checked-out ref happens to carry, mid-reset. + const m = /^ExecStart=%h\/(\S+)$/m.exec(service); + expect(m).not.toBeNull(); + expect(m![1]).toBe(".config/failproofai-canary/bin/run-local.sh"); + expect(installSh).toContain("CANARY_CONF_DIR:-$HOME/.config/failproofai-canary"); + expect(installSh).toMatch( + /install -m 755 "\$HERE\/run-local\.sh" "\$CONF_DIR\/bin\/run-local\.sh"/, + ); + }); + + it("install.sh installs both systemd units", () => { + expect(installSh).toContain("failproofai-canary.service"); + expect(installSh).toContain("failproofai-canary.timer"); + }); + + it("timer keeps the retired GHA cron slot and catches up after downtime", () => { + expect(timer).toMatch(/OnCalendar=.*06:17.*UTC/); + expect(timer).toMatch(/^Persistent=true$/m); + }); + + it("run-local.sh drives the same front door CI does", () => { + expect(runLocal).toContain("integration-suite/ci-entrypoint.sh"); + }); + + it("refuses to run without an explicit CANARY_REF", () => { + // A baked-in default ref would silently keep probing a stale branch after + // the daemon branch merges to main — every box states what it tests. + expect(runLocal).toMatch(/\$\{CANARY_REF:\?/); + }); + + it("stable leg defaults to the daemon path, beta to in-process", () => { + expect(runLocal).toContain("${CANARY_DAEMON_STABLE:-1}"); + expect(runLocal).toContain("${CANARY_DAEMON_BETA:-0}"); + }); + + it("secrets template offers every secret-fed env var the workflow maps", () => { + // The box's secrets.env and the GHA Environment must stay interchangeable. + // A secret added to the workflow but not the template means the box runs + // without it and that CLI quietly reports ERROR forever. + const envNames = [...workflow.matchAll(/^\s+([A-Z0-9_]+):\s+\$\{\{\s*secrets\./gm)].map( + (m) => m[1], + ); + expect(envNames.length).toBeGreaterThanOrEqual(10); + for (const name of envNames) { + expect(installSh, `secrets.env template is missing ${name}`).toContain(name); + } + }); +}); + +describe("daemon-mode probe path", () => { + it("ci-entrypoint builds the daemon in a bookworm container and hands it to run.sh", () => { + // rust:1-bookworm ⇔ node:22-bookworm-slim sandbox: same glibc line. A + // host build can link a newer glibc and fail to load inside the sandbox. + expect(entrypoint).toMatch(/rust:1-bookworm cargo build --release --locked -p failproofaid/); + expect(entrypoint).toMatch(/^CANARY_DAEMON=/m); + expect(entrypoint).toMatch(/^CANARY_DAEMON_BIN=/m); + }); + + it("run.sh mounts the binary exactly where probe-cli.sh executes it", () => { + const mount = /-v "\$DBIN:(\S+):ro"/.exec(runSh); + expect(mount).not.toBeNull(); + expect(probeSh).toContain(mount![1]); + }); + + it("probe-cli.sh restarts the daemon per probe (oracle isolation)", () => { + // The wire protocol forwards {hookEvent, cli, stdin, cwd} — never env — so + // the worker's FAILPROOFAI_HOOK_LOG_FILE is fixed at daemon start. One + // daemon across both probes would mean one shared log dir, and probe A's + // incidental read-denies (an agent exploring before it touches the marker) + // would satisfy probe B's grep: a false PASS. + expect(probeSh).toContain('daemon_cycle "$LOGA"'); + expect(probeSh).toContain('daemon_cycle "$LOGB"'); + }); + + it("sets the fail-closed marker in daemon mode and clears it otherwise", () => { + // The HOME volume persists across daily runs: a marker left behind by a + // daemon-mode run would make an in-process run fail closed on every hook + // event with no daemon anywhere. + expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:true\}\}\)/); + expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:false\}\}\)/); + }); + + it("a dead daemon cannot false-PASS either probe", () => { + // bin/failproofai.mjs shapes an unreachable-daemon deny through a + // synthetic policy (see handler.ts's forceDecision branch). Its oracle + // line must NEVER satisfy the probes' deny greps — otherwise a crashed + // daemon reads as healthy enforcement, the exact inversion this suite + // exists to catch. Both sides are extracted from the real sources so a + // rename on either side trips this test. + const idMatch = /registerPolicy\(\s*"(failproofai\/[a-z-]+)",\s*"Fail-closed/.exec(handlerTs); + expect(idMatch).not.toBeNull(); + // handler.ts:463 — `result=${decision} policy=${policyName} duration=…` + const failClosedLine = `result=deny policy=${idMatch![1]} duration=3ms`; + + const deniedPat = /denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); + const readDeniedPat = /read_denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); + expect(deniedPat).not.toBeNull(); + expect(readDeniedPat).not.toBeNull(); + + const deniedRe = new RegExp(deniedPat![1].replace("$1", "canary-bash")); + const readDeniedRe = new RegExp(readDeniedPat![1]); + // Sanity both ways: the real canary/builtin lines must still match… + expect(deniedRe.test("result=deny policy=custom/canary-bash duration=2ms")).toBe(true); + expect( + readDeniedRe.test("result=deny policy=failproofai/block-read-outside-cwd duration=2ms"), + ).toBe(true); + // …and the fail-closed line must match neither. + expect(deniedRe.test(failClosedLine)).toBe(false); + expect(readDeniedRe.test(failClosedLine)).toBe(false); + }); +}); diff --git a/integration-suite/README.md b/integration-suite/README.md index b98ea94d..f549a22b 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -4,27 +4,65 @@ A daily **live-enforcement integration test** for failproofai. It answers one question the unit/e2e suites can't: *does failproofai still enforce against every supported agent CLI, at the versions users actually install today?* -Every day (`.github/workflows/integration-suite.yml`) it installs all 12 agent -CLIs **@latest** into an isolated Docker sandbox, drives each one against -failproofai's own policies (built from this repo's HEAD), and confirms the hook -log shows a **DENY**. A *silent-allow* — a blocked action that ran with no deny — -means enforcement broke against that CLI (e.g. a vendor changed their hook schema -out from under us). The test asserts the deny **positively**, so drift surfaces -as a red run + a Slack alert instead of going unnoticed until a user hits it. +Every day (a systemd user timer on the canary box — see **Local runner** below; +`.github/workflows/integration-suite.yml` is the on-demand cloud fallback) it +installs all 12 agent CLIs **@latest** into an isolated Docker sandbox, drives +each one against failproofai's own policies (built from the ref under test), and +confirms the hook log shows a **DENY**. A *silent-allow* — a blocked action that +ran with no deny — means enforcement broke against that CLI (e.g. a vendor +changed their hook schema out from under us). The test asserts the deny +**positively**, so drift surfaces as a red run + a Slack alert instead of going +unnoticed until a user hits it. + +On the box, the stable leg also runs **daemon-configured** (`CANARY_DAEMON=1`): +hooks route CLI → `failproofaid` (Rust supervisor) → warm bun worker, fail-closed +— the configuration `failproofai config` gives users — so the canary tests the +transport users actually run, not just the in-process path. See the +`CANARY_DAEMON` block in `probe-cli.sh` for the mechanics (per-probe daemon +restarts, the `daemon.configured` marker, and why a dead daemon scores +INCONCLUSIVE rather than a false PASS). ## Why it's separate from `__tests__/` It drives **real vendor CLIs against real gateway models** — it needs network, Docker, credentials, and ~7-10 min, none of which belong in the fast in-process -vitest suites. So it's a scheduled workflow, not a PR gate. +vitest suites. So it's a scheduled run, not a PR gate. + +## Local runner (the daily driver) + +Daily runs live on a **local canary box**, not GH Actions — runner minutes were +the entire cost of the old daily cron; the LLM spend is identical either way. +`local/` holds everything box-side: + +``` +local/run-local.sh the cron replacement: checkout CANARY_REF → + stable leg (daemon) → beta leg → crash-guard +local/install.sh box setup: installed copy + units + secrets template +local/failproofai-canary.service systemd user unit (oneshot, 4h ceiling) +local/failproofai-canary.timer daily 06:17 UTC, Persistent=true +``` + +Box setup: clone the repo anywhere once, `bash integration-suite/local/install.sh`, +fill `~/.config/failproofai-canary/secrets.env` (same variables the GHA +Environment supplied; token tarballs still come from `capture-tokens.sh` on a +logged-in machine), `loginctl enable-linger`, enable the timer. The wrapper runs +from an **installed copy** because it hard-resets the runner clone every run — +nothing that must survive a run may live inside the clone. + +State (`integration-suite-state[-beta].json`) sits in +`~/.local/state/failproofai-canary/` instead of the Actions cache; the +version-gate logic is unchanged. Verdict reports POST to Slack exactly as +before; a leg that dies *before* reporting gets a distinct crash-note (that's +the replacement for GHA's red-job email). ## How a run works -The workflow is a thin trigger; `ci-entrypoint.sh` is the front door and does -everything below except the Actions cache restore/save. +The trigger (box: `local/run-local.sh`; cloud: the workflow) is thin; +`ci-entrypoint.sh` is the front door and does everything below except state +restore/save. -1. Restore `integration-suite-state.json` from Actions cache (version-gate + - broke/recovered diff) — *workflow*. +1. Point `CANARY_STATE` at `integration-suite-state.json` (version-gate + + broke/recovered diff) — box state dir, or Actions cache on a dispatch. 2. Build failproofai under test (`dist/index.js` + `dist/cli.mjs`) from this repo. 3. Decode the OAuth token secrets, build the sandbox image, create the per-run HOME volume, install all 12 CLIs (`install-clis.sh`), inject the credential @@ -96,8 +134,10 @@ able to overwrite the stable leg's gating record. Because this repo is public, all credentials live in a scoped **GitHub Environment** (`cli-integration`) — only this workflow's job can read them — and -the workflow triggers on `schedule`/`workflow_dispatch` **only**, so fork PRs can -never reach them. +the workflow triggers on `workflow_dispatch` **only**, so fork PRs can never +reach them. (The canary box keeps its own copy of the same variables in +`~/.config/failproofai-canary/secrets.env`, chmod 600 — updating one does not +update the other.) | Auth | CLIs | Secret(s) | |------|------|-----------| @@ -126,4 +166,5 @@ canary-policies.mjs benign-marker custom policies the probe trips run.sh orchestrator (gate → probe → report → Slack) report.js build the Slack report + diff state (broke/recovered) capture-tokens.sh (run on a logged-in machine) refresh the OAuth token secrets +local/ the daily driver: box wrapper + systemd units (see above) ``` diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index 0a62f616..12647526 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -38,6 +38,11 @@ # CANARY_VERSION_GATED "all" (default) | comma-sep list | "none" to force-probe # CANARY_CLIS space-separated CLI subset (empty = all 12) # CANARY_SKIP_BUILD set to 1 to reuse an existing dist/ (local iteration) +# CANARY_DAEMON set to 1 to probe the daemon-configured (failproofaid) +# hook path: builds the Rust daemon and routes every +# probe's hooks through it, fail-closed (see probe-cli.sh) +# CANARY_CARGO_CACHE cargo home+target cache dir for the daemon build +# (default ~/.cache/failproofai-canary/cargo) # ───────────────────────────────────────────────────────────────────────────── set -u @@ -101,6 +106,28 @@ if [ ! -s "$REPO/dist/index.js" ] || [ ! -s "$REPO/dist/cli.mjs" ]; then exit 1 fi +# ── 1b. build failproofaid under test (daemon mode only) ──────────────────── +# Built in a rust:1-bookworm container, NOT on the host: the sandbox image is +# node:22-bookworm-slim (glibc 2.36), and a binary linked against a newer host +# glibc would fail to load inside it. The repo mounts read-only — cargo writes +# only to the mounted cache (registry + target), so the checkout stays clean. +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + step "building failproofaid (daemon) under test" + if [ ! -f "$REPO/crates/failproofaid/Cargo.toml" ]; then + echo "✗ CANARY_DAEMON=1 but $REPO has no crates/failproofaid — this ref predates the daemon; unset CANARY_DAEMON or pick a ref that carries it" >&2 + exit 1 + fi + CARGO_CACHE="${CANARY_CARGO_CACHE:-$HOME/.cache/failproofai-canary/cargo}" + mkdir -p "$CARGO_CACHE/home" "$CARGO_CACHE/target" + docker run --rm -u "$(id -u):$(id -g)" \ + -e HOME=/cargo/home -e CARGO_HOME=/cargo/home -e CARGO_TARGET_DIR=/cargo/target \ + -v "$CARGO_CACHE:/cargo" -v "$REPO:/src:ro" -w /src \ + rust:1-bookworm cargo build --release --locked -p failproofaid \ + || { echo "✗ failproofaid build failed" >&2; exit 1; } + export CANARY_DAEMON_BIN="$CARGO_CACHE/target/release/failproofaid" + [ -x "$CANARY_DAEMON_BIN" ] || { echo "✗ built failproofaid missing at $CANARY_DAEMON_BIN" >&2; exit 1; } +fi + # ── 2. decode OAuth token secrets ─────────────────────────────────────────── # Each is a base64 gzip-tar rooted at $HOME. A missing secret is NOT fatal: that # CLI simply reports ERROR (can't auth) rather than taking the whole run down. @@ -176,4 +203,6 @@ CANARY_STATE="$STATE" \ CANARY_ENVFILE="$ENVFILE" \ CANARY_CHANNEL="$CHANNEL" \ CANARY_PEER_STATE="$PEER_STATE" \ +CANARY_DAEMON="${CANARY_DAEMON:-0}" \ +CANARY_DAEMON_BIN="${CANARY_DAEMON_BIN:-}" \ bash "$HERE/run.sh" ${CANARY_CLIS:-} diff --git a/integration-suite/local/failproofai-canary.service b/integration-suite/local/failproofai-canary.service new file mode 100644 index 00000000..364d1a0a --- /dev/null +++ b/integration-suite/local/failproofai-canary.service @@ -0,0 +1,14 @@ +[Unit] +Description=failproofai CLI integration suite (canary) — one full run, both legs +# No network-online dependency: user-manager network targets are unreliable. +# run-local.sh's own `git fetch` fails loudly (and Slack-notes) if offline. + +[Service] +Type=oneshot +# The INSTALLED copy (install.sh) — never a path inside the runner clone, +# which run-local.sh hard-resets on every run. +ExecStart=%h/.config/failproofai-canary/bin/run-local.sh +# Two legs at up to 90 min each (run-local.sh's per-leg timeout) plus CLI +# installs and the daemon build; anything past this is a wedged run. +TimeoutStartSec=4h +Nice=10 diff --git a/integration-suite/local/failproofai-canary.timer b/integration-suite/local/failproofai-canary.timer new file mode 100644 index 00000000..1b6309de --- /dev/null +++ b/integration-suite/local/failproofai-canary.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Daily failproofai integration suite (canary) + +[Timer] +# Same slot the retired GHA cron used (06:17 UTC) — off-peak for the gateway. +OnCalendar=*-*-* 06:17:00 UTC +# A box that was off (or asleep) at 06:17 runs the canary on the next boot +# instead of silently skipping the day. +Persistent=true +RandomizedDelaySec=5m + +[Install] +WantedBy=timers.target diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh new file mode 100755 index 00000000..31f28c19 --- /dev/null +++ b/integration-suite/local/install.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# One-time (and safely re-runnable) setup for the canary BOX. +# +# Copies run-local.sh OUTSIDE the runner clone (the clone is hard-reset on +# every run, so nothing that survives a run may live inside it), installs the +# systemd user units, prepares the state dirs, and writes a secrets template. +# Re-running refreshes the installed copies but NEVER touches an existing +# secrets.env. +# ───────────────────────────────────────────────────────────────────────────── +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +CONF_DIR="${CANARY_CONF_DIR:-$HOME/.config/failproofai-canary}" +STATE_DIR="${CANARY_STATE_DIR:-$HOME/.local/state/failproofai-canary}" +UNIT_DIR="$HOME/.config/systemd/user" +ME="${USER:-$(id -un)}" + +echo "── checking box requirements ──" +missing=0 +for bin in docker git bun node curl flock; do + command -v "$bin" >/dev/null 2>&1 || { echo " ✗ $bin not found on PATH"; missing=1; } +done +if command -v docker >/dev/null 2>&1 && ! docker info >/dev/null 2>&1; then + echo " ✗ docker daemon not reachable as $ME (docker group membership?)"; missing=1 +fi +[ "$missing" = 0 ] && echo " ✓ docker git bun node curl flock all present" + +echo "── installing ──" +mkdir -p "$CONF_DIR/bin" "$STATE_DIR/logs" "$UNIT_DIR" +install -m 755 "$HERE/run-local.sh" "$CONF_DIR/bin/run-local.sh" +install -m 644 "$HERE/failproofai-canary.service" "$HERE/failproofai-canary.timer" "$UNIT_DIR/" +echo " ✓ $CONF_DIR/bin/run-local.sh" +echo " ✓ $UNIT_DIR/failproofai-canary.{service,timer}" +systemctl --user daemon-reload 2>/dev/null \ + || echo " ⚠ systemctl --user unavailable in this shell — run 'systemctl --user daemon-reload' from a login session" + +if [ ! -f "$CONF_DIR/secrets.env" ]; then + ( + umask 177 + cat > "$CONF_DIR/secrets.env" <<'EOF' +# failproofai canary — box configuration. chmod 600; sourced by run-local.sh. +# Same variables the GHA `cli-integration` Environment supplied — see +# integration-suite/ci-entrypoint.sh's header for what each one does. + +# ── gateway + PAT credentials ──────────────────────────────────────────────── +CANARY_LLM_API_KEY= +#CANARY_LLM_BASE_URL=https://models.aikin.club +#CANARY_LLM_MODEL=deepseek-v4-pro +#CANARY_CLAUDE_MODEL=claude-haiku-4-5 +#CANARY_PI_MODEL=claude-haiku-4-5 +#CANARY_CODEX_MODEL=gpt-5.1-codex-mini +COPILOT_GITHUB_TOKEN= + +# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── +# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh +# and paste the output here; the box itself never needs vendor logins. An empty +# value just makes that CLI report ERROR (can't auth), not a failed run. +CURSOR_TOKEN_TGZ_B64= +DEVIN_TOKEN_TGZ_B64= +ANTIGRAVITY_TOKEN_TGZ_B64= + +# ── reporting ──────────────────────────────────────────────────────────────── +CANARY_SLACK_WEBHOOK= + +# ── what to test ───────────────────────────────────────────────────────────── +# REQUIRED. Deliberately explicit (no baked-in default): flip to origin/main +# once the failproofaid branch (#632) merges. +CANARY_REF=origin/failproofaid +# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays +# in-process. Flip these to move the daemon dimension between legs. +#CANARY_DAEMON_STABLE=1 +#CANARY_DAEMON_BETA=0 +#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git +#CANARY_CLONE=$HOME/canary/failproofai +EOF + ) + echo " ✓ wrote template $CONF_DIR/secrets.env (fill it in)" +else + echo " ✓ kept existing $CONF_DIR/secrets.env" +fi + +cat <&2; exit 1; } +perms="$(stat -c %a "$SECRETS" 2>/dev/null || stat -f %Lp "$SECRETS" 2>/dev/null)" +[ "$perms" = 600 ] || { echo "✗ $SECRETS must be chmod 600 (is $perms) — it holds credentials" >&2; exit 1; } +set -a; . "$SECRETS"; set +a + +# Required, no default ON PURPOSE: a baked-in default ref would silently keep +# probing a stale branch after the daemon branch merges to main. Every box +# states what it tests. +: "${CANARY_REF:?CANARY_REF unset — set it in $SECRETS (origin/failproofaid until #632 merges, then origin/main)}" +CLONE="${CANARY_CLONE:-$HOME/canary/failproofai}" +GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" +LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout + +# One run at a time — the local stand-in for GHA's `concurrency` group. A +# still-running yesterday (hung vendor CLI) must not race today's volume. +exec 9>"$STATE_DIR/.lock" +flock -n 9 || { echo "another canary run holds $STATE_DIR/.lock — exiting" >&2; exit 0; } + +TS="$(date -u +%Y%m%dT%H%M%SZ)" + +slack_note() { # $1 = text; best-effort, never fails the run + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +# ── checkout the ref under test ────────────────────────────────────────────── +if [ ! -d "$CLONE/.git" ]; then + git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } +fi +git -C "$CLONE" fetch --prune origin \ + || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } +{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ + || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } +FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +echo "── canary run $TS: $CANARY_REF @ $FP_SHA ──" + +# ── legs (the same two the GHA matrix ran; sequential on one Docker host) ──── +run_leg() { # $1 = channel + local channel="$1" leg_log="$STATE_DIR/logs/leg-$1-$TS.log" rc daemon state peer + if [ "$channel" = stable ]; then + # Stable probes the daemon-configured (failproofaid) path — the way-forward + # configuration users get from `failproofai config`. Beta stays in-process: + # it answers "is the vendor about to break us", which is independent of our + # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. + daemon="${CANARY_DAEMON_STABLE:-1}" + state="$STATE_DIR/integration-suite-state.json" + peer="" + else + daemon="${CANARY_DAEMON_BETA:-0}" + state="$STATE_DIR/integration-suite-state-$channel.json" + peer="$STATE_DIR/integration-suite-state.json" + fi + echo "── leg: $channel (daemon=$daemon) ──" + GITHUB_WORKSPACE="$CLONE" \ + CANARY_CHANNEL="$channel" \ + CANARY_STATE="$state" \ + CANARY_PEER_STATE="$peer" \ + CANARY_FP_SHA="$FP_SHA" \ + CANARY_DAEMON="$daemon" \ + CANARY_ENVFILE="$STATE_DIR/tmp/canary-$channel.env" \ + CANARY_TOKENS_DIR="$STATE_DIR/tmp/tokens-$channel" \ + timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" + rc=${PIPESTATUS[0]} + # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack + # already carries the story); non-zero WITHOUT one means the harness died + # before reporting, which on GHA surfaced as a red-job email and here would + # otherwise be silence. + if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then + slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — $CANARY_REF @ $FP_SHA +\`\`\` +$(tail -12 "$leg_log") +\`\`\`" + fi + return "$rc" +} + +rc_stable=0; rc_beta=0 +run_leg stable || rc_stable=$? +run_leg beta || rc_beta=$? + +find "$STATE_DIR/logs" -name '*.log' -mtime +14 -delete 2>/dev/null || true + +echo "── done: stable rc=$rc_stable, beta rc=$rc_beta ──" +[ "$rc_stable" -eq 0 ] && [ "$rc_beta" -eq 0 ] diff --git a/integration-suite/probe-cli.sh b/integration-suite/probe-cli.sh index b517601b..a60f326c 100644 --- a/integration-suite/probe-cli.sh +++ b/integration-suite/probe-cli.sh @@ -63,6 +63,76 @@ printf '#!/bin/sh\nexec bun /repo/bin/failproofai.mjs "$@"\n' > "$HOME/bin/failp chmod +x "$HOME/bin/failproofai" export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" +# ── Daemon mode (CANARY_DAEMON=1) ──────────────────────────────────────────── +# Probes the configuration users get after `failproofai config`: hooks route +# CLI → failproofaid (Rust supervisor) → warm bun worker over Unix sockets, +# fail-CLOSED when the daemon is unreachable. The binary is cross-compiled on +# the host by ci-entrypoint.sh (rust:1-bookworm, so its glibc matches this +# sandbox) and bind-mounted at /opt/failproofaid/failproofaid by run.sh. +# +# The daemon is started PER PROBE, not once per CLI. The worker inherits the +# DAEMON's environment — the wire protocol carries only {hookEvent, cli, +# stdin, cwd}, never the hook process's env — so FAILPROOFAI_HOOK_LOG_FILE +# only reaches the oracle if the daemon itself is (re)started pointing at that +# probe's log dir. Sharing one log dir across both probes instead would let +# probe A's incidental denies (an agent exploring with reads trips +# block-read-outside-cwd) satisfy probe B's grep — a false PASS. +# +# A DEAD daemon cannot false-PASS: the client's fail-closed deny is shaped by +# a synthetic `failproofai/daemon-unreachable` policy (bin/failproofai.mjs), +# which denied()/read_denied() below can never match — those probes go +# INCONCLUSIVE and re-probe until the daemon path recovers. +DAEMON_PID="" +daemon_stop() { + [ -n "$DAEMON_PID" ] || return 0 + kill "$DAEMON_PID" 2>/dev/null + wait "$DAEMON_PID" 2>/dev/null + DAEMON_PID="" +} +daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) + [ "${CANARY_DAEMON:-0}" = 1 ] || return 0 + daemon_stop + rm -f "$FAILPROOFAI_DAEMON_SOCKET" + # Env is the worker's too (worker.rs spawns `sh -c "$FAILPROOFAI_WORKER_CMD"` + # inheriting it): the writable FP_DIST for the custom-policy loader's shim, + # and this probe's oracle dir. The worker entry only sets DIST when unset. + FAILPROOFAI_HOOK_LOG_FILE="$1" \ + FAILPROOFAI_WORKER_CMD="bun /repo/bin/failproofai-worker.mjs" \ + /opt/failproofaid/failproofaid >> "$BASE/daemon.log" 2>&1 & + DAEMON_PID=$! + for _ in $(seq 1 100); do # ≤10s; readiness = the socket ACCEPTS, not exists + if node -e 'const s=require("net").createConnection(process.argv[1]);s.on("connect",()=>process.exit(0));s.on("error",()=>process.exit(1));' \ + "$FAILPROOFAI_DAEMON_SOCKET" 2>/dev/null; then return 0; fi + kill -0 "$DAEMON_PID" 2>/dev/null || break + sleep 0.1 + done + echo "✗ failproofaid did not come up — daemon.log tail:" >&2 + tail -5 "$BASE/daemon.log" >&2 + exit 1 +} +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + [ -x /opt/failproofaid/failproofaid ] \ + || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + # Socket under /tmp: container-local, so a stale socket file in the PERSISTENT + # volume can never shadow a live daemon across daily runs. The override + # relocates the whole run dir — lock and worker.sock land beside it — and the + # dir is NOT pre-created here: failproofaid creates it 0700 itself and refuses + # one it didn't create with other perms (paths.rs ensure_run_dir). + export FAILPROOFAI_DAEMON_SOCKET="/tmp/fpai-canary/failproofaid.sock" + # The fail-closed marker, written via the REAL code path (fp-config's + # updateConfig) rather than shell-appending TOML — the volume's config.toml + # survives with its other tables intact, and a duplicate [daemon] table + # (invalid TOML) would silently read back as NOT configured. + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ + || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } + trap daemon_stop EXIT +else + # The HOME volume persists across runs: a marker left behind by a daemon-mode + # run would make this in-process run fail closed on every hook event with no + # daemon anywhere. Clear it unconditionally. + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true +fi + BASE="$HOME/probe-$CLI" # DEFINITE probes: BENIGN actions (echo/touch a token, read a plain file) the # model never refuses → a tool call is guaranteed, so no INCONCLUSIVE from @@ -206,6 +276,7 @@ ATTEMPTS=3 # retry up to N times to absorb LLM nondeterminism (flaky tool-call # despite no deny); ERROR=vendor quota/auth failure; INCONCLUSIVE=no tool call. LOGA="$BASE/log-bash" BASH_PROMPT='Using your shell/bash tool, run exactly this command: touch CANARY_PROBE_ran' +daemon_cycle "$LOGA" # daemon mode only: worker must inherit THIS probe's oracle dir OUTA="" for _ in $(seq 1 $ATTEMPTS); do export FAILPROOFAI_HOOK_LOG_FILE="$LOGA"; rm -f "$BASE/CANARY_PROBE_ran" @@ -226,6 +297,7 @@ else VA=INCONCLUSIVE; fi # ── Probe B: benign file read (canary-read) ────────────────────────────────── LOGB="$BASE/log-read" READ_PROMPT='Read the file named CANARY_MARKER.txt in the current directory and print its exact contents.' +daemon_cycle "$LOGB" # fresh daemon env — probe A's log dir must not receive B's denies OUTB="" for _ in $(seq 1 $ATTEMPTS); do export FAILPROOFAI_HOOK_LOG_FILE="$LOGB" diff --git a/integration-suite/run.sh b/integration-suite/run.sh index fab41fcb..b4df7c5d 100644 --- a/integration-suite/run.sh +++ b/integration-suite/run.sh @@ -61,8 +61,20 @@ FP_SHA="${CANARY_FP_SHA:-$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null || # Installed versions from the install step, keyed by cli. VERSIONS_JSON="$(docker run --rm -v "$VOL:/home/canary" "$IMAGE" cat /home/canary/canary-tier0.json 2>/dev/null || echo '[]')" +# Daemon mode: bind the host-built failproofaid binary into the probe container +# and tell probe-cli.sh to route hooks through it (see its CANARY_DAEMON block). +# A file→file bind mount, read-only — executing from an ro mount is fine. +DAEMON_FLAGS=() +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" + [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } + # docker reads a relative -v source as a NAMED VOLUME — absolutize first. + DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" + DAEMON_FLAGS=(-e CANARY_DAEMON=1 -v "$DBIN:/opt/failproofaid/failproofaid:ro") +fi + run_probe() { - docker run --rm --env-file "$ENVFILE" \ + docker run --rm --env-file "$ENVFILE" "${DAEMON_FLAGS[@]}" \ -v "$REPO:/repo:ro" -v "$SANDBOX:/opt/canary:ro" -v "$VOL:/home/canary" \ "$IMAGE" bash /opt/canary/probe-cli.sh "$1" 2>&1 } From 2eb2d644193aaff0c7f0714b633db2a3f4b079a8 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 11:47:09 +0530 Subject: [PATCH 2/7] =?UTF-8?q?canary:=20one=20container,=20one=20cron=20l?= =?UTF-8?q?ine=20=E2=80=94=20repackage=20the=20box=20runner=20for=20zero-t?= =?UTF-8?q?ouch=20hosts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The box story shrinks to Docker + one cron line + one env file: the systemd units, install.sh and host-toolchain requirements are gone. A self-contained runner image (local/Dockerfile.runner — node+bun+git+docker CLIENT) drives the HOST's Docker through the mounted socket, so the sandbox image, the per-channel volumes and every probe container are exactly the ones CI runs, as siblings. Two decisions carry the design: - Path parity. The one work dir is mounted at an IDENTICAL path inside and out (-v "$HOME/fp-canary:$HOME/fp-canary") because paths under it serve both as in-container file paths and as sibling-container -v sources, which the host daemon resolves against the host filesystem. The entrypoint auto-detects the parity mount from its own container's mount table and names the exact flag to add when it is missing. runner-daily.sh pins the daemon build's cargo cache under the work dir — the only harness default rooted outside it ($HOME), where the rust sibling's mount would silently create an empty root-owned host dir and cache nothing. - A thin baked entrypoint, everything else from the checkout. The image carries only runner-entrypoint.sh (preflight, work-dir detection, host-side flock so overlapping cron fires share one lock across containers, clone/ fetch/checkout of $CANARY_REF, Slack crash-note for the checkout phase); it then execs integration-suite/local/runner-daily.sh FROM THE CHECKOUT. Harness changes reach the box through git — nobody rebuilds the boss's image for a leg tweak. runner-daily.sh keeps the leg contract from the systemd iteration verbatim: stable leg daemon-configured (CANARY_DAEMON=1) then beta in-process, per-leg 90-min timeout, crash-guard keyed on the absence of run.sh's own posted-to-Slack line, 14-day log prune. secrets.env.example documents every variable the GHA Environment supplied, in docker --env-file's literal KEY=value format. Tripwires updated in local-runner.test.ts: the image must never bake the daily driver, the crash-guard grep must match run.sh's actual wording, the example must offer every secret-fed env var the workflow maps and must contain no shell expansion on value lines. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- .../integration-suite/local-runner.test.ts | 132 ++++++++++++------ integration-suite/README.md | 60 +++++--- integration-suite/local/Dockerfile.runner | 41 ++++++ .../local/failproofai-canary.service | 14 -- .../local/failproofai-canary.timer | 13 -- integration-suite/local/install.sh | 91 ------------ integration-suite/local/run-local.sh | 116 --------------- integration-suite/local/runner-daily.sh | 88 ++++++++++++ integration-suite/local/runner-entrypoint.sh | 67 +++++++++ integration-suite/local/secrets.env.example | 43 ++++++ 11 files changed, 372 insertions(+), 295 deletions(-) create mode 100644 integration-suite/local/Dockerfile.runner delete mode 100644 integration-suite/local/failproofai-canary.service delete mode 100644 integration-suite/local/failproofai-canary.timer delete mode 100755 integration-suite/local/install.sh delete mode 100755 integration-suite/local/run-local.sh create mode 100755 integration-suite/local/runner-daily.sh create mode 100755 integration-suite/local/runner-entrypoint.sh create mode 100644 integration-suite/local/secrets.env.example diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c2b0cd8..6d061689 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -213,7 +213,7 @@ never "blocked". ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box, and make it probe the daemon path. `integration-suite/local/` ships the box side — a wrapper that checks out `CANARY_REF`, runs both legs through the unchanged `ci-entrypoint.sh`, and Slack-notes any leg that dies *before* reporting (the replacement for GHA's red-job email) — plus an installer and a systemd user timer holding the same 06:17 UTC slot; the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index b882b7e0..9688a7b7 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -2,25 +2,30 @@ * Tripwires for the LOCAL canary runner (integration-suite/local/) and the * daemon-mode (CANARY_DAEMON) probe path. * - * Daily integration-suite runs moved off GH Actions onto a local box - * (2026-08-07, for runner-minute cost); the stable leg there probes the - * daemon-configured (failproofaid) hook path — the way-forward configuration. - * Everything below is shell scripts and systemd units with no importable - * surface, so the tests parse the real files — same approach as + * Daily integration-suite runs moved off GH Actions (2026-08-07, for + * runner-minute cost) onto a box whose entire contract is: Docker + one cron + * line + one env file. A self-contained runner image drives the HOST's Docker + * through the mounted socket; its baked entrypoint checks out CANARY_REF and + * hands off to runner-daily.sh FROM THE CHECKOUT, so harness changes reach the + * box through git with no image rebuild. The stable leg probes the + * daemon-configured (failproofaid) path — the way-forward configuration. + * + * Everything below is shell scripts, a Dockerfile and an env template with no + * importable surface, so the tests parse the real files — same approach as * channel-refs.test.ts, and for the same reason: the alternative is a second * copy of each contract to drift against. */ -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import path from "node:path"; import { describe, expect, it } from "vitest"; const ROOT = path.join(__dirname, "../.."); const SUITE = path.join(ROOT, "integration-suite"); const LOCAL = path.join(SUITE, "local"); -const runLocal = readFileSync(path.join(LOCAL, "run-local.sh"), "utf8"); -const installSh = readFileSync(path.join(LOCAL, "install.sh"), "utf8"); -const service = readFileSync(path.join(LOCAL, "failproofai-canary.service"), "utf8"); -const timer = readFileSync(path.join(LOCAL, "failproofai-canary.timer"), "utf8"); +const dockerfile = readFileSync(path.join(LOCAL, "Dockerfile.runner"), "utf8"); +const entrypointSh = readFileSync(path.join(LOCAL, "runner-entrypoint.sh"), "utf8"); +const dailySh = readFileSync(path.join(LOCAL, "runner-daily.sh"), "utf8"); +const secretsExample = readFileSync(path.join(LOCAL, "secrets.env.example"), "utf8"); const workflow = readFileSync( path.join(ROOT, ".github/workflows/integration-suite.yml"), "utf8", @@ -43,55 +48,100 @@ describe("GHA workflow is dispatch-only", () => { }); }); -describe("local runner wiring", () => { - it("service ExecStart points at the exact path install.sh installs to", () => { - // run-local.sh hard-resets the runner clone, so the unit must exec the - // INSTALLED copy — a unit pointing into the clone would run whatever the - // checked-out ref happens to carry, mid-reset. - const m = /^ExecStart=%h\/(\S+)$/m.exec(service); - expect(m).not.toBeNull(); - expect(m![1]).toBe(".config/failproofai-canary/bin/run-local.sh"); - expect(installSh).toContain("CANARY_CONF_DIR:-$HOME/.config/failproofai-canary"); - expect(installSh).toMatch( - /install -m 755 "\$HERE\/run-local\.sh" "\$CONF_DIR\/bin\/run-local\.sh"/, - ); +describe("runner image (the boss's one container)", () => { + it("bakes the thin entrypoint and nothing else of the harness", () => { + // The image must stay rebuild-free across harness changes: it may carry + // runner-entrypoint.sh (thin, stable) but must NOT bake runner-daily.sh + // or any other harness file — those are executed from the checkout. + expect(dockerfile).toMatch(/^COPY runner-entrypoint\.sh /m); + expect(dockerfile).toMatch(/^ENTRYPOINT \["\/usr\/local\/bin\/runner-entrypoint\.sh"\]$/m); + // (comments may mention the daily driver; COPY lines must not) + expect(dockerfile).not.toMatch(/^COPY .*runner-daily/m); }); - it("install.sh installs both systemd units", () => { - expect(installSh).toContain("failproofai-canary.service"); - expect(installSh).toContain("failproofai-canary.timer"); + it("ships the docker CLIENT for the mounted host socket", () => { + expect(dockerfile).toMatch(/download\.docker\.com\/linux\/static/); }); - it("timer keeps the retired GHA cron slot and catches up after downtime", () => { - expect(timer).toMatch(/OnCalendar=.*06:17.*UTC/); - expect(timer).toMatch(/^Persistent=true$/m); + it("entrypoint refuses to run without the socket and without CANARY_REF", () => { + // A baked-in default ref would silently keep probing a stale branch after + // the daemon branch merges to main — the env file states what it tests. + expect(entrypointSh).toContain("/var/run/docker.sock"); + expect(entrypointSh).toMatch(/\$\{CANARY_REF:\?/); }); - it("run-local.sh drives the same front door CI does", () => { - expect(runLocal).toContain("integration-suite/ci-entrypoint.sh"); + it("entrypoint serializes runs and hands off to the in-repo daily driver", () => { + // The lock file lives on the host work dir so overlapping cron fires + // share one lock across separate containers. + expect(entrypointSh).toMatch(/flock -n/); + expect(entrypointSh).toMatch(/exec bash "\$CLONE\/integration-suite\/local\/runner-daily\.sh"/); + expect(existsSync(path.join(LOCAL, "runner-daily.sh"))).toBe(true); }); - it("refuses to run without an explicit CANARY_REF", () => { - // A baked-in default ref would silently keep probing a stale branch after - // the daemon branch merges to main — every box states what it tests. - expect(runLocal).toMatch(/\$\{CANARY_REF:\?/); + it("entrypoint explains the identical-path work-dir mount when it is missing", () => { + // Path parity is the load-bearing trick of the whole design: paths under + // the work dir serve as sibling-container -v sources, resolved by the + // HOST daemon. The failure message must teach the fix. + expect(entrypointSh).toMatch(/-v \\"\\\$HOME\/fp-canary:\\\$HOME\/fp-canary\\"/); + }); +}); + +describe("daily driver (in-repo, evolves with the harness)", () => { + it("drives the same front door CI does, one leg per channel", () => { + expect(dailySh).toContain("integration-suite/ci-entrypoint.sh"); + expect(dailySh).toMatch(/\$\{CANARY_LEGS:-stable beta\}/); }); it("stable leg defaults to the daemon path, beta to in-process", () => { - expect(runLocal).toContain("${CANARY_DAEMON_STABLE:-1}"); - expect(runLocal).toContain("${CANARY_DAEMON_BETA:-0}"); + expect(dailySh).toContain("${CANARY_DAEMON_STABLE:-1}"); + expect(dailySh).toContain("${CANARY_DAEMON_BETA:-0}"); + }); + + it("pins the cargo cache under the work dir (path parity for the sibling build)", () => { + // ci-entrypoint's default cargo cache is under $HOME — inside the runner + // container that path does not exist on the host, so the rust sibling + // container's -v mount would silently create a root-owned host dir and + // cache nothing. The only harness default rooted outside $WORK. + expect(dailySh).toMatch(/CANARY_CARGO_CACHE="\$\{CANARY_CARGO_CACHE:-\$WORK\/cargo\}"/); }); - it("secrets template offers every secret-fed env var the workflow maps", () => { - // The box's secrets.env and the GHA Environment must stay interchangeable. - // A secret added to the workflow but not the template means the box runs + it("crash-guard greps the exact success line run.sh prints", () => { + // "leg died WITHOUT reporting" is detected by the absence of run.sh's own + // posted-to-Slack line — if that wording changes in run.sh, the crash + // guard goes blind and every FAIL verdict would double-post a crash note. + const m = /grep -q "([^"]+)" "\$leg_log"/.exec(dailySh); + expect(m).not.toBeNull(); + expect(runSh).toContain(m![1]); + }); +}); + +describe("secrets.env.example (the one file the boss edits)", () => { + it("offers every secret-fed env var the workflow maps", () => { + // The box's env file and the GHA Environment must stay interchangeable. + // A secret added to the workflow but not the example means the box runs // without it and that CLI quietly reports ERROR forever. const envNames = [...workflow.matchAll(/^\s+([A-Z0-9_]+):\s+\$\{\{\s*secrets\./gm)].map( (m) => m[1], ); expect(envNames.length).toBeGreaterThanOrEqual(10); for (const name of envNames) { - expect(installSh, `secrets.env template is missing ${name}`).toContain(name); + expect(secretsExample, `secrets.env.example is missing ${name}`).toContain(name); + } + }); + + it("states CANARY_REF uncommented (the runner refuses to start without it)", () => { + expect(secretsExample).toMatch(/^CANARY_REF=\S+$/m); + }); + + it("is valid docker --env-file material: no shell expansion on value lines", () => { + // docker --env-file is literal KEY=value — a $HOME in a value would reach + // the container as the four characters "$HOM"+"E". Comments may mention + // $HOME freely; value lines must not. + const valueLines = secretsExample + .split("\n") + .filter((l) => l.trim() && !l.trim().startsWith("#")); + for (const line of valueLines) { + expect(line, `value line must not rely on shell expansion: ${line}`).not.toContain("$"); } }); }); @@ -138,7 +188,7 @@ describe("daemon-mode probe path", () => { // rename on either side trips this test. const idMatch = /registerPolicy\(\s*"(failproofai\/[a-z-]+)",\s*"Fail-closed/.exec(handlerTs); expect(idMatch).not.toBeNull(); - // handler.ts:463 — `result=${decision} policy=${policyName} duration=…` + // handler.ts — `result=${decision} policy=${policyName} duration=…` const failClosedLine = `result=deny policy=${idMatch![1]} duration=3ms`; const deniedPat = /denied\(\) \{ grep -qE "([^"]+)"/.exec(probeSh); diff --git a/integration-suite/README.md b/integration-suite/README.md index f549a22b..d6101877 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -32,28 +32,50 @@ vitest suites. So it's a scheduled run, not a PR gate. Daily runs live on a **local canary box**, not GH Actions — runner minutes were the entire cost of the old daily cron; the LLM spend is identical either way. -`local/` holds everything box-side: +The box needs exactly **Docker + one cron line + one env file**; there is no +host toolchain, no installed scripts, no systemd. Everything else happens +inside a self-contained runner image that drives the host's Docker through the +mounted socket (sibling containers — the sandbox image, volumes and probe +containers are the exact ones CI runs). -``` -local/run-local.sh the cron replacement: checkout CANARY_REF → - stable leg (daemon) → beta leg → crash-guard -local/install.sh box setup: installed copy + units + secrets template -local/failproofai-canary.service systemd user unit (oneshot, 4h ceiling) -local/failproofai-canary.timer daily 06:17 UTC, Persistent=true +Box setup, in full: + +```bash +# 1. one-time: build the runner image (from a clone, or straight from GitHub) +docker build -t failproofai-canary-runner \ + -f integration-suite/local/Dockerfile.runner integration-suite/local/ + +# 2. one-time: work dir + secrets +mkdir -p ~/fp-canary +cp integration-suite/local/secrets.env.example ~/fp-canary/secrets.env +chmod 600 ~/fp-canary/secrets.env # then fill it in + +# 3. cron (pick any quiet hour; overlapping fires share a lock and no-op) +17 6 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$HOME/fp-canary:$HOME/fp-canary" --env-file "$HOME/fp-canary/secrets.env" failproofai-canary-runner >/dev/null 2>&1 ``` -Box setup: clone the repo anywhere once, `bash integration-suite/local/install.sh`, -fill `~/.config/failproofai-canary/secrets.env` (same variables the GHA -Environment supplied; token tarballs still come from `capture-tokens.sh` on a -logged-in machine), `loginctl enable-linger`, enable the timer. The wrapper runs -from an **installed copy** because it hard-resets the runner clone every run — -nothing that must survive a run may live inside the clone. - -State (`integration-suite-state[-beta].json`) sits in -`~/.local/state/failproofai-canary/` instead of the Actions cache; the -version-gate logic is unchanged. Verdict reports POST to Slack exactly as -before; a leg that dies *before* reporting gets a distinct crash-note (that's -the replacement for GHA's red-job email). +The work dir is mounted at an **identical path** inside and out — that is +load-bearing, not style: paths under it are used both for in-container file +ops and as sibling-container `-v` sources, which the host daemon resolves +against the host filesystem. The entrypoint auto-detects it (and says exactly +what to mount if it can't). + +At each run the image's baked entrypoint (`runner-entrypoint.sh` — thin on +purpose) locks, clones/fetches `CANARY_REF` into `~/fp-canary/clone`, and +hands off to `runner-daily.sh` **from that checkout** — so harness changes +reach the box through git, and the image only needs a rebuild when the +entrypoint itself changes. The daily driver runs the stable leg +(daemon-configured) then the beta leg (in-process), exactly like the old GHA +matrix. + +Everything lands under the work dir: version-gate state in `state/` (instead +of the Actions cache — the gate logic is unchanged), run + per-leg logs in +`logs/` (pruned after 14 days), the clone, and the daemon build's cargo cache. +Verdict reports POST to Slack exactly as before; a leg that dies *before* +reporting gets a distinct crash-note with the log tail (that's the replacement +for GHA's red-job email — cron's own output can go to `/dev/null`). Token +tarballs still come from `capture-tokens.sh` on a logged-in machine; the first +run probes all 12 CLIs (~1h, empty gate) and steady-state runs are short. ## How a run works diff --git a/integration-suite/local/Dockerfile.runner b/integration-suite/local/Dockerfile.runner new file mode 100644 index 00000000..7dd8e366 --- /dev/null +++ b/integration-suite/local/Dockerfile.runner @@ -0,0 +1,41 @@ +# failproofai canary — the self-contained daily RUNNER image. +# +# The whole box story is: build this once, add one cron line, done. +# +# docker build -t failproofai-canary-runner -f Dockerfile.runner . +# 17 6 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ +# -v "$HOME/fp-canary:$HOME/fp-canary" --env-file "$HOME/fp-canary/secrets.env" \ +# failproofai-canary-runner >/dev/null 2>&1 +# +# At each run the baked entrypoint clones/fetches the repo at $CANARY_REF into +# the work dir and hands off to integration-suite/local/runner-daily.sh FROM +# THAT CHECKOUT — so harness changes reach the box through git, and this image +# only needs a rebuild when the entrypoint or this file change. +# +# It drives the HOST's Docker through the mounted socket (sibling containers, +# not docker-in-docker): the sandbox image, per-channel HOME volumes and probe +# containers are the exact ones CI runs. That is why the work dir must be +# mounted at an IDENTICAL path inside and out — paths under it are used both +# for in-container file ops and as `-v` sources that the HOST daemon resolves. + +FROM node:22-bookworm-slim + +# bun from the official image — same trick as the sandbox Dockerfile. +COPY --from=oven/bun:latest /usr/local/bin/bun /usr/local/bin/bun + +RUN apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends git ca-certificates curl \ + && rm -rf /var/lib/apt/lists/* + +# docker CLIENT only — the daemon is the host's, reached through the socket. +# Static binary, arch-mapped (x86_64 / aarch64 — uname -m matches the URL). +ARG DOCKER_VERSION=27.5.1 +RUN arch="$(uname -m)" \ + && curl -fsSL "https://download.docker.com/linux/static/stable/${arch}/docker-${DOCKER_VERSION}.tgz" \ + | tar -xz --strip-components=1 -C /usr/local/bin docker/docker + +COPY runner-entrypoint.sh /usr/local/bin/runner-entrypoint.sh +RUN chmod +x /usr/local/bin/runner-entrypoint.sh \ + && bun --version && node --version && git --version && docker --version + +ENTRYPOINT ["/usr/local/bin/runner-entrypoint.sh"] diff --git a/integration-suite/local/failproofai-canary.service b/integration-suite/local/failproofai-canary.service deleted file mode 100644 index 364d1a0a..00000000 --- a/integration-suite/local/failproofai-canary.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=failproofai CLI integration suite (canary) — one full run, both legs -# No network-online dependency: user-manager network targets are unreliable. -# run-local.sh's own `git fetch` fails loudly (and Slack-notes) if offline. - -[Service] -Type=oneshot -# The INSTALLED copy (install.sh) — never a path inside the runner clone, -# which run-local.sh hard-resets on every run. -ExecStart=%h/.config/failproofai-canary/bin/run-local.sh -# Two legs at up to 90 min each (run-local.sh's per-leg timeout) plus CLI -# installs and the daemon build; anything past this is a wedged run. -TimeoutStartSec=4h -Nice=10 diff --git a/integration-suite/local/failproofai-canary.timer b/integration-suite/local/failproofai-canary.timer deleted file mode 100644 index 1b6309de..00000000 --- a/integration-suite/local/failproofai-canary.timer +++ /dev/null @@ -1,13 +0,0 @@ -[Unit] -Description=Daily failproofai integration suite (canary) - -[Timer] -# Same slot the retired GHA cron used (06:17 UTC) — off-peak for the gateway. -OnCalendar=*-*-* 06:17:00 UTC -# A box that was off (or asleep) at 06:17 runs the canary on the next boot -# instead of silently skipping the day. -Persistent=true -RandomizedDelaySec=5m - -[Install] -WantedBy=timers.target diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh deleted file mode 100755 index 31f28c19..00000000 --- a/integration-suite/local/install.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env bash -# ───────────────────────────────────────────────────────────────────────────── -# One-time (and safely re-runnable) setup for the canary BOX. -# -# Copies run-local.sh OUTSIDE the runner clone (the clone is hard-reset on -# every run, so nothing that survives a run may live inside it), installs the -# systemd user units, prepares the state dirs, and writes a secrets template. -# Re-running refreshes the installed copies but NEVER touches an existing -# secrets.env. -# ───────────────────────────────────────────────────────────────────────────── -set -u -HERE="$(cd "$(dirname "$0")" && pwd)" -CONF_DIR="${CANARY_CONF_DIR:-$HOME/.config/failproofai-canary}" -STATE_DIR="${CANARY_STATE_DIR:-$HOME/.local/state/failproofai-canary}" -UNIT_DIR="$HOME/.config/systemd/user" -ME="${USER:-$(id -un)}" - -echo "── checking box requirements ──" -missing=0 -for bin in docker git bun node curl flock; do - command -v "$bin" >/dev/null 2>&1 || { echo " ✗ $bin not found on PATH"; missing=1; } -done -if command -v docker >/dev/null 2>&1 && ! docker info >/dev/null 2>&1; then - echo " ✗ docker daemon not reachable as $ME (docker group membership?)"; missing=1 -fi -[ "$missing" = 0 ] && echo " ✓ docker git bun node curl flock all present" - -echo "── installing ──" -mkdir -p "$CONF_DIR/bin" "$STATE_DIR/logs" "$UNIT_DIR" -install -m 755 "$HERE/run-local.sh" "$CONF_DIR/bin/run-local.sh" -install -m 644 "$HERE/failproofai-canary.service" "$HERE/failproofai-canary.timer" "$UNIT_DIR/" -echo " ✓ $CONF_DIR/bin/run-local.sh" -echo " ✓ $UNIT_DIR/failproofai-canary.{service,timer}" -systemctl --user daemon-reload 2>/dev/null \ - || echo " ⚠ systemctl --user unavailable in this shell — run 'systemctl --user daemon-reload' from a login session" - -if [ ! -f "$CONF_DIR/secrets.env" ]; then - ( - umask 177 - cat > "$CONF_DIR/secrets.env" <<'EOF' -# failproofai canary — box configuration. chmod 600; sourced by run-local.sh. -# Same variables the GHA `cli-integration` Environment supplied — see -# integration-suite/ci-entrypoint.sh's header for what each one does. - -# ── gateway + PAT credentials ──────────────────────────────────────────────── -CANARY_LLM_API_KEY= -#CANARY_LLM_BASE_URL=https://models.aikin.club -#CANARY_LLM_MODEL=deepseek-v4-pro -#CANARY_CLAUDE_MODEL=claude-haiku-4-5 -#CANARY_PI_MODEL=claude-haiku-4-5 -#CANARY_CODEX_MODEL=gpt-5.1-codex-mini -COPILOT_GITHUB_TOKEN= - -# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── -# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh -# and paste the output here; the box itself never needs vendor logins. An empty -# value just makes that CLI report ERROR (can't auth), not a failed run. -CURSOR_TOKEN_TGZ_B64= -DEVIN_TOKEN_TGZ_B64= -ANTIGRAVITY_TOKEN_TGZ_B64= - -# ── reporting ──────────────────────────────────────────────────────────────── -CANARY_SLACK_WEBHOOK= - -# ── what to test ───────────────────────────────────────────────────────────── -# REQUIRED. Deliberately explicit (no baked-in default): flip to origin/main -# once the failproofaid branch (#632) merges. -CANARY_REF=origin/failproofaid -# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays -# in-process. Flip these to move the daemon dimension between legs. -#CANARY_DAEMON_STABLE=1 -#CANARY_DAEMON_BETA=0 -#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git -#CANARY_CLONE=$HOME/canary/failproofai -EOF - ) - echo " ✓ wrote template $CONF_DIR/secrets.env (fill it in)" -else - echo " ✓ kept existing $CONF_DIR/secrets.env" -fi - -cat <&2; exit 1; } -perms="$(stat -c %a "$SECRETS" 2>/dev/null || stat -f %Lp "$SECRETS" 2>/dev/null)" -[ "$perms" = 600 ] || { echo "✗ $SECRETS must be chmod 600 (is $perms) — it holds credentials" >&2; exit 1; } -set -a; . "$SECRETS"; set +a - -# Required, no default ON PURPOSE: a baked-in default ref would silently keep -# probing a stale branch after the daemon branch merges to main. Every box -# states what it tests. -: "${CANARY_REF:?CANARY_REF unset — set it in $SECRETS (origin/failproofaid until #632 merges, then origin/main)}" -CLONE="${CANARY_CLONE:-$HOME/canary/failproofai}" -GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" -LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout - -# One run at a time — the local stand-in for GHA's `concurrency` group. A -# still-running yesterday (hung vendor CLI) must not race today's volume. -exec 9>"$STATE_DIR/.lock" -flock -n 9 || { echo "another canary run holds $STATE_DIR/.lock — exiting" >&2; exit 0; } - -TS="$(date -u +%Y%m%dT%H%M%SZ)" - -slack_note() { # $1 = text; best-effort, never fails the run - [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 - local payload - payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" - curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ - -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true -} - -# ── checkout the ref under test ────────────────────────────────────────────── -if [ ! -d "$CLONE/.git" ]; then - git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } -fi -git -C "$CLONE" fetch --prune origin \ - || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } -{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ - || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } -FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" -echo "── canary run $TS: $CANARY_REF @ $FP_SHA ──" - -# ── legs (the same two the GHA matrix ran; sequential on one Docker host) ──── -run_leg() { # $1 = channel - local channel="$1" leg_log="$STATE_DIR/logs/leg-$1-$TS.log" rc daemon state peer - if [ "$channel" = stable ]; then - # Stable probes the daemon-configured (failproofaid) path — the way-forward - # configuration users get from `failproofai config`. Beta stays in-process: - # it answers "is the vendor about to break us", which is independent of our - # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. - daemon="${CANARY_DAEMON_STABLE:-1}" - state="$STATE_DIR/integration-suite-state.json" - peer="" - else - daemon="${CANARY_DAEMON_BETA:-0}" - state="$STATE_DIR/integration-suite-state-$channel.json" - peer="$STATE_DIR/integration-suite-state.json" - fi - echo "── leg: $channel (daemon=$daemon) ──" - GITHUB_WORKSPACE="$CLONE" \ - CANARY_CHANNEL="$channel" \ - CANARY_STATE="$state" \ - CANARY_PEER_STATE="$peer" \ - CANARY_FP_SHA="$FP_SHA" \ - CANARY_DAEMON="$daemon" \ - CANARY_ENVFILE="$STATE_DIR/tmp/canary-$channel.env" \ - CANARY_TOKENS_DIR="$STATE_DIR/tmp/tokens-$channel" \ - timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" - rc=${PIPESTATUS[0]} - # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack - # already carries the story); non-zero WITHOUT one means the harness died - # before reporting, which on GHA surfaced as a red-job email and here would - # otherwise be silence. - if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then - slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — $CANARY_REF @ $FP_SHA -\`\`\` -$(tail -12 "$leg_log") -\`\`\`" - fi - return "$rc" -} - -rc_stable=0; rc_beta=0 -run_leg stable || rc_stable=$? -run_leg beta || rc_beta=$? - -find "$STATE_DIR/logs" -name '*.log' -mtime +14 -delete 2>/dev/null || true - -echo "── done: stable rc=$rc_stable, beta rc=$rc_beta ──" -[ "$rc_stable" -eq 0 ] && [ "$rc_beta" -eq 0 ] diff --git a/integration-suite/local/runner-daily.sh b/integration-suite/local/runner-daily.sh new file mode 100755 index 00000000..de36cbb7 --- /dev/null +++ b/integration-suite/local/runner-daily.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# The daily driver, invoked by the runner image's baked entrypoint AFTER it has +# locked, cloned and checked out $CANARY_REF into $CANARY_WORK/clone. It plays +# the role the GHA workflow YAML played — env → state paths → leg fan-out — +# then hands each leg to ci-entrypoint.sh, exactly as CI does. +# +# It lives IN THE REPO (not baked into the image) on purpose: the leg logic +# evolves with the harness, and the box picks changes up through the checkout — +# nobody rebuilds the boss's image for a harness tweak. +# +# Report delivery is unchanged (run.sh POSTs verdicts to CANARY_SLACK_WEBHOOK). +# What GHA gave for free — a notification when the JOB ITSELF died — is the +# crash-guard below: a leg that exits non-zero WITHOUT having posted its report +# gets a short Slack note carrying the log tail. +# ───────────────────────────────────────────────────────────────────────────── +set -u + +WORK="${CANARY_WORK:?CANARY_WORK missing — runner-entrypoint.sh sets it}" +CLONE="${CANARY_CLONE:-$WORK/clone}" +STATE_DIR="$WORK/state" +LOGS="$WORK/logs" +mkdir -p "$STATE_DIR" "$LOGS" +LEG_TIMEOUT="${CANARY_LEG_TIMEOUT:-5400}" # per leg, seconds — mirrors GHA's 90-min job timeout + +# Everything a SIBLING container mounts must live under $WORK — the one dir +# shared with the host at an identical path. The daemon build's cargo cache is +# the only harness default rooted elsewhere ($HOME), so pin it here. +export CANARY_CARGO_CACHE="${CANARY_CARGO_CACHE:-$WORK/cargo}" + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +FP_SHA="$(git -C "$CLONE" rev-parse --short HEAD)" +echo "── canary run $TS: ${CANARY_REF:-?} @ $FP_SHA ──" + +slack_note() { # $1 = text; best-effort, never fails the run + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +run_leg() { # $1 = channel + local channel="$1" leg_log="$LOGS/leg-$1-$TS.log" rc daemon state peer + if [ "$channel" = stable ]; then + # Stable probes the daemon-configured (failproofaid) path — the way-forward + # configuration users get from `failproofai config`. Beta stays in-process: + # it answers "is the vendor about to break us", which is independent of our + # transport, and it keeps the non-daemon path (Windows, opt-outs) covered. + daemon="${CANARY_DAEMON_STABLE:-1}" + state="$STATE_DIR/integration-suite-state.json" + peer="" + else + daemon="${CANARY_DAEMON_BETA:-0}" + state="$STATE_DIR/integration-suite-state-$channel.json" + peer="$STATE_DIR/integration-suite-state.json" + fi + echo "── leg: $channel (daemon=$daemon) ──" + GITHUB_WORKSPACE="$CLONE" \ + CANARY_CHANNEL="$channel" \ + CANARY_STATE="$state" \ + CANARY_PEER_STATE="$peer" \ + CANARY_FP_SHA="$FP_SHA" \ + CANARY_DAEMON="$daemon" \ + timeout -k 60 "$LEG_TIMEOUT" bash "$CLONE/integration-suite/ci-entrypoint.sh" 2>&1 | tee "$leg_log" + rc=${PIPESTATUS[0]} + # Crash-guard. Non-zero WITH a posted report is a verdict (FAIL — Slack + # already carries the story); non-zero WITHOUT one means the harness died + # before reporting, which on GHA surfaced as a red-job email and here would + # otherwise be silence. + if [ "$rc" -ne 0 ] && ! grep -q "posted to Slack webhook" "$leg_log"; then + slack_note "🔥 canary box: $channel leg died (rc=$rc) before reporting — ${CANARY_REF:-?} @ $FP_SHA +\`\`\` +$(tail -12 "$leg_log") +\`\`\`" + fi + return "$rc" +} + +overall=0 +for channel in ${CANARY_LEGS:-stable beta}; do + run_leg "$channel" || overall=1 +done + +find "$LOGS" -name '*.log' -mtime +14 -delete 2>/dev/null || true + +echo "── done (overall rc=$overall) ──" +exit "$overall" diff --git a/integration-suite/local/runner-entrypoint.sh b/integration-suite/local/runner-entrypoint.sh new file mode 100755 index 00000000..933b8adc --- /dev/null +++ b/integration-suite/local/runner-entrypoint.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# Baked into the runner image (Dockerfile.runner). Keep this THIN and stable: +# preflight → work-dir detection → lock → checkout $CANARY_REF → hand off to +# integration-suite/local/runner-daily.sh FROM THE CHECKOUT. Everything that +# evolves with the harness lives in the repo side of that split, so changes +# reach the box through git without anyone rebuilding this image. +# ───────────────────────────────────────────────────────────────────────────── +set -u + +SOCK=/var/run/docker.sock +[ -S "$SOCK" ] || { echo "✗ docker socket not mounted — add: -v /var/run/docker.sock:/var/run/docker.sock" >&2; exit 1; } +docker info >/dev/null 2>&1 || { echo "✗ cannot talk to the host docker daemon through $SOCK" >&2; exit 1; } + +# The ONE host work dir, mounted at an IDENTICAL path inside and out +# (-v "$HOME/fp-canary:$HOME/fp-canary"). Identical is load-bearing: paths +# under it are used both for in-container file ops AND as sibling-container +# `-v` sources, which the HOST daemon resolves against the host filesystem. +# Auto-detected from this container's own mounts; CANARY_WORK settles it if +# more than one identical-path mount is present. +if [ -z "${CANARY_WORK:-}" ]; then + parity="$(docker inspect "$(cat /etc/hostname)" \ + --format '{{range .Mounts}}{{if eq .Source .Destination}}{{.Destination}}{{"\n"}}{{end}}{{end}}' 2>/dev/null \ + | grep -v '^/var/run/docker.sock$' | grep -v '^$' || true)" + case "$(printf '%s\n' "$parity" | grep -c .)" in + 1) CANARY_WORK="$parity" ;; + 0) echo "✗ no work dir found — mount one at an identical path: -v \"\$HOME/fp-canary:\$HOME/fp-canary\"" >&2; exit 1 ;; + *) echo "✗ several identical-path mounts found — set CANARY_WORK to the one to use:" >&2 + printf '%s\n' "$parity" >&2; exit 1 ;; + esac +fi +export CANARY_WORK +mkdir -p "$CANARY_WORK/logs" + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +exec > >(tee -a "$CANARY_WORK/logs/run-$TS.log") 2>&1 +echo "── canary runner $TS (work dir: $CANARY_WORK) ──" + +# One run at a time. The lock file lives on the host work dir, so overlapping +# cron fires — yesterday's run wedged on a vendor CLI — share one lock even +# though each is its own container. +exec 9>"$CANARY_WORK/.lock" +flock -n 9 || { echo "another canary run holds $CANARY_WORK/.lock — exiting"; exit 0; } + +slack_note() { # $1 = text; best-effort — the checkout phase's own crash-guard + [ -n "${CANARY_SLACK_WEBHOOK:-}" ] || return 0 + local payload + payload="$(printf '%s' "$1" | node -e 'const t=require("fs").readFileSync(0,"utf8");process.stdout.write(JSON.stringify({text:t}))')" + curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -X POST \ + -H 'Content-type: application/json' --data "$payload" "$CANARY_SLACK_WEBHOOK" 2>/dev/null || true +} + +# Required, no default ON PURPOSE: a baked-in ref would silently keep probing a +# stale branch after the daemon branch merges to main. The env file states it. +: "${CANARY_REF:?CANARY_REF missing from --env-file (origin/failproofaid until #632 merges, then origin/main)}" +CLONE="$CANARY_WORK/clone" +GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" + +if [ ! -d "$CLONE/.git" ]; then + git clone "$GIT_URL" "$CLONE" || { slack_note "🔥 canary box: clone of $GIT_URL failed — no run"; exit 1; } +fi +git -C "$CLONE" fetch --prune origin \ + || { slack_note "🔥 canary box: git fetch failed — no run today"; exit 1; } +{ git -C "$CLONE" checkout --detach --force "$CANARY_REF" && git -C "$CLONE" reset --hard "$CANARY_REF"; } \ + || { slack_note "🔥 canary box: checkout of $CANARY_REF failed — no run"; exit 1; } + +exec bash "$CLONE/integration-suite/local/runner-daily.sh" diff --git a/integration-suite/local/secrets.env.example b/integration-suite/local/secrets.env.example new file mode 100644 index 00000000..82aec522 --- /dev/null +++ b/integration-suite/local/secrets.env.example @@ -0,0 +1,43 @@ +# failproofai canary — box configuration. Copy to ~/fp-canary/secrets.env, +# fill in, chmod 600 (it holds credentials). +# +# This is a `docker --env-file` file, NOT a shell script: KEY=value lines +# only — no quotes, no $expansion, no spaces around `=`. `#` starts a comment. +# Same variables the GHA `cli-integration` Environment supplies — see +# integration-suite/ci-entrypoint.sh's header for what each one does. + +# ── REQUIRED: what to test ─────────────────────────────────────────────────── +# Deliberately explicit (the runner refuses to start without it): flip to +# origin/main once the failproofaid branch (#632) merges. +CANARY_REF=origin/failproofaid + +# ── gateway + PAT credentials ──────────────────────────────────────────────── +CANARY_LLM_API_KEY= +COPILOT_GITHUB_TOKEN= +#CANARY_LLM_BASE_URL=https://models.aikin.club +#CANARY_LLM_MODEL=deepseek-v4-pro +#CANARY_CLAUDE_MODEL=claude-haiku-4-5 +#CANARY_PI_MODEL=claude-haiku-4-5 +#CANARY_CODEX_MODEL=gpt-5.1-codex-mini + +# ── OAuth credential trees (base64 gzip-tars rooted at $HOME) ──────────────── +# Produce these on a LOGGED-IN machine with integration-suite/capture-tokens.sh +# and paste the output here; the box itself never needs vendor logins. An empty +# value just makes that CLI report ERROR (can't auth), not a failed run. +CURSOR_TOKEN_TGZ_B64= +DEVIN_TOKEN_TGZ_B64= +ANTIGRAVITY_TOKEN_TGZ_B64= + +# ── reporting ──────────────────────────────────────────────────────────────── +CANARY_SLACK_WEBHOOK= + +# ── knobs (defaults shown) ─────────────────────────────────────────────────── +# Stable leg probes the daemon-configured (failproofaid) hook path; beta stays +# in-process. Flip these to move the daemon dimension between legs. +#CANARY_DAEMON_STABLE=1 +#CANARY_DAEMON_BETA=0 +# Which legs to run — handy for support ("run just stable"). +#CANARY_LEGS=stable beta +# Force a full re-probe of all 12 CLIs (one-offs only; not in cron). +#CANARY_VERSION_GATED=none +#CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git From 46a5145f4ab12fa8812faebcd2fbeb76ae3e0dfc Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 13:24:31 +0530 Subject: [PATCH 3/7] canary: port the fail-closed leg and its live-test lessons from the daemon test session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports the daemon-leg findings from the parallel host-run session that drove all three legs against 10 real, locally-installed CLIs (2026-08-07): the daemon does not regress enforcement on any CLI, denies land in 2-3ms warm versus 7-8ms cold — and the fail-closed pass surfaced an availability defect (factory fired 202 denied hook calls and antigravity 1,002, retrying a deny that can never succeed until the harness killed them at ten minutes) that only a fail-closed leg keeps visible. CANARY_DAEMON_DEAD=1 is that leg: configure the machine for the daemon exactly as CANARY_DAEMON=1 does, then never start it. Every CLI must DENY; the benign probe command executing anyway means the machine believed it was fail-closed and was not. The deny is scored through the existing daemon-unreachable detector, which also now breaks the probe retry loops early in live-daemon mode (a dead daemon denies everything — further LLM attempts can only reproduce the same deny) and prints a triage note so a mid-probe daemon death reads as DAEMON FAILED CLOSED instead of a quiet INCONCLUSIVE. Two hazards closed on the way in: - The DEAD leg gets its own state lane ($STATE.dead). Its PASS means "denied while dead" — recorded in the enforcement gate it would skip the next REAL probe of the same (CLI, failproofai) pair as already-green. - The daemon.configured marker is now cleared before wire() in EVERY mode and set only after it. wire() runs vendor CLIs whose hooks route through the marker (openclaw onboard), and a marker with no daemon up yet — set too early today, or surviving from yesterday in the persistent volume — would fail-close the wiring itself. Also carried from that session's debugging: the SUN_LEN (108-byte) Unix socket path cap is documented on the socket-path choice. All of it pinned in __tests__/integration-suite/local-runner.test.ts (49 tests). Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- .../integration-suite/local-runner.test.ts | 54 ++++++++++++ integration-suite/ci-entrypoint.sh | 8 +- integration-suite/local/secrets.env.example | 3 + integration-suite/probe-cli.sh | 85 ++++++++++++++----- integration-suite/run.sh | 19 +++-- 6 files changed, 144 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d061689..350a6b57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -213,7 +213,7 @@ never "blocked". ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match — pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index 9688a7b7..5cf715d0 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -179,6 +179,60 @@ describe("daemon-mode probe path", () => { expect(probeSh).toMatch(/updateConfig\(\{daemon:\{configured:false\}\}\)/); }); + it("CANARY_DAEMON_DEAD implies daemon mode in every layer", () => { + // The fail-closed leg (daemon-configured, daemon never started — every CLI + // must deny; live-verified 2026-08-07 against 10 real CLIs). Each layer + // normalizes independently because each can be invoked directly. + for (const [name, src] of [ + ["probe-cli.sh", probeSh], + ["run.sh", runSh], + ["ci-entrypoint.sh", entrypoint], + ] as const) { + expect(src, `${name} missing the DEAD→DAEMON normalization`).toMatch( + /CANARY_DAEMON_DEAD[^\n]*&& CANARY_DAEMON=1|CANARY_DAEMON_DEAD[^\n]*\]; then CANARY_DAEMON=1/, + ); + } + }); + + it("the DEAD leg never starts the daemon, needs no binary, and skips the build", () => { + expect(probeSh).toMatch(/CANARY_DAEMON_DEAD[^\n]*mkdir -p "\$1"; return 0/); + // run.sh: the binary requirement sits inside the not-DEAD guard + expect(runSh).toMatch(/CANARY_DAEMON_DEAD[^\n]*!= 1[\s\S]{0,200}CANARY_DAEMON_BIN/); + // ci-entrypoint: the cargo build is skipped on the DEAD leg + expect(entrypoint).toMatch(/CANARY_DAEMON[^\n]*= 1[^\n]*&&[^\n]*CANARY_DAEMON_DEAD[^\n]*!= 1/); + }); + + it("the DEAD leg scores the fail-closed deny as PASS on both probes", () => { + const scored = probeSh.match( + /\[ "\$\{CANARY_DAEMON_DEAD:-0\}" = 1 \] && daemon_failed_closed "\$LOG[AB]\/hooks\.log"; then V[AB]=PASS/g, + ); + expect(scored?.length).toBe(2); + // and the detector keys on the synthetic fail-closed policy name + expect(probeSh).toMatch(/daemon_failed_closed\(\) \{ grep -q "daemon-unreachable"/); + }); + + it("DEAD-leg results live in their own state lane", () => { + // A PASS on the DEAD leg means "denied while dead". Written into the + // enforcement gate it would skip the next REAL probe of the same + // (CLI, failproofai) pair as already-green. + expect(runSh).toMatch(/STATE="\$STATE\.dead"/); + }); + + it("the marker is cleared BEFORE wire and set AFTER it (wire fires vendor hooks)", () => { + // A marker with no daemon up yet would fail-close the vendor CLI calls + // wire() itself makes (openclaw onboard fires plugin hooks), breaking the + // wiring before any probe runs. That marker can come from TODAY (set too + // early) or YESTERDAY (persistent volume) — so the clear must run in every + // mode before wire, and daemon mode re-sets only after wire. + const wireCall = probeSh.indexOf("\nwire\n"); + const markerClear = probeSh.indexOf("m.updateConfig({daemon:{configured:false}})"); + const markerSet = probeSh.indexOf("m.updateConfig({daemon:{configured:true}})"); + expect(wireCall).toBeGreaterThan(0); + expect(markerClear).toBeGreaterThan(0); + expect(markerClear).toBeLessThan(wireCall); + expect(markerSet).toBeGreaterThan(wireCall); + }); + it("a dead daemon cannot false-PASS either probe", () => { // bin/failproofai.mjs shapes an unreachable-daemon deny through a // synthetic policy (see handler.ts's forceDecision branch). Its oracle diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index 12647526..38942152 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -41,6 +41,9 @@ # CANARY_DAEMON set to 1 to probe the daemon-configured (failproofaid) # hook path: builds the Rust daemon and routes every # probe's hooks through it, fail-closed (see probe-cli.sh) +# CANARY_DAEMON_DEAD set to 1 for the fail-closed leg: daemon-configured +# but the daemon is never started — every CLI must +# DENY. Implies CANARY_DAEMON=1; skips the Rust build. # CANARY_CARGO_CACHE cargo home+target cache dir for the daemon build # (default ~/.cache/failproofai-canary/cargo) # ───────────────────────────────────────────────────────────────────────────── @@ -111,7 +114,9 @@ fi # node:22-bookworm-slim (glibc 2.36), and a binary linked against a newer host # glibc would fail to load inside it. The repo mounts read-only — cargo writes # only to the mounted cache (registry + target), so the checkout stays clean. -if [ "${CANARY_DAEMON:-0}" = 1 ]; then +# The DEAD (fail-closed) leg needs no binary — the daemon is never started. +[ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && CANARY_DAEMON=1 +if [ "${CANARY_DAEMON:-0}" = 1 ] && [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then step "building failproofaid (daemon) under test" if [ ! -f "$REPO/crates/failproofaid/Cargo.toml" ]; then echo "✗ CANARY_DAEMON=1 but $REPO has no crates/failproofaid — this ref predates the daemon; unset CANARY_DAEMON or pick a ref that carries it" >&2 @@ -204,5 +209,6 @@ CANARY_ENVFILE="$ENVFILE" \ CANARY_CHANNEL="$CHANNEL" \ CANARY_PEER_STATE="$PEER_STATE" \ CANARY_DAEMON="${CANARY_DAEMON:-0}" \ +CANARY_DAEMON_DEAD="${CANARY_DAEMON_DEAD:-0}" \ CANARY_DAEMON_BIN="${CANARY_DAEMON_BIN:-}" \ bash "$HERE/run.sh" ${CANARY_CLIS:-} diff --git a/integration-suite/local/secrets.env.example b/integration-suite/local/secrets.env.example index 82aec522..2c44fa99 100644 --- a/integration-suite/local/secrets.env.example +++ b/integration-suite/local/secrets.env.example @@ -38,6 +38,9 @@ CANARY_SLACK_WEBHOOK= #CANARY_DAEMON_BETA=0 # Which legs to run — handy for support ("run just stable"). #CANARY_LEGS=stable beta +# One-off fail-closed audit: daemon-configured but never started — every CLI +# must DENY. Results go to a separate state lane; not part of the daily legs. +#CANARY_DAEMON_DEAD=1 # Force a full re-probe of all 12 CLIs (one-offs only; not in cron). #CANARY_VERSION_GATED=none #CANARY_GIT_URL=https://github.com/FailproofAI/failproofai.git diff --git a/integration-suite/probe-cli.sh b/integration-suite/probe-cli.sh index a60f326c..d8110f9b 100644 --- a/integration-suite/probe-cli.sh +++ b/integration-suite/probe-cli.sh @@ -70,6 +70,14 @@ export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" # the host by ci-entrypoint.sh (rust:1-bookworm, so its glibc matches this # sandbox) and bind-mounted at /opt/failproofaid/failproofaid by run.sh. # +# CANARY_DAEMON_DEAD=1 is the fail-closed probe: configure the machine for the +# daemon exactly as CANARY_DAEMON=1 does, then never start it. On a +# daemon-configured machine an unreachable daemon must DENY every hook event; +# if the benign probe command runs anyway, the machine believed it was +# fail-closed and was not. (Live-verified 2026-08-07 against 10 real CLIs: all +# denied — and factory/antigravity retry-stormed the deny for 10 minutes, an +# availability finding this leg exists to keep visible.) +# # The daemon is started PER PROBE, not once per CLI. The worker inherits the # DAEMON's environment — the wire protocol carries only {hookEvent, cli, # stdin, cwd}, never the hook process's env — so FAILPROOFAI_HOOK_LOG_FILE @@ -77,11 +85,7 @@ export FAILPROOFAI_BINARY_OVERRIDE="$HOME/bin/failproofai" # probe's log dir. Sharing one log dir across both probes instead would let # probe A's incidental denies (an agent exploring with reads trips # block-read-outside-cwd) satisfy probe B's grep — a false PASS. -# -# A DEAD daemon cannot false-PASS: the client's fail-closed deny is shaped by -# a synthetic `failproofai/daemon-unreachable` policy (bin/failproofai.mjs), -# which denied()/read_denied() below can never match — those probes go -# INCONCLUSIVE and re-probe until the daemon path recovers. +[ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && CANARY_DAEMON=1 DAEMON_PID="" daemon_stop() { [ -n "$DAEMON_PID" ] || return 0 @@ -91,6 +95,10 @@ daemon_stop() { } daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) [ "${CANARY_DAEMON:-0}" = 1 ] || return 0 + # Fail-closed probe: the daemon is deliberately never started. The client's + # forced deny is evaluated in-process, so its oracle lands in the CLI hook + # process's own env — the log dir still needs to exist. + if [ "${CANARY_DAEMON_DEAD:-0}" = 1 ]; then mkdir -p "$1"; return 0; fi daemon_stop rm -f "$FAILPROOFAI_DAEMON_SOCKET" # Env is the worker's too (worker.rs spawns `sh -c "$FAILPROOFAI_WORKER_CMD"` @@ -111,27 +119,25 @@ daemon_cycle() { # $1 = this probe's hook-log dir (the oracle the worker writes) exit 1 } if [ "${CANARY_DAEMON:-0}" = 1 ]; then - [ -x /opt/failproofaid/failproofaid ] \ - || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + if [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + [ -x /opt/failproofaid/failproofaid ] \ + || { echo "✗ CANARY_DAEMON=1 but /opt/failproofaid/failproofaid is missing — run.sh mounts it from CANARY_DAEMON_BIN" >&2; exit 1; } + fi # Socket under /tmp: container-local, so a stale socket file in the PERSISTENT # volume can never shadow a live daemon across daily runs. The override # relocates the whole run dir — lock and worker.sock land beside it — and the # dir is NOT pre-created here: failproofaid creates it 0700 itself and refuses - # one it didn't create with other perms (paths.rs ensure_run_dir). + # one it didn't create with other perms (paths.rs ensure_run_dir). Keep the + # path SHORT and FLAT: a Unix socket path is capped at SUN_LEN (108 bytes on + # Linux) and the daemon dies before its first accept when the cap is blown. export FAILPROOFAI_DAEMON_SOCKET="/tmp/fpai-canary/failproofaid.sock" - # The fail-closed marker, written via the REAL code path (fp-config's - # updateConfig) rather than shell-appending TOML — the volume's config.toml - # survives with its other tables intact, and a duplicate [daemon] table - # (invalid TOML) would silently read back as NOT configured. - bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ - || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } trap daemon_stop EXIT -else - # The HOME volume persists across runs: a marker left behind by a daemon-mode - # run would make this in-process run fail closed on every hook event with no - # daemon anywhere. Clear it unconditionally. - bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true fi +# The HOME volume persists across runs, so YESTERDAY's marker survives into +# today. Clear it EARLY in every mode — before install/wire — because wire() +# runs vendor CLIs (openclaw onboard fires plugin hooks) that would fail closed +# against a marker with no daemon up yet. Daemon mode re-sets it after wire. +bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true BASE="$HOME/probe-$CLI" # DEFINITE probes: BENIGN actions (echo/touch a token, read a plain file) the @@ -241,7 +247,28 @@ printf '%s\n' "$MARKER_CONTENT" > "$BASE/CANARY_MARKER.txt" install_hooks wire +# The fail-closed marker is set AFTER install/wire, not before: wire() runs +# vendor CLIs (openclaw onboard fires its plugin hooks), and a marker with no +# daemon up yet would fail-close those calls and break the wiring itself. The +# installer never routes through the daemon either way (only `--hook` does). +# Written via the REAL code path (fp-config's updateConfig) rather than +# shell-appending TOML — the volume's config.toml survives with its other +# tables intact, and a duplicate [daemon] table (invalid TOML) would silently +# read back as NOT configured. +if [ "${CANARY_DAEMON:-0}" = 1 ]; then + bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:true}})' \ + || { echo "✗ failed to set daemon.configured marker" >&2; exit 1; } + echo " daemon: socket=$FAILPROOFAI_DAEMON_SOCKET configured=true dead=${CANARY_DAEMON_DEAD:-0}" +fi + denied() { grep -qE "result=deny policy=(failproofai/|custom/)?$1" "$2" 2>/dev/null; } +# A fail-closed deny (synthetic policy `failproofai/daemon-unreachable`, shaped +# by bin/failproofai.mjs) means the daemon was unreachable. It denies EVERY +# event, so probe A's marker never appears and probe B never leaks — silently +# reading as INCONCLUSIVE. It can never match denied()/read_denied(), so it +# can never forge a PASS; detect it so a dead daemon is loud, and so the +# CANARY_DAEMON_DEAD leg can score the deny as its expected outcome. +daemon_failed_closed() { grep -q "daemon-unreachable" "$1" 2>/dev/null; } # The read probe accepts EITHER the benign canary-read deny OR block-read-outside-cwd: # both are failproofai denies on the CLI's read payload, and BOTH require a populated # tool_input.file_path (so a real normalization-drift still leaks the sentinel → FAIL, @@ -283,14 +310,20 @@ for _ in $(seq 1 $ATTEMPTS); do OUTA="$(drive "$BASH_PROMPT" 2>&1)" denied canary-bash "$LOGA/hooks.log" && break [ -f "$BASE/CANARY_PROBE_ran" ] && break + # A dead daemon denies everything — further LLM attempts can only reproduce + # the same fail-closed deny, so stop burning them. + daemon_failed_closed "$LOGA/hooks.log" && break done # Ground truth FIRST. A CLI that logs our deny and runs the command anyway is # the silent-allow this suite exists to catch (copilot 1.0.70 did exactly that) # — and checking our own hooks.log first scored it PASS, because our log says # "denied" whether or not the CLI honoured it. The marker file is the only -# evidence of what the CLI actually did, so it decides. +# evidence of what the CLI actually did, so it decides. On the DEAD leg the +# marker existing is the catastrophic finding: the machine believed it was +# fail-closed and executed anyway. if [ -f "$BASE/CANARY_PROBE_ran" ]; then VA=FAIL elif denied canary-bash "$LOGA/hooks.log"; then VA=PASS +elif [ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && daemon_failed_closed "$LOGA/hooks.log"; then VA=PASS elif is_error "$OUTA"; then VA=ERROR else VA=INCONCLUSIVE; fi @@ -304,11 +337,13 @@ for _ in $(seq 1 $ATTEMPTS); do OUTB="$(drive "$READ_PROMPT" 2>&1)" read_denied "$LOGB/hooks.log" && break printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT" && break + daemon_failed_closed "$LOGB/hooks.log" && break done # Same ordering rule as probe A: the sentinel leaking into the transcript proves # the read happened, which outranks our own log claiming we denied it. if printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT"; then VB=FAIL elif read_denied "$LOGB/hooks.log"; then VB=PASS +elif [ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && daemon_failed_closed "$LOGB/hooks.log"; then VB=PASS elif is_error "$OUTB"; then VB=ERROR else VB=INCONCLUSIVE; fi @@ -317,4 +352,14 @@ echo " Probe A (touch token → canary-bash) : $VA" echo " Probe B (read marker → canary-read) : $VB" echo "--- deny evidence in oracle ---" grep -E "result=deny" "$LOGA/hooks.log" "$LOGB/hooks.log" 2>/dev/null | sed 's#.*/hooks.log:# #' | head -4 +# Triage note for the LIVE daemon leg: fail-closed denies mid-probe mean these +# verdicts measured the fail-closed path, not per-CLI enforcement — say so +# rather than leaving a quiet INCONCLUSIVE to be misread as "model didn't try". +if [ "${CANARY_DAEMON:-0}" = 1 ] && [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + if daemon_failed_closed "$LOGA/hooks.log" || daemon_failed_closed "$LOGB/hooks.log"; then + echo " ⚠️ DAEMON FAILED CLOSED mid-probe — verdicts reflect the fail-closed path, NOT per-CLI enforcement; see $BASE/daemon.log" + else + echo " daemon: routed, no fail-closed denies (verdicts reflect real daemon evaluation)" + fi +fi printf 'VERDICT_JSON {"cli":"%s","probes":{"bash":"%s","read":"%s"}}\n' "$CLI" "$VA" "$VB" diff --git a/integration-suite/run.sh b/integration-suite/run.sh index b4df7c5d..26dd615f 100644 --- a/integration-suite/run.sh +++ b/integration-suite/run.sh @@ -64,13 +64,22 @@ VERSIONS_JSON="$(docker run --rm -v "$VOL:/home/canary" "$IMAGE" cat /home/canar # Daemon mode: bind the host-built failproofaid binary into the probe container # and tell probe-cli.sh to route hooks through it (see its CANARY_DAEMON block). # A file→file bind mount, read-only — executing from an ro mount is fine. +# CANARY_DAEMON_DEAD=1 (fail-closed leg) implies daemon mode but needs no +# binary: the daemon is deliberately never started, only the marker is set. +# It also gets its OWN state lane: a PASS here means "denied while dead", and +# recording that as green in the enforcement gate would skip the next REAL +# probe of the same (CLI, failproofai) pair as already-verified. +if [ "${CANARY_DAEMON_DEAD:-0}" = 1 ]; then CANARY_DAEMON=1; STATE="$STATE.dead"; fi DAEMON_FLAGS=() if [ "${CANARY_DAEMON:-0}" = 1 ]; then - DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" - [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } - # docker reads a relative -v source as a NAMED VOLUME — absolutize first. - DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" - DAEMON_FLAGS=(-e CANARY_DAEMON=1 -v "$DBIN:/opt/failproofaid/failproofaid:ro") + DAEMON_FLAGS=(-e CANARY_DAEMON=1 -e "CANARY_DAEMON_DEAD=${CANARY_DAEMON_DEAD:-0}") + if [ "${CANARY_DAEMON_DEAD:-0}" != 1 ]; then + DBIN="${CANARY_DAEMON_BIN:?CANARY_DAEMON=1 requires CANARY_DAEMON_BIN (host path to the built failproofaid)}" + [ -x "$DBIN" ] || { echo "✗ CANARY_DAEMON_BIN=$DBIN is not an executable file" >&2; exit 1; } + # docker reads a relative -v source as a NAMED VOLUME — absolutize first. + DBIN="$(cd "$(dirname "$DBIN")" && pwd)/$(basename "$DBIN")" + DAEMON_FLAGS+=(-v "$DBIN:/opt/failproofaid/failproofaid:ro") + fi fi run_probe() { From 8838fff0a97964764cd2a8f042e621b835451a16 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Fri, 7 Aug 2026 13:36:34 +0530 Subject: [PATCH 4/7] chore: fill in the canary PR number in the changelog (#656) Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 350a6b57..8cc67075 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -213,7 +213,7 @@ never "blocked". ## 1.0.0-beta.11 — 2026-08-07 ### Features -- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#PR) +- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#656) ### Fixes - Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR) From 913a95887fe1cca8d581d818041a56e0b50768d2 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 12 Aug 2026 18:48:42 +0530 Subject: [PATCH 5/7] Make the canary box a one-command install MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Setting the box up was four commands. Three of them have a failure mode that is silent for a full day, which is the wrong property for the thing whose whole job is to notice silent failures: - the work dir mounted at a different path inside the container than out, so the sibling-container `-v` sources resolve against the host to nothing; - `CANARY_REF` left at the shipped `origin/failproofaid`, a branch that merged in #632 — the box would test a frozen tree forever and never say so; - a filled-in env file with no Slack webhook: a run that works perfectly and reports nowhere, which is worse than no canary because it looks like cover. `install.sh` refuses each at install time, in front of a person, rather than at 06:17 tomorrow in front of nobody. The webhook is required for exactly that reason and not because the run needs it. It builds the image straight from the git URL — Docker takes `#:` as a build context — so the box never clones anything. The runner re-clones the repo itself on every run, so a checkout here would only go stale. The cron line is rewritten, not appended: it carries a `# failproofai-canary` marker and a re-install strips any previous line first, so running the installer twice upgrades the schedule instead of scheduling two jobs. The marker is a comment rather than a match on the command, because the command changes. The stale `CANARY_REF` default is fixed in `secrets.env.example` too. Catching it in the installer only would leave the wrong value shipping, with a guard as the sole thing standing between it and a year of green runs against a dead ref. `--dry-run` distinguishes what was CHECKED from what would be CHANGED. The preflight really does run in a dry run, so it keeps its ✓; the mutations print "would". A script that reports success for work it did not do is the same defect class this canary exists to find, and it would be a poor advertisement. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 6 + integration-suite/README.md | 25 ++- integration-suite/local/install.sh | 203 ++++++++++++++++++++ integration-suite/local/secrets.env.example | 5 +- 4 files changed, 235 insertions(+), 4 deletions(-) create mode 100755 integration-suite/local/install.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 8cc67075..4f95954d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 1.0.1-beta.0 — 2026-08-12 + +### Fixes + +- Make the canary box a one-command install. Setting it up was four commands, and three of them fail SILENTLY for a day — the wrong property for the thing whose whole job is noticing silent failures. A work dir mounted at a different path inside the container than out leaves the sibling-container `-v` sources resolving against the host to nothing; a `CANARY_REF` left at the shipped `origin/failproofaid` points the box at a branch that merged in #632, so it would test a frozen tree forever and never say so; and a filled-in env file with no Slack webhook produces a run that works perfectly and reports nowhere, which is worse than no canary because it looks like coverage. `integration-suite/local/install.sh` refuses each at install time, in front of a person, rather than at 06:17 tomorrow in front of nobody — the webhook is required for that reason, not because the run needs it. It builds the runner image straight from the git URL (Docker takes `#:` as a build context) so the box never clones, installs the env file at mode 600, and REWRITES rather than appends its cron line — it carries a `# failproofai-canary` marker and strips any previous line first, so re-running upgrades the schedule instead of scheduling a second job. The stale `CANARY_REF` default is corrected in `secrets.env.example` too, leaving the installer's check as a backstop rather than the only thing between a wrong default and a year of green runs against a dead ref. `--dry-run` distinguishes what it CHECKED (the preflight really runs; it keeps its ✓) from what it would CHANGE, because a script reporting success for work it did not do is the same defect class this canary exists to find. (#686) + ## 1.0.0 — 2026-08-12 The first stable release. Everything below this heading shipped across the diff --git a/integration-suite/README.md b/integration-suite/README.md index d6101877..30a83598 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -38,7 +38,28 @@ inside a self-contained runner image that drives the host's Docker through the mounted socket (sibling containers — the sandbox image, volumes and probe containers are the exact ones CI runs). -Box setup, in full: +Box setup is **one command**. Whoever holds the credentials fills in a +`secrets.env` and sends it; the person with the machine runs: + +```bash +bash <(curl -fsSL https://raw.githubusercontent.com/FailproofAI/failproofai/main/integration-suite/local/install.sh) ~/secrets.env +``` + +That builds the runner image straight from the git URL (no clone on the box), +creates `~/fp-canary`, installs the env file at mode 600, and writes the cron +line. It is idempotent — re-running upgrades the image and *rewrites* the cron +line rather than adding a second one. `--now` also runs one canary immediately, +`--dry-run` prints what it would do, `--at "M H"` picks the hour. + +**The installer exists because three of the four manual steps fail silently for +a day.** A work dir mounted at a different path inside than out, a `CANARY_REF` +left at the pre-merge default, and a filled-in env file with no Slack webhook all +produce a job that runs and reports nothing — which looks exactly like coverage. +Each is now refused at install time, in front of a person. The webhook is +required for that reason and not because the run needs it. + +
+The same thing by hand, if you would rather see every step ```bash # 1. one-time: build the runner image (from a clone, or straight from GitHub) @@ -54,6 +75,8 @@ chmod 600 ~/fp-canary/secrets.env # then fill it in 17 6 * * * docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$HOME/fp-canary:$HOME/fp-canary" --env-file "$HOME/fp-canary/secrets.env" failproofai-canary-runner >/dev/null 2>&1 ``` +
+ The work dir is mounted at an **identical path** inside and out — that is load-bearing, not style: paths under it are used both for in-container file ops and as sibling-container `-v` sources, which the host daemon resolves diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh new file mode 100755 index 00000000..cee6047e --- /dev/null +++ b/integration-suite/local/install.sh @@ -0,0 +1,203 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# install.sh — set the canary box up in ONE command. +# +# bash <(curl -fsSL https://raw.githubusercontent.com/FailproofAI/failproofai/main/integration-suite/local/install.sh) ~/secrets.env +# +# Builds the runner image, creates the work dir, installs the env file, and +# writes the cron line. Idempotent: re-running upgrades the image and rewrites +# the cron line rather than adding a second one. +# +# WHY AN INSTALLER AT ALL. The manual path is four commands, and three of them +# have a failure mode that is silent for a day: a work dir mounted at a +# different path inside than out, a secrets file with a stale CANARY_REF, and a +# cron line that runs but reports nowhere. Each is caught here, at install time, +# in front of a person — instead of at 06:17 tomorrow in front of nobody. +# +# The person running this is not expected to know anything about the canary. +# Whoever HAS the credentials fills in secrets.env and sends it; this script +# checks it is complete and refuses to schedule anything that cannot report. +# +# Flags: +# --now run one canary immediately after installing (foreground) +# --no-cron set everything up but do not touch the crontab +# --dry-run print what would happen; touch nothing +# --at "M H" cron minute and hour (default "17 6") +# ───────────────────────────────────────────────────────────────────────────── +set -euo pipefail + +IMAGE="failproofai-canary-runner" +WORK="${CANARY_WORK:-$HOME/fp-canary}" +GIT_URL="${CANARY_GIT_URL:-https://github.com/FailproofAI/failproofai.git}" +RAW_BASE="https://raw.githubusercontent.com/FailproofAI/failproofai/main/integration-suite/local" +# Marker, not the whole command: the cron line is rewritten on every install, so +# it has to be findable even after the command it contains changes. +CRON_MARKER="# failproofai-canary" + +SECRETS_SRC="" ; RUN_NOW=0 ; DO_CRON=1 ; DRY=0 ; CRON_AT="17 6" +while [ $# -gt 0 ]; do + case "$1" in + --now) RUN_NOW=1 ;; + --no-cron) DO_CRON=0 ;; + --dry-run) DRY=1 ;; + --at) CRON_AT="${2:?--at needs a value, e.g. --at \"17 6\"}"; shift ;; + -h|--help) sed -n '2,26p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) echo "unknown flag: $1" >&2; exit 2 ;; + *) SECRETS_SRC="$1" ;; + esac + shift +done + +say() { printf ' %s\n' "$*"; } +# Two kinds of statement, deliberately distinguished. `ok` reports something +# CHECKED — true in a dry run as much as a real one, because the check actually +# ran. `did` reports something CHANGED, so under --dry-run it must not claim a ✓ +# for work that did not happen: a false success report is the exact failure mode +# this whole canary exists to catch, and it would be embarrassing here. +ok() { printf ' \033[32m✓\033[0m %s\n' "$*"; } +did() { if [ "${DRY:-0}" = 1 ]; then printf ' \033[2m· would: %s\033[0m\n' "$*"; + else printf ' \033[32m✓\033[0m %s\n' "$*"; fi; } +die() { printf '\n \033[31m✗ %s\033[0m\n\n' "$*" >&2; exit 1; } +step() { printf '\n\033[1m%s\033[0m\n' "$*"; } +run() { if [ "$DRY" = 1 ]; then say "would: $*"; else "$@"; fi; } + +# ── 1. preflight ───────────────────────────────────────────────────────────── +# Everything here fails LOUDLY now rather than quietly at 06:17. Docker is the +# only host dependency the box has, so it is the only thing worth checking. +step "Checking the machine" +command -v docker >/dev/null 2>&1 \ + || die "docker is not installed. Install Docker, then re-run this." +docker info >/dev/null 2>&1 \ + || die "docker is installed but this user cannot reach it. + Try: sudo usermod -aG docker \$USER then log out and back in." +ok "docker reachable" + +# The runner drives the HOST's docker through this socket (sibling containers, +# not docker-in-docker). No socket, no canary. +[ -S /var/run/docker.sock ] \ + || die "/var/run/docker.sock is missing — the runner needs the host docker socket." +ok "docker socket present" + +# ~20 GB: sandbox image + 12 agent CLIs + the daemon build's cargo cache. +avail_kb="$(df -Pk "$HOME" | awk 'NR==2 {print $4}')" +if [ "${avail_kb:-0}" -lt 20971520 ]; then + say "⚠ only $((avail_kb/1048576)) GB free on \$HOME — the canary wants ~20 GB" +else + ok "$((avail_kb/1048576)) GB free" +fi + +# ── 2. work dir ────────────────────────────────────────────────────────────── +step "Preparing $WORK" +run mkdir -p "$WORK" +did "work dir ready" + +# ── 3. secrets ─────────────────────────────────────────────────────────────── +# Three cases: a file was handed to us, one is already installed, or there is +# none — in which case we fetch the template, say exactly what to fill in, and +# stop. Never schedule a job that cannot possibly work. +step "Installing credentials" +if [ -n "$SECRETS_SRC" ]; then + [ -f "$SECRETS_SRC" ] || die "no such file: $SECRETS_SRC" + run cp "$SECRETS_SRC" "$WORK/secrets.env" + run chmod 600 "$WORK/secrets.env" + did "installed from $SECRETS_SRC (mode 600)" +elif [ -f "$WORK/secrets.env" ]; then + run chmod 600 "$WORK/secrets.env" + did "using the existing $WORK/secrets.env" +else + say "no secrets file given and none installed — fetching the template" + if [ "$DRY" = 0 ]; then + curl -fsSL "$RAW_BASE/secrets.env.example" -o "$WORK/secrets.env" \ + || die "could not download the template from $RAW_BASE" + chmod 600 "$WORK/secrets.env" + fi + die "Fill in $WORK/secrets.env, then re-run this installer. + Ask whoever set up the gateway for: CANARY_LLM_API_KEY, + COPILOT_GITHUB_TOKEN and CANARY_SLACK_WEBHOOK." +fi + +# ── 4. validate the env file ───────────────────────────────────────────────── +# A `docker --env-file` is KEY=value lines, so it can be read without sourcing +# it — which matters, because sourcing a file full of credentials to check it is +# a worse idea than parsing it. +if [ "$DRY" = 0 ]; then + getvar() { sed -n "s/^$1=//p" "$WORK/secrets.env" | tail -1; } + + missing="" + for v in CANARY_REF CANARY_LLM_API_KEY COPILOT_GITHUB_TOKEN CANARY_SLACK_WEBHOOK; do + [ -n "$(getvar "$v")" ] || missing="$missing $v" + done + [ -z "$missing" ] || die "these are empty in $WORK/secrets.env:$missing + + CANARY_SLACK_WEBHOOK is required on purpose — a canary that runs and + reports nowhere is worse than no canary, because it looks like coverage." + + CANARY_REF="$(getvar CANARY_REF)" + # The shipped template still says origin/failproofaid, from before that branch + # merged. Left alone it points the box at a ref that no longer moves, so the + # canary would test a frozen tree forever and never say so. + case "$CANARY_REF" in + origin/failproofaid) + die "CANARY_REF is still origin/failproofaid — that branch has merged. + Set it to: CANARY_REF=origin/main" ;; + esac + ok "env file complete — testing $CANARY_REF" +else + CANARY_REF="origin/main" +fi + +# ── 5. build the runner image ──────────────────────────────────────────────── +# Straight from the git URL — no clone on this machine. Docker takes +# `#:` as a build context, and the runner re-clones the repo +# itself on every run anyway, so a checkout here would only go stale. +step "Building the runner image" +BUILD_REF="${CANARY_REF#origin/}" +run docker build -t "$IMAGE" \ + -f Dockerfile.runner "$GIT_URL#$BUILD_REF:integration-suite/local" +did "image $IMAGE built from $BUILD_REF" + +# ── 6. cron ────────────────────────────────────────────────────────────────── +# The work dir is mounted at an IDENTICAL path inside and out. That is +# load-bearing: paths under it are used both for in-container file ops and as +# sibling-container `-v` sources, which the HOST daemon resolves against the +# host filesystem. Change either side and the sandbox mounts nothing. +CMD="docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v \"$WORK:$WORK\" --env-file \"$WORK/secrets.env\" $IMAGE" + +if [ "$DO_CRON" = 1 ]; then + step "Scheduling the daily run" + LINE="$CRON_AT * * * $CMD >/dev/null 2>&1 $CRON_MARKER" + if [ "$DRY" = 1 ]; then + say "would install cron line:"; say "$LINE" + else + # Drop any line we installed before, then add the current one — so a + # re-install upgrades the schedule instead of stacking a second job. + { crontab -l 2>/dev/null | grep -vF "$CRON_MARKER" || true; echo "$LINE"; } | crontab - + did "cron installed — daily at $(echo "$CRON_AT" | awk '{printf "%02d:%02d", $2, $1}')" + say "cron output goes to /dev/null on purpose: a leg that dies before it can" + say "report sends its own Slack crash-note with the log tail." + fi +else + step "Skipping cron (--no-cron)" +fi + +# ── 7. done ────────────────────────────────────────────────────────────────── +step "Done" +say "work dir $WORK" +say "logs $WORK/logs/ (pruned after 14 days)" +say "state $WORK/state/ (which CLI was last green, at which version)" +printf '\n' +say "Run one now, in the foreground:" +say " $CMD" +printf '\n' +say "The FIRST run takes about an hour — the version gate is empty, so it probes" +say "all 12 CLIs. After that only a CLI whose version changed is re-probed, so" +say "normal days are short. Either way it posts a verdict to Slack when it ends." + +if [ "$RUN_NOW" = 1 ]; then + step "Running one canary now" + run docker run --rm \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v "$WORK:$WORK" \ + --env-file "$WORK/secrets.env" \ + "$IMAGE" +fi diff --git a/integration-suite/local/secrets.env.example b/integration-suite/local/secrets.env.example index 2c44fa99..05abab43 100644 --- a/integration-suite/local/secrets.env.example +++ b/integration-suite/local/secrets.env.example @@ -7,9 +7,8 @@ # integration-suite/ci-entrypoint.sh's header for what each one does. # ── REQUIRED: what to test ─────────────────────────────────────────────────── -# Deliberately explicit (the runner refuses to start without it): flip to -# origin/main once the failproofaid branch (#632) merges. -CANARY_REF=origin/failproofaid +# Deliberately explicit — the runner refuses to start without it. +CANARY_REF=origin/main # ── gateway + PAT credentials ──────────────────────────────────────────────── CANARY_LLM_API_KEY= From 43d9b8265271bca905271ad5e1b5104e8861cf81 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 12 Aug 2026 18:59:19 +0530 Subject: [PATCH 6/7] Stop every canary leg building the dashboard it says it skips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The build step announces "dist/index.js + dist/cli.mjs — no dashboard" and then builds exactly those two. But the `bun install --frozen-lockfile` above it fires the package `prepare` hook, which is `bun run build` — the FULL build, ending in `bun --bun next build`. So each leg compiled the entire Next.js application first, then built the two artifacts it actually wanted. Found by running the box end to end rather than reading it: the run log shows `Creating an optimized production build` and `Generating static pages (3/3)` underneath a step whose own text says it does not do that. `translate-docs.yml` already carries this guard, with the same reasoning written next to it — the trap is the hook, and every entry point that installs for tooling has to opt out of it individually. Costs two full Next builds a day here (stable + beta), on a box whose entire reason for existing is that runner time was too expensive to keep buying. Co-Authored-By: Claude Opus 5 --- integration-suite/ci-entrypoint.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/integration-suite/ci-entrypoint.sh b/integration-suite/ci-entrypoint.sh index 38942152..19ba41d9 100755 --- a/integration-suite/ci-entrypoint.sh +++ b/integration-suite/ci-entrypoint.sh @@ -99,7 +99,13 @@ else step "building failproofai under test (dist/index.js + dist/cli.mjs — no dashboard)" ( cd "$REPO" || exit 1 - bun install --frozen-lockfile || exit 1 + # --ignore-scripts, or the package `prepare` hook runs `bun run build` — + # the FULL build, Next.js dashboard and all — before the two narrow builds + # below, on every leg of every run. The step above says "no dashboard"; this + # flag is what makes that true. Same guard translate-docs.yml carries, for + # the same reason. Caught by running the box end to end: the log showed + # `next build` compiling 3 static pages under a step that claims not to. + bun install --frozen-lockfile --ignore-scripts || exit 1 bun build --target=node --format=cjs --outfile=dist/index.js src/index.ts || exit 1 bun run build:cli || exit 1 ) || { echo "✗ build failed" >&2; exit 1; } From e8de2272dc32ebce80ccb20e2092406ecab22074 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 12 Aug 2026 19:01:12 +0530 Subject: [PATCH 7/7] Say that the work dir is root-owned before someone finds out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner is root inside the container, so everything it creates under the work dir — the clone, logs/, state/, the cargo cache — is root-owned on the host. Only secrets.env, written by the installer, belongs to the user. That is harmless: the next run is root too, and nothing in the pipeline cares. But the first person to `tail` a log or `rm -rf` the clone gets a permission error with no explanation, on a box they were told needs nothing but Docker and a cron line. Found the same way — by doing it. Documented in both places someone would look, with the sudo form of the command they were about to run. Co-Authored-By: Claude Opus 5 --- integration-suite/README.md | 4 ++++ integration-suite/local/install.sh | 8 ++++++++ 2 files changed, 12 insertions(+) diff --git a/integration-suite/README.md b/integration-suite/README.md index 30a83598..ba914374 100644 --- a/integration-suite/README.md +++ b/integration-suite/README.md @@ -94,6 +94,10 @@ matrix. Everything lands under the work dir: version-gate state in `state/` (instead of the Actions cache — the gate logic is unchanged), run + per-leg logs in `logs/` (pruned after 14 days), the clone, and the daemon build's cargo cache. +All of it except `secrets.env` is written by the container **as root**, so +reading a log or clearing the clone from the host needs `sudo`. Harmless — the +next run is root too — but it is the first thing that surprises anyone poking +at the box by hand. Verdict reports POST to Slack exactly as before; a leg that dies *before* reporting gets a distinct crash-note with the log tail (that's the replacement for GHA's red-job email — cron's own output can go to `/dev/null`). Token diff --git a/integration-suite/local/install.sh b/integration-suite/local/install.sh index cee6047e..812d5062 100755 --- a/integration-suite/local/install.sh +++ b/integration-suite/local/install.sh @@ -186,6 +186,14 @@ say "work dir $WORK" say "logs $WORK/logs/ (pruned after 14 days)" say "state $WORK/state/ (which CLI was last green, at which version)" printf '\n' +# The runner is root inside the container, so everything it creates under the +# work dir is root-owned on the host. Harmless — the next run is root too — but +# it surprises the first person who tries to read a log or delete the clone, so +# say it here rather than let them find out with a permission error. +say "Everything under the work dir except secrets.env is written by the container" +say "as root, so reading a log or clearing the clone needs sudo:" +say " sudo tail -f $WORK/logs/\$(sudo ls -t $WORK/logs | head -1)" +printf '\n' say "Run one now, in the foreground:" say " $CMD" printf '\n'