Skip to content

Canary box: move the daily suite off Actions, and make setting it up one command #686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chhhee10 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Canary box: move the daily suite off Actions, and make setting it up one command #686

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f36d9ac
canary: move the daily integration suite to a local box and probe the…
chhhee10 Aug 7, 2026
2eb2d64
canary: one container, one cron line — repackage the box runner for z…
chhhee10 Aug 7, 2026
46a5145
canary: port the fail-closed leg and its live-test lessons from the d…
chhhee10 Aug 7, 2026
8838fff
chore: fill in the canary PR number in the changelog (#656)
chhhee10 Aug 7, 2026
913a958
Make the canary box a one-command install
chhhee10 Aug 12, 2026
43d9b82
Stop every canary leg building the dashboard it says it skips
chhhee10 Aug 12, 2026
e8de227
Say that the work dir is root-owned before someone finds out
chhhee10 Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 17 additions & 11 deletions .github/workflows/integration-suite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,32 @@
name: Integration Suite

# Daily integration test: does failproofai still ENFORCE against every supported
# agent CLI @latest? Installs all 12 CLIs into an isolated Docker sandbox, drives
# each one against failproofai's OWN policies (built from THIS repo's HEAD), and
# asserts the hook log shows a DENY. A silent-allow — a blocked action that ran
# with no deny — means enforcement broke against that CLI (e.g. a vendor changed
# their hook schema out from under us), and turns the run red. Reports only
# CHANGES (broke/recovered) plus a daily heartbeat to Slack.
# ON-DEMAND FALLBACK for the integration suite: does failproofai still ENFORCE
# against every supported agent CLI @latest? Installs all 12 CLIs into an
# isolated Docker sandbox, drives each one against failproofai's OWN policies
# (built from THIS repo's HEAD), and asserts the hook log shows a DENY. A
# silent-allow — a blocked action that ran with no deny — means enforcement
# broke against that CLI (e.g. a vendor changed their hook schema out from
# under us), and turns the run red. Reports only CHANGES (broke/recovered)
# plus a heartbeat to Slack.
#
# The DAILY runs moved off Actions to a local canary box for cost —
# integration-suite/local/ carries the systemd timer + wrapper that replaced
# the cron that used to live here (same 06:17 UTC slot). This workflow stays
Comment on lines +12 to +14

@coderabbitai coderabbitai Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Documentation still describes the replaced systemd-timer design. This PR schedules the daily canary with a single crontab line written by integration-suite/local/install.sh, and integration-suite/local/ contains no systemd unit and no wrapper script. Two documents still name systemd as the box's scheduler.

  • .github/workflows/integration-suite.yml#L12-L14: replace "carries the systemd timer + wrapper that replaced the cron that used to live here" with a description of the cron line plus the runner image, keeping the 06:17 UTC slot.
  • integration-suite/README.md#L7-L8: replace "a systemd user timer on the canary box" with "one cron line on the canary box", which agrees with line 35 of the same file.
📍 Affects 2 files
  • .github/workflows/integration-suite.yml#L12-L14 (this comment)
  • integration-suite/README.md#L7-L8
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/integration-suite.yml around lines 12 - 14, Update the
documentation to describe the daily canary as cron-based rather than
systemd-based: in .github/workflows/integration-suite.yml lines 12-14, mention
the single cron line, runner image, and 06:17 UTC slot; in
integration-suite/README.md lines 7-8, replace the systemd user timer wording
with one cron line on the canary box.

# dispatch-only: the cloud escape hatch for when the box is down or a clean
# cloud reproduction is wanted. Its Actions-cache state is separate from the
# box's state dir, so a dispatch may re-probe CLIs the box already gated.
#
# Unlike the unit/e2e suites, this drives REAL vendor CLIs against real gateway
# models, so it needs credentials. They live in the `cli-integration` Environment
# and the workflow runs ONLY on schedule / manual dispatch — never on pull_request
# — so fork PRs can never reach the secrets.
# and the workflow runs ONLY on manual dispatch — never on pull_request — so
# fork PRs can never reach the secrets.
#
# This file is a THIN TRIGGER on purpose. Everything beyond the GitHub-specific
# wiring (checkout, bun, cache, secret->env mapping) lives in
# integration-suite/ci-entrypoint.sh, so the harness is readable — and runnable —
# without opening this YAML. See integration-suite/README.md.

on:
schedule:
- cron: "17 6 * * *" # ~06:17 UTC daily
workflow_dispatch:
inputs:
clis:
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Changelog

## 1.0.1-beta.0 — 2026-08-12

### Fixes

- Make the canary box a one-command install. Setting it up was four commands, and three of them fail SILENTLY for a day — the wrong property for the thing whose whole job is noticing silent failures. A work dir mounted at a different path inside the container than out leaves the sibling-container `-v` sources resolving against the host to nothing; a `CANARY_REF` left at the shipped `origin/failproofaid` points the box at a branch that merged in #632, so it would test a frozen tree forever and never say so; and a filled-in env file with no Slack webhook produces a run that works perfectly and reports nowhere, which is worse than no canary because it looks like coverage. `integration-suite/local/install.sh` refuses each at install time, in front of a person, rather than at 06:17 tomorrow in front of nobody — the webhook is required for that reason, not because the run needs it. It builds the runner image straight from the git URL (Docker takes `<repo>#<ref>:<subdir>` as a build context) so the box never clones, installs the env file at mode 600, and REWRITES rather than appends its cron line — it carries a `# failproofai-canary` marker and strips any previous line first, so re-running upgrades the schedule instead of scheduling a second job. The stale `CANARY_REF` default is corrected in `secrets.env.example` too, leaving the installer's check as a backstop rather than the only thing between a wrong default and a year of green runs against a dead ref. `--dry-run` distinguishes what it CHECKED (the preflight really runs; it keeps its ✓) from what it would CHANGE, because a script reporting success for work it did not do is the same defect class this canary exists to find. (#686)

## 1.0.0 — 2026-08-12

The first stable release. Everything below this heading shipped across the
Expand Down Expand Up @@ -212,6 +218,9 @@ never "blocked".

## 1.0.0-beta.11 — 2026-08-07

### Features
- Move the daily CLI integration suite off GH Actions onto a local canary box whose entire contract is Docker + one cron line + one env file, and make it probe the daemon path. `integration-suite/local/` ships a self-contained runner image (`Dockerfile.runner`) that drives the *host's* Docker through the mounted socket — sibling containers, with the work dir mounted at an identical path inside and out so the harness's `-v` sources resolve on both sides — whose baked entrypoint stays deliberately thin: lock, clone/fetch `CANARY_REF`, then hand off to `runner-daily.sh` *from the checkout*, so harness changes reach the box through git with no image rebuild. A leg that dies *before* posting its report gets a Slack crash-note with the log tail (the replacement for GHA's red-job email); the workflow keeps `workflow_dispatch` as the cloud fallback and loses its cron, which was the entire Actions cost. On the box the stable leg runs `CANARY_DAEMON=1`: the harness cross-compiles `failproofaid` in a `rust:1-bookworm` container (glibc-matched to the sandbox), sets the `daemon.configured` fail-closed marker through the real `updateConfig` path, and restarts the daemon per probe — the wire protocol carries no env, so the warm worker's oracle log dir is fixed at daemon start, and sharing one dir across probes would let probe A's incidental read-denies false-PASS probe B. A dead daemon cannot false-PASS either: its fail-closed deny is shaped by the synthetic `failproofai/daemon-unreachable` policy, which the probes' greps never match. `CANARY_DAEMON_DEAD=1` adds the complementary fail-closed leg — daemon-configured, daemon deliberately never started, every CLI must deny — which live-testing against 10 real CLIs proved out (all denied; factory and antigravity retry-stormed the deny for the full 10-minute timeout, an availability finding now kept visible by this leg), with its results kept in a separate state lane so a "denied while dead" PASS can never gate-skip a real enforcement probe; the marker is set only after `wire()`, whose vendor onboarding fires hooks that a marker-without-daemon would fail-close. All pinned, along with the marker hygiene, the env-file↔workflow secret parity, and the workflow staying cron-free, in `__tests__/integration-suite/local-runner.test.ts`. (#656)

### Fixes
- Stop `handler.test.ts` reading the developer's own machine. It set no `FAILPROOFAI_HOME`, and `handler.ts` resolves cloud-managed policies from disk — so once cloud policy started working, anyone with a real deployment saw the suite fail with their own artifacts as the unexpected argument (`["/home/…/cloud-policies/generations/4/block-curl-simple.mjs"]` where the assertion wanted `undefined`). Nothing was broken; the test was reading their laptop. That is worse than flakiness: CI is green, so the red is only ever seen locally, by exactly the people who most need to trust the suite. Each test now runs against a throwaway home, and the variable is restored rather than deleted so one test cannot hand the real home to the next. (#PR)
- Make the Rust daemon enforce the same cloud-URL rule the TS side does. `CloudClient::new()` checked only that the scheme was `http` or `https`, so `http://internal-host` was accepted and `spawn_maintenance()` then put the org-scoped `policies:pull` bearer token on the wire **in clear, every 30 seconds**. `validateCloudUrl()` in `cloud-enrollment.ts` has always blocked non-loopback `http`, and `configure-wizard.ts` carries a comment asserting the daemon enforces the same rule — it did not. It matters most on the path the TS validator cannot cover: `FAILPROOFAI_CLOUD_URL` takes precedence over the credentials file and is a documented CI/container knob, so it reaches the constructor without passing through the wizard. (#PR)
Expand Down
Loading