diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..87c0f6d --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,148 @@ +name: CI + +# Backlog §D1: gate every pull request. Jobs reuse the same scripts a +# developer runs locally, exactly like release.yml does, so CI never becomes a +# separate build path that can drift from the documented workflow. + +on: + pull_request: + push: + branches: [main] + schedule: + # Nightly slot for the macOS E2E suite, which needs a GUI session and + # mutates com.headless.app user defaults. + - cron: "0 6 * * *" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ci-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + static: + name: Static checks + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Install zsh + run: sudo apt-get update && sudo apt-get install -y --no-install-recommends zsh + - name: Shell syntax + run: | + set -eu + status=0 + for script in $(git ls-files '*.sh'); do + case "$(head -n 1 "$script")" in + *zsh*) checker="zsh -n" ;; + *bash*) checker="bash -n" ;; + *) checker="sh -n" ;; + esac + if ! $checker "$script"; then + echo "syntax error: $script" >&2 + status=1 + fi + done + exit "$status" + - name: QA evidence checksums + # docs/qa/evidence is the only binary media allowed in git; prove the + # committed bundle still matches its manifest. + run: cd docs/qa/evidence && sha256sum -c SHA256SUMS + - name: Whitespace + if: github.event_name == 'pull_request' + run: git diff --check "origin/${{ github.base_ref }}...HEAD" + + runtime: + name: Agent runtime suite + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: 20 + cache: pnpm + - run: pnpm install --frozen-lockfile --filter @headless/app + - name: Context pruning suite + run: pnpm test:runtime + + protocol: + name: Protocol suite + runs-on: ubuntu-latest + timeout-minutes: 30 + container: swift:6.1-bookworm + steps: + - uses: actions/checkout@v4 + - name: Prepare SwiftPM exclude paths + # Package.swift excludes build/ and Headless.app from the host target; + # Dockerfile.linux creates them for the same reason. + run: mkdir -p apps/headless/build apps/headless/Headless.app + - name: Protocol and security tests + run: ./apps/headless/test.sh + + web: + name: Web app + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: 20 + cache: pnpm + - run: pnpm install --frozen-lockfile --filter @headless/web + - run: pnpm --filter @headless/web lint + - run: pnpm --filter @headless/web build + + linux-e2e: + name: Linux E2E (Docker) + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - name: Sandboxed Chromium E2E + run: ./apps/headless/Tests/linux-docker.sh + - name: Upload QA evidence + if: always() + uses: actions/upload-artifact@v4 + with: + name: linux-e2e-evidence + path: apps/headless/build/qa-evidence + if-no-files-found: ignore + retention-days: 7 + + macos: + name: macOS build and protocol suite + runs-on: macos-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - name: Build app + run: ./apps/headless/build.sh + - name: Protocol and security tests + run: ./apps/headless/test.sh + + macos-e2e: + name: macOS E2E (WKWebView) + # Opens real windows and mutates com.headless.app user defaults, so it runs + # nightly, on demand, or on a pull request labelled macos-e2e. + if: >- + github.event_name == 'schedule' || + github.event_name == 'workflow_dispatch' || + contains(github.event.pull_request.labels.*.name, 'macos-e2e') + runs-on: macos-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + - name: Install FFmpeg + run: brew install ffmpeg + - name: Build app + run: ./apps/headless/build.sh + - name: E2E + run: zsh ./apps/headless/Tests/macos-e2e.sh diff --git a/AGENTS.md b/AGENTS.md index c02f229..ba92eda 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -53,8 +53,14 @@ pnpm build:linux # Linux binaries via Docker (no local Swift needed) Minimum bar for a PR touching `apps/headless`: `pnpm test` and `pnpm test:runtime` pass; run the Linux E2E if you changed host, transport, -runtime JS, or artifact code. There is currently no PR CI (backlog §D1) — do -not rely on the machine to catch you. +runtime JS, or artifact code. + +`.github/workflows/ci.yml` gates every PR with the same scripts: static checks +(shell syntax, QA evidence checksums), the agent-runtime suite, the protocol +suite on Linux (Swift container) and macOS, the web lint/build, and the Linux +Docker E2E. The macOS WKWebView E2E is heavier — it runs nightly, on +`workflow_dispatch`, or when a PR carries the `macos-e2e` label. Run it that +way before merging changes to `main.swift`, `Host/`, or capture code. ## Hard rules (host-enforced contracts — never weaken) diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 30251f7..35859a7 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -103,8 +103,9 @@ Honest snapshot, so newcomers know what is real: command reference, and a Docker sandbox wrapper. **Not yet real** -- No CI on pull requests or `main` — only tag pushes. Merged PRs #4–#8 had no - automated gate. +- ~~No CI on pull requests or `main`~~ — PR CI landed (`.github/workflows/ci.yml`, + backlog §D1/§D3). Correctness fixes in §A are still outstanding, and the + macOS E2E is nightly/label-gated rather than a per-PR gate. - No signing/notarization, no Homebrew/apt/npm, no published Docker image, no checksums on release assets. "Install" means "download from GitHub Releases and click through a Gatekeeper warning". @@ -186,8 +187,8 @@ root. The project cannot absorb contributors or agents while merges are ungated and known races exist. -- PR/`main` CI: protocol tests, `agent-runtime.test.mjs`, Linux Docker E2E on - every PR; macOS E2E on a schedule or label (backlog §D). +- ~~PR/`main` CI: protocol tests, `agent-runtime.test.mjs`, Linux Docker E2E on + every PR; macOS E2E on a schedule or label (backlog §D1, §D3).~~ **Done.** - Fix the correctness bugs that can crash or corrupt a running host: the shutdown data race on `LinuxBrowserHost`, force-unwraps in `visual compare`, the oversized `qa report`/`artifact.list` responses vs the 1 MiB frame, diff --git a/docs/roadmap/improvements-backlog.md b/docs/roadmap/improvements-backlog.md index a549203..df6cd1e 100644 --- a/docs/roadmap/improvements-backlog.md +++ b/docs/roadmap/improvements-backlog.md @@ -205,13 +205,20 @@ response-body inspection stays denied (P1.md:134) unless a gated design lands. ## §D — CI & testing (Phase 1) -**D1. PR/`main` CI is absent** — only tag-triggered release exists -(`.github/workflows/release.yml`; PRs #4–#8 merged ungated). Add a `ci.yml`: -protocol suite + `pnpm test:runtime` + `linux-docker.sh` on every PR (Linux -runners); macOS build + `test.sh` on PR, full `macos-e2e.sh` nightly/label -(it mutates user defaults and needs a GUI session — see -`Tests/macos-e2e.sh:26-40`). Reuse the same scripts as release (keep that -property). +**D1. PR/`main` CI** — ~~absent; only tag-triggered release existed +(`.github/workflows/release.yml`; PRs #4–#8 merged ungated).~~ **Done:** +`.github/workflows/ci.yml` runs on every PR and `main` push — static checks +(shell syntax across sh/bash/zsh, `docs/qa/evidence` checksum verification, +whitespace), `pnpm test:runtime`, the protocol suite on Linux +(`swift:6.1-bookworm` container) and macOS, and `linux-docker.sh` with the QA +evidence uploaded as an artifact. `macos-e2e.sh` is gated to nightly cron, +`workflow_dispatch`, or the `macos-e2e` PR label because it needs a GUI +session and mutates user defaults (`Tests/macos-e2e.sh:26-40`). All jobs +invoke the same scripts a developer runs locally, matching release.yml. + +Follow-ups: mark the jobs required in branch protection (repo setting, not +code); consider arm64 Linux E2E on PRs (release covers it on tags); revisit +whether the macOS E2E can become a per-PR gate once runtime is measured. **D2. Known test gaps (from code audit).** No tests for: peer-UID rejection (A9), MCP (C3), `Recording.swift` (ffmpeg args per format/quality, discovery @@ -228,7 +235,9 @@ flow start-stop/network emulate/mock clear/session ops/status/stop/start/help). Also tighten `rejectsUnexpectedRequestFields` (`ProtocolTests.swift:109-114`) which currently passes for the wrong reason. -**D3. Web CI:** `next build` + eslint on PR (site can break invisibly today). +**D3. Web CI:** ~~`next build` + eslint on PR (site can break invisibly today).~~ +**Done** — the `web` job in `ci.yml` runs `pnpm --filter @headless/web lint` +and `build`. **D4. Cross-engine conformance runner** (architecture §14) replacing drifted hand-mirrored E2E assertions (e.g. `macos-e2e.sh:231` vs `linux-e2e.sh:157`).