ci: make fork-PR benchmarks work via trusted workflow_run#113
Merged
Conversation
Fork PRs (e.g. #103) fail benchmark CI because they cannot access the private mosaic/* GHCR packages: the read-only fork token is denied `packages: write` (build push fails) and cannot read the private package either (every image pull returns `manifest unknown`). Both the changed solver's head-SHA image and the unchanged solvers' base-SHA images are unreachable, so the benchmark run jobs fail with "no image found". Split benchmark CI along a trust boundary, mirroring the existing docs-publish workflow_run pattern: - benchmark.yml (on pull_request, fork context, read-only, no secrets) now only plans and hands off. `plan-gate` (renamed from the old `bench-ok` merge-gate job) enforces the label policy and seeds the `bench-ok` commit status: failure on a policy violation, immediate success when nothing will run, else pending. Seeds are best-effort so a fork token that can't write a status doesn't cascade. A new `handoff` job uploads the validated plan outputs as an artifact. - benchmark-execute.yml (new, on workflow_run of "Benchmark", trusted base-repo context) recovers + validates the handoff, then builds and pushes the changed solver image to private GHCR, runs the benchmarks (pulling the private images), renders the docs preview, and posts the authoritative `bench-ok` commit status plus the PR comment / RTD preview. Its `finalize` job always resolves a seeded pending — posting `error` if the handoff can't be recovered on a successful planner run — so the gate can never deadlock in `pending`. The status step is gated on the planner having concluded success, so it never clobbers a correct terminal status set by plan-gate. - build-tesseracts.yml gains `ref`/`image_sha` inputs so the trusted run builds the PR's HEAD and tags the image to match what the benchmark run pulls; the fork-tolerant `continue-on-error` push is removed (both remaining callers hold packages:write). - docs-publish.yml is retired: workflow_run cannot chain, so its RTD + PR-comment steps fold into benchmark-execute's finalize job. The reusable build/benchmark calls drop `secrets: inherit`: the jobs that build and run fork-authored solver code see only the ephemeral GITHUB_TOKEN, keeping RTD_TOKEN / the bot PAT out of reach. Branch protection keeps requiring the `bench-ok` context unchanged — it is now a commit status rather than a job, posted only by the two paths above, so a PR cannot merge until real benchmarks pass. workflow_run changes only take effect once on main, so the fork path must be verified by re-running a fork PR after merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork PRs (e.g. #103) fail benchmark CI because they cannot access the private mosaic/* GHCR packages: the read-only fork token is denied
packages: write(build push fails) and cannot read the private package either (every image pull returnsmanifest unknown). Both the changed solver's head-SHA image and the unchanged solvers' base-SHA images are unreachable, so the benchmark run jobs fail with "no image found".Split benchmark CI along a trust boundary, mirroring the existing docs-publish workflow_run pattern:
benchmark.yml (on pull_request, fork context, read-only, no secrets) now only plans and hands off.
plan-gate(renamed from the oldbench-okmerge-gate job) enforces the label policy and seeds thebench-okcommit status: failure on a policy violation, immediate success when nothing will run, else pending. Seeds are best-effort so a fork token that can't write a status doesn't cascade. A newhandoffjob uploads the validated plan outputs as an artifact.benchmark-execute.yml (new, on workflow_run of "Benchmark", trusted base-repo context) recovers + validates the handoff, then builds and pushes the changed solver image to private GHCR, runs the benchmarks (pulling the private images), renders the docs preview, and posts the authoritative
bench-okcommit status plus the PR comment / RTD preview. Itsfinalizejob always resolves a seeded pending — postingerrorif the handoff can't be recovered on a successful planner run — so the gate can never deadlock inpending. The status step is gated on the planner having concluded success, so it never clobbers a correct terminal status set by plan-gate.build-tesseracts.yml gains
ref/image_shainputs so the trusted run builds the PR's HEAD and tags the image to match what the benchmark run pulls; the fork-tolerantcontinue-on-errorpush is removed (both remaining callers hold packages:write).docs-publish.yml is retired: workflow_run cannot chain, so its RTD + PR-comment steps fold into benchmark-execute's finalize job.
The reusable build/benchmark calls drop
secrets: inherit: the jobs that build and run fork-authored solver code see only the ephemeral GITHUB_TOKEN, keeping RTD_TOKEN / the bot PAT out of reach.Branch protection keeps requiring the
bench-okcontext unchanged — it is now a commit status rather than a job, posted only by the two paths above, so a PR cannot merge until real benchmarks pass. workflow_run changes only take effect once on main, so the fork path must be verified by re-running a fork PR after merge.