Skip to content

ci: make fork-PR benchmarks work via trusted workflow_run#113

Merged
dionhaefner merged 1 commit into
mainfrom
dion/fork-friendly-benchmarks
Jul 21, 2026
Merged

ci: make fork-PR benchmarks work via trusted workflow_run#113
dionhaefner merged 1 commit into
mainfrom
dion/fork-friendly-benchmarks

Conversation

@dionhaefner

Copy link
Copy Markdown
Contributor

Fork PRs (e.g. #103) fail benchmark CI because they cannot access the private mosaic/* GHCR packages: the read-only fork token is denied packages: write (build push fails) and cannot read the private package either (every image pull returns manifest unknown). Both the changed solver's head-SHA image and the unchanged solvers' base-SHA images are unreachable, so the benchmark run jobs fail with "no image found".

Split benchmark CI along a trust boundary, mirroring the existing docs-publish workflow_run pattern:

  • benchmark.yml (on pull_request, fork context, read-only, no secrets) now only plans and hands off. plan-gate (renamed from the old bench-ok merge-gate job) enforces the label policy and seeds the bench-ok commit status: failure on a policy violation, immediate success when nothing will run, else pending. Seeds are best-effort so a fork token that can't write a status doesn't cascade. A new handoff job uploads the validated plan outputs as an artifact.

  • benchmark-execute.yml (new, on workflow_run of "Benchmark", trusted base-repo context) recovers + validates the handoff, then builds and pushes the changed solver image to private GHCR, runs the benchmarks (pulling the private images), renders the docs preview, and posts the authoritative bench-ok commit status plus the PR comment / RTD preview. Its finalize job always resolves a seeded pending — posting error if the handoff can't be recovered on a successful planner run — so the gate can never deadlock in pending. The status step is gated on the planner having concluded success, so it never clobbers a correct terminal status set by plan-gate.

  • build-tesseracts.yml gains ref/image_sha inputs so the trusted run builds the PR's HEAD and tags the image to match what the benchmark run pulls; the fork-tolerant continue-on-error push is removed (both remaining callers hold packages:write).

  • docs-publish.yml is retired: workflow_run cannot chain, so its RTD + PR-comment steps fold into benchmark-execute's finalize job.

The reusable build/benchmark calls drop secrets: inherit: the jobs that build and run fork-authored solver code see only the ephemeral GITHUB_TOKEN, keeping RTD_TOKEN / the bot PAT out of reach.

Branch protection keeps requiring the bench-ok context unchanged — it is now a commit status rather than a job, posted only by the two paths above, so a PR cannot merge until real benchmarks pass. workflow_run changes only take effect once on main, so the fork path must be verified by re-running a fork PR after merge.

Fork PRs (e.g. #103) fail benchmark CI because they cannot access the
private mosaic/* GHCR packages: the read-only fork token is denied
`packages: write` (build push fails) and cannot read the private package
either (every image pull returns `manifest unknown`). Both the changed
solver's head-SHA image and the unchanged solvers' base-SHA images are
unreachable, so the benchmark run jobs fail with "no image found".

Split benchmark CI along a trust boundary, mirroring the existing
docs-publish workflow_run pattern:

- benchmark.yml (on pull_request, fork context, read-only, no secrets)
  now only plans and hands off. `plan-gate` (renamed from the old
  `bench-ok` merge-gate job) enforces the label policy and seeds the
  `bench-ok` commit status: failure on a policy violation, immediate
  success when nothing will run, else pending. Seeds are best-effort so
  a fork token that can't write a status doesn't cascade. A new `handoff`
  job uploads the validated plan outputs as an artifact.

- benchmark-execute.yml (new, on workflow_run of "Benchmark", trusted
  base-repo context) recovers + validates the handoff, then builds and
  pushes the changed solver image to private GHCR, runs the benchmarks
  (pulling the private images), renders the docs preview, and posts the
  authoritative `bench-ok` commit status plus the PR comment / RTD
  preview. Its `finalize` job always resolves a seeded pending — posting
  `error` if the handoff can't be recovered on a successful planner run —
  so the gate can never deadlock in `pending`. The status step is gated
  on the planner having concluded success, so it never clobbers a correct
  terminal status set by plan-gate.

- build-tesseracts.yml gains `ref`/`image_sha` inputs so the trusted run
  builds the PR's HEAD and tags the image to match what the benchmark run
  pulls; the fork-tolerant `continue-on-error` push is removed (both
  remaining callers hold packages:write).

- docs-publish.yml is retired: workflow_run cannot chain, so its RTD +
  PR-comment steps fold into benchmark-execute's finalize job.

The reusable build/benchmark calls drop `secrets: inherit`: the jobs that
build and run fork-authored solver code see only the ephemeral
GITHUB_TOKEN, keeping RTD_TOKEN / the bot PAT out of reach.

Branch protection keeps requiring the `bench-ok` context unchanged — it is
now a commit status rather than a job, posted only by the two paths above,
so a PR cannot merge until real benchmarks pass. workflow_run changes only
take effect once on main, so the fork path must be verified by re-running
a fork PR after merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dionhaefner
dionhaefner merged commit ad4353a into main Jul 21, 2026
12 checks passed
@dionhaefner
dionhaefner deleted the dion/fork-friendly-benchmarks branch July 21, 2026 15:26
@pasteurlabs pasteurlabs locked and limited conversation to collaborators Jul 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant