CI: provision EC2 runners through shared workflows, with retry and backoff

[harrism]

Replaces #758. CI keeps failing because us-east-2 has no capacity for the requested instance type at the moment the runner is requested. machulav/ec2-github-runner sweeps the availability zones in availability-zones-config within one attempt, but it never retries the sweep, so a transient region-wide shortage fails the whole run. Retrying twice, 90s and then 180s later, turns most of those into a slow pass instead of a red build.

#758 inlined that retry ladder at every call site. That is ~30 lines of near-identical YAML in nine places that can drift independently, which is why this replaces it: the ladder goes in a reusable workflow instead.

What changed

Two new local reusable workflows hold the retry policy, the pinned action SHA, the AWS OIDC step and the teardown:

• .github/workflows/start-ec2-runner.yml
• .github/workflows/stop-ec2-runner.yml

tests.yml, cu130.yml, cu132.yml, publish.yml and nightly-publish.yml now call them — nine start jobs and nine stop jobs. 401 lines of duplicated YAML removed, 171 added.

The reusable workflows absorb both call shapes the repo already had:

• runner-label / aws-resource-tags for the matrix fan-outs in publish.yml and nightly-publish.yml, which need a predictable label because they find the instance again by tag at teardown.
• stagger-seconds, replacing the hand-rolled RANDOM % 16 sleep those same matrix jobs used to spread out runner registrations.

stop-ec2-runner.yml takes the instance id directly when the caller has one from a start job's output, and otherwise resolves it from the RunnerLabel tag — the lookup the publish workflows had to do inline, because a matrix job has no single output to read.

Two details worth a reviewer's eye

• aws-resource-tags defaults to '[]', not ''. The pinned action does a bare JSON.parse() on that input (src/config.js) with no fallback, so an empty string throws and would break every caller that does not set tags. An empty label, by contrast, is explicitly fine — the action generates a unique one.
• Retry semantics are unchanged from CI: retry EC2 runner start with backoff on capacity failures #758: only one attempt can succeed, since each is skipped unless every earlier one failed, so at most one of the three step outputs is non-empty and || picks it.

Testing

• check_runner_token_policy.py (with CI: let the runner token be forwarded to a local reusable workflow #759 applied): OK on all 17 workflow files.
• Policy unit tests: 26 passed.
• actionlint 1.7.7: clean. zizmor 1.26.1 --persona=regular: no findings.
• Parsed every reusable-workflow caller job to confirm none carries a key incompatible with uses: (runs-on, steps, env, container, …).

Because CI runs on pull_request_target from the base branch, this change cannot exercise itself on its own PR — the first real run of the new provisioning path is the one after merge.

🤖 Generated with Claude Code

Update: retry-path fix folded in

#759 has merged, so the Workflow Security gate now passes here.

While confirming the ladder had not yet fired in production (it hasn't — every start since the merge succeeded on attempt 1), I audited the never-executed path and found a defect, now fixed in c90f2d7:

The action publishes its outputs as soon as the instance launches, before waiting for the runner to register. So a "no capacity" failure leaves outputs empty, but a registration failure leaves them populated. Selecting outputs with a1 || a2 || a3 takes the first non-empty value rather than the successful one — which in that second case hands back a dead label (dependent job queues until timeout instead of failing fast) and the wrong instance id (the retry's instance leaks).

Fixed by selecting on outcome, and by stopping the instance a failed attempt left running before retrying. Verified against all five outcome combinations. Same fix for frc is openvdb/fvdb-reality-capture#330.

Not a regression from this refactor — the inline version in #758 had the same || chain. The refactor is why the fix is one file rather than nine.