CI: reuse acquired GPU runners instead of demanding three per PR event

[harrism]

Problem

PR CI goes red when EC2 refuses g6.xlarge in us-east-2. Every PR event fires three
workflows — tests.yml, cu130.yml, cu132.yml — and each demands its own GPU runner
within seconds of the others. When the region can't serve three at once, the losers fail
the whole workflow.

Data (54 GPU-runner start jobs, both repos, 2026-08-27 .. 09-03)

39 succeeded, 14 failed. Every failure was InsufficientInstanceCapacity; none had
another cause. Splitting the failures by whether capacity existed nearby:

	count
a sibling job on the same commit succeeded within ±5 min	6/14
no GPU start succeeded anywhere within ±5 min	6/14
other (success nearby, different commit)	2/14

So roughly half self-inflicted contention, half genuine scarcity. Two supporting facts:

• We race ourselves. The self-contention cases are three adjacent run IDs from one
  event. Example, commit 8dfa68e: Conda succeeded 04:32:01, cu130 failed 04:33:12,
  cu132 succeeded 04:33:17 — capacity existed 5 s after the refusal.
• Scarcity is real too. All six no-capacity-anywhere cases are fvdb-reality-capture,
  mostly 09-01/09-02 while it still requested us-east-2b only. Whole-region (all three AZ)
  exhaustion was observed 6 times: 2× on 08-31, 4× on 09-02. 09-03 was clean.

Already landed: multi-AZ fallback (#705, openvdb/fvdb-reality-capture#326) and
retry-with-backoff (#760, openvdb/fvdb-reality-capture#329). Neither reduces peak demand.

Suggested approach: reuse the runners we did get

Merge the three workflows into one run, attempt three starts with continue-on-error,
then assign the three test tasks across whichever labels actually came up:

  start:                       # 3 attempts, failures tolerated
    strategy: { fail-fast: false, matrix: { n: [1, 2, 3] } }
    uses: ./.github/workflows/start-ec2-runner.yml

  plan:                        # round-robin tasks over the labels that exist
    needs: start               # e.g. [{task: conda, label: L1},
                               #       {task: cu130, label: L1},
                               #       {task: cu132, label: L2}]
  test:
    needs: plan
    strategy: { fail-fast: false, matrix: ${{ fromJSON(needs.plan.outputs.matrix) }} }
    runs-on: ${{ matrix.label }}

No scheduling code is needed: a self-hosted runner takes one job at a time, so two matrix
entries sharing a label serialize automatically. 3 runners → full parallelism; 2 → one
pair serializes; 1 → all three serialize; 0 → fail, as today. Instances remain ephemeral
and per-event, so the pull_request_target isolation properties are unchanged.

This must be one workflow rather than three sharing a label: independent runs can share
a label, but none of them knows when it is safe to terminate, and coordinating teardown
across runs needs an external reaper. Inside one run, needs: expresses it exactly.

Note cu130.yml and cu132.yml are byte-identical modulo the CUDA version string (zero
diff after normalising 13.0↔13.2), so folding them into a matrix is worth doing on its
own merits.

Cost: trades wall clock for reliability. Under pressure a PR's GPU phase takes up to 3×
longer instead of going red.

Not worth pursuing: a capacity reservation

Measured GPU usage over 6.7 days (37 instance lifetimes, both repos):

total g6.xlarge instance-hours	13.4
average per day	2.0 h
median instance lifetime	18 min
time with zero GPU instances running	96%
peak concurrency	6, momentarily

An On-Demand Capacity Reservation bills the on-demand rate 24/7 whether or not anything
runs in it. Against 2.0 h/day that is 8.4% utilisation for a single reserved instance
(which would not even cover peak) and 1.4% if we reserved six. At roughly $0.80/hr for
g6.xlarge — verify before quoting — we currently spend on the order of $50/month on GPU
runners; one 24/7 reservation would be ~$590/month. Not proportionate to ~14 red jobs in a
week. Capacity Blocks are prepaid windows aimed at large training clusters and may not even
list g6 as eligible.

Cheaper lever: allow g5.xlarge as a fallback instance type

Doubling the pool we can draw from costs nothing and needs no AWS negotiation. Nothing in
core or frc requires SM 8.9:

• the CUTLASS convolution backend targets arch::Sm80 (PredGatherIGemm.cu);
• the only capability gate is __CUDA_ARCH__ >= 800 (AtomicAdd.cuh);
• Flash Attention needs >= 8.0.

g5.xlarge is SM 8.6 and satisfies all of that. The blocker is the PR build architecture.

The status quo cannot run on g5 at all

arch_list_pr is "8.9+PTX". On an SM 8.6 device that has no compatible cubin (binary
compatibility runs forward across minor revisions, not backward) and its embedded PTX is
compute_89, which cannot be JIT-compiled down to sm_86. It fails outright — this is not
a slow-path case.

Note also that no PTX JIT is involved in either option below. JIT only happens when no
compatible cubin exists; an sm_86 cubin loads natively on sm_89.

Build cost per architecture (measured from CI, successful fVDB Build jobs)

arch list	n	mean	min
8.9+PTX (1 arch, tests.yml)	12	769 s	673 s
7.5;8.0;8.6;9.0;10.0;12.0+PTX (6 archs, publish.yml)	106	1365 s	835 s

Six architectures cost +78%, not 6x, since config, host compilation, dependency fetch and
link are shared — roughly +120 s per additional architecture. (The two build in
different environments, conda vs the wheel Docker image, so treat this as order-of-magnitude.)

Two options

A. arch_list_pr: "8.6+PTX" — one architecture, no build-time change, loads natively on
both g5 and g6. Also increases fidelity with what we ship: arch_list_publish contains no
8.9, so users on Ada already run the 8.6 cubin, and publish validation exercises exactly
that path on g6.xlarge. PR CI currently tests an 8.9-native binary no user receives.

B. arch_list_pr: "8.6;8.9+PTX" — exact-match cubin on both, at roughly +2 min on a
~13 min build (+15%), about $0.15 of m6a.8xlarge time per push.

The case for B is Ada-native codegen for performance-sensitive tests; the case for A is
that PR CI is correctness testing and matching the shipped artifact is worth more.
Unmeasured: whether any test is meaningfully slower under the 8.6 cubin on Ada. If that
matters to the decision it is worth measuring before choosing.

Either way, add g5.xlarge as a second instance type in the retry ladder, so an attempt
sweeps g6 across three AZs and then g5 across three before giving up.