CI: retry EC2 runner start with backoff on capacity failures

[harrism]

Problem

Multi-AZ configuration makes the runner action walk every availability zone before giving up — but that walk takes seconds. When a whole region is short of an instance type, every zone fails in one burst and the job dies:

Attempting to start EC2 instance using 3 availability zone configuration(s)
Trying availability zone configuration 1/3 ... 2/3 ... 3/3
##[error]All availability zone configurations failed

g6.xlarge is currently unavailable across all three us-east-2 zones, so multi-AZ alone doesn't help. Capacity typically frees within minutes.

The action has no launch retry of its own — in either v2.4.3 or v2.6.1, startup-retry-interval-seconds only governs runner registration after an instance already exists.

Change

Each start step becomes three attempts with 90s then 180s backoff:

• each attempt is continue-on-error: true, so a failure falls through to the next
• each attempt still walks every configured AZ
• a final step fails the job only if all three attempts found no capacity
• jobs exposing the runner label as a job output select it with || across the three attempt ids; jobs passing an explicit label: input need no change

Why inline rather than a shared reusable workflow

I first built this as a reusable workflow (much DRYer), then found fvdb-core's check_runner_token_policy.py forbids exactly that:

Rule 2: "…already forbids putting the token in … a reusable-workflow secrets: block"
Rule 4: "A job that references the token must not … no local actions (uses: ./...)"

Calling a local reusable workflow is uses: ./... and passes the token via secrets: — two violations, and the policy anticipates the pattern by name. Inline expansion keeps every occurrence as the exact github-token: ${{ secrets.<TOKEN> }} action input. The same shape is used in both repos for consistency, even though only fvdb-core enforces the policy.

Testing

CI provisioning can't be fully proven without a capacity failure to retry against, but everything statically checkable was verified:

• all workflows parse; every start job has exactly 3 attempts, all continue-on-error
• every attempt carries the exact policy-required token line
• label propagation intact via ||-joined outputs or an explicit label: input
• mode: stop steps untouched
• fvdb-core: check_runner_token_policy.py passes (✅ EC2 runner token policy: OK)

Note the failing GPU checks on this PR are the capacity outage itself, not a defect here; and for pull_request_target workflows these changes only take effect after merge.

🤖 Generated with Claude Code

[harrism]

Superseded. The inline retry ladder was ~30 lines of near-identical YAML repeated at nine call sites, which can drift independently — so it is now a reusable workflow instead:

• CI: let the runner token be forwarded to a local reusable workflow #759 — amends the runner-token policy so the token may be forwarded by name to a local reusable workflow, and rejects secrets: inherit (which the current checker lets through silently). Strictly tighter than today's rule on its own merits.
• CI: provision EC2 runners through shared workflows, with retry and backoff #760 — the actual change: start-ec2-runner.yml / stop-ec2-runner.yml, called from all nine start and nine stop jobs. 401 lines removed, 171 added. Draft until CI: let the runner token be forwarded to a local reusable workflow #759 merges, because workflow-security.yml runs the policy script from the base branch.

Retry semantics are unchanged from this PR: three attempts, 90s then 180s backoff.