From 1a3e8109030253df3ae39c539fea5ad4edfae569 Mon Sep 17 00:00:00 2001 From: Mark Harris Date: Thu, 3 Sep 2026 02:06:19 +0000 Subject: [PATCH 1/3] CI: let the runner token be forwarded to a local reusable workflow The runner-token policy required every mention of EC2_RUNNER_TOKEN to be literally `github-token: ${{ secrets.EC2_RUNNER_TOKEN }}`, which forbade a `secrets:` block outright. That made every EC2 start/stop job inline the machulav action, and it is the reason the same nine-line block is copy-pasted across tests.yml, cu130.yml, cu132.yml, publish.yml and nightly-publish.yml. The prohibition was too blunt in one direction and not blunt enough in the other. The rule's purpose (#672) is to keep the token's reachable surface enumerable by grep, so that a PR cannot quietly move it somewhere it can escape and have that land on main. But: * `secrets: inherit` never spells the token's name, so it slipped through the textual scan entirely -- including a forward to a *remote* reusable workflow, which takes the token out of the repo with the check reporting OK; * forwarding to a *local* reusable workflow is safe by the rule's own reasoning. The callee is a file in .github/workflows/, so this same scan covers it and rules 2-4 apply to it directly; under pull_request_target it is base-controlled like every other workflow; and a caller job has no steps and no workspace, so rule 4 ("no untrusted code beside the token") has nothing to bite on. So the checker rejected the careful form and accepted the blanket one. Make the rule structural instead of textual: * allow `EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }}` in a job's `secrets:` block, but only when that job's `uses:` is `./.github/workflows/.yml` and that file exists here; * require the forwarded name to be the token's own, so a single `git grep EC2_RUNNER_TOKEN` still enumerates every file that touches it; * allow the matching `on.workflow_call.secrets` declaration on the callee; * reject `secrets: inherit` anywhere (new rule 6); * check every remaining mention structurally, so shapes that merely look like a forward -- a workflow-level `env:`, a job-level `with:` -- are still rejected. Net effect is strictly tighter than before: it closes the `inherit` hole and the remote-callee hole, and opens only the one shape this scan can actually verify. Signed-off-by: Mark Harris Co-authored-by: Claude Opus 5 (1M context) Signed-off-by: Mark Harris --- .github/scripts/check_runner_token_policy.py | 237 +++++++++++++++--- .../scripts/test_check_runner_token_policy.py | 178 ++++++++++++- 2 files changed, 367 insertions(+), 48 deletions(-) diff --git a/.github/scripts/check_runner_token_policy.py b/.github/scripts/check_runner_token_policy.py index 83333f2bd..ad6a5467a 100755 --- a/.github/scripts/check_runner_token_policy.py +++ b/.github/scripts/check_runner_token_policy.py @@ -9,20 +9,43 @@ tooling under ``.github/scripts/`` (this enforcement script and its tests). It must not leak into product source, docs, etc. - 2. Every textual occurrence in a workflow must be EXACTLY the action input: - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - (whitespace inside ``${{ }}`` is tolerated). This single rule already - forbids putting the token in ``env:``, ``GH_TOKEN``/``GITHUB_TOKEN``, - ``with.token``, a ``run:`` script, or a reusable-workflow ``secrets:`` - block, because none of those match this pattern. - - 3. The step that consumes the token must ``uses: machulav/ec2-github-runner``. - - 4. A job that references the token must not pull untrusted code into its - workspace alongside the privileged context: no local actions + 2. Every textual occurrence in a workflow must be one of exactly three forms + (whitespace inside ``${{ }}`` is tolerated): + + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} # the action input + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} # caller forwarding + EC2_RUNNER_TOKEN: # callee declaration + + Each is a bare interpolation that is the *whole* value of a scalar key, so + this rule alone forbids the token in a ``run:`` script or in any + concatenated string. Which of the three a line is allowed to be is then + settled structurally by rules 3 and 4 -- e.g. an ``env:`` entry happens to + have the same shape as the forwarding form, and is rejected below. + + The forwarded name is required to be ``EC2_RUNNER_TOKEN`` itself, so the + token keeps one name across caller and callee and a single + ``git grep EC2_RUNNER_TOKEN`` still enumerates every file that touches it. + + 3. The token may be consumed in exactly two ways: + + a. by a *step* whose ``uses:`` is ``machulav/ec2-github-runner``, via its + ``github-token`` input; or + b. by a *job* whose ``uses:`` is a local reusable workflow + (``./.github/workflows/.yml``), via a ``secrets:`` mapping. + + (b) is safe precisely because the callee is itself a file in + ``.github/workflows/``: this same scan covers it, so rules 2-4 apply to it + directly, and under ``pull_request_target`` it is base-controlled like + every other workflow. Forwarding to anything else -- a remote reusable + workflow, or a callee that is not present in this directory -- would send + the token somewhere the scan cannot see, and is rejected. + + 4. A job that references the token *in a step* must not pull untrusted code + into its workspace alongside the privileged context: no local actions (``uses: ./...``) and no ``actions/checkout``. (Sibling ``run:`` steps are - fine -- rule 2 guarantees they can never reference the token, since it - only ever appears as the ``github-token`` input.) + fine -- rule 2 guarantees they can never reference the token.) A + reusable-workflow caller job under 3(b) has no ``steps:`` and no workspace + at all, so there is nothing for this rule to bite on. 5. No dynamic secret access (``secrets[...]``) anywhere in a workflow. Rules 1-4 are textual: they key on the literal name ``EC2_RUNNER_TOKEN``. Dynamic @@ -32,6 +55,12 @@ reject it outright (this check runs on every workflow, even ones that never mention the token by name). + 6. No ``secrets: inherit`` anywhere in a workflow. It forwards *every* secret, + including the admin token, without ever spelling a name -- so it is + invisible to rules 1-3 and would let a caller hand the token to a remote + reusable workflow with this scan reporting OK. Forward the one secret + explicitly instead (rule 3b), which is both narrower and checkable. + Usage: check_runner_token_policy.py [WORKFLOW_DIR] [--repo-root DIR] @@ -52,9 +81,25 @@ TOKEN_REF = f"secrets.{TOKEN_NAME}" ALLOWED_ACTION = "machulav/ec2-github-runner" -# The one and only allowed textual form, e.g. -# github-token: ${{ secrets.EC2_RUNNER_TOKEN }} -ALLOWED_LINE = re.compile(r"^\s*github-token:\s*\$\{\{\s*secrets\." + re.escape(TOKEN_NAME) + r"\s*\}\}\s*$") +# Rule 2's three allowed textual forms. Each is a bare interpolation that is the +# whole value of a scalar key, so none of them can hide inside a `run:` script or +# a concatenated string. +_INTERP = r"\$\{\{\s*secrets\." + re.escape(TOKEN_NAME) + r"\s*\}\}" +# github-token: ${{ secrets.EC2_RUNNER_TOKEN }} -- the action input +ACTION_INPUT_LINE = re.compile(r"^\s*github-token:\s*" + _INTERP + r"\s*$") +# EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} -- caller forwarding +FORWARD_LINE = re.compile(r"^\s*" + re.escape(TOKEN_NAME) + r":\s*" + _INTERP + r"\s*$") +# EC2_RUNNER_TOKEN: -- callee declaration +DECLARE_LINE = re.compile(r"^\s*" + re.escape(TOKEN_NAME) + r":\s*$") +ALLOWED_LINES = (ACTION_INPUT_LINE, FORWARD_LINE, DECLARE_LINE) + +# The only `uses:` a job may have if it is forwarded the token (rule 3b): a +# reusable workflow in this repo's own .github/workflows, which this same scan +# therefore covers. The capture group is the callee's file name, checked to exist. +LOCAL_REUSABLE = re.compile(r"^\./\.github/workflows/([^/]+\.ya?ml)$") + +# Rule 6: blanket secret forwarding, which names nothing and so evades rules 1-3. +INHERIT_LINE = re.compile(r"^\s*secrets:\s*inherit\s*$") # Dynamic secret access, e.g. secrets['X'], secrets[matrix.y], secrets[format(...)]. # The \b avoids matching identifiers that merely end in "secrets" (e.g. mysecrets[0]). @@ -85,12 +130,45 @@ def iter_steps(job: dict): yield step +def _on_block(data: dict) -> dict: + """The workflow's trigger mapping. + + YAML 1.1 parses a bare ``on:`` key as the boolean ``True``, so accept both + spellings rather than depending on how the file happens to quote it. + """ + for key in ("on", True): + value = data.get(key) + if isinstance(value, dict): + return value + return {} + + +def _token_secret_declaration(data: dict) -> dict | None: + """The ``on.workflow_call.secrets`` mapping, if it declares the token. + + This is the callee side of rule 3b -- the one legitimate mention of the + token name outside a job. Returns the enclosing mapping so the caller can + remove the declaration before scanning what is left. + """ + call = _on_block(data).get("workflow_call") + if not isinstance(call, dict): + return None + secrets = call.get("secrets") + if isinstance(secrets, dict) and TOKEN_NAME in secrets: + return secrets + return None + + +def _mentions_token(value) -> bool: + return TOKEN_NAME in yaml.safe_dump(value) + + def check_workflow_file(path: Path, violations: list[str]) -> None: text = path.read_text() - # --- Rule 5: no dynamic secret access (runs on every workflow). ----------- - # Must run before the token-name early return below: the whole point is that - # dynamic indexing can reach the token without naming it. + # --- Rules 5 & 6: blanket / unnamed secret access (every workflow). ------- + # These must run before the token-name early return below: the whole point + # of both is that they can reach the token without ever naming it. for lineno, line in enumerate(text.splitlines(), start=1): if DYNAMIC_SECRET.search(line): fail( @@ -101,21 +179,34 @@ def check_workflow_file(path: Path, violations: list[str]) -> None: f"'secrets.', not 'secrets[...]' (it can resolve " f"'{TOKEN_NAME}' without naming it): {line.strip()!r}", ) + if INHERIT_LINE.match(line): + fail( + violations, + path, + None, + f"line {lineno}: 'secrets: inherit' is not allowed -- it forwards " + f"every secret, including '{TOKEN_NAME}', without naming one, so no " + f"textual check can see where the token goes. Forward the single " + f"secret explicitly: '{TOKEN_NAME}: ${{{{ secrets.{TOKEN_NAME} }}}}'.", + ) if TOKEN_REF not in text and TOKEN_NAME not in text: return - # --- Rule 2: every line mentioning the token must be the exact input. ----- + # --- Rule 2: every line naming the token must be one of the three forms. -- for lineno, line in enumerate(text.splitlines(), start=1): if TOKEN_NAME not in line: continue - if not ALLOWED_LINE.match(line): + if not any(rx.match(line) for rx in ALLOWED_LINES): fail( violations, path, None, f"line {lineno}: '{TOKEN_NAME}' may only appear as " - f"'github-token: ${{{{ secrets.{TOKEN_NAME} }}}}', got: {line.strip()!r}", + f"'github-token: ${{{{ secrets.{TOKEN_NAME} }}}}' (action input), " + f"'{TOKEN_NAME}: ${{{{ secrets.{TOKEN_NAME} }}}}' (forward to a local " + f"reusable workflow), or '{TOKEN_NAME}:' (workflow_call declaration), " + f"got: {line.strip()!r}", ) # --- Structural rules 3 & 4 via parsed YAML. ------------------------------ @@ -125,22 +216,73 @@ def check_workflow_file(path: Path, violations: list[str]) -> None: fail(violations, path, None, f"could not parse YAML: {exc}") return + # Outside of `jobs:`, the only legitimate mention is the workflow_call + # declaration; drop it and anything left over is a leak (workflow-level + # `env:`, a default, a trigger expression, ...). + declaration = _token_secret_declaration(data) + if declaration is not None: + declaration.pop(TOKEN_NAME) + outside_jobs = {k: v for k, v in data.items() if k != "jobs"} + if _mentions_token(outside_jobs): + fail( + violations, + path, + None, + "token referenced outside any job; the only legitimate mention there " + "is an 'on.workflow_call.secrets' declaration", + ) + jobs = data.get("jobs") or {} for job_name, job in jobs.items(): if not isinstance(job, dict): continue - job_text = yaml.safe_dump(job) - if TOKEN_NAME not in job_text: + if not _mentions_token(job): continue + # --- Rule 3b: forwarding to a local reusable workflow. --------------- + secrets_block = job.get("secrets") + forwarded = {} + if isinstance(secrets_block, dict): + forwarded = {k: v for k, v in secrets_block.items() if _mentions_token({k: v})} + + if forwarded: + renamed = sorted(k for k in forwarded if k != TOKEN_NAME) + if renamed: + fail( + violations, + path, + job_name, + f"token forwarded under a different name {renamed}; it must be " + f"forwarded as '{TOKEN_NAME}' so that a single " + f"'git grep {TOKEN_NAME}' still finds every file that touches it", + ) + uses = job.get("uses") or "" + callee = LOCAL_REUSABLE.match(uses) + if callee is None: + fail( + violations, + path, + job_name, + f"token forwarded via 'secrets:' to {uses or ''!r}; " + f"it may only be forwarded to a local reusable workflow " + f"('./.github/workflows/.yml'), which this same scan covers", + ) + elif not (path.parent / callee.group(1)).is_file(): + fail( + violations, + path, + job_name, + f"token forwarded to {uses!r}, which does not exist in " + f"{path.parent}; the callee must be a workflow this scan covers", + ) + + # --- Rule 3a: only the EC2 runner action may consume it in a step. ---- token_steps = [] for step in iter_steps(job): - step_text = yaml.safe_dump(step) - if TOKEN_NAME not in step_text: + if not _mentions_token(step): continue token_steps.append(step) - # Rule 3: the consuming step must be the EC2 runner action. uses = (step.get("uses") or "").split("@")[0] if uses != ALLOWED_ACTION: fail( @@ -153,9 +295,7 @@ def check_workflow_file(path: Path, violations: list[str]) -> None: # Rule 2 (structural backstop): only via the github-token input. with_block = step.get("with") or {} - offending = { - k: v for k, v in with_block.items() if k != "github-token" and TOKEN_NAME in yaml.safe_dump({k: v}) - } + offending = {k: v for k, v in with_block.items() if k != "github-token" and _mentions_token({k: v})} if offending: fail( violations, @@ -164,19 +304,30 @@ def check_workflow_file(path: Path, violations: list[str]) -> None: f"token passed via disallowed input(s): {sorted(offending)}", ) - if not token_steps: - # Token appears in the job but in no step (e.g. job-level env: or a - # reusable-workflow `secrets:` block). That is never allowed. + # Anything left -- job-level `env:`, a `with:` input, a `if:` expression + # -- is a mention in a position neither rule 3a nor 3b sanctions. + leftover = {k: v for k, v in job.items() if k not in ("steps", "secrets")} + if _mentions_token(leftover): fail( violations, path, job_name, - "token referenced at job level (env/secrets/with), not as an " f"'{ALLOWED_ACTION}' step input", + f"token referenced at job level (env/with/...), not as an " + f"'{ALLOWED_ACTION}' step input or a 'secrets:' forward to a local " + f"reusable workflow", + ) + elif not token_steps and not forwarded: + # The job names the token but in none of the shapes above (e.g. a + # `secrets:` block that is not a mapping). Fail closed. + fail( + violations, + path, + job_name, + "token referenced in this job in an unrecognised position", ) - continue - # Rule 4: a privileged job must not pull untrusted code into its - # workspace (no local actions, no checkout) next to the token. + # --- Rule 4: no untrusted code in a job that touches the token. ------- + # A rule 3b caller job has no steps, so this only ever applies to 3a. for step in iter_steps(job): uses = step.get("uses") or "" bare = uses.split("@")[0] @@ -291,18 +442,22 @@ def main() -> int: for v in violations: print(f" - {v}", file=sys.stderr) print( - "\nThe admin-scoped runner token may ONLY be used as:\n" + "\nThe admin-scoped runner token may ONLY be used in one of two ways:\n" f" github-token: ${{{{ secrets.{TOKEN_NAME} }}}}\n" f" in a step that uses '{ALLOWED_ACTION}', inside a job that does\n" - " not check out code or run local actions. See " - ".github/scripts/check_runner_token_policy.py.", + " not check out code or run local actions; or\n" + f" secrets:\n {TOKEN_NAME}: ${{{{ secrets.{TOKEN_NAME} }}}}\n" + " on a job whose 'uses:' is a local './.github/workflows/.yml'\n" + " (which this same scan covers). 'secrets: inherit' is never allowed.\n" + " See .github/scripts/check_runner_token_policy.py.", file=sys.stderr, ) return 1 print( f"✅ EC2 runner token policy: OK ({len(files)} workflow file(s) scanned; " - f"token used only as '{ALLOWED_ACTION}' input)." + f"token reaches only '{ALLOWED_ACTION}', directly or via a local " + f"reusable workflow)." ) return 0 diff --git a/.github/scripts/test_check_runner_token_policy.py b/.github/scripts/test_check_runner_token_policy.py index 3ef549070..b07141577 100644 --- a/.github/scripts/test_check_runner_token_policy.py +++ b/.github/scripts/test_check_runner_token_policy.py @@ -3,9 +3,11 @@ """Unit tests for the EC2 runner-token CI policy. Exercises check_runner_token_policy.py -- the security gate that enforces that -``secrets.EC2_RUNNER_TOKEN`` is only ever used as the ``github-token`` input to -``machulav/ec2-github-runner``. Run by .github/workflows/workflow-security.yml -on every PR (it needs only pyyaml + pytest, no fvdb build). +``secrets.EC2_RUNNER_TOKEN`` only ever reaches ``machulav/ec2-github-runner``, +either as that action's ``github-token`` input or forwarded by name to a *local* +reusable workflow that this same scan covers. Run by +.github/workflows/workflow-security.yml on every PR (it needs only pyyaml + +pytest, no fvdb build). """ from __future__ import annotations @@ -34,8 +36,15 @@ def _load_policy_module(): policy = _load_policy_module() -def _check(tmp_path: Path, yaml_text: str) -> list[str]: - """Write a workflow file and return the policy violations it produces.""" +def _check(tmp_path: Path, yaml_text: str, callees: tuple[str, ...] = ()) -> list[str]: + """Write a workflow file and return the policy violations it produces. + + ``callees`` names sibling workflow files to create first, for the rule-3b + check that a forwarding target actually exists in this directory (and is + therefore covered by the same scan). + """ + for name in callees: + (tmp_path / name).write_text("name: callee\n") wf = tmp_path / "wf.yml" wf.write_text(textwrap.dedent(yaml_text)) violations: list[str] = [] @@ -272,7 +281,54 @@ def test_local_action_in_token_job_is_rejected(tmp_path): assert any("LOCAL action" in v for v in violations) -def test_job_level_token_with_no_step_is_rejected(tmp_path): +# --- rule 3b: forwarding to a local reusable workflow ------------------------ + + +def test_forward_to_local_reusable_workflow_is_allowed(tmp_path): + """The DRY shape: the callee is a workflow file this same scan covers.""" + violations = _check( + tmp_path, + """ + name: ok + on: [push] + jobs: + call: + uses: ./.github/workflows/start-ec2-runner.yml + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} + """, + callees=("start-ec2-runner.yml",), + ) + assert violations == [] + + +def test_callee_declaring_the_token_secret_is_allowed(tmp_path): + """The other half of 3b: `on.workflow_call.secrets` naming the token.""" + violations = _check( + tmp_path, + """ + name: ok + on: + workflow_call: + secrets: + EC2_RUNNER_TOKEN: + required: true + jobs: + start: + runs-on: ubuntu-latest + steps: + - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef + with: + mode: start + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + """, + ) + assert violations == [] + + +def test_forward_to_remote_reusable_workflow_is_rejected(tmp_path): + """The hole `secrets: inherit` used to walk through: the token leaves the + repo, where no scan of ours can see what the callee does with it.""" violations = _check( tmp_path, """ @@ -280,14 +336,122 @@ def test_job_level_token_with_no_step_is_rejected(tmp_path): on: [push] jobs: call: - uses: ./.github/workflows/reusable.yml + uses: other-org/other-repo/.github/workflows/x.yml@main + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} + """, + ) + assert any("only be forwarded to a local reusable workflow" in v for v in violations) + + +def test_forward_to_missing_local_workflow_is_rejected(tmp_path): + """A callee that is not in this directory would not be scanned.""" + violations = _check( + tmp_path, + """ + name: bad + on: [push] + jobs: + call: + uses: ./.github/workflows/nope.yml + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} + """, + ) + assert any("does not exist" in v for v in violations) + + +def test_forward_under_a_different_name_is_rejected(tmp_path): + """One name everywhere, so a single grep still enumerates every file.""" + violations = _check( + tmp_path, + """ + name: bad + on: [push] + jobs: + call: + uses: ./.github/workflows/start-ec2-runner.yml secrets: gh-token: ${{ secrets.EC2_RUNNER_TOKEN }} """, + callees=("start-ec2-runner.yml",), ) assert violations +def test_forward_from_a_job_with_no_uses_is_rejected(tmp_path): + violations = _check( + tmp_path, + """ + name: bad + on: [push] + jobs: + call: + runs-on: ubuntu-latest + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} + steps: + - run: echo hi + """, + ) + assert any("no job-level uses" in v for v in violations) + + +def test_workflow_level_env_is_rejected(tmp_path): + """Same textual shape as a legitimate forward -- caught structurally.""" + violations = _check( + tmp_path, + """ + name: bad + on: [push] + env: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} + jobs: + leak: + runs-on: ubuntu-latest + steps: + - run: env + """, + ) + assert any("outside any job" in v for v in violations) + + +# --- rule 6: no blanket secret forwarding ------------------------------------ + + +def test_secrets_inherit_is_rejected(tmp_path): + """`inherit` forwards the admin token while naming nothing, so no textual + check can see where it goes -- including a forward out of the repo.""" + violations = _check( + tmp_path, + """ + name: bad + on: [push] + jobs: + call: + uses: ./.github/workflows/start-ec2-runner.yml + secrets: inherit + """, + callees=("start-ec2-runner.yml",), + ) + assert any("secrets: inherit" in v for v in violations) + + +def test_secrets_inherit_is_rejected_even_to_a_remote_workflow(tmp_path): + violations = _check( + tmp_path, + """ + name: bad + on: [push] + jobs: + call: + uses: other-org/other-repo/.github/workflows/x.yml@main + secrets: inherit + """, + ) + assert any("secrets: inherit" in v for v in violations) + + # --- the real fvdb workflows must all pass ----------------------------------- From 53bdb4c54c9d7d150bdcd5df81ad2ac6a6599c64 Mon Sep 17 00:00:00 2001 From: Mark Harris Date: Thu, 3 Sep 2026 02:06:34 +0000 Subject: [PATCH 2/3] CI: provision EC2 runners through shared workflows, with retry and backoff CI keeps failing because us-east-2 has no capacity for the requested instance type at the moment the runner is requested. machulav/ec2-github-runner sweeps the availability zones in `availability-zones-config` within one attempt, but it never retries the sweep, so a transient region-wide shortage fails the whole run. Retrying twice, 90s and then 180s later, turns most of those into a slow pass instead of a red build. Rather than paste that retry ladder into all nine start jobs, put it in a reusable workflow. `start-ec2-runner.yml` and `stop-ec2-runner.yml` now hold the retry policy, the pinned action SHA, the AWS OIDC step and the teardown, and the five workflows call them. The retry ladder exists once and cannot drift between tests.yml, cu130.yml, cu132.yml, publish.yml and nightly-publish.yml -- which is also why the previous inline version of this change was not worth landing. The reusable workflows absorb the two call shapes the repo already had: * `runner-label` / `aws-resource-tags` for the matrix fan-outs in publish.yml and nightly-publish.yml, which need a predictable label because they find the instance again by tag at teardown. `aws-resource-tags` defaults to '[]' rather than '': the action runs a bare JSON.parse() on it, which throws on an empty string. * `stagger-seconds`, replacing the hand-rolled `RANDOM % 16` sleep those same matrix jobs used to spread out runner registrations. `stop-ec2-runner.yml` takes the instance id directly when the caller has one from a start job's output, and otherwise resolves it from the RunnerLabel tag -- the lookup the publish workflows had to do inline, because a matrix job has no single output to read. Net: 401 lines of duplicated YAML removed, 171 added, plus the two shared workflows. Forwarding the token this way is what the accompanying policy change permits; the checker, actionlint and zizmor all pass on the result. Signed-off-by: Mark Harris Co-authored-by: Claude Opus 5 (1M context) Signed-off-by: Mark Harris --- .github/workflows/cu130.yml | 96 ++++-------- .github/workflows/cu132.yml | 96 ++++-------- .github/workflows/nightly-publish.yml | 94 ++++-------- .github/workflows/publish.yml | 190 +++++++----------------- .github/workflows/start-ec2-runner.yml | 183 +++++++++++++++++++++++ .github/workflows/stop-ec2-runner.yml | 90 +++++++++++ .github/workflows/tests.yml | 96 ++++-------- .github/workflows/workflow-security.yml | 6 +- 8 files changed, 448 insertions(+), 403 deletions(-) create mode 100644 .github/workflows/start-ec2-runner.yml create mode 100644 .github/workflows/stop-ec2-runner.yml diff --git a/.github/workflows/cu130.yml b/.github/workflows/cu130.yml index 7ba89345f..af29e5cb5 100644 --- a/.github/workflows/cu130.yml +++ b/.github/workflows/cu130.yml @@ -33,29 +33,19 @@ jobs: start-build-runner: name: Start CPU-only EC2 runner for build needs: [check-changes, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC if: >- needs.check-changes.outputs.should_test == 'true' && (github.event.pull_request.draft == false || github.event_name != 'pull_request_target') - outputs: - label: ${{ steps.start-build-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-build-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 runner - id: start-build-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: m6a.8xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: m6a.8xlarge + az-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # BUILD FVDB @@ -205,25 +195,19 @@ jobs: - start-build-runner # required to get output from the start-build-runner job - fvdb-build # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs # but only if the start-build-runner job was not skipped if: ${{ always() && needs.start-build-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-build-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-build-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # START FVDB TESTS GPU RUNNER @@ -231,26 +215,16 @@ jobs: start-tests-gpu-runner: name: Start EC2 GPU runner for gtests needs: [fvdb-build, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC - outputs: - label: ${{ steps.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-tests-gpu-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 GPU runner - id: start-tests-gpu-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: g6.xlarge # 4 CPU-core, L4 GPU - availability-zones-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: g6.xlarge # 4 CPU-core, L4 GPU + az-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # RUN FVDB GTESTS @@ -573,22 +547,16 @@ jobs: - fvdb-unit-test # required to wait when the main job is done - fvdb-docs-test # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs # but only if the start-tests-gpu-runner job was not skipped if: ${{ always() && needs.start-tests-gpu-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-tests-gpu-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} diff --git a/.github/workflows/cu132.yml b/.github/workflows/cu132.yml index 59e55ce3f..7430d6b10 100644 --- a/.github/workflows/cu132.yml +++ b/.github/workflows/cu132.yml @@ -33,29 +33,19 @@ jobs: start-build-runner: name: Start CPU-only EC2 runner for build needs: [check-changes, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC if: >- needs.check-changes.outputs.should_test == 'true' && (github.event.pull_request.draft == false || github.event_name != 'pull_request_target') - outputs: - label: ${{ steps.start-build-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-build-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 runner - id: start-build-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: m6a.8xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: m6a.8xlarge + az-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # BUILD FVDB @@ -205,25 +195,19 @@ jobs: - start-build-runner # required to get output from the start-build-runner job - fvdb-build # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs # but only if the start-build-runner job was not skipped if: ${{ always() && needs.start-build-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-build-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-build-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # START FVDB TESTS GPU RUNNER @@ -231,26 +215,16 @@ jobs: start-tests-gpu-runner: name: Start EC2 GPU runner for gtests needs: [fvdb-build, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC - outputs: - label: ${{ steps.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-tests-gpu-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 GPU runner - id: start-tests-gpu-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: g6.xlarge # 4 CPU-core, L4 GPU - availability-zones-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: g6.xlarge # 4 CPU-core, L4 GPU + az-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # RUN FVDB GTESTS @@ -573,22 +547,16 @@ jobs: - fvdb-unit-test # required to wait when the main job is done - fvdb-docs-test # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs # but only if the start-tests-gpu-runner job was not skipped if: ${{ always() && needs.start-tests-gpu-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-tests-gpu-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} diff --git a/.github/workflows/nightly-publish.yml b/.github/workflows/nightly-publish.yml index 0e11e941a..f332d1e25 100644 --- a/.github/workflows/nightly-publish.yml +++ b/.github/workflows/nightly-publish.yml @@ -59,42 +59,29 @@ jobs: name: Start CPU-only EC2 runner for nightly build needs: [check-new-commits, versions] if: needs.check-new-commits.outputs.should_build == 'true' - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Stagger job starts to avoid API rate limits - run: | - DELAY=$((RANDOM % 16)) - echo "Delaying start by ${DELAY} seconds for Python ${MATRIX_PYTHON_VERSION}" - sleep $DELAY - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 runner - id: start-build-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: m6a.xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-cpu-az-config }} - label: nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - aws-resource-tags: > - [ - {"Key": "RunnerLabel", "Value": "nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, - {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, - {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, - {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, - {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} - ] + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: m6a.xlarge + az-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + runner-label: nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-resource-tags: > + [ + {"Key": "RunnerLabel", "Value": "nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, + {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, + {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, + {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, + {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} + ] + stagger-seconds: 15 + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # BUILD NIGHTLY WHEELS @@ -360,47 +347,16 @@ jobs: - start-build-runner - nightly-build - versions - runs-on: ubuntu-latest if: ${{ always() && needs.start-build-runner.result != 'skipped' }} permissions: id-token: write # Required for AWS OIDC strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Find EC2 instance ID by label - id: find-instance - run: | - LABEL="nightly-${MATRIX_PYTHON_VERSION}-pt${MATRIX_TORCH_VERSION}-cu${MATRIX_CUDA_VERSION}-${{ github.run_id }}" - echo "Looking for instance with RunnerLabel: $LABEL" - - INSTANCE_ID=$(aws ec2 describe-instances \ - --filters "Name=tag:RunnerLabel,Values=$LABEL" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ - --query 'Reservations[0].Instances[0].InstanceId' \ - --output text) - - if [ "$INSTANCE_ID" == "None" ] || [ -z "$INSTANCE_ID" ]; then - echo "ERROR: No instance found with RunnerLabel=$LABEL" - echo "instance-id=" >> $GITHUB_OUTPUT - exit 1 - else - echo "Found instance: $INSTANCE_ID" - echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT - fi - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - MATRIX_TORCH_VERSION: ${{ matrix.torch-version }} - MATRIX_CUDA_VERSION: ${{ matrix.cuda-version }} - - name: Stop EC2 runner - if: steps.find-instance.outputs.instance-id != '' - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - ec2-instance-id: ${{ steps.find-instance.outputs.instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: nightly-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e020a2a44..baf956175 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -119,44 +119,30 @@ jobs: start-build-runner: name: Start CPU-only EC2 runner for build - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC needs: versions strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Stagger job starts to avoid API rate limits - run: | - # Random delay between 0 and 15 seconds to stagger runner registrations - DELAY=$((RANDOM % 16)) - echo "Delaying start by ${DELAY} seconds for Python ${MATRIX_PYTHON_VERSION}" - sleep $DELAY - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 runner - id: start-build-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: m6a.8xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-cpu-az-config }} - label: ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - aws-resource-tags: > - [ - {"Key": "RunnerLabel", "Value": "ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, - {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, - {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, - {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, - {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} - ] + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: m6a.8xlarge + az-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + runner-label: ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-resource-tags: > + [ + {"Key": "RunnerLabel", "Value": "ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, + {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, + {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, + {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, + {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} + ] + stagger-seconds: 15 + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} fvdb-build: name: fVDB Build @@ -533,42 +519,29 @@ jobs: name: Start GPU EC2 runner for validation needs: [pr-flags, fvdb-build, versions] if: ${{ !cancelled() && needs.fvdb-build.result == 'success' && (needs.pr-flags.outputs.release_push == 'true' || needs.pr-flags.outputs.release == 'true' || inputs.run_validation == true) }} - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Stagger job starts to avoid API rate limits - run: | - DELAY=$((RANDOM % 16)) - echo "Delaying start by ${DELAY} seconds for Python ${MATRIX_PYTHON_VERSION}" - sleep $DELAY - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 GPU runner - id: start-validation-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: g6.xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-gpu-az-config }} - label: val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - aws-resource-tags: > - [ - {"Key": "RunnerLabel", "Value": "val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, - {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, - {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, - {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, - {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} - ] + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: g6.xlarge + az-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + runner-label: val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-resource-tags: > + [ + {"Key": "RunnerLabel", "Value": "val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }}"}, + {"Key": "PythonVersion", "Value": "${{ matrix.python-version }}"}, + {"Key": "TorchVersion", "Value": "${{ matrix.torch-version }}"}, + {"Key": "CudaVersion", "Value": "${{ matrix.cuda-version }}"}, + {"Key": "GitHubRunId", "Value": "${{ github.run_id }}"} + ] + stagger-seconds: 15 + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} validate-smoke-test: name: Smoke test built wheel @@ -701,50 +674,19 @@ jobs: - validate-smoke-test - validate-unit-tests - versions - runs-on: ubuntu-latest if: ${{ always() && needs.start-validation-runner.result != 'skipped' }} permissions: id-token: write # Required for AWS OIDC strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Find EC2 instance ID by label - id: find-instance - run: | - LABEL="val-${MATRIX_PYTHON_VERSION}-pt${MATRIX_TORCH_VERSION}-cu${MATRIX_CUDA_VERSION}-${{ github.run_id }}" - echo "Looking for instance with RunnerLabel: $LABEL" - - INSTANCE_ID=$(aws ec2 describe-instances \ - --filters "Name=tag:RunnerLabel,Values=$LABEL" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ - --query 'Reservations[0].Instances[0].InstanceId' \ - --output text) - - if [ "$INSTANCE_ID" == "None" ] || [ -z "$INSTANCE_ID" ]; then - echo "ERROR: No instance found with RunnerLabel=$LABEL" - echo "instance-id=" >> $GITHUB_OUTPUT - exit 1 - else - echo "Found instance: $INSTANCE_ID" - echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT - fi - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - MATRIX_TORCH_VERSION: ${{ matrix.torch-version }} - MATRIX_CUDA_VERSION: ${{ matrix.cuda-version }} - - name: Stop EC2 runner - if: steps.find-instance.outputs.instance-id != '' - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - ec2-instance-id: ${{ steps.find-instance.outputs.instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: val-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} fvdb-build-stop-runner: name: Stop CPU-only EC2 runner for build @@ -752,48 +694,16 @@ jobs: - start-build-runner # required to get output from the start-build-runner job - fvdb-build # required to wait when the main job is done - versions - runs-on: ubuntu-latest if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs permissions: id-token: write # Required for AWS OIDC strategy: fail-fast: false matrix: ${{ fromJSON(needs.versions.outputs.publish-matrix) }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Find EC2 instance ID by label - id: find-instance - run: | - LABEL="ec2-${MATRIX_PYTHON_VERSION}-pt${MATRIX_TORCH_VERSION}-cu${MATRIX_CUDA_VERSION}-${{ github.run_id }}" - echo "Looking for instance with RunnerLabel: $LABEL" - - # Query by the custom RunnerLabel tag we set during instance creation - INSTANCE_ID=$(aws ec2 describe-instances \ - --filters "Name=tag:RunnerLabel,Values=$LABEL" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ - --query 'Reservations[0].Instances[0].InstanceId' \ - --output text) - - if [ "$INSTANCE_ID" == "None" ] || [ -z "$INSTANCE_ID" ]; then - echo "ERROR: No instance found with RunnerLabel=$LABEL" - echo "instance-id=" >> $GITHUB_OUTPUT - exit 1 - else - echo "Found instance: $INSTANCE_ID" - echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT - fi - env: - MATRIX_PYTHON_VERSION: ${{ matrix.python-version }} - MATRIX_TORCH_VERSION: ${{ matrix.torch-version }} - MATRIX_CUDA_VERSION: ${{ matrix.cuda-version }} - - name: Stop EC2 runner - if: steps.find-instance.outputs.instance-id != '' - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} - ec2-instance-id: ${{ steps.find-instance.outputs.instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ec2-${{ matrix.python-version }}-pt${{ matrix.torch-version }}-cu${{ matrix.cuda-version }}-${{ github.run_id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} diff --git a/.github/workflows/start-ec2-runner.yml b/.github/workflows/start-ec2-runner.yml new file mode 100644 index 000000000..f90fd4980 --- /dev/null +++ b/.github/workflows/start-ec2-runner.yml @@ -0,0 +1,183 @@ +# Copyright Contributors to the OpenVDB Project +# SPDX-License-Identifier: Apache-2.0 +# +# Start a self-hosted EC2 runner, retrying with backoff when the region has no +# capacity for the requested instance type. +# +# Every workflow that needs an EC2 runner calls this instead of inlining the +# machulav action, so the retry policy (attempt count, backoff, the AZ sweep, +# the pinned action SHA) lives in exactly one place and cannot drift between +# tests.yml, cu130.yml, cu132.yml, publish.yml and nightly-publish.yml. +# +# Security note: this workflow holds the admin-scoped runner token, so it +# deliberately has no `actions/checkout` and no local actions -- see +# .github/scripts/check_runner_token_policy.py, which enforces that and which +# also covers this file, because it is a *local* reusable workflow. Callers must +# forward that one secret explicitly by name; `secrets: inherit` is rejected by +# the same check. (The token's name is deliberately not spelled in this comment: +# the policy allows it to appear only in the three positions it documents, so a +# single grep for the name enumerates every place it can actually reach.) +name: Start EC2 Runner + +on: + workflow_call: + inputs: + instance-type: + description: EC2 instance type to launch, e.g. m6a.8xlarge or g6.xlarge. + type: string + required: true + az-config: + description: >- + JSON array of availability-zone configs, tried in order by the action + within a single attempt. Usually needs.versions.outputs.aws-*-az-config. + type: string + required: true + aws-role: + description: IAM role to assume via OIDC. + type: string + required: true + aws-region: + description: AWS region to launch in. + type: string + required: true + runner-label: + description: >- + Explicit runner label. Leave empty to let the action generate one -- + only the publish workflows need a predictable label, because they find + the instance again by tag at teardown. + type: string + required: false + default: '' + aws-resource-tags: + description: >- + JSON array of EC2 tags to apply to the instance. The publish workflows + set a RunnerLabel tag here so stop-ec2-runner.yml can find the instance + again without a job output. + type: string + required: false + default: '' + stagger-seconds: + description: >- + Upper bound (inclusive) on a random delay before the first attempt. + Matrix fan-outs set this so N simultaneous registrations do not hit the + GitHub API rate limit at the same instant. 0 disables the delay. + type: number + required: false + default: 0 + backoff-seconds: + description: >- + Seconds to wait after the first failed attempt. The second wait is + twice this, so the default gives 90s then 180s between three attempts. + type: number + required: false + default: 90 + secrets: + EC2_RUNNER_TOKEN: + description: Admin-scoped token used to register the runner with this repo. + required: true + outputs: + label: + description: Runner label to pass as `runs-on` in the job being provisioned. + value: ${{ jobs.start.outputs.label }} + ec2-instance-id: + description: Instance id, to be handed back to stop-ec2-runner.yml. + value: ${{ jobs.start.outputs.ec2-instance-id }} + +permissions: + id-token: write # Required for AWS OIDC + +jobs: + start: + name: Start EC2 runner + runs-on: ubuntu-latest + # Exactly one attempt can succeed: each is skipped unless every earlier one + # failed, so at most one of these outputs is non-empty and `||` picks it. + outputs: + label: ${{ steps.a1.outputs.label || steps.a2.outputs.label || steps.a3.outputs.label }} + ec2-instance-id: ${{ steps.a1.outputs.ec2-instance-id || steps.a2.outputs.ec2-instance-id || steps.a3.outputs.ec2-instance-id }} + steps: + - name: Stagger job start to avoid API rate limits + if: inputs.stagger-seconds > 0 + env: + STAGGER: ${{ inputs.stagger-seconds }} + run: | + DELAY=$((RANDOM % (STAGGER + 1))) + echo "Delaying start by ${DELAY}s to spread out runner registrations." + sleep "${DELAY}" + + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 + with: + role-to-assume: ${{ inputs.aws-role }} + aws-region: ${{ inputs.aws-region }} + + - name: Start EC2 runner (attempt 1) + id: a1 + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: start + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + ec2-instance-type: ${{ inputs.instance-type }} + availability-zones-config: ${{ inputs.az-config }} + # Empty label => the action generates a unique one (its documented + # behaviour). aws-resource-tags is NOT so forgiving: the action does a + # bare JSON.parse() on it, which throws on '', so default to '[]'. + label: ${{ inputs.runner-label }} + aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} + + - name: Back off before attempt 2 + if: steps.a1.outcome == 'failure' + env: + BACKOFF: ${{ inputs.backoff-seconds }} + run: | + echo "::warning::Runner start attempt 1 found no capacity; retrying in ${BACKOFF}s." + sleep "${BACKOFF}" + + - name: Start EC2 runner (attempt 2) + id: a2 + if: steps.a1.outcome == 'failure' + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: start + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + ec2-instance-type: ${{ inputs.instance-type }} + availability-zones-config: ${{ inputs.az-config }} + # Empty label => the action generates a unique one (its documented + # behaviour). aws-resource-tags is NOT so forgiving: the action does a + # bare JSON.parse() on it, which throws on '', so default to '[]'. + label: ${{ inputs.runner-label }} + aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} + + - name: Back off before attempt 3 + if: steps.a1.outcome == 'failure' && steps.a2.outcome == 'failure' + env: + BACKOFF: ${{ inputs.backoff-seconds }} + run: | + echo "::warning::Runner start attempt 2 found no capacity; retrying in $((BACKOFF * 2))s." + sleep "$((BACKOFF * 2))" + + - name: Start EC2 runner (attempt 3) + id: a3 + if: steps.a1.outcome == 'failure' && steps.a2.outcome == 'failure' + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: start + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + ec2-instance-type: ${{ inputs.instance-type }} + availability-zones-config: ${{ inputs.az-config }} + # Empty label => the action generates a unique one (its documented + # behaviour). aws-resource-tags is NOT so forgiving: the action does a + # bare JSON.parse() on it, which throws on '', so default to '[]'. + label: ${{ inputs.runner-label }} + aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} + + - name: Fail if no attempt obtained capacity + if: steps.a1.outcome == 'failure' && steps.a2.outcome == 'failure' && steps.a3.outcome == 'failure' + env: + INSTANCE_TYPE: ${{ inputs.instance-type }} + run: | + echo "::error::Could not obtain a ${INSTANCE_TYPE} instance after 3 attempts across every configured availability zone; the region is likely out of this instance type." + exit 1 diff --git a/.github/workflows/stop-ec2-runner.yml b/.github/workflows/stop-ec2-runner.yml new file mode 100644 index 000000000..7e8b3b6ab --- /dev/null +++ b/.github/workflows/stop-ec2-runner.yml @@ -0,0 +1,90 @@ +# Copyright Contributors to the OpenVDB Project +# SPDX-License-Identifier: Apache-2.0 +# +# De-register and terminate a self-hosted EC2 runner started by +# start-ec2-runner.yml. Callers keep their own `needs:` and +# `if: always() && needs..result != 'skipped'` guards; this workflow +# is only the teardown itself. +# +# Two call shapes, matching the two ways the repo provisions runners: +# * pass `ec2-instance-id` straight from the start job's output (tests.yml, +# cu130.yml, cu132.yml, nightly-publish.yml); +# * leave it empty and the instance is looked up by its `RunnerLabel` tag +# (publish.yml, whose start job fans out over a matrix, so there is no +# single job output to read). +# +# Security note: like its start counterpart, this workflow holds the +# admin-scoped runner token and therefore never checks out code or runs a +# local action -- see .github/scripts/check_runner_token_policy.py. +name: Stop EC2 Runner + +on: + workflow_call: + inputs: + label: + description: Runner label, as emitted or as given to start-ec2-runner.yml. + type: string + required: true + ec2-instance-id: + description: >- + Instance id emitted by start-ec2-runner.yml. Leave empty to resolve it + from the instance's RunnerLabel tag instead. + type: string + required: false + default: '' + aws-role: + description: IAM role to assume via OIDC. + type: string + required: true + aws-region: + description: AWS region the instance is running in. + type: string + required: true + secrets: + EC2_RUNNER_TOKEN: + description: Admin-scoped token used to de-register the runner. + required: true + +permissions: + id-token: write # Required for AWS OIDC + +jobs: + stop: + name: Stop EC2 runner + runs-on: ubuntu-latest + steps: + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 + with: + role-to-assume: ${{ inputs.aws-role }} + aws-region: ${{ inputs.aws-region }} + + - name: Find EC2 instance ID by label + id: find-instance + if: inputs.ec2-instance-id == '' + env: + LABEL: ${{ inputs.label }} + run: | + echo "Looking for instance with RunnerLabel: $LABEL" + INSTANCE_ID=$(aws ec2 describe-instances \ + --filters "Name=tag:RunnerLabel,Values=$LABEL" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ + --query 'Reservations[0].Instances[0].InstanceId' \ + --output text) + + if [ "$INSTANCE_ID" == "None" ] || [ -z "$INSTANCE_ID" ]; then + echo "ERROR: No instance found with RunnerLabel=$LABEL" + echo "instance-id=" >> "$GITHUB_OUTPUT" + exit 1 + fi + echo "Found instance: $INSTANCE_ID" + echo "instance-id=$INSTANCE_ID" >> "$GITHUB_OUTPUT" + + - name: Stop EC2 runner + # The lookup step is skipped when the caller supplied an id, so its + # output is empty then; `||` picks whichever of the two we actually have. + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: stop + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + label: ${{ inputs.label }} + ec2-instance-id: ${{ inputs.ec2-instance-id || steps.find-instance.outputs.instance-id }} diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 360cf95e0..9fbf12404 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -39,29 +39,19 @@ jobs: start-build-runner: name: Start CPU-only EC2 runner for build needs: [check-changes, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC if: >- needs.check-changes.outputs.should_test == 'true' && (github.event.pull_request.draft == false || github.event_name != 'pull_request_target') - outputs: - label: ${{ steps.start-build-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-build-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 runner - id: start-build-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: m6a.8xlarge - availability-zones-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: m6a.8xlarge + az-config: ${{ needs.versions.outputs.aws-cpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} fvdb-build: name: fVDB Build (Conda) needs: [start-build-runner, versions] # required to start the main job when the runner is ready @@ -159,25 +149,19 @@ jobs: - start-build-runner # required to get output from the start-build-runner job - fvdb-build # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs, but only if the # start-build-runner job was not skipped if: ${{ always() && needs.start-build-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-build-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-build-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## @@ -186,26 +170,16 @@ jobs: start-tests-gpu-runner: name: Start EC2 GPU runner for tests needs: [fvdb-build, versions] - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC - outputs: - label: ${{ steps.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ steps.start-tests-gpu-runner.outputs.ec2-instance-id }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Start EC2 GPU runner - id: start-tests-gpu-runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: start - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - ec2-instance-type: g6.xlarge # 4 CPU-core, L4 GPU - availability-zones-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + uses: ./.github/workflows/start-ec2-runner.yml + with: + instance-type: g6.xlarge # 4 CPU-core, L4 GPU + az-config: ${{ needs.versions.outputs.aws-gpu-az-config }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} ############################################################################## # RUN FVDB GTESTS @@ -446,22 +420,16 @@ jobs: - fvdb-unit-tests # required to wait when the main job is done - fvdb-docs-test # required to wait when the main job is done - versions - runs-on: ubuntu-latest permissions: id-token: write # Required for AWS OIDC # required to stop the runner even if the error happened in the previous jobs # but only if the start-tests-gpu-runner job was not skipped if: ${{ always() && needs.start-tests-gpu-runner.result != 'skipped' }} - steps: - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3 - with: - role-to-assume: ${{ needs.versions.outputs.aws-role }} - aws-region: ${{ needs.versions.outputs.aws-region }} - - name: Stop EC2 runner - uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 - with: - mode: stop - github-token: ${{ secrets.EC2_RUNNER_TOKEN }} - label: ${{ needs.start-tests-gpu-runner.outputs.label }} - ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + uses: ./.github/workflows/stop-ec2-runner.yml + with: + label: ${{ needs.start-tests-gpu-runner.outputs.label }} + ec2-instance-id: ${{ needs.start-tests-gpu-runner.outputs.ec2-instance-id }} + aws-role: ${{ needs.versions.outputs.aws-role }} + aws-region: ${{ needs.versions.outputs.aws-region }} + secrets: + EC2_RUNNER_TOKEN: ${{ secrets.EC2_RUNNER_TOKEN }} diff --git a/.github/workflows/workflow-security.yml b/.github/workflows/workflow-security.yml index 49cb16077..61ae282ab 100644 --- a/.github/workflows/workflow-security.yml +++ b/.github/workflows/workflow-security.yml @@ -64,8 +64,10 @@ jobs: echo "Scanning PR workflow files:" ls -1 .github/workflows - # 3a. Repo-specific policy: the admin runner token may only be used as the - # github-token input to machulav/ec2-github-runner. The policy script + # 3a. Repo-specific policy: the admin runner token may only reach + # machulav/ec2-github-runner -- either as that action's github-token + # input, or forwarded by name to a local reusable workflow that this + # same scan covers (never via secrets: inherit). The policy script # and its tests run from the trusted base checkout (not the PR). # Rules 2-4 scan the overlaid PR workflow files; the Rule 1 leak check # runs against the PR head commit tree (read-only) so it also catches From c90f2d7c808839d888813feba74a36363e978f8b Mon Sep 17 00:00:00 2001 From: Mark Harris Date: Thu, 3 Sep 2026 04:15:21 +0000 Subject: [PATCH 3/3] CI: pick the winning runner attempt by outcome, and clean up the losers The retry ladder handled only one of the two ways a start attempt can fail. machulav/ec2-github-runner publishes its outputs as soon as the instance launches, BEFORE it waits for the runner to register: const result = await aws.startEc2Instance(label, ...); setOutput(label, ec2InstanceId, region); // <-- here await aws.waitForInstanceRunning(ec2InstanceId, region); await gh.waitForRunnerRegistered(label, pollCallback); // 5-min timeout So a "no capacity" failure throws before setOutput and leaves the outputs empty, but a registration failure -- timeout, bad AMI, userdata error -- fails with the outputs already populated. The ladder picked outputs with ${{ steps.a1.outputs.label || steps.a2.outputs.label || ... }} which selects the first NON-EMPTY value, not the successful one. In the second failure class that hands back the dead attempt's label, so the dependent job queues against a runner that never appears instead of failing fast, and the instance the retry actually provisioned is never stopped, because teardown was given the wrong instance id. Fix both halves: * Select on the step outcome instead of on emptiness, in an explicit step that also produces the terminal error when no attempt succeeded. Keying on `outcome` is what makes "the attempt that worked" and "the first attempt with an output" the same thing again. * Stop the instance a failed attempt left running before retrying, rather than leaking it. This is needed even with correct selection: nothing else knows those instances exist, since only the winner reaches the stop job. Verified the selection logic against all five combinations, including the case that motivated this: attempt 1 launches but never registers, attempt 2 succeeds -- previously resolved to attempt 1's dead label, now resolves to attempt 2. Not a regression from the reusable-workflow refactor; the inline version had the same `||` chain. The refactor is why the fix is one file instead of nine. Signed-off-by: Mark Harris Co-authored-by: Claude Opus 5 (1M context) Signed-off-by: Mark Harris --- .github/workflows/start-ec2-runner.yml | 81 +++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 8 deletions(-) diff --git a/.github/workflows/start-ec2-runner.yml b/.github/workflows/start-ec2-runner.yml index f90fd4980..7d9e2c498 100644 --- a/.github/workflows/start-ec2-runner.yml +++ b/.github/workflows/start-ec2-runner.yml @@ -90,11 +90,11 @@ jobs: start: name: Start EC2 runner runs-on: ubuntu-latest - # Exactly one attempt can succeed: each is skipped unless every earlier one - # failed, so at most one of these outputs is non-empty and `||` picks it. + # Taken from the attempt that actually SUCCEEDED -- see the select step + # below for why "first non-empty" is not the same thing. outputs: - label: ${{ steps.a1.outputs.label || steps.a2.outputs.label || steps.a3.outputs.label }} - ec2-instance-id: ${{ steps.a1.outputs.ec2-instance-id || steps.a2.outputs.ec2-instance-id || steps.a3.outputs.ec2-instance-id }} + label: ${{ steps.select.outputs.label }} + ec2-instance-id: ${{ steps.select.outputs.ec2-instance-id }} steps: - name: Stagger job start to avoid API rate limits if: inputs.stagger-seconds > 0 @@ -126,6 +126,19 @@ jobs: label: ${{ inputs.runner-label }} aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} + # The action publishes its outputs as soon as the instance launches, i.e. + # BEFORE it waits for the runner to register. So a failed attempt can still + # have left an instance running -- terminate it rather than leak it. + - name: Clean up the instance left behind by attempt 1 + if: steps.a1.outcome == 'failure' && steps.a1.outputs.ec2-instance-id != '' + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: stop + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + label: ${{ steps.a1.outputs.label }} + ec2-instance-id: ${{ steps.a1.outputs.ec2-instance-id }} + - name: Back off before attempt 2 if: steps.a1.outcome == 'failure' env: @@ -150,6 +163,19 @@ jobs: label: ${{ inputs.runner-label }} aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} + # The action publishes its outputs as soon as the instance launches, i.e. + # BEFORE it waits for the runner to register. So a failed attempt can still + # have left an instance running -- terminate it rather than leak it. + - name: Clean up the instance left behind by attempt 2 + if: steps.a2.outcome == 'failure' && steps.a2.outputs.ec2-instance-id != '' + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: stop + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + label: ${{ steps.a2.outputs.label }} + ec2-instance-id: ${{ steps.a2.outputs.ec2-instance-id }} + - name: Back off before attempt 3 if: steps.a1.outcome == 'failure' && steps.a2.outcome == 'failure' env: @@ -174,10 +200,49 @@ jobs: label: ${{ inputs.runner-label }} aws-resource-tags: ${{ inputs.aws-resource-tags || '[]' }} - - name: Fail if no attempt obtained capacity - if: steps.a1.outcome == 'failure' && steps.a2.outcome == 'failure' && steps.a3.outcome == 'failure' + # The action publishes its outputs as soon as the instance launches, i.e. + # BEFORE it waits for the runner to register. So a failed attempt can still + # have left an instance running -- terminate it rather than leak it. + - name: Clean up the instance left behind by attempt 3 + if: steps.a3.outcome == 'failure' && steps.a3.outputs.ec2-instance-id != '' + continue-on-error: true + uses: machulav/ec2-github-runner@343a1b2ae682e681c3cec9a235d882da17ff04ef # v2.6.1 + with: + mode: stop + github-token: ${{ secrets.EC2_RUNNER_TOKEN }} + label: ${{ steps.a3.outputs.label }} + ec2-instance-id: ${{ steps.a3.outputs.ec2-instance-id }} + + # Pick the outputs of the attempt that succeeded. This must key on the + # step OUTCOME, not on which output is non-empty: an attempt whose + # instance launched but never registered a runner fails with its outputs + # already populated, so "first non-empty" would hand back a dead label -- + # the dependent job would then queue against a runner that never appears + # instead of failing fast. + - name: Select the successful attempt + id: select env: INSTANCE_TYPE: ${{ inputs.instance-type }} + A1_OUTCOME: ${{ steps.a1.outcome }} + A1_LABEL: ${{ steps.a1.outputs.label }} + A1_ID: ${{ steps.a1.outputs.ec2-instance-id }} + A2_OUTCOME: ${{ steps.a2.outcome }} + A2_LABEL: ${{ steps.a2.outputs.label }} + A2_ID: ${{ steps.a2.outputs.ec2-instance-id }} + A3_OUTCOME: ${{ steps.a3.outcome }} + A3_LABEL: ${{ steps.a3.outputs.label }} + A3_ID: ${{ steps.a3.outputs.ec2-instance-id }} run: | - echo "::error::Could not obtain a ${INSTANCE_TYPE} instance after 3 attempts across every configured availability zone; the region is likely out of this instance type." - exit 1 + if [ "${A1_OUTCOME}" = "success" ]; then + LABEL="${A1_LABEL}"; ID="${A1_ID}"; WON=1 + elif [ "${A2_OUTCOME}" = "success" ]; then + LABEL="${A2_LABEL}"; ID="${A2_ID}"; WON=2 + elif [ "${A3_OUTCOME}" = "success" ]; then + LABEL="${A3_LABEL}"; ID="${A3_ID}"; WON=3 + else + echo "::error::Could not obtain a ${INSTANCE_TYPE} instance after 3 attempts across every configured availability zone; the region is likely out of this instance type." + exit 1 + fi + echo "Runner provisioned on attempt ${WON}: ${LABEL} (${ID})" + echo "label=${LABEL}" >> "$GITHUB_OUTPUT" + echo "ec2-instance-id=${ID}" >> "$GITHUB_OUTPUT"