From b80ab49e4018f170705856f6d4527219aac58e41 Mon Sep 17 00:00:00 2001 From: Merill Fernando Date: Fri, 24 Jul 2026 19:41:11 +1000 Subject: [PATCH 1/2] Gate smoke tests only for pull requests --- .../workflows/maester-action-smoke-test.md | 61 ++++++++++++------- .../workflows/maester-action-smoke-test.yaml | 58 ++++++++++++++---- 2 files changed, 83 insertions(+), 36 deletions(-) diff --git a/.github/workflows/maester-action-smoke-test.md b/.github/workflows/maester-action-smoke-test.md index f658e2711..d3d35e025 100644 --- a/.github/workflows/maester-action-smoke-test.md +++ b/.github/workflows/maester-action-smoke-test.md @@ -16,15 +16,13 @@ fail the workflow. ## Protected environment configuration In `maester365/maester`, create a GitHub Actions environment named exactly -`maester-smoke-test`. Configure its deployment protection rules before adding -credentials: +`maester-smoke-test`. This is the credentials environment used by automatic +trusted-branch runs. Configure it before adding credentials: -1. Add `maester365/core-module`, the core Maester maintainer GitHub team, as - the only required reviewer. Do not add outside collaborators, bots, or - broad contributor teams. -2. Enable **Prevent self-review**. -3. Disable **Allow administrators to bypass configured protection rules**. -4. Restrict deployment branches and tags to the `main` branch. +1. Do not configure required reviewers. Runs from `main`, including every merge, + must start automatically. +2. Disable **Allow administrators to bypass configured protection rules**. +3. Restrict deployment branches and tags to the `main` branch. Create these **environment secrets** on `maester-smoke-test`: @@ -46,10 +44,22 @@ supports workload identity federation and authenticates with GitHub's short-lived OIDC token. The GitHub App private key is used only to mint a short-lived installation token scoped to the private report repository. -GitHub evaluates the environment's required-reviewer and branch rules before -starting each credentialed matrix job. The job cannot read the environment -secrets or request an OIDC token until a core maintainer approves the pending -deployment. +Create a second environment named exactly `maester-smoke-test-pr`. This is an +approval-only gate and must not contain secrets or variables: + +1. Add `maester365/core-module`, the core Maester maintainer GitHub team, as + the only required reviewer. Do not add outside collaborators, bots, or broad + contributor teams. +2. Disable **Prevent self-review** so an authorized core maintainer can approve + a pull request smoke run they initiated. +3. Disable **Allow administrators to bypass configured protection rules**. +4. Restrict deployment branches and tags to the `main` branch. + +GitHub evaluates `maester-smoke-test-pr` before starting the approval job for a +pull request. Only after a core maintainer approves that job can the workflow +continue to the credentialed matrix. Runs caused by pushes to `main`, the +schedule, or a manual dispatch from `main` skip the approval environment and +start automatically. ## Private GitHub report storage @@ -128,25 +138,30 @@ result. ## Trigger and disclosure strategy -- A successful `publish-module-preview` run starts the cross-platform smoke - test against the newly published preview module, then waits for a core - maintainer to approve the protected-environment deployment. +- Every push to `main`, including a merged pull request, starts the + cross-platform smoke test automatically. +- A pull request targeting `main` starts a pending smoke run when it is opened, + updated, reopened, or marked ready for review. A core maintainer must approve + `maester-smoke-test-pr` before the workflow can access the tenant. - A weekly Monday 05:17 UTC run catches action, runner-image, dependency, and - tenant-authentication drift, subject to the same approval. + tenant-authentication drift automatically. - A manual run from `main` can exercise either the `preview` or `latest` module - channel. Use this as the explicit pre-release check; it also requires approval. -- Pull requests and ordinary pushes are not workflow triggers. Manual runs from - another ref fail in the uncredentialed preflight job without requesting - approval or reading secrets. + channel without approval. Manual runs from another ref fail in the + uncredentialed preflight job without reading secrets. +- Pull request runs use `pull_request_target`, so GitHub loads the workflow and + checkout from the trusted default branch. The workflow never checks out, + builds, or executes pull request code. The event associates the published + action smoke test with the pull request; it does not test the pull request's + unmerged implementation. - The federated credential trusts only jobs assigned to the `maester-smoke-test` environment. The environment restricts deployments to - `main` and its only required reviewer is the core maintainer team. + `main`; the separate approval-only environment has no tenant or report + credentials. The workflow has no default token permissions. Only the protected, credentialed job receives `contents: read` and `id-token: write`; the uncredentialed preflight job cannot mint an OIDC token. No trigger or broader -permission is needed for the environment gate: GitHub applies it to manual, -scheduled, and `workflow_run` executions before the job starts. +permission is needed for the pull request gate. The action is pinned to the immutable commit for `maester-action` v1.2.0. Public result summaries, public artifacts, and telemetry remain disabled. diff --git a/.github/workflows/maester-action-smoke-test.yaml b/.github/workflows/maester-action-smoke-test.yaml index 70348320c..dbcbaa4c6 100644 --- a/.github/workflows/maester-action-smoke-test.yaml +++ b/.github/workflows/maester-action-smoke-test.yaml @@ -1,6 +1,11 @@ name: Maester action tenant smoke test on: + push: + branches: [main] + pull_request_target: + branches: [main] + types: [opened, reopened, synchronize, ready_for_review] workflow_dispatch: inputs: maester_version: @@ -15,31 +20,30 @@ on: # Exercise the published preview regularly, even when no module preview was # published during the week. - cron: "17 5 * * 1" - workflow_run: - # Test each successfully published module preview against a real tenant. - workflows: ["publish-module-preview"] - types: [completed] - branches: [main] permissions: {} concurrency: - group: maester-action-tenant-smoke-test - cancel-in-progress: false + # A pull request waiting for approval must not block trusted main runs. + group: maester-action-tenant-smoke-test-${{ github.event.pull_request.number || 'trusted' }} + cancel-in-progress: ${{ github.event_name == 'pull_request_target' }} env: SMOKE_MAESTER_VERSION: ${{ inputs.maester_version || 'preview' }} jobs: preflight: - name: Validate trusted ref - if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' + name: Validate trusted execution context + if: github.event_name != 'pull_request_target' || !github.event.pull_request.draft runs-on: ubuntu-latest timeout-minutes: 5 steps: - - name: Validate trusted ref + - name: Validate trusted execution context shell: bash + env: + EVENT_NAME: ${{ github.event_name }} + PULL_REQUEST_BASE: ${{ github.event.pull_request.base.ref || '' }} run: | set -euo pipefail @@ -47,14 +51,39 @@ jobs: echo "::error title=Untrusted smoke-test ref::Run this workflow from the main branch." exit 1 fi + if [[ "${EVENT_NAME}" == "pull_request_target" && "${PULL_REQUEST_BASE}" != "main" ]]; then + echo "::error title=Unexpected pull request base::Pull request smoke tests must target main." + exit 1 + fi + + approve-pr: + name: Approve tenant access for pull request + needs: preflight + if: github.event_name == 'pull_request_target' + runs-on: ubuntu-latest + timeout-minutes: 5 + environment: + name: maester-smoke-test-pr + deployment: false + permissions: {} + steps: + - name: Record approval + shell: bash + env: + PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} + run: echo "A core maintainer approved tenant access for pull request #${PULL_REQUEST_NUMBER}." smoke: name: Run published action (${{ matrix.os }}) - needs: preflight + needs: [preflight, approve-pr] + if: >- + always() && + needs.preflight.result == 'success' && + (needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped') runs-on: ${{ matrix.os }} timeout-minutes: 30 - # Required reviewers on this environment must approve the deployment before - # the job starts, its secrets are exposed, or an OIDC token can be requested. + # This credentials environment is restricted to main but has no reviewer + # gate. Pull request runs pass the approval-only job above before reaching it. environment: name: maester-smoke-test permissions: @@ -96,6 +125,9 @@ jobs: - name: Check out private report publisher uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: + # pull_request_target resolves github.sha to trusted default-branch + # code. Never check out the pull request head in this privileged job. + ref: ${{ github.sha }} persist-credentials: false - name: Create private report uploader token From 8b43193d61f877f1645105cc51d3db2873069ad6 Mon Sep 17 00:00:00 2001 From: Merill Fernando Date: Fri, 24 Jul 2026 19:57:00 +1000 Subject: [PATCH 2/2] Handle duplicate smoke test results --- .github/workflows/maester-action-smoke-test.md | 4 +++- .github/workflows/maester-action-smoke-test.yaml | 14 ++++++++------ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/.github/workflows/maester-action-smoke-test.md b/.github/workflows/maester-action-smoke-test.md index d3d35e025..46722dbda 100644 --- a/.github/workflows/maester-action-smoke-test.md +++ b/.github/workflows/maester-action-smoke-test.md @@ -11,7 +11,9 @@ review. The check intentionally accepts either `Passed` or `Failed` for `MT.1068`. The test tenant's policy state is not a release gate. Authentication errors, Graph errors, skipped/not-run tests, missing results, or the wrong test/tenant/OS -fail the workflow. +fail the workflow. Maester may emit more than one result instance for the +selected check; every executed result must still be `MT.1068` and must complete +normally. ## Protected environment configuration diff --git a/.github/workflows/maester-action-smoke-test.yaml b/.github/workflows/maester-action-smoke-test.yaml index dbcbaa4c6..f53f43162 100644 --- a/.github/workflows/maester-action-smoke-test.yaml +++ b/.github/workflows/maester-action-smoke-test.yaml @@ -179,19 +179,21 @@ jobs: $tests = @($results.Tests) $executedTests = @($tests | Where-Object Result -ne 'NotRun') $smokeTests = @($tests | Where-Object Id -eq 'MT.1068') + $unexpectedTests = @($executedTests | Where-Object Id -ne 'MT.1068') + $incompleteSmokeTests = @( + $smokeTests | Where-Object { $_.Result -notin @('Passed', 'Failed') } + ) if ([int]$env:TESTS_TOTAL -le 0 -or [int]$env:TESTS_TOTAL -ne [int]$results.TotalCount -or [int]$results.TotalCount -ne $tests.Count) { throw "Maester result counts are inconsistent (action=$($env:TESTS_TOTAL), JSON=$($results.TotalCount), tests=$($tests.Count))." } - if ($executedTests.Count -ne 1 -or $smokeTests.Count -ne 1 -or - $executedTests[0].Id -ne 'MT.1068') { - throw "Expected only MT.1068 to execute, but Maester executed $($executedTests.Count) test(s)." + if ($smokeTests.Count -lt 1 -or $unexpectedTests.Count -gt 0) { + throw "Expected only MT.1068 to execute, but Maester produced $($executedTests.Count) executed result(s), including $($unexpectedTests.Count) unexpected result(s)." } - if ($smokeTests[0].Result -notin @('Passed', 'Failed') -or - [int]$results.ErrorCount -ne 0) { - throw "MT.1068 did not complete normally (result=$($smokeTests[0].Result), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions." + if ($incompleteSmokeTests.Count -gt 0 -or [int]$results.ErrorCount -ne 0) { + throw "MT.1068 did not complete normally (incomplete=$($incompleteSmokeTests.Count), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions." } if ([string]::IsNullOrWhiteSpace($results.TenantName) -or $results.TenantId -ne $env:EXPECTED_TENANT_ID) {