-
Notifications
You must be signed in to change notification settings - Fork 265
Gate tenant smoke tests only for pull requests #1988
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,6 +1,11 @@ | ||||||||||||||||||||
| name: Maester action tenant smoke test | ||||||||||||||||||||
|
|
||||||||||||||||||||
| on: | ||||||||||||||||||||
| push: | ||||||||||||||||||||
| branches: [main] | ||||||||||||||||||||
| pull_request_target: | ||||||||||||||||||||
| branches: [main] | ||||||||||||||||||||
| types: [opened, reopened, synchronize, ready_for_review] | ||||||||||||||||||||
| workflow_dispatch: | ||||||||||||||||||||
| inputs: | ||||||||||||||||||||
| maester_version: | ||||||||||||||||||||
|
|
@@ -15,46 +20,70 @@ on: | |||||||||||||||||||
| # Exercise the published preview regularly, even when no module preview was | ||||||||||||||||||||
| # published during the week. | ||||||||||||||||||||
| - cron: "17 5 * * 1" | ||||||||||||||||||||
| workflow_run: | ||||||||||||||||||||
| # Test each successfully published module preview against a real tenant. | ||||||||||||||||||||
| workflows: ["publish-module-preview"] | ||||||||||||||||||||
| types: [completed] | ||||||||||||||||||||
| branches: [main] | ||||||||||||||||||||
|
|
||||||||||||||||||||
| permissions: {} | ||||||||||||||||||||
|
|
||||||||||||||||||||
| concurrency: | ||||||||||||||||||||
| group: maester-action-tenant-smoke-test | ||||||||||||||||||||
| cancel-in-progress: false | ||||||||||||||||||||
| # A pull request waiting for approval must not block trusted main runs. | ||||||||||||||||||||
| group: maester-action-tenant-smoke-test-${{ github.event.pull_request.number || 'trusted' }} | ||||||||||||||||||||
| cancel-in-progress: ${{ github.event_name == 'pull_request_target' }} | ||||||||||||||||||||
|
|
||||||||||||||||||||
| env: | ||||||||||||||||||||
| SMOKE_MAESTER_VERSION: ${{ inputs.maester_version || 'preview' }} | ||||||||||||||||||||
|
|
||||||||||||||||||||
| jobs: | ||||||||||||||||||||
| preflight: | ||||||||||||||||||||
| name: Validate trusted ref | ||||||||||||||||||||
| if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' | ||||||||||||||||||||
| name: Validate trusted execution context | ||||||||||||||||||||
| if: github.event_name != 'pull_request_target' || !github.event.pull_request.draft | ||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||
| timeout-minutes: 5 | ||||||||||||||||||||
|
|
||||||||||||||||||||
| steps: | ||||||||||||||||||||
| - name: Validate trusted ref | ||||||||||||||||||||
| - name: Validate trusted execution context | ||||||||||||||||||||
| shell: bash | ||||||||||||||||||||
| env: | ||||||||||||||||||||
| EVENT_NAME: ${{ github.event_name }} | ||||||||||||||||||||
| PULL_REQUEST_BASE: ${{ github.event.pull_request.base.ref || '' }} | ||||||||||||||||||||
| run: | | ||||||||||||||||||||
| set -euo pipefail | ||||||||||||||||||||
|
|
||||||||||||||||||||
| if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then | ||||||||||||||||||||
| echo "::error title=Untrusted smoke-test ref::Run this workflow from the main branch." | ||||||||||||||||||||
| exit 1 | ||||||||||||||||||||
| fi | ||||||||||||||||||||
| if [[ "${EVENT_NAME}" == "pull_request_target" && "${PULL_REQUEST_BASE}" != "main" ]]; then | ||||||||||||||||||||
| echo "::error title=Unexpected pull request base::Pull request smoke tests must target main." | ||||||||||||||||||||
| exit 1 | ||||||||||||||||||||
| fi | ||||||||||||||||||||
|
|
||||||||||||||||||||
| approve-pr: | ||||||||||||||||||||
| name: Approve tenant access for pull request | ||||||||||||||||||||
| needs: preflight | ||||||||||||||||||||
| if: github.event_name == 'pull_request_target' | ||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||
| timeout-minutes: 5 | ||||||||||||||||||||
| environment: | ||||||||||||||||||||
| name: maester-smoke-test-pr | ||||||||||||||||||||
| deployment: false | ||||||||||||||||||||
| permissions: {} | ||||||||||||||||||||
| steps: | ||||||||||||||||||||
| - name: Record approval | ||||||||||||||||||||
| shell: bash | ||||||||||||||||||||
| env: | ||||||||||||||||||||
| PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} | ||||||||||||||||||||
| run: echo "A core maintainer approved tenant access for pull request #${PULL_REQUEST_NUMBER}." | ||||||||||||||||||||
|
|
||||||||||||||||||||
| smoke: | ||||||||||||||||||||
| name: Run published action (${{ matrix.os }}) | ||||||||||||||||||||
| needs: preflight | ||||||||||||||||||||
| needs: [preflight, approve-pr] | ||||||||||||||||||||
| if: >- | ||||||||||||||||||||
| always() && | ||||||||||||||||||||
| needs.preflight.result == 'success' && | ||||||||||||||||||||
| (needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped') | ||||||||||||||||||||
|
Comment on lines
+79
to
+82
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== locate workflow =="
fd -a 'maester-action-smoke-test.yaml' . || true
echo "== workflow around gated step =="
if [ -f .github/workflows/maester-action-smoke-test.yaml ]; then
wc -l .github/workflows/maester-action-smoke-test.yaml
sed -n '1,140p' .github/workflows/maester-action-smoke-test.yaml | cat -n
fi
echo "== related workflow files =="
fd -a '\.yaml$|\.yml$' .github/workflows | sed 's#^\./##' | sort
echo "== search approve and tenant access references =="
rg -n "approve-pr|approve|maester-action|tenant|secrets|permissions:|pull_request_target|cancel-in-progress|if: >-|cancelled\(" .github/workflows || trueRepository: maester365/maester Length of output: 19797 🌐 Web query:
💡 Result: In GitHub Actions, jobs are skipped by default if any of their dependencies (defined in the Citations:
🌐 Web query:
💡 Result: GitHub Docs say If you want, I can show the exact YAML pattern to avoid this. Citations: Make the credentialed gate fail closed and cancellation-aware.
Proposed fix- if: >-
- always() &&
- needs.preflight.result == 'success' &&
- (needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped')
+ if: >-
+ ${{ !cancelled() &&
+ needs.preflight.result == 'success' &&
+ (github.event_name != 'pull_request_target' ||
+ needs.approve-pr.result == 'success') }}📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||
| runs-on: ${{ matrix.os }} | ||||||||||||||||||||
| timeout-minutes: 30 | ||||||||||||||||||||
| # Required reviewers on this environment must approve the deployment before | ||||||||||||||||||||
| # the job starts, its secrets are exposed, or an OIDC token can be requested. | ||||||||||||||||||||
| # This credentials environment is restricted to main but has no reviewer | ||||||||||||||||||||
| # gate. Pull request runs pass the approval-only job above before reaching it. | ||||||||||||||||||||
| environment: | ||||||||||||||||||||
| name: maester-smoke-test | ||||||||||||||||||||
| permissions: | ||||||||||||||||||||
|
|
@@ -96,6 +125,9 @@ jobs: | |||||||||||||||||||
| - name: Check out private report publisher | ||||||||||||||||||||
| uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 | ||||||||||||||||||||
| with: | ||||||||||||||||||||
| # pull_request_target resolves github.sha to trusted default-branch | ||||||||||||||||||||
| # code. Never check out the pull request head in this privileged job. | ||||||||||||||||||||
| ref: ${{ github.sha }} | ||||||||||||||||||||
| persist-credentials: false | ||||||||||||||||||||
|
|
||||||||||||||||||||
| - name: Create private report uploader token | ||||||||||||||||||||
|
|
@@ -147,19 +179,21 @@ jobs: | |||||||||||||||||||
| $tests = @($results.Tests) | ||||||||||||||||||||
| $executedTests = @($tests | Where-Object Result -ne 'NotRun') | ||||||||||||||||||||
| $smokeTests = @($tests | Where-Object Id -eq 'MT.1068') | ||||||||||||||||||||
| $unexpectedTests = @($executedTests | Where-Object Id -ne 'MT.1068') | ||||||||||||||||||||
| $incompleteSmokeTests = @( | ||||||||||||||||||||
| $smokeTests | Where-Object { $_.Result -notin @('Passed', 'Failed') } | ||||||||||||||||||||
| ) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| if ([int]$env:TESTS_TOTAL -le 0 -or | ||||||||||||||||||||
| [int]$env:TESTS_TOTAL -ne [int]$results.TotalCount -or | ||||||||||||||||||||
| [int]$results.TotalCount -ne $tests.Count) { | ||||||||||||||||||||
| throw "Maester result counts are inconsistent (action=$($env:TESTS_TOTAL), JSON=$($results.TotalCount), tests=$($tests.Count))." | ||||||||||||||||||||
| } | ||||||||||||||||||||
| if ($executedTests.Count -ne 1 -or $smokeTests.Count -ne 1 -or | ||||||||||||||||||||
| $executedTests[0].Id -ne 'MT.1068') { | ||||||||||||||||||||
| throw "Expected only MT.1068 to execute, but Maester executed $($executedTests.Count) test(s)." | ||||||||||||||||||||
| if ($smokeTests.Count -lt 1 -or $unexpectedTests.Count -gt 0) { | ||||||||||||||||||||
| throw "Expected only MT.1068 to execute, but Maester produced $($executedTests.Count) executed result(s), including $($unexpectedTests.Count) unexpected result(s)." | ||||||||||||||||||||
| } | ||||||||||||||||||||
| if ($smokeTests[0].Result -notin @('Passed', 'Failed') -or | ||||||||||||||||||||
| [int]$results.ErrorCount -ne 0) { | ||||||||||||||||||||
| throw "MT.1068 did not complete normally (result=$($smokeTests[0].Result), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions." | ||||||||||||||||||||
| if ($incompleteSmokeTests.Count -gt 0 -or [int]$results.ErrorCount -ne 0) { | ||||||||||||||||||||
| throw "MT.1068 did not complete normally (incomplete=$($incompleteSmokeTests.Count), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions." | ||||||||||||||||||||
| } | ||||||||||||||||||||
| if ([string]::IsNullOrWhiteSpace($results.TenantName) -or | ||||||||||||||||||||
| $results.TenantId -ne $env:EXPECTED_TENANT_ID) { | ||||||||||||||||||||
|
|
||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: maester365/maester
Length of output: 50375
🏁 Script executed:
Repository: maester365/maester
Length of output: 8501
Align draft PR handling with the documented trigger contract.
pull_request_targetincludes draft PRs (github.event.pull_request.draftexists), but this condition skipspreflight, andapprove-prdepends on it vianeeds. Draft PRs therefore do not create the approval/smoke run described for opened or synchronized external PRs. Remove!github.event.pull_request.draftor explicitly document that draft PRs are intentionally excluded.🤖 Prompt for AI Agents