Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 41 additions & 24 deletions .github/workflows/maester-action-smoke-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,20 @@ review.
The check intentionally accepts either `Passed` or `Failed` for `MT.1068`.
The test tenant's policy state is not a release gate. Authentication errors,
Graph errors, skipped/not-run tests, missing results, or the wrong test/tenant/OS
fail the workflow.
fail the workflow. Maester may emit more than one result instance for the
selected check; every executed result must still be `MT.1068` and must complete
normally.

## Protected environment configuration

In `maester365/maester`, create a GitHub Actions environment named exactly
`maester-smoke-test`. Configure its deployment protection rules before adding
credentials:
`maester-smoke-test`. This is the credentials environment used by automatic
trusted-branch runs. Configure it before adding credentials:

1. Add `maester365/core-module`, the core Maester maintainer GitHub team, as
the only required reviewer. Do not add outside collaborators, bots, or
broad contributor teams.
2. Enable **Prevent self-review**.
3. Disable **Allow administrators to bypass configured protection rules**.
4. Restrict deployment branches and tags to the `main` branch.
1. Do not configure required reviewers. Runs from `main`, including every merge,
must start automatically.
2. Disable **Allow administrators to bypass configured protection rules**.
3. Restrict deployment branches and tags to the `main` branch.

Create these **environment secrets** on `maester-smoke-test`:

Expand All @@ -46,10 +46,22 @@ supports workload identity federation and authenticates with GitHub's
short-lived OIDC token. The GitHub App private key is used only to mint a
short-lived installation token scoped to the private report repository.

GitHub evaluates the environment's required-reviewer and branch rules before
starting each credentialed matrix job. The job cannot read the environment
secrets or request an OIDC token until a core maintainer approves the pending
deployment.
Create a second environment named exactly `maester-smoke-test-pr`. This is an
approval-only gate and must not contain secrets or variables:

1. Add `maester365/core-module`, the core Maester maintainer GitHub team, as
the only required reviewer. Do not add outside collaborators, bots, or broad
contributor teams.
2. Disable **Prevent self-review** so an authorized core maintainer can approve
a pull request smoke run they initiated.
3. Disable **Allow administrators to bypass configured protection rules**.
4. Restrict deployment branches and tags to the `main` branch.

GitHub evaluates `maester-smoke-test-pr` before starting the approval job for a
pull request. Only after a core maintainer approves that job can the workflow
continue to the credentialed matrix. Runs caused by pushes to `main`, the
schedule, or a manual dispatch from `main` skip the approval environment and
start automatically.

## Private GitHub report storage

Expand Down Expand Up @@ -128,25 +140,30 @@ result.

## Trigger and disclosure strategy

- A successful `publish-module-preview` run starts the cross-platform smoke
test against the newly published preview module, then waits for a core
maintainer to approve the protected-environment deployment.
- Every push to `main`, including a merged pull request, starts the
cross-platform smoke test automatically.
- A pull request targeting `main` starts a pending smoke run when it is opened,
updated, reopened, or marked ready for review. A core maintainer must approve
`maester-smoke-test-pr` before the workflow can access the tenant.
- A weekly Monday 05:17 UTC run catches action, runner-image, dependency, and
tenant-authentication drift, subject to the same approval.
tenant-authentication drift automatically.
- A manual run from `main` can exercise either the `preview` or `latest` module
channel. Use this as the explicit pre-release check; it also requires approval.
- Pull requests and ordinary pushes are not workflow triggers. Manual runs from
another ref fail in the uncredentialed preflight job without requesting
approval or reading secrets.
channel without approval. Manual runs from another ref fail in the
uncredentialed preflight job without reading secrets.
- Pull request runs use `pull_request_target`, so GitHub loads the workflow and
checkout from the trusted default branch. The workflow never checks out,
builds, or executes pull request code. The event associates the published
action smoke test with the pull request; it does not test the pull request's
unmerged implementation.
- The federated credential trusts only jobs assigned to the
`maester-smoke-test` environment. The environment restricts deployments to
`main` and its only required reviewer is the core maintainer team.
`main`; the separate approval-only environment has no tenant or report
credentials.

The workflow has no default token permissions. Only the protected,
credentialed job receives `contents: read` and `id-token: write`; the
uncredentialed preflight job cannot mint an OIDC token. No trigger or broader
permission is needed for the environment gate: GitHub applies it to manual,
scheduled, and `workflow_run` executions before the job starts.
permission is needed for the pull request gate.

The action is pinned to the immutable commit for `maester-action` v1.2.0.
Public result summaries, public artifacts, and telemetry remain disabled.
Expand Down
72 changes: 53 additions & 19 deletions .github/workflows/maester-action-smoke-test.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
name: Maester action tenant smoke test

on:
push:
branches: [main]
pull_request_target:
branches: [main]
types: [opened, reopened, synchronize, ready_for_review]
workflow_dispatch:
inputs:
maester_version:
Expand All @@ -15,46 +20,70 @@ on:
# Exercise the published preview regularly, even when no module preview was
# published during the week.
- cron: "17 5 * * 1"
workflow_run:
# Test each successfully published module preview against a real tenant.
workflows: ["publish-module-preview"]
types: [completed]
branches: [main]

permissions: {}

concurrency:
group: maester-action-tenant-smoke-test
cancel-in-progress: false
# A pull request waiting for approval must not block trusted main runs.
group: maester-action-tenant-smoke-test-${{ github.event.pull_request.number || 'trusted' }}
cancel-in-progress: ${{ github.event_name == 'pull_request_target' }}

env:
SMOKE_MAESTER_VERSION: ${{ inputs.maester_version || 'preview' }}

jobs:
preflight:
name: Validate trusted ref
if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
name: Validate trusted execution context
if: github.event_name != 'pull_request_target' || !github.event.pull_request.draft

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow file =="
if [ -f .github/workflows/maester-action-smoke-test.yaml ]; then
  cat -n .github/workflows/maester-action-smoke-test.yaml
else
  echo "workflow file not found"
fi

echo "== relevant docs references in repo =="
rg -n "draft|pending approval|approval|opened|synchronized|pull_request_target" -S . || true

echo "== git diff stat/name =="
git diff --stat || true
git diff -- .github/workflows/maester-action-smoke-test.yaml || true

Repository: maester365/maester

Length of output: 50375


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow existence =="
git ls-files | grep -F '.github/workflows/maester-action-smoke-test.yaml' || true

echo "== workflow file first 140 lines =="
if [ -f .github/workflows/maester-action-smoke-test.yaml ]; then
  sed -n '1,140p' .github/workflows/maester-action-smoke-test.yaml | cat -n
fi

echo "== job dependency/dependency-related docs references in workflow =="
if [ -f .github/workflows/maester-action-smoke-test.yaml ]; then
  rg -n "needs:|if:|always\\(|preflight|approval|smoke|pull_request_target|draft|on:" .github/workflows/maester-action-smoke-test.yaml
fi

Repository: maester365/maester

Length of output: 8501


Align draft PR handling with the documented trigger contract.

pull_request_target includes draft PRs (github.event.pull_request.draft exists), but this condition skips preflight, and approve-pr depends on it via needs. Draft PRs therefore do not create the approval/smoke run described for opened or synchronized external PRs. Remove !github.event.pull_request.draft or explicitly document that draft PRs are intentionally excluded.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/maester-action-smoke-test.yaml at line 37, Update the
job-level if condition in the workflow to stop excluding draft pull requests by
removing the github.event.pull_request.draft check. Preserve the existing
pull_request_target versus other-event logic so preflight runs for opened or
synchronized external PRs and allows approve-pr to proceed.

runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- name: Validate trusted ref
- name: Validate trusted execution context
shell: bash
env:
EVENT_NAME: ${{ github.event_name }}
PULL_REQUEST_BASE: ${{ github.event.pull_request.base.ref || '' }}
run: |
set -euo pipefail

if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then
echo "::error title=Untrusted smoke-test ref::Run this workflow from the main branch."
exit 1
fi
if [[ "${EVENT_NAME}" == "pull_request_target" && "${PULL_REQUEST_BASE}" != "main" ]]; then
echo "::error title=Unexpected pull request base::Pull request smoke tests must target main."
exit 1
fi

approve-pr:
name: Approve tenant access for pull request
needs: preflight
if: github.event_name == 'pull_request_target'
runs-on: ubuntu-latest
timeout-minutes: 5
environment:
name: maester-smoke-test-pr
deployment: false
permissions: {}
steps:
- name: Record approval
shell: bash
env:
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
run: echo "A core maintainer approved tenant access for pull request #${PULL_REQUEST_NUMBER}."

smoke:
name: Run published action (${{ matrix.os }})
needs: preflight
needs: [preflight, approve-pr]
if: >-
always() &&
needs.preflight.result == 'success' &&
(needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped')
Comment on lines +79 to +82

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate workflow =="
fd -a 'maester-action-smoke-test.yaml' . || true

echo "== workflow around gated step =="
if [ -f .github/workflows/maester-action-smoke-test.yaml ]; then
  wc -l .github/workflows/maester-action-smoke-test.yaml
  sed -n '1,140p' .github/workflows/maester-action-smoke-test.yaml | cat -n
fi

echo "== related workflow files =="
fd -a '\.yaml$|\.yml$' .github/workflows | sed 's#^\./##' | sort

echo "== search approve and tenant access references =="
rg -n "approve-pr|approve|maester-action|tenant|secrets|permissions:|pull_request_target|cancel-in-progress|if: >-|cancelled\(" .github/workflows || true

Repository: maester365/maester

Length of output: 19797


🌐 Web query:

GitHub Actions workflow syntax always() skipped jobs cancel-in-progress jobs.context needs result skipped

💡 Result:

In GitHub Actions, jobs are skipped by default if any of their dependencies (defined in the needs property) fail or are skipped [1][2]. This behavior applies to the entire dependency chain from the point of failure or skip onwards [1][3][2]. To force a job to run regardless of the status of its dependencies—including when they are skipped—you must use the always() conditional function in the job-level if expression [1][2]. Key Considerations: 1. Default Behavior: By default, if a job has needs dependencies, an implicit success() status check is applied [4][5]. If any job in the dependency chain is skipped, the dependent job is also skipped because the default success() check evaluates to false [1][2][5]. 2. Using always: When you include if: ${{ always() }}, it overrides this default behavior, allowing the job to execute even if preceding jobs were skipped or failed [1][3][2]. 3. Context Access: You can check the specific status of dependencies using the needs context, which includes the result property [6]. Possible values for needs.<job_id>.result are success, failure, cancelled, or skipped [6]. If you need a job to run only under specific conditions (e.g., only if a previous job was skipped), you can combine always() with a check on the result property [7][4][5]: if: ${{ always && needs.job_a.result == 'skipped' }} 4. Dependency Chains: If you have a long chain of jobs, simply using always() on the final job is often insufficient if intermediate jobs in the chain were also skipped [3]. Every job in a dependency chain that you intend to run after a skip or failure typically requires its own if: ${{ always() }} condition to ensure it is not automatically skipped by the dependency chain rules [3]. Top results supporting this information: [1][3][2][6][4]

Citations:


🌐 Web query:

site:docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow always() cancel-in-progress

💡 Result:

GitHub Docs say always() can make a dependent job run even if earlier jobs fail, but during workflow cancellation GitHub re-evaluates if conditions and a job with if: always() will continue running instead of being canceled. cancel-in-progress: true is part of concurrency and cancels in-progress runs in the same concurrency group. So always() and cancel-in-progress can conflict if you expect a job to stop on cancellation. (docs.github.com)

If you want, I can show the exact YAML pattern to avoid this.

Citations:


Make the credentialed gate fail closed and cancellation-aware.

smoke currently reaches tenant secrets whenever approve-pr is skipped, which the workflow currently allows for non-PR events. Include always() only for non-PR runs, require needs.approve-pr.result == 'success' for pull_request_target, and guard the whole condition with !cancelled() so PR cancellation does not let privileged jobs continue or start.

Proposed fix
-    if: >-
-      always() &&
-      needs.preflight.result == 'success' &&
-      (needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped')
+    if: >-
+      ${{ !cancelled() &&
+        needs.preflight.result == 'success' &&
+        (github.event_name != 'pull_request_target' ||
+         needs.approve-pr.result == 'success') }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if: >-
always() &&
needs.preflight.result == 'success' &&
(needs.approve-pr.result == 'success' || needs.approve-pr.result == 'skipped')
if: >-
${{ !cancelled() &&
needs.preflight.result == 'success' &&
(github.event_name != 'pull_request_target' ||
needs.approve-pr.result == 'success') }}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/maester-action-smoke-test.yaml around lines 79 - 82,
Update the smoke job’s credentialed gate condition to be cancellation-aware by
requiring !cancelled(), allow always() only for non-pull_request_target runs,
and require needs.approve-pr.result == 'success' for pull_request_target events.
Preserve the existing preflight success requirement while ensuring skipped
approval cannot grant credential access to PR-triggered runs.

runs-on: ${{ matrix.os }}
timeout-minutes: 30
# Required reviewers on this environment must approve the deployment before
# the job starts, its secrets are exposed, or an OIDC token can be requested.
# This credentials environment is restricted to main but has no reviewer
# gate. Pull request runs pass the approval-only job above before reaching it.
environment:
name: maester-smoke-test
permissions:
Expand Down Expand Up @@ -96,6 +125,9 @@ jobs:
- name: Check out private report publisher
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
# pull_request_target resolves github.sha to trusted default-branch
# code. Never check out the pull request head in this privileged job.
ref: ${{ github.sha }}
persist-credentials: false

- name: Create private report uploader token
Expand Down Expand Up @@ -147,19 +179,21 @@ jobs:
$tests = @($results.Tests)
$executedTests = @($tests | Where-Object Result -ne 'NotRun')
$smokeTests = @($tests | Where-Object Id -eq 'MT.1068')
$unexpectedTests = @($executedTests | Where-Object Id -ne 'MT.1068')
$incompleteSmokeTests = @(
$smokeTests | Where-Object { $_.Result -notin @('Passed', 'Failed') }
)

if ([int]$env:TESTS_TOTAL -le 0 -or
[int]$env:TESTS_TOTAL -ne [int]$results.TotalCount -or
[int]$results.TotalCount -ne $tests.Count) {
throw "Maester result counts are inconsistent (action=$($env:TESTS_TOTAL), JSON=$($results.TotalCount), tests=$($tests.Count))."
}
if ($executedTests.Count -ne 1 -or $smokeTests.Count -ne 1 -or
$executedTests[0].Id -ne 'MT.1068') {
throw "Expected only MT.1068 to execute, but Maester executed $($executedTests.Count) test(s)."
if ($smokeTests.Count -lt 1 -or $unexpectedTests.Count -gt 0) {
throw "Expected only MT.1068 to execute, but Maester produced $($executedTests.Count) executed result(s), including $($unexpectedTests.Count) unexpected result(s)."
}
if ($smokeTests[0].Result -notin @('Passed', 'Failed') -or
[int]$results.ErrorCount -ne 0) {
throw "MT.1068 did not complete normally (result=$($smokeTests[0].Result), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions."
if ($incompleteSmokeTests.Count -gt 0 -or [int]$results.ErrorCount -ne 0) {
throw "MT.1068 did not complete normally (incomplete=$($incompleteSmokeTests.Count), errors=$($results.ErrorCount)). Check OIDC trust and Graph application permissions."
}
if ([string]::IsNullOrWhiteSpace($results.TenantName) -or
$results.TenantId -ne $env:EXPECTED_TENANT_ID) {
Expand Down