.github: tell a CI-Gate timeout apart from a cancellation - #23203
Open
AskAlexSharov wants to merge 2 commits into
Open
.github: tell a CI-Gate timeout apart from a cancellation#23203AskAlexSharov wants to merge 2 commits into
AskAlexSharov wants to merge 2 commits into
Conversation
A job killed by `timeout-minutes` reports conclusion `cancelled`, the same value an external cancel or a merge-queue reshuffle produces, and its reporting step still runs. The gate therefore blamed the reporting step for the overrun, and a timeout whose reporting step was skipped was absorbed by the reshuffle fast-path and passed the gate. Read the runner's annotation for every cancelled leaf: the message "The job has exceeded the maximum execution time of ..." is the only reliable discriminator. Report those jobs as timeouts with the step that overran, exclude them from the reshuffle fast-path, and keep them out of the root-cause list.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves the CI gate’s handling of cancelled jobs by distinguishing timeout-minutes terminations from external cancellations/merge-queue reshuffles using runner annotations, so the gate can report timeouts accurately and avoid incorrectly passing merge-queue runs.
Changes:
- Add a new test harness helper and new cases to validate timeout-vs-cancel behavior.
- Extend
ci-gate-check.shto read per-job annotations and surface timeouts with the step(s) that overran, while excluding timeouts from the reshuffle fast-path and root-cause list. - Grant
checks: readpermissions to allow fetching check-run annotations.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/workflows/scripts/ci-gate-check.test.sh | Adds output/assertion helper and new scenarios covering timeout annotation parsing and reshuffle fast-path exclusion. |
| .github/workflows/scripts/ci-gate-check.sh | Adds annotation lookup and timeout classification/reporting logic for cancelled leaf jobs. |
| .github/workflows/ci-gate.yml | Adds checks: read permission needed to query annotations. |
Suppressed comments (1)
.github/workflows/scripts/ci-gate-check.sh:67
- The loop passes the Actions job
.idintoannotations_of, butannotations_ofqueries the Checks API (/check-runs/{id}/annotations). On GitHub, workflow-job IDs and check-run IDs are different; the job payload includescheck_run_urlfor the corresponding check run. As written, timeout annotations may never be found, reintroducing the exact misclassification this PR is trying to fix.
timed_out_names="${timed_out_names}${job_name}"$'\n'
timeouts="${timeouts}::error title=CI timeout::${job_name} — exceeded its timeout-minutes budget${overran:+ while running: $overran}"$'\n'
done < <(jq -r '.jobs[] | select(.name != "ci-gate") | select(.conclusion == "cancelled") | "\(.id)\t\(.name)"' <<<"$jobs" 2>/dev/null || true)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+42
to
+48
| annotations_of() { | ||
| if [ -n "${CI_GATE_NO_FETCH:-}" ]; then | ||
| jq -r --arg id "$1" '.[$id] // [] | .[]' <<<"${CI_GATE_ANNOTATIONS_JSON:-{\}}" 2>/dev/null || true | ||
| return 0 | ||
| fi | ||
| gh api "repos/${GITHUB_REPOSITORY}/check-runs/$1/annotations" --jq '.[].message' 2>/dev/null || true | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A job killed by
timeout-minutesreports conclusioncancelled— the same value an external cancel or a merge-queue reshuffle produces — and its reporting step still runs afterwards. Two consequences:The runner's annotation is the only reliable discriminator:
The job has exceeded the maximum execution time of 1h0m0s. The gate now reads it for every cancelled leaf, reports those jobs as timeouts naming the step that overran, excludes them from the reshuffle fast-path, and keeps them out of the root-cause list.checks: readis added for the annotations endpoint.Two of yesterday's nine merge-queue evictions were timeouts (PR 23181
hive-eest (glamsterdam-devnet, serial), 54 min of tests inside a 60 min budget; PR 23164tests-mac-linux (windows-2025, parallel)). Replaying 23181's real run through the new script:against what it printed at the time:
Replaying a runner-drop eviction (PR 23165, annotation
A task was canceled.) confirms it is not mislabelled.bash .github/workflows/scripts/ci-gate-check.test.sh— 14 passed, 0 failed; three cases are new.shellcheckclean.