ci: file an issue when a scheduled run goes red, and close it when it clears - #257
Conversation
… clears A red cron is invisible in a way a red pull request is not: nothing is blocked, nobody is requested, and the run sits in the Actions tab looking exactly like the green one above it. The placeholder e2e failure was red on `main` for six consecutive nights (2026-08-26 through 2026-08-31) before anyone opened the run, and the scheduled release of 2026-08-24 failed the same silent way. GitHub does email the cron's owner, which is one person, one message per failure, and the sixth is indistinguishable from the first. So both scheduled workflows now end in an alarm job that files the failure where the project already looks every day -- the issue tracker. `.github/alarm.sh` opens one issue per title, comments on it rather than opening a second when the next run also fails, and closes it when a scheduled run goes green. That last part is what keeps it a signal: an alarm that only ever fires is the same background noise as the emails, and an open issue always means "the last scheduled run was red" rather than "somebody forgot to tidy up". The script owns exactly one issue, matched on exact title through the REST list rather than `gh issue list --search`, which ranks and would eventually hand back somebody's bug report. Pull requests are filtered out of that list because they share the number space. Only `schedule` raises it -- a run you dispatched from the Actions tab is one you are already watching -- and a `cancelled` or `skipped` job leaves the alarm untouched, since neither is evidence either way. One script for both workflows, for the reason ci.yml already shells out to `scripts/verify.mjs` instead of listing its steps: two copies drift. Documented in CONTRIBUTING.md (what a red nightly does now), docs/releasing.md (step 7 of the weekly run), and docs/agents/issue-tracker.md -- the last because an agent triaging the tracker will meet these two issues and must not retitle or hand-close them. Signed-off-by: Peyton Nowlin <peytonn98@googlemail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 897e87fc4c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,98 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
The full message for commit 769b35920ba89767cd02f3ecc88cdf5f4032624c has no Signed-off-by: trailer, so the repository's dco job will fail this change regardless of the workflow implementation; recreate the commit using the contributor's real, author-matching name and email and include the sign-off.
AGENTS.md reference: AGENTS.md:L7-L11
Useful? React with 👍 / 👎.
| alarm: | ||
| name: Nightly alarm |
There was a problem hiding this comment.
This new job changes contributor-facing repository behavior by automatically creating, updating, and closing tracker issues, but the commit leaves CHANGELOG.md unchanged. Add an entry under ## Unreleased as required for behavior changes rather than limiting the documentation to the contributor and release guides.
AGENTS.md reference: AGENTS.md:L59-L69
Useful? React with 👍 / 👎.
| gh api "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \ | ||
| --jq '[.[] | select(.pull_request == null) | select(.title == env.TITLE) | .number] | first // empty' |
There was a problem hiding this comment.
Once the repository has more than 100 open issues and the existing alarm falls outside the first REST page, this lookup returns empty, causing a red run to create a duplicate and a later green run to leave the original alarm open. I checked gh help api; --paginate is the flag that makes additional requests for all result pages, so the query needs pagination before concluding that no exact-title issue exists.
Useful? React with 👍 / 👎.
1 participant
Follow-up to #256, which fixed the test that was red. This is about the six nights nobody noticed.
The gap
A failing cron is invisible in a way a failing pull request is not — nothing is blocked, nobody is requested, and the run sits in the Actions tab looking exactly like the green one above it.
mainCI was red on six consecutive nightlies, 2026-08-26 through 2026-08-31, on one Firefox assertion (e2e: [firefox] placeholder assertion reads attr() unresolved from getComputedStyle, and it is why main is red #254).GitHub does email the cron's owner. That is one person, one message per failure, and the sixth is indistinguishable from the first.
What this adds
An
alarmjob at the end of both scheduled workflows, sharing.github/alarm.sh:failure, no alarm openNightly CI is redorThe Monday release is redfailure, alarm already opensuccesscancelled/skippedSelf-clearing is the point. An alarm that only ever fires becomes the same background noise as the emails, and noise is what let six nights pass. While an issue is open, the last scheduled run was red; the comment count is how many runs have failed since.
Design notes
gh issue list --search, which ranks by relevance and would eventually hand back somebody's bug report. Pull requests are filtered out of that list because they share the number space with issues.ci.ymlalready shells out toscripts/verify.mjsrather than listing its own steps: two hand-maintained copies drift.scheduleonly. A run dispatched from the Actions tab is one somebody is already watching; filing an issue at them would be the noise this is trying to avoid.if: always()on the job, because the case it exists for is precisely the one where the job itneedsfailed — without it the dependency is skipped exactly when it has something to say.issues: writeandcontents: readfor itself; neither workflow's other jobs gain anything.github.tokenand theghCLI already present on the runner.Verification
The issue lookup was exercised read-only against this repo: the jq query returns empty for a title with no open issue, and
243for the exact title of #243.bash -non the script, and both workflows parse.The alarm paths themselves cannot run on a pull request by construction (
github.event_name == 'schedule'). The first real exercise is tomorrow's nightly — which, with #256 merged, should be the green path that closes nothing because nothing is open. If you would rather see it fire once first, say so and I will temporarily widen the guard on a throwaway branch.Documentation
CONTRIBUTING.md— what a red nightly does now, next to the existing note that Firefox and WebKit only run nightly.docs/releasing.md— step 7 of the weekly run.docs/agents/issue-tracker.md— a new section, because an agent triaging this tracker will meet these two issues and must not retitle or hand-close them.No
CHANGELOG.mdentry: nothing ships to npm and no published behaviour changes.