fix(ops): make the alert steps actually run - #143
Open
g5n-dev wants to merge 2 commits into
Open
Conversation
Run 30798868809 detected a real outage and told nobody. The publication heartbeat correctly exited 1 after 101 hours without a new post, and the delivery step was skipped. GitHub's default shell is `bash --noprofile --norc -eo pipefail`, which already sets -e. The steps declared `set -uo pipefail`, which adds -u but does not remove -e, so the non-zero exit aborted the step before `code=$?` and the `echo "exit_code=..." >> $GITHUB_OUTPUT` ever ran. The delivery step guards on `steps.<id>.outputs.exit_code != ''`, which was therefore false, so it skipped. The check worked. The notification never existed. That is precisely the failure mode this monitoring was built to eliminate, reintroduced one layer up. The same run shows a second defect: the release-state check was skipped too, because a failed heartbeat aborted the remaining steps. One broken check hid whatever else was wrong. Disable errexit around each measurement so the code is recorded, and let the checks run independently of each other. Four tests pin both properties, since the symptom of getting this wrong is silence — which is indistinguishable from health, and cannot be noticed by watching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
告警通道从未真正工作过
监控运行
30798868809正确检测到了故障，然后没有通知任何人：Issue 一个都没创建。检查是对的，通知从来不存在。
根因是我写的 bug
GitHub Actions 的默认 shell 是
bash --noprofile --norc -eo pipefail——已经带-e。我写的
set -uo pipefail只是追加-u，没有移除-e。于是：投递步骤的条件是
steps.heartbeat.outputs.exit_code != ''，output 从未写入，条件为假，skipped。这正是这套监控要消灭的失败模式，被我在上一层原样重现了一遍。
第二个缺陷
同一次运行显示：心跳失败后，发布状态检查也被跳过。一个检查坏掉会掩盖其余所有问题。
改动
set +e，让退出码能被记录if: always()，检查之间互不阻断为什么要测
这个 bug 的症状是沉默——而沉默与健康无法区分，靠看是发现不了的。所以加了四条测试固定这两个属性：
exit_code的步骤必须有set +eDeliver步骤必须always()全量 1,377 项通过。
说明
这是今天第四次「我自己造出要修的病」：容量报告曾把
CRITICAL_RATIO硬编码成 retention 主动维持的 0.85；这次是让告警在检测成功后静默。共同点是——只验证了检查逻辑，没验证信号能否送达。🤖 Generated with Claude Code