fix: stop the README capture snippet from hiding failed builds - #5
Merged
Conversation
The documented usage was `make build | tee build.log` / `pytest -q | tee test.log`. GitHub's default `run:` shell is `bash -e` without pipefail, so a pipeline reports `tee`'s status: a failing build exits 0, the step goes green, `if: failure()` never fires and the action never runs. Copying the snippet made a red build report as passing. Capture with `shell: bash` (which is `bash -eo pipefail`) and `2>&1`, so the step stays red and stderr-only tools land in the log. The same missing `2>&1` also produced an empty log, where `ci explain` exits 2 and `set -euo pipefail` killed the step with a bare exit code -- a second red step on top of the failure the user was already debugging. A missing or empty log is now a warning annotation naming the cause and the fix, with empty outputs and a green step; only a workflow that configures no log at all still errors. Guards: tests/test_readme_snippets.py parses every README snippet and fails if a capture step loses pipefail or `2>&1`, or tees a file it never triages; tests/test_no_classification.py covers the new path; a smoke-no-log CI job runs the action against an empty and a never-written log on a real runner, and gates the v1 tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The README's flagship snippet is what a maintainer copies into their workflow:
GitHub's default
run:shell isbash -e— no pipefail — so a pipeline reports the status of its last command (tee). A failing test suite exits 0: the step goes green,if: failure()never fires, the action never runs, and the user's red build is reported as passing. Verified:bash -e -c 'false | tee x'→ exit 0;bash -eo pipefail -c 'false | tee x'→ exit 1.The missing
2>&1is the second half. Tools that report only on stderr (compilers, linters, mypy, cargo) leave the log empty,patchrail ci explainexits 2 without writing--out, andset -euo pipefailinaction.ymlkilled the triage step withProcess completed with exit code 2— a second, meaningless red step stacked on the failure the user actually came to debug. A log-path pointing at a file that was never written was worse still: it fell through to::error::provide either log-path or log-text, which is wrong — they did provide it.The fix
shell: bash(bash -eo pipefail) and2>&1, plus a short section explaining why each half matters.guide-url= guide index), and leaves the step green. A workflow that configures no log at all still fails, as it should — the action can never work.Guards
tests/test_readme_snippets.py— parses every YAML snippet in the README and fails if a capture step drops pipefail or2>&1, or tees a file it never hands to the action. Mutation-tested: each of the three regressions turns exactly one test red.tests/test_no_classification.py— the no-log path: exit 0, one-line annotation, empty outputs, summary explains the capture.smoke-no-logCI job — runs the action on a real runner against an empty log and a never-written log, asserts neither fails the job. It's the shell inaction.yml, not the Python, that decides whether the step survives, so this is checked end-to-end. Added tosync-v1'sneeds, so thev1tag only moves when it passes.Happy path is unchanged (verified locally: same class, confidence and guide URL).