ci(release): check the release people can actually download - #146
Merged
Conversation
The verify command in both READMEs did not work, and 0.5.0 shipped it. `gh attestation verify <zip> -R donislawdev/BeanNetworkTester` answers HTTP 404: it defaults to asking for build provenance, and the archive a user downloads deliberately has none, because a person signs it on their own machine and the attestation workflow refuses to claim a runner built it. What travels with the release is the SBOM attestation over the signed bytes, so the command has to say so. All three occurrences across the two READMEs now name the predicate type, and the prose no longer promises a proof the file does not carry. That is the second time this command has been wrong, and both times a human found it afterwards, because nothing in the suite runs `gh`. So the fix is not only the text: - A new workflow runs on `release: published` and executes the README's own commands, verbatim, against the published assets - then the checksums, the Authenticode signature, the presence of a timestamp, the signing certificate against the pin in beantester/legal.py, and, for a full release, that /releases/latest is this tag, because the site's download button points there. It holds `contents: read` and nothing else. - The existing guard now reads EVERY documented verify command instead of only the one containing `--bundle`. It found the third occurrence in each README a minute after being widened. Two more manual rules become gates while the subject is open: - release.yml refuses to build unless CI succeeded on exactly the tagged commit. It builds and publishes but does not test, so this was the last large step of the recipe kept by memory alone. It asks for the CI workflow by name rather than for a clean sweep of every check, because one check here fails for a licensing reason of its own and would otherwise block every release. - sign_release.py acts on the certificate's expiry date instead of printing it: a warning inside 90 days, a refusal after it, both naming CODESIGN_SHA256, because a renewal is a different certificate and the pin moves with it. - sign_release.py also gained a seventh step that waits for the attestation workflow and confirms the draft is complete - four assets, the published checksums naming the digest that was signed, still a draft. It used to end at "dispatched, go look". Verified against the published v0.5.0 rather than in the abstract: both README commands exit 0 with the predicate type and 404 without it, and the PowerShell step's logic run over the real archive reports Valid, timestamped, and a certificate matching the pin. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…mmand Two guards caught the previous commit, and both were right. The mutation registry pins the documented verify command by removing its predicate type and expecting a test to redden. Its pattern was the flag on its own, which matched exactly once while only one command carried it. Adding the flag to the other two occurrences made that pattern match three places, and a pattern that matches three places proves nothing about any of them. There are two entries now, anchored on `.sigstore.json` and on `-R` - each unique, each version-independent, one for the offline command and one for the online one that actually broke. Both were run for real and both are caught. The other is the no-semicolons rule for README prose: the sentence explaining why `--predicate-type` is required used one. It is two sentences now, in both languages. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The verify command in both READMEs did not work, and 0.5.0 shipped it.
ghdefaults to asking for build provenance. The archive a user downloads deliberately hasnone: a person signs it on their own machine, and
attest-release.ymlrefuses to claim arunner built it. What ships beside the release is the SBOM attestation over the signed bytes,
so the command has to say so. Measured, not assumed - adding
--predicate-type https://spdx.dev/Document/v2.3makes both the online and the offline formexit 0 against the real published release.
This is the second time that command has been wrong, and both times a person found it after
the fact, because nothing in the suite runs
gh. So the text is only half the fix.What changed
The documentation
README.mdandREADME.pl.mdnow name the predicate type, andthe prose no longer promises a proof the file does not carry.
A workflow that checks what people download (
verify-release.yml, onrelease: published)the documentation cannot drift away from something that works.
sha256sum -c,Get-AuthenticodeSignaturewith status Validand a timestamp, the signing certificate's SHA-256 against the pin read out of
beantester/legal.py, and, for a full release, that/releases/latestis this tag - thesite's download button points there.
contents: readand nothing else. Re-runnable byworkflow_dispatchfor any published tag.Two manual rules become gates
release.ymlrefuses to build unless CI succeeded on exactly the tagged commit. It builds andpublishes but does not test, so this was the last large step of the recipe kept by memory. It
asks for the CI workflow by name rather than for a clean sweep, because one check in this
repository fails for a licensing reason of its own and would otherwise block every release.
sign_release.pyacts on the certificate's expiry instead of printing it: a warning inside 90days, a refusal after it, both naming
CODESIGN_SHA256because a renewal is a differentcertificate and the pin moves with it.
And a seventh step for phase B
sign_release.pynow waits for the attestation workflow and confirms the draft is complete:four assets, the checksums downloaded back from the release naming the digest that was
actually signed, still a draft. It used to end at "dispatched, go look" - and a draft missing
one file looks almost exactly like a finished one.
Verification
tests/test_version_and_release.py- 28 passed (25 before; three new guards).a minute of being written, which is the shape of guard worth having.
v0.5.0: exit 0 with the predicate type,HTTP 404without it.Valid, timestamped byCertum, certificate SHA-256 equal to the pin.
release: published,holds
contents: read, and runs on Windows because Authenticode exists nowhere else.ruffandmypyclean on the changed Python.The full suite runs on CI rather than locally.