Skip to content

fix(release): close the three ways this repo's releases lied about themselves - #45

Merged
NSchatz merged 4 commits into
mainfrom
fix/release-pipeline-honesty
Aug 4, 2026
Merged

fix(release): close the three ways this repo's releases lied about themselves#45
NSchatz merged 4 commits into
mainfrom
fix/release-pipeline-honesty

Conversation

@NSchatz

@NSchatz NSchatz commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

config was not on the shared cosyte/.github release pipeline, and its releases showed it. Three
defects, all the same class: a mechanism asserting coverage it did not have.

1. An empty changeset made the pipeline publish nothing and exit green

changesets/action given only changesets that resolve to zero package bumps logs
All changesets are empty; not creating PR, opens no Version PR, publishes nothing, and exits 0.
Run 30640138565 (2026-07-31) did exactly that: approved through the protected release environment
as a real publish, reported success, and shipped none of the six packages whose manifests were
already a patch ahead of the registry. cf07086 (#41) deleted the offending file and wrote the
finding down. That fixed the instance and left the class.

scripts/changeset-guard.mjs now refuses it, per file, and runs in both ci.yml and release.yml.
Three shapes bump nothing and only the first is what the action calls empty, all three read off
the vendored @changesets/parse@0.4.3 source rather than inferred from the log line:

  • frontmatter declaring no packages. The parser does not throw: yaml.load of an empty block
    returns falsy and it sets releases = []. The file parses cleanly and carries a human summary, so
    it looks entirely normal in a diff.
  • every entry typed none. none is a valid type, so the releases list is non-empty and the
    action's own emptiness check never fires. It opens a Version PR that changes no version.
  • a misspelled package name.

none alongside a real bump is still accepted, because that is what none is for.

Negative control

test/changeset-guard.test.ts, 16 cases, committed rather than run once by hand. It drives the
shipped CLI with execFileSync and asserts the process exit code, because that is what the
workflow depends on and an exported function returning { ok: false } proves nothing about it. This
repo has already been bitten by exactly that gap (#42, where attw's wrapper exited 0 on an untyped
pack). Both directions are pinned, and the pair is the evidence:

input required
inert changeset (empty frontmatter) exit 1
real changeset exit 0
inert alongside a real one exit 1 (graded per file, not per repo)
all-none, including "none" and 'none' exit 1
none next to a real bump exit 0
empty .changeset/ (the publish arm) exit 0
guard cannot run exit 2, never 1

A guard that refuses everything fails the second row; one that refuses nothing fails the first. The
2-versus-1 split matters because a broken gate and a caught defect must not be one signal in CI.

2. Release bodies were the raw CHANGELOG file

changesets/action defaults createGithubReleases to true and builds each body by finding a
## <version> heading in that package's CHANGELOG.md. Every package here sets "changelog": false
and hand-maintains its changelog, so changeset version writes no such heading, the action finds
none, and its fallback is the whole file. On 2026-07-31 all six bodies published as the raw
CHANGELOG.md, # Changelog preamble and ## [Unreleased] included. They were corrected by hand,
which is not a gate.

The flag is now false, and scripts/release-notes.mjs supplies the replacement, derived from the
changesets the version commit consumed. Not from the changelog, deliberately: that would need a
## [0.0.6] heading for a version that does not exist when the changeset is written, and with the
generator disabled nothing writes it, so the gate would refuse every release until a human predicted
the next version by hand.

Every refusal runs before npm is touched, because the changesets are in the tree at HEAD^. A
published version is permanent (ADR 0001), so a check after the publish is a complaint, not a gate.

3. The six packages/*/CHANGELOG.md headed shipped content [Unreleased]

Each was byte-identical to its own latest release tag, so every byte under that heading had
already shipped and would have republished. Each section now carries the version it shipped in, dated
from that release's tag. @cosyte/vitest-config needed a two-way split, its boundary read off
the @cosyte/vitest-config@0.0.2 tag rather than guessed.

The generator flag was NOT flipped. "changelog": false in .changeset/config.json is
untouched. The six files were corrected by hand; the mechanism that produced them is unchanged.
I am aware the founder has since decided the generator goes on across the eight repos, and that is
deliberately still one separate act, not this PR.

Why this is not a thin caller of the shared workflow

Putting config on the shared pipeline was the intended remedy. It cannot serve this repo,
measured on cosyte/.github's main at 1e634f0, the ref a caller would pin, and re-checked as
that repo moved during this slice:

  1. It would withhold every config publish, permanently, on a green run. Its gate answers "is a
    release pending" from the root package.json's version. This repo's root is cosyte-config,
    private: true, pinned at 0.0.0, and Changesets does not version a private root package, so it
    has never moved. Running the shared prepare against this repo returns is-release=false, code
    never-versioned, and the shared workflow supplies publish: only when that is true. That is a
    strictly worse instance of defect 1.
  2. It hardcodes tag="v${version}", and says why: "Every caller of this workflow is a
    single-package repo." This repo's fourteen tags are all <pkg>@<version>, and six packages
    publishing in one run would collide on a single tag.

Neither reason depends on an input: it offers no multi-package mode and no tag override. So the two
portable halves were ported instead: the
RELEASE_PR_TOKEN wiring and a notes gate rebuilt for six packages. release.yml records what to
delete if the shared workflow ever grows a multi-package mode.

RELEASE_PR_TOKEN

Both halves, because fixing only the first works until the second changeset. The token goes in the
action's env (it reads process.env.GITHUB_TOKEN || core.getInput("github-token"), so the env wins
and the input alone would be a silent no-op), and the checkout sets persist-credentials: false,
because the version commit is pushed by git push out of that checkout and the persisted
http.<host>.extraheader otherwise outranks the ~/.netrc the action writes. Absent, it falls back
to GITHUB_TOKEN and warns loudly rather than failing closed, which would take the release path
down to protect against a state the repo is already in.

What the gate caught

gate-refuter REFUTED the first draft. The finding was the same defect class this PR exists to
close, arrived at through the fix for it:

createGithubReleases: false means git.pushTag is never reached, so the post-publish step is the
only thing that creates a tag. Keyed on published == 'true': six packages publish,
gh release create fails on the third, the run reds, an operator re-runs, changeset publish finds
everything already on npm and exits 0, so published is false, so the step is skipped, so the
run goes GREEN with four packages on npm carrying no tag and no release, permanently.

Not a base regression: with the flag at its default the action pushed the tag inside its own call, so
base lost a release body. Losing the tag was new. The step is now driven by what the version
commit bumped, runs on !cancelled(), and asks the registry whether each package is actually
there, so a re-run completes the job instead of skipping it. Exercised with stubbed npm/gh across
six cases (happy path, the refuted re-run, partial publish, mid-loop gh failure, empty package
list, missing body); only the first two are green.

Pass 2 returned NOT REFUTED, and still caught the correction overreaching: the trailing-comment
strip had turned a safe exit 2 into an exit 0 on an inert file, because "none" was compared as a
raw token while js-yaml strips the quotes. Fixed and pinned by a control over three spellings.

Blast radius

Every other cosyte repo inherits this toolchain, so this is the number that matters: nothing any
consumer runs changes.
No dependency, no lockfile line, no rule, setting, compiler option, build
option or runner behaviour. The only bytes that change in a published tarball are CHANGELOG.md
inside the six, which is what the changeset describes. package.json gains two script entries and
nothing else.

changesets/action is pinned to a45c4d5, the sha cosyte/.github pins and the sha the pushTag
behaviour was measured at, because this workflow now depends on that internal.

Verification

  • 82 root + 75 package tests pass. actionlint clean. check:no-emdash, format:check,
    typecheck, lint, build, attw green.
  • verify.sh config is red on pnpm audit and pnpm licenses only, both of which are red on
    base and neither of which CI runs. The delta is provably zero: those two read the lockfile and
    dependency manifests, and this branch changes neither. The failures come from tsup > postcss and
    CC-BY-3.0 in spdx-exceptions.
  • The full path was rehearsed end to end in a throwaway clone: real changeset version (six bumps,
    changeset consumed), then real prepare, producing six correct bodies with zero CHANGELOG-dump
    fingerprints.

Deliberately deferred

  • format:check's glob covers .js but not .mjs, so scripts/*.mjs is unformatted on base.
    Widening it would reformat scaffold-parser.mjs and add blast radius to this PR.
  • The guard exits 2 on a YAML flow map or a block scalar splitting the pair across lines. Loud, in
    the safe direction, and closing it means taking a YAML dependency for a shape nothing here writes.
  • The npm view retry budget is 3 attempts over 10s. Thin for a CDN-fronted packument, but a re-run
    is idempotent and green, so this was left rather than hardened after a passing verdict.
  • A catch-up publish on a non-version commit now reds and creates no tag, where base would have
    tagged it with a bad body. Loud and recoverable, but a real capability change worth a backlog line.

NSchatz added 4 commits August 4, 2026 14:19
…emselves

`config` is not on the shared `cosyte/.github` pipeline, and its releases
showed it. Three defects, all of the same class: a mechanism asserting
coverage it did not have.

1. AN EMPTY CHANGESET MAKES `changesets/action` PUBLISH NOTHING AND EXIT 0.
   Run 30640138565 was approved through the protected `release` environment
   as a real publish, reported success, and shipped none of the six packages
   already a patch ahead of the registry. `cf07086` deleted the offending
   file; the class stayed open. `scripts/changeset-guard.mjs` now refuses it,
   per file, before install and before an approver is asked for anything.
   Three shapes bump nothing and only the first is what the action calls
   empty: frontmatter declaring no packages (@changesets/parse does not throw
   on this, it sets `releases = []`), every entry typed `none` (a VALID type,
   so the action's own emptiness check never fires), and a misspelled name.
   The negative control is committed, drives the shipped CLI, and asserts the
   process exit code in both directions.

2. RELEASE BODIES WERE THE RAW CHANGELOG FILE. With `"changelog": false`,
   `changeset version` writes no `## <version>` heading, so
   `createGithubReleases: true` found none and fell back to the whole file:
   all six bodies published with the `# Changelog` preamble and
   `## [Unreleased]` included. The flag is now false and
   `scripts/release-notes.mjs` derives one body per package from the
   changesets the version commit consumed, all of it before npm is touched.

3. THE SIX `packages/*/CHANGELOG.md` HEADED SHIPPED CONTENT `[Unreleased]`.
   Each was byte-identical to its own latest release tag, so every byte under
   that heading had already shipped and would have republished. Corrected by
   hand; `vitest-config` needed a two-way split read off its 0.0.2 tag.

`"changelog": false` is UNTOUCHED. Turning the generator on is the
founder-owned call tracked as CHANGELOG-PREAMBLE-FUTURE-TENSE, and it changes
the shape of a shipped file in eight repos at once. The six files were fixed;
the mechanism that produced them was not, and that call stays open.

PUTTING `config` ON THE SHARED PIPELINE WAS THE INTENDED REMEDY AND IT CANNOT
SERVE THIS REPO, measured against `cosyte/.github` at `b222d21`:

  * Its gate reads the ROOT package.json's version. This repo's root is
    `cosyte-config`, private, pinned at 0.0.0, and Changesets does not version
    a private root package, so it has never moved. Running the shared
    `prepare` here returns `is-release=false`, code `never-versioned`, and the
    shared workflow supplies `publish:` only when that is true. Adopting it
    would withhold EVERY config publish, permanently, on a green run: a
    strictly worse instance of defect 1.
  * It hardcodes `tag="v${version}"`, and says why: "Every caller of this
    workflow is a single-package repo." This repo's fourteen tags are all
    `<pkg>@<version>`, and six packages in one run would collide.

So the two portable halves were ported instead: the `RELEASE_PR_TOKEN` wiring
(both halves, including `persist-credentials: false`, without which the fix
works only until the second changeset) and a notes gate rebuilt for six
packages. Both gates also run in `ci.yml`'s required `verify` job, which is
the half that can be made blocking from inside the repo.

Verified: 78 root + 75 package tests pass. `verify.sh config` is red only on
`pnpm audit` and `pnpm licenses`, both reproducing on base, neither run by CI,
and this diff touches no dependency or lockfile line.
…nd go green

`gate-refuter` REFUTED the first draft, and the finding was the same defect
class this slice exists to close: a green run that did nothing.

Turning `createGithubReleases` off means `git.pushTag` is never reached (it
sits inside `if (createGithubReleases)` in the action's own run.ts), so the
post-publish step is the ONLY thing that creates a tag. Keyed on
`published == 'true'`, that produced:

  Six packages publish. `gh release create` succeeds for two and fails on the
  third. The step reds, so an operator re-runs it. On the re-run
  `changeset publish` finds all six already on the registry, publishes nothing
  and exits 0, so `published` is false, so the step is SKIPPED, so the run goes
  GREEN with four packages on npm carrying no tag and no release, permanently.

Not a base regression: with the flag at its default of true the action pushed
the tag inside its own call, so base lost a release BODY. Losing the TAG is
new, and this repo now dates its changelog headings from tags.

The step is now driven by what the version commit BUMPED (which the notes gate
already emits), runs on `!cancelled()` so a partial publish still tags what
reached npm, and asks the REGISTRY whether each package is actually there
rather than trusting a per-run output. A re-run completes the job instead of
skipping it; a bumped package that never published is named and reds the run.
Exercised with stubbed `npm`/`gh` across six cases: happy path, the refuted
re-run, partial publish, a mid-loop `gh` failure, an empty package list, and a
missing body. Only the first two are green.

Three smaller findings, all applied:

* A comment claimed the guards run "before an approver is asked for anything".
  FALSE in `release.yml`: `environment:` is declared at job level, so the whole
  job waits for approval before step one. The `ci.yml` copy is the one that
  genuinely spares the approver. Corrected in both the workflow and CHANGELOG.
* The guard exited 2 on `"@cosyte/x": patch # comment`, valid YAML a human
  plausibly writes. Trailing comments are now stripped, with a committed
  control proving an inert file carrying a comment is still refused.
* The guard bans an idiom this repo used deliberately three times
  (`changeset add --empty` for a repo-level note). `RELEASING.md` now says so
  and names the root CHANGELOG as the only home for such an entry.

`changesets/action` is pinned to `a45c4d5`, the sha cosyte/.github pins and the
sha the `pushTag` behaviour was measured at. A floating `@v1` could move the
internals this file now depends on with no diff here.

80 root + 75 package tests pass. actionlint clean.
`gate-refuter` pass 2 returned NOT REFUTED and still found that the trailing
comment strip added one commit earlier had turned a safe exit 2 into an
exit 0 ON AN INERT FILE:

    "@cosyte/tsconfig": "none" # keep pinned

`js-yaml` strips the quotes, so `@changesets/parse` reads that as the
all-`none` shape the guard exists to refuse, while the guard compared the raw
token and cleared it. A green guard over a changeset that bumps nothing,
introduced by the commit that fixed green runs which did nothing.

The type is now unquoted before the comparison and validated against
`validVersionTypes`, so an unknown type (`patch#nospace`, which the comment
rule deliberately does not strip) takes exit 2 here instead of throwing inside
`changesets/action` halfway through a release. Three spellings of `none` are
pinned by a control.

Also from pass 2, all diagnostic or prose:

* The "bumped but never published" annotation asserted "the ones that did
  publish have been tagged and released" unconditionally, including when an
  earlier step failed and nothing published at all. It now names both readings
  and only claims the successful ones when there are any.
* Deleted "a cancelled job" from the list of `gh release create` failure
  causes: on a cancelled run `!cancelled()` is false and the step does not run.
* "changelog headings are derived from tags" described a manual practice as
  automation. Now says "dated from tags by hand".
* Deleted a stale sentence claiming the post-publish step keys off the action's
  own `published` output, which stopped being true when that step was rewritten.

The shared `cosyte/.github` release workflow was re-read a third time, at
`5896185` (PR #31, the post-publish install gate). Both reasons config cannot
call it are unchanged: its gate still answers "is a release pending" from the
ROOT package.json (this repo's is private, pinned at 0.0.0, and returns
`never-versioned`, which would withhold every publish), and it still hardcodes
`tag="v${version}"`. Its inputs offer no multi-package mode and no tag
override. That comment is now one sentence instead of an accreting list of
shas.

82 root + 75 package tests. actionlint clean.
… overstatements

`gate-refuter` pass 3 returned NOT REFUTED and found that the comment
justifying why this repo cannot call the shared workflow measured an UNMERGED
branch. It cited sha `5896185` and an `expect-unpublished-deps` input, both of
which exist only on `cosyte/.github`'s `feat/post-publish-install-gate`
(PR #31, still open), while the workflow next door names `release.yml@main` as
the alternative. `main` is at `1e634f0` and has three inputs.

The conclusion was never affected: both blockers were re-verified on `main`
itself, so "config cannot call it" holds either way. But a citation reaching
for a moving third-party branch is the third round running where a CLAIM about
a measurement was the weak point rather than the code, so the fix is to delete
the moving parts rather than update them. The comment now names `1e634f0` on
`main` and says the reasons do not depend on an input at all.

Also deleted, same reason: "halfway through a release run" and "mid-release".
`@changesets/read` parses every changeset up front and lets the error
propagate, so a bad release type kills the action BEFORE `changeset version`
or `changeset publish`. Nothing would be half-released. The error string inside
the guard was already accurate and is unchanged.

Deliberately NOT changed, both raised by pass 3 as minor and neither a false
green: two exit-code mis-assignments on inputs no tooling can produce
(`"none"#x` takes 2 where the contract says 1; `"none'` with asymmetric quotes
takes 1 where it says 2), and duplicate frontmatter keys passing the guard
while js-yaml throws. The first is hardening after a passing verdict, which
this repo family has been refuted for before; the second is pre-existing and
loud. Both are backlog lines.
@NSchatz
NSchatz merged commit 89cc33f into main Aug 4, 2026
5 of 6 checks passed
@NSchatz
NSchatz deleted the fix/release-pipeline-honesty branch August 4, 2026 17:01
@NSchatz NSchatz mentioned this pull request Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant