Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 160 additions & 0 deletions .dev/features/ci-matrix-required-checks/GRILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# GRILL — ci-matrix-required-checks

Plan under interrogation: `.dev/features/ci-matrix-required-checks/PLAN.md` (approved at GATE 1).
**Spec-hash check (content-hash floor primitive, surfaced not blocking):** recomputed
`sha256(ARCHITECTURE.md)` = `bca940a5ad247c120e6d8a3acba119d0d8df51dca275964d0e54c48d729d3c4e` —
**matches** the plan's `spec_content_hash`. No drift. (`/pharn-dev-build`'s fix #4 gate is where drift
would actually block; this line only reports.)

**Griller discovery (FLOOR — enum/regex membership, `.dev/floor/count-grillers.mjs .`):**
`{"registered":0,"grillers":[]}` — zero `role: griller` capabilities are registered in this repo, so
the axes below are the inline Step-2 set only. No griller findings exist to fold in. Stated so the
absence reads as measured, not skipped.

---

## Findings

### Axis: guarantee-audit completeness (P0)

```yaml
- type: FINDING
rule_id: 'P0'
severity: important
file: '.dev/features/ci-matrix-required-checks/PLAN.md:88'
problem: 'The plan''s only floor reduction rests on a regex extractor whose parsing rule is left unspecified, and `name:` appears at three different levels of a GitHub workflow file — so the extractor can silently over- or under-match and still report green.'
evidence: 'extract every `name:` under `jobs:` in `.github/workflows/ci.yml` -> assert **set equality** with the six expected contexts'
```

`ci.yml` will contain `name: ci` at column 0 (the workflow name), six job-level `name:` keys, and one
`- name:` per step (`Install`, the gate step, …). "Every `name:` under `jobs:`" is not a rule a regex
can apply without an indentation convention that nothing in this repo enforces — prettier does not
format YAML here and `markdownlint` does not see it. If the extractor matches step names it fails
loudly (harmless); if it matches too few — e.g. someone reformats to 2-space job indentation — the
set-equality still passes on a subset only if the expected list is also edited, but a **rename plus a
test edit in the same commit** is exactly the drift this test claims to catch. The reduction is
weaker than "floor: enum/regex" suggests unless the extractor anchors on the job-level indentation
**and** asserts an exact count of six **and** explicitly excludes the workflow-level `name: ci`.

```yaml
- type: FINDING
rule_id: 'P0'
severity: minor
file: '.dev/features/ci-matrix-required-checks/PLAN.md:96'
problem: 'The floor claim guards the workflow side of a two-sided invariant, while the failure it exists to prevent lives entirely on the side the test cannot read — a gap the plan labels advisory but does not weigh.'
evidence: '"Those six names equal what the GitHub ruleset requires" -> **advisory.** The expected list is a checked-in copy; nothing in this repo reads the live ruleset.'
```

The plan is honest here (it labels the gap advisory and even names it as the mechanism of the current
breakage), so this is not the P0 disease. It is raised so the human weighs it consciously: after this
increment the repo still has **no** check that would catch this exact incident recurring. A follow-up
that reads the live ruleset via `gh api` in a non-blocking job would close it — deliberately **not**
proposed as part of this increment (P7).

### Axis: eval coverage and the structural/semantic split (P1, `pharn-contracts/eval-format.md`)

```yaml
- type: FINDING
rule_id: 'P1'
severity: minor
file: '.dev/features/ci-matrix-required-checks/PLAN.md:81'
problem: 'The section is titled "Evals to write (P1)" but contains a vitest test, not an eval — there is no `case`/`expected` pair and no `structural[]`/`semantic[]` split, because this increment has no Capability under test.'
evidence: '## Evals to write (P1)'
```

Cite, don't restate (P4): `pharn-contracts/eval-format.md` defines an eval as a `{case, expected}`
pair whose `expected.assertions` splits into `structural[]` and `semantic[]`. Nothing in this
increment is a Capability, so no eval in that sense is owed. The vitest test is the right artifact —
it is entirely `structural[]`-class (string set equality, no judge), so it does **not** launder a
floor-checkable assertion through an LLM. The finding is terminological: relabel the heading so a
later reader does not go looking for `evals/cases/*.md` that were never owed.

### Axis: honest scope / no speculation (P7)

```yaml
- type: FINDING
rule_id: 'P7'
severity: important
file: '.dev/features/ci-matrix-required-checks/PLAN.md:74'
problem: 'The rollback path for the one irreversible, un-revertable part of the increment — the GitHub ruleset mutation — is a session-scoped scratchpad file that will not exist tomorrow.'
evidence: 'The current ruleset JSON is captured to the session scratchpad first so it can be restored verbatim.'
```

Ruleset 18605288 is repo **settings**, not a file: `git revert` cannot undo it, no writes-scope hook
gates it, and no test covers it. Its pre-change JSON is the only record of what 33 contexts were
required. A scratchpad under `/private/tmp/claude-501/...` is deleted with the session. Either the
backup belongs somewhere durable, or `SHIP.md` must record the exact restore command with the
verbatim prior context list inline.

```yaml
- type: FINDING
rule_id: 'P7'
severity: important
file: '.dev/features/ci-matrix-required-checks/PLAN.md:71'
problem: 'The plan does not account for the two open dependabot PRs, which were opened against the old workflow and will not report the six newly-required contexts until they are rebased onto the merged change.'
evidence: '**Out of repo, done as a separate deliberate step, not a file write:** update ruleset 18605288''s `required_status_checks` to the 9 contexts above'
```

Verified live this run: PR #88 is `mergeable_state: "behind"`, and the ruleset sets
`strict_required_status_checks_policy: true` — so both dependabot PRs already require an update-to-
`main` before merging, which will pick up the new `ci.yml` and produce the six contexts. The exposure
is therefore **bounded and self-healing**, not a trap — but it should be stated, because between the
ruleset edit and their rebase both PRs display six permanently-pending required checks, which looks
identical to the failure being fixed. Naming it now prevents diagnosing it twice.

### Axis: docs cite code (P4)

```yaml
- type: FINDING
rule_id: 'P4'
severity: important
file: '.dev/features/ci-matrix-required-checks/PLAN.md:47'
problem: 'The plan accepts that node 20 and 22 lose all CI coverage but plans no corresponding change to the `engines.node` range or to any doc that advertises it, leaving a published support claim that nothing exercises.'
evidence: '`package.json` declares `engines.node: ">=20"`, and after this change **no CI gate exercises node 20 or 22 at all**'
```

This is the sharpest concern in the plan and the plan itself raises it — credit where due — but it
stops at recording the limit. `@pharn-dev/pharn` ships `engines.node: ">=20"` to npm; after this
change that range is asserted, published, and untested at both its lower bound and its midpoint. Two
coherent resolutions exist, and the choice is the human's (P5/P6 terminal fallback = ask): narrow
`engines.node` to what CI actually gates, or keep the range and add back a node-20 `Test` job only.
Doing neither is defensible for one increment; doing neither **silently** is what P4 objects to.

### Axis: one axis of change (P3), determinism (P5), trust (P2)

No findings.

- **P3** — each planned file changes for one reason: `ci.yml` for job topology, the test for the
name invariant, the two docs for describing it. No sibling-leaf import is introduced; nothing in
`src/` is touched.
- **P5** — the test's decision is set equality over extracted strings; there is no classifier, no
matrix product, no exclusion list, and no fallback that ends in a guess.
- **P2** — no untrusted artifact is ingested. The plan preserves `permissions: contents: read` and
`persist-credentials: false` on `pull_request`, so splitting one job into six does not widen the
token surface a fork PR can reach.

---

## Summary

The plan is internally honest — it labels its own advisory gaps rather than dressing them as
guarantees, and it records the node-24 coverage decision as a named limit instead of quietly
narrowing what CI proves. Its guarantee audit does not contain the P0 disease.

Four concerns are worth the human's attention before `/pharn-dev-build`. Two are about the increment's
**edges rather than its core**: the ruleset mutation has no durable rollback record, and the two open
dependabot PRs will show six pending required checks until they rebase (bounded — `strict` policy
already forces that rebase). One is about the **strength of the only floor reduction**: the regex
extractor's parsing rule is unspecified, and `name:` occurs at three levels of a workflow file, so
the test must anchor on indentation, assert an exact count, and exclude the workflow-level `name: ci`
or it guards less than it advertises. The fourth is the **published-but-untested `engines.node`
range**, which the plan surfaces and then leaves unresolved.

None of these argues against building. Three are satisfied by tightening the test and recording the
rollback verbatim; the fourth is a decision the human should make explicitly rather than inherit.

**ADVISORY VERDICT: 6 concerns raised (0 blocking-severity, 4 important, 2 minor) — for the human to
weigh before `/pharn-dev-build`.** This grill-log gates nothing: every finding above rests on model
judgment, the severities are LLM-assigned and advisory (fix #3), and the only floor-grade facts in
this run are the spec-hash match, the `count-grillers.mjs` membership result, and the writes-scope
hook that pinned this file. `/pharn-dev-build`'s own floor-gates remain the deterministic backstop.
147 changes: 147 additions & 0 deletions .dev/features/ci-matrix-required-checks/PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# PLAN — ci-matrix-required-checks

> **Revised after GATE 1 feedback.** The first draft proposed building the 30-context OS/node matrix
> the ruleset demands. The human rejected that direction: _"leave one check per each — we don't need
> to have one for node 20 one for node 22 etc., like we have before these changes."_ This plan now
> does the opposite: **one required check per gate, no matrix**, and the **ruleset** is what gets
> corrected. The slug is kept so the folder stays stable.

- spec_content_hash: bca940a5ad247c120e6d8a3acba119d0d8df51dca275964d0e54c48d729d3c4e # fix #4
- increment: Split `ci.yml`'s single `check` job into six independently-reporting gate jobs
(`Format check`, `Lint`, `Markdown lint`, `Typecheck`, `Test`, `Build`) on one platform/node
version, and reduce the ruleset's required contexts from the 30 never-reported matrix names to
those six plus the three already-passing external contexts.
- layer(s): repo infrastructure (CI + repo config) — not a `ARCHITECTURE.md §4` product layer; no
`src/` behavior changes.
- constitution_refs: [P0, P1, P4, P5, P6, P7]

## Live state this run (P6)

Read/verified in this run, not from memory:

- Ruleset `main protection` (id 18605288, `enforcement: active`, updated 2026-08-12T11:58) requires
**33** contexts: 30 of the shape `<Gate> (<os> / node <ver>)` over
`{Format check, Lint, Markdown lint, Typecheck, Test, Build}` ×
`{ubuntu-latest 20, ubuntu-latest 22, ubuntu-latest 24, windows-latest 24, macos-latest 24}`,
plus `floor`, `Analyze (javascript-typescript)`, `gitleaks`.
- `.github/workflows/ci.yml` defines **one** job, `check` (ubuntu-latest, node 20), running the six
gates as `if: always()` steps. It reports the context `check` — which the ruleset does not list.
- Consequence on PR #92: `gh pr checks 92` → all 8 reported contexts pass; the 30 matrix contexts sit
at "Expected — waiting for status to be reported" and never arrive. Merge is blocked.
- `floor`, `gitleaks`, `Analyze (javascript-typescript)` are produced by `floor.yml`, `gitleaks.yml`,
`codeql.yml` and already pass. Their contexts are **bare job names** (`codeql.yml` job `analyze` is
`name: Analyze (${{ matrix.language }})`) — confirming a required context is the check-run name,
not `workflow / job`. No other workflow defines a job named `Build`, `Test`, `Lint`, `Typecheck`,
`Format check`, or `Markdown lint`, so the six new names cannot collide.
- `package.json` `engines.node` is `>=20`; the six gate scripts are `format:check`, `lint`,
`lint:md`, `typecheck`, `test:coverage`, `build`.
- `.prettierrc` sets `"endOfLine": "lf"` and the repo has no `.gitattributes` — this mattered only
for the rejected Windows cell and is therefore **out of scope now** (P7: no speculative addition).

## Decision (records the human's answers at GATE 1)

Six required contexts, one per gate, on **ubuntu-latest / node 24** — one job per gate, no matrix.
Multi-OS coverage is explicitly **not** part of this increment.

Node 24 was chosen over today's node 20 (human's answer). Recorded consequence, so it is a decision
and not an accident: `package.json` declares `engines.node: ">=20"`, and after this change **no CI
gate exercises node 20 or 22 at all**. A node-24-only pipeline can go green on code that breaks the
declared minimum. That is an accepted, named limit of this increment (P7) — not a claim that node 20
is supported-and-verified.

Resulting required-status-check list (9 contexts, all of which actually report):

`Format check`, `Lint`, `Markdown lint`, `Typecheck`, `Test`, `Build`, `floor`,
`Analyze (javascript-typescript)`, `gitleaks`.

## Files

- `.github/workflows/ci.yml` — rewrite: six jobs (`format-check`, `lint`, `markdown-lint`,
`typecheck`, `test`, `build`), each `runs-on: ubuntu-latest` with node 24, each with an explicit
`name:` equal to its required context; keep the existing pinned action SHAs and
`persist-credentials: false`; the aggregate `check` job and its `if: always()` step guards are
removed — job-level independence supersedes them — layer: CI infra.
- `tests/ci-workflow.test.ts` — new vitest test pinning the six job names and their gate commands
(see Evals) — layer: tests.
- `.dev/floor/check-run-pins.test.mjs` — **added by human-approved amendment after the first
`/pharn-dev-regress` STOP**, not present in the originally-approved `## Files`. Its live-repo
assertion `assert.equal(d.skipped, 2)` counts lockfile (`npm ci`) installs across every workflow;
six gate jobs raise that count to 7, so the exact-count tripwire fires by design — its own comment
says a changed count means "a lockfile install was added or removed on purpose", which is precisely
the case here. The edit is the one number plus a comment naming the new arithmetic. `d.violations`
stays `[]` — every added line is `npm ci`, so **no floating install is introduced** and the rule the
checker actually enforces is untouched — layer: floor tests.
- `CLAUDE.md` — update the CI paragraph: the six gates now run as six **jobs**, each reporting its
own required status check, rather than six steps in one `check` job (P4).
- `docs/contributing.md` — update "Quality gates": name the six jobs and add the missing
`npm run build` gate (P4).

**Out of repo, done as a separate deliberate step, not a file write:** update ruleset 18605288's
`required_status_checks` to the 9 contexts above via `gh api`. This is a GitHub settings mutation,
not a plan file — it is called out here so the increment is not mistaken for complete without it.
The current ruleset JSON is captured to the session scratchpad first so it can be restored verbatim.

## Contracts satisfied

- None in `pharn-contracts` — this increment adds no Capability, finding, or install surface. It is
repo infrastructure. Named explicitly so the omission is not read as an oversight (P0/P7).

## Evals to write (P1)

`tests/ci-workflow.test.ts` — deterministic, dependency-free (no YAML parser is a direct
devDependency; a regex extractor over the raw file is the floor primitive here, `ARCHITECTURE.md §2`
#3). It encodes exactly the defect this increment fixes — a workflow job name drifting away from the
required context that names it:

- extract every `name:` under `jobs:` in `.github/workflows/ci.yml` → assert **set equality** with
the six expected contexts (both directions: a rename, an addition, or a deletion fails);
- assert each job's `run:` line is its expected npm script — `Format check` → `npm run format:check`,
`Lint` → `npm run lint`, `Markdown lint` → `npm run lint:md`, `Typecheck` → `npm run typecheck`,
`Test` → `npm run test:coverage`, `Build` → `npm run build`;
- assert every job declares `runs-on: ubuntu-latest` and `node-version: 24`, so a silent
platform/version change cannot slip in unreviewed.

## Guarantee audit (P0)

- "`ci.yml` defines exactly the six named gate jobs, each running its own gate" → **floor:
enum/regex** — `tests/ci-workflow.test.ts` set-equality + per-job command assertions.
- "Those six names equal what the GitHub ruleset requires" → **advisory.** The expected list is a
checked-in copy; nothing in this repo reads the live ruleset. If the ruleset is edited on
github.com again, the test still passes and PRs block again. This is precisely how the current
breakage happened; it is a stated limit, not a solved problem.
- "The six gates are independent — one failure can't mask another" → **advisory**, but structurally
stronger than before: previously six `if: always()` steps in one job (one context, first failure
reported); now six jobs, each with its own context and its own pass/fail.
- "Merging PR #92 becomes possible" → **advisory, and false on its own.** The ruleset's
`required_signatures` rule independently blocks: commit `4b8a0be` is `"verified": false,
"reason": "unsigned"`, and local git has no `commit.gpgsign` / `gpg.format` / `user.signingkey`.
Per the human's answer this increment **reports that blocker and changes no signing config** —
the rule stays enforced and the branch stays unmergeable until the human signs it.
- **Honest scope note (P7):** this increment deliberately *reduces* CI coverage relative to the
ruleset's stated ambition — no Windows, macOS, node 20, or node 22 job. That is the human's
explicit decision, recorded above. It should not later be described as "cross-platform CI", and
node 20 must not be described as CI-verified while `engines.node` still claims `>=20`.

## Trust audit (P2)

No untrusted artifact is ingested: the change touches only repo-owned files. `ci.yml` keeps
`permissions: contents: read` and `persist-credentials: false` on `pull_request`, so a fork PR's
untrusted content still cannot exfiltrate or mutate anything from these jobs — the property
`floor.yml` documents is preserved, not extended. Splitting one job into six does not widen the
token or permission surface.

## Determinism audit (P5)

- The test's decision is set equality over extracted strings — a membership test, no classification.
- Each job is an explicit literal; there is no matrix, no computed product, and no exclusion list, so
the emitted context set is a readable enumeration.
- No fallback branch is introduced; a malformed `ci.yml` fails extraction and the test fails loudly
rather than degrading to a partial match.

## Open questions (HALT)

None outstanding — all three were resolved at GATE 1:

1. Shape → **six contexts, one per gate** (not the single `check` job).
2. Node version → **node 24** (limit recorded under Decision and Guarantee audit).
3. Signing → **report only**; no git signing config is touched by this increment.
Loading