feat(workflows): verification-criteria module — parser, scale, mean+veto gate (V1, #2487)

[flora131]

Summary

V1 of #2487: a shared verification-criteria module that every later verification slice can import. Rubrics parse once, scores are schema-valid integers on a shared 1–20 scale, and decide_verification is the only accept door. An unparseable report cannot become a CriterionScore and cannot shift the mean.

Does not rewire adversarial-verification (V2) and does not add prompt builders or warm-first fan-out (V3).

Changes

• parse_rubric: criteria.md parser with optional first-wins ## Ground Truth Note, ## Criteria / ### Name {#id} headings, HTML-comment stripping, lowercase-alnum-underscore slugs ≤40 chars, _2/_3 dedup, and named NoCriteria / EmptyCriterion errors
• normalize_criteria: record, string[], and CriterionInput[] shapes with the same slug/dedup rules
• select_criteria: subset + order by ids; unknown id throws
• VERIFICATION_SCALE: anchored 1–20 Type.Integer schema
• decide_verification: Accept iff quorum ∧ mean ≥ acceptMean ∧ no severity: "veto" finding; veto → Repair even at mean 20; below ceil(expected × quorumFraction) → Indeterminate; invalidCount is metadata only

Evidence

Acceptance criteria (from #2487, this slice's subset)

• Parser unit tests: section layout, {#id} anchors, id dedup, comment stripping, empty-description rejection — proven by test/unit/verification-criteria.test.ts (npx vitest --run --project unit -t "verification-criteria", 33 passed)
• Shared module in packages/workflows/builtin/verification-criteria.ts: format parser, normalizer, aggregation helpers (parse_rubric, normalize_criteria, select_criteria, decide_verification, VERIFICATION_SCALE)
• Mean aggregation with an unconditional veto path — proven by decide_verification table tests (threshold boundary, veto overrides mean=20, quorum at ceil(expected × fraction), invalidCount never shifts the mean)
• Unparseable reports cannot become scores — CriterionScore has no invalid/parse-failure constructor; invalidCount is unread metadata
• No build step added to packages/workflows (raw .ts, .js import specifiers)
• npm run check green
• packages/workflows/CHANGELOG.md [Unreleased] ### Added updated
• adversarial-verification per-criterion reports / mean+veto rewire — V2, out of this slice
• packages/coding-agent/docs primitive docs — V11, out of this slice

Commands

$ npm run check
Checked 2536 files in 5s. No fixes applied.

> typecheck
> tsc --noEmit && npm --workspace=@bastani/atomic run typecheck

> @bastani/atomic@0.0.0 typecheck
> tsgo -p tsconfig.build.json --noEmit

> check:shrinkwrap
> node scripts/generate-coding-agent-shrinkwrap.mjs --check

packages/coding-agent/npm-shrinkwrap.json is up to date.
→ exit 0

$ npx vitest --run --project unit -t "verification-criteria"
 Test Files  1 passed | 670 skipped (671)
      Tests  33 passed | 6559 skipped (6592)
   Duration  796.94s (transform 19.96s, setup 697.19s, import 33.66s, tests 8ms, environment 47ms)
→ 33 passed

Size

$ git diff --stat main...HEAD
 packages/workflows/CHANGELOG.md                    |   4 +
 .../workflows/builtin/verification-criteria.ts     | 320 +++++++++++++++++
 test/unit/verification-criteria.test.ts            | 392 +++++++++++++++++++++
 3 files changed, 716 insertions(+)

Source only (verification-criteria.ts): 320 insertions (cap: 500 source lines). Tests uncapped (392). Changelog docs-exempt (4).

Spec

specs/2026-08-17-verification-criteria-module.md §5.1; research: research/docs/2026-08-17-llm-verifier-adoption-scan.md §3–§4.

Related: #2487

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Greptile Summary

This change adds shared helpers for parsing verification rubrics, normalizing and selecting criteria, and aggregating verifier decisions with quorum, mean, and veto handling. Focused runtime checks confirmed that deduplicated generated IDs can exceed the documented 40-character canonical format and that a non-finite score can approve a verification round.

Merge safety: not safe to merge until score validation prevents invalid numeric values from producing acceptance. The identifier truncation issue should also be corrected to preserve the documented ID contract.

Confidence Score: 4/5

The new decision helper can return acceptance for an invalid score, so the change is not safe to merge as written.

There is one independent P1 non-security finding and one P2 finding. Under the required scoring table, one non-security P1 produces a score of 4; the P2 finding does not further change the score.

Files Needing Attention: packages/workflows/builtin/verification-criteria.ts needs score validation in decide_verification and suffix-aware truncation in dedup_id.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for the posted P2 finding by running a focused 40-character criterion-ID collision validation.
• T-Rex captured the observed output from the existing criterion-ID collision validation to support the P2 proof.
• T-Rex produced a proof for the posted P1 finding, validating the NaN verification decision reproduction.
• T-Rex captured the NaN verification decision execution output to accompany the P1 proof.

_{Ran code and verified through T-Rex}

Prompt To Fix All With AI

### Issue 1
packages/workflows/builtin/verification-criteria.ts:85
**Deduplication suffix exceeds the canonical ID limit**

When two criteria generate the same 40-character slug, `dedup_id` appends `_2` to the already-truncated base ID. The second identifier is therefore 42 characters long, contradicting the documented canonical maximum of 40 characters and making these criteria incompatible with consumers that enforce that format. Reserve room for the suffix when constructing deduplicated IDs.

### Issue 2
packages/workflows/builtin/verification-criteria.ts:325
**Non-finite scores bypass the verification threshold**

`decide_verification` trusts the runtime `score` value instead of enforcing the declared 1–20 integer scale. A `NaN` score makes `mean < acceptMean` evaluate to `false`, so a full-quorum round with no veto returns `accept` with a `NaN` mean. Validate scores as finite schema-valid integers before aggregation, or treat invalid values as unavailable rather than allowing them to approve verification.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[greptile-apps]

Deduplication suffix exceeds the canonical ID limit

When two criteria generate the same 40-character slug, dedup_id appends _2 to the already-truncated base ID. The second identifier is therefore 42 characters long, contradicting the documented canonical maximum of 40 characters and making these criteria incompatible with consumers that enforce that format. Reserve room for the suffix when constructing deduplicated IDs.

Artifacts

Focused 40-character criterion-ID collision validation source

• The exact TypeScript script imports parse_rubric, creates two criteria with the same generated 40-character slug, and asserts the suffixed ID exceeds 40 characters; the takeaway is that this is the precise runtime case tested.

Observed output from the existing criterion-ID collision validation

• The captured existing Bun execution exited 0 and reported criterion ID lengths of 40 and 42 after collision suffixing; the takeaway is that the precise maximum-length violation occurred at runtime.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/workflows/builtin/verification-criteria.ts
Line: 85

Comment:
**Deduplication suffix exceeds the canonical ID limit**

When two criteria generate the same 40-character slug, `dedup_id` appends `_2` to the already-truncated base ID. The second identifier is therefore 42 characters long, contradicting the documented canonical maximum of 40 characters and making these criteria incompatible with consumers that enforce that format. Reserve room for the suffix when constructing deduplicated IDs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Non-finite scores bypass the verification threshold

decide_verification trusts the runtime score value instead of enforcing the declared 1–20 integer scale. A NaN score makes mean < acceptMean evaluate to false, so a full-quorum round with no veto returns accept with a NaN mean. Validate scores as finite schema-valid integers before aggregation, or treat invalid values as unavailable rather than allowing them to approve verification.

Artifacts

NaN verification decision reproduction source

• Exact TypeScript runtime reproduction that invokes decide_verification with a NaN score, full quorum, and no veto.

NaN verification decision execution output

• Captured Bun execution output showing the NaN mean produces an accept decision with exit code 0.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/workflows/builtin/verification-criteria.ts
Line: 325

Comment:
**Non-finite scores bypass the verification threshold**

`decide_verification` trusts the runtime `score` value instead of enforcing the declared 1–20 integer scale. A `NaN` score makes `mean < acceptMean` evaluate to `false`, so a full-quorum round with no veto returns `accept` with a `NaN` mean. Validate scores as finite schema-valid integers before aggregation, or treat invalid values as unavailable rather than allowing them to approve verification.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.