gate: a deterministic test-integrity gate, so green CI must be earned

[open-coder-ai]

What

Adds a test_integrity gate kind: a deterministic check that blocks a change from winning
green CI by weakening the tests instead of fixing the code. It catches three shapes: a
deleted test file, a net loss of assertions across the whole change, and a vacuous
assertion (assert True, expect(true)) added in their place. chock: test-removal-reviewed
on the removing line is the reviewed escape hatch.

This closes the honestly-carried gap in docs/agentic-risk-coverage.md:

…an agent deleting tests or assertions to get green? → agent-discipline
(assertion_deletion: block) → advisory today — a deterministic test-integrity gate is
a named catalog candidate, not yet a roadmap issue, so this row stays advisory until it is.

Before → after (docs/agentic-risk-coverage.md):

- | …an agent deleting tests or assertions to get green? | `agent-discipline` (`assertion_deletion: block`) | `advisory` today — a deterministic test-integrity gate is a named catalog candidate, not yet a roadmap issue, so this row stays `advisory` until it is |
+ | …an agent deleting tests or assertions to get green? | `test-integrity` gate, `agent-discipline` (`assertion_deletion: block`) | `enforced-at-commit` (gate) — blocks a deleted test file, a net loss of assertions across the change, or a vacuous assertion (`assert True`, `expect(true)`) added in their place; re-checked in CI on the PR head via the `ci-gate` backstop, so it holds for an inbound contributor whose agent never ran a hook. The pragma `chock: test-removal-reviewed` on the removing line is the reviewed escape hatch. No `pre-tool-use` binding exists, so this is not claimed at the in-agent tier. The ambient rule text remains `advisory` |

The claim is scoped to exactly what the tests demonstrate: enforced-at-commit and
re-checked in CI on the PR head. No pre-tool-use tier is claimed — this gate has no
PreToolUse binding.

The engine work (already landed on this branch before this PR)

• GateContext.removed_lines() — the missing primitive; the three existing kinds read only
  the added side of a diff.
• _kind_test_integrity + registration in KINDS (src/chock/gate/runner.py).
• Closed-object param schema (test_path_regex, assertion_pattern,
  dummy_assertion_pattern, allowlist_pragma) in src/chock/gate/schema.py.
• Baseline policy manifest, .agents/policies/test-integrity/, on: [commit], action: block.
• 7 behavioural tests over real git repos, tests/test_test_integrity_gate.py.

This PR: full-suite regression, wiring, and three real bugs it surfaced

Only five test files were run before this PR (45 passed). Running the whole suite and the
whole lint tree surfaced seven failures — all caused by this branch, all fixed here:

1. Vendored runner was stale. .chock/bin/gate.py (the stdlib-only copy the git-hook
   actually executes) hadn't been re-synced against src/chock/gate/runner.py. Fixed via
   chock sync.
2. The manifest declared an event nothing reads. on: [commit, ci] — but CI coverage is
   derived from "commit" being present (runner.py's event == "ci" branch), and no
   other manifest or spec/gate-dsl.md lists "ci" as a value inside on. Fixed to
   on: [commit].
3. The structural JSON-schema layer never learned the new kind.
   src/chock/validation/schemas/manifest.hook.json's kind enum (separate from
   KINDS/KIND_PARAM_SCHEMAS, which already knew about test_integrity) still only listed
   the three original kinds, so any manifest declaring test_integrity failed structural
   validation before the per-kind param check ever ran. Added.
4. No eval suite. Every other catalog policy ships evals/suite.yaml; this one didn't,
   tripping the eval_first check. Added six hand-authored cases (trigger ×2,
   negative_trigger, behavior, edge ×2).
5. spec/gate-dsl.md was never updated with the new kind's params.
6. Two files were unformatted — ruff check was clean but ruff format --check (also
   part of CI) was not.
7. A real false positive, found live while writing the docs for this PR:
   test_path_regex's directory branch matched any file under a tests?/spec/__tests__
   directory regardless of extension. This repo's own spec/ directory holds design docs,
   not RSpec tests — so documenting the gate's dummy_assertion_pattern param in
   spec/gate-dsl.md, using assert True as the worked example, tripped the gate's own
   vacuous-assertion check on itself. Fixed by excluding .md/.rst/.txt from the
   directory branch (a real improvement for any consumer, not a repo-specific hack); a
   regression case was added to both the eval suite and the pytest suite.

Wiring into every place that enumerates kinds

• src/chock/validation/checks_gate_shape.py — already read KINDS/KIND_PARAM_SCHEMAS
  generically; needed no code change. Added tests confirming a test_integrity manifest
  validates and a bad param shape is rejected (tests/test_manifest_validation.py).
• src/chock/gate/schema.py::GATEWAY_ONLY_KINDS — confirmed test_integrity is not
  gateway-only (it needs a git diff, which the gateway doesn't have).
• src/chock/compile/emitters/mcp_gateway.py::RUNTIME_KINDS — confirmed test_integrity is
  absent (same reason); chock sync emits no gateway-gate.json for it, and nothing breaks.
• src/chock/eval/derive.py::DERIVABLE_KINDS — judgement call: not added. The existing
  derivers (forbidden_ref, dependency_allowlist) synthesize cases from an enumerable list
  (branch names, manifest formats); test_integrity's params are open regexes over diff
  content, with no such list to enumerate from. Hand-authored evals/suite.yaml instead.

Docs and registry

• .agents/policies/INDEX.md — picked up automatically by chock sync.
• docs/baseline-policies.md — added to the "Deterministic guards" table and the
  declarative-kinds list.
• docs/policies/, docs/cli-reference.md, llms.txt — checked
  (grep -rl "verify-dependency-exists" docs/ *.md llms.txt); none enumerate individual
  policies, so none needed changes.
• CHANGELOG.md — entry added under Unreleased.

Test evidence

$ pytest -q
1055 passed, 6 skipped

$ ruff check .
All checks passed!

$ ruff format --check .
242 files already formatted

$ chock check --repo .
26 policies: 90 pass, 96 skipped (agent-mode / no executable form)
== enforcement matrix
OK: all 29 spec invariant IDs are present in the enforcement matrix.

$ chock sync --repo . --check
Compiled artifacts match their manifests.

$ chock check --repo . --only verify
verify: all packs match lockfile

$ pytest -q acceptance/ -c acceptance/pytest.ini --rootdir=acceptance
21 passed

test-integrity's own eval suite (chock check --only evals): 6/6 pass, score 1.00.

Definition of done

• chock check → 0 errors, 0 warnings, 0 infos
• chock check --only matrix passes; matrix updated in this PR if behavior changed (no
  invariant IDs changed — 29 present before and after)
• chock sync --repo . --check clean (compiled artifacts match their manifests)
• chock check --only verify clean (lockfile matches packs and compiled artifacts)
• Registry rescanned; no stale entries
• pytest -q green; new checks have attack + ordinary-data tests
• pytest acceptance/ -c acceptance/pytest.ini --rootdir=acceptance green
• Existing artifacts migrated in this PR if a check was added/extended (n/a — no existing
  artifact's schema changed)
• Touched manifests: version bump + changelog entry (n/a for test-integrity itself — it
  is brand new in this PR, still 0.0.1/draft, one "Initial" changelog entry per the
  convention other pre-release policies use, e.g. scan-secrets)
• ruff check . and ruff format --check . clean

Claims

• No surface is described as enforcing more than it installs. test-integrity uses the
  existing git-hook + ci-gate surfaces (already in INSTALLED_SURFACES); no new
  surface type was introduced, so nothing there needed a change. docs/enforcement-surfaces.md
  doesn't enumerate policies individually (checked), so nothing there needed a change.
  docs/agentic-risk-coverage.md is updated as shown above.

Authority note

Per the dispatching brief (chock-g1/W1), authorized paths for this work are
src/chock/gate/, src/chock/validation/, .agents/policies/test-integrity/, tests/,
docs/, CHANGELOG.md, .agents/policies/INDEX.md. Two touched files sit just outside that
literal list, both load-bearing for "full suite green" (chock sync's own generated diff, and
the doc fix a genuine test failure required):

• spec/gate-dsl.md — the gate DSL reference; test_documented_gate_kinds_match_the_code
  fails without it (every other kind is documented there).
• .chock/bin/gate.py, .chock/compiled/test-integrity/, .chock/coverage.json,
  .chock/registry.json, chock.lock — regenerated by chock sync, never hand-edited.

---

Generated by Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[open-coder-ai]

Superseded by #108: same tree (confirmed byte-identical diff between branches), one commit amended to add a missing Signed-off-by trailer that was failing dco. Closing in favor of #108 rather than force-pushing over this branch's history.

---

Generated by Claude Code

---

Generated by Claude Code