Scope deployability to declared harness/model pairs (spec revision 11) - #57
Merged
Conversation
…sks 3 and 10 Spec revision 11. SC-03 required the fits-proof against every pair whose limit could be established, which silently promoted Claude Code at 200k to a gate. The operator runs only 1M sessions. Demand does not move with the window, so the existing measurements rescore without re-probing: 41/51/58% against a 90% ceiling, corroborated by the captured 1M session in the fixtures, which emits no budget warning at all. Deployability now turns on whether a pair is declared in use. Established but undeclared pairs are measured and reported, never gating -- a session that does land at 200k is told it is non-conformant rather than discovering bare skill names on its own -- and declaring a pair is a maintainer edit, so the gate cannot widen because someone selected a different model. Nothing the operator runs is now in breach except viral at 95% on Codex. That is the third revision running where better information reduced urgency. What survives is stated plainly because it is now the whole case: the Claude margin is one model selection wide, the figures moved 2x in a day twice with no distribution decision, cross-machine agreement broke silently within 24 hours of verification, and the curated set is installed state rather than a declaration. Plan: Task 3's detector goes from three shapes to five -- Task 0 found Codex's whole-skill omission tier and that its two truncations differ (the 1,024-char pre-cap appends "...", budget clipping appends nothing) -- and is told to import probe_codex's primitives rather than write second implementations that can disagree. Task 3's fit proof drops to two declared pairs plus a reported-not-gated case. Task 10 is rescoped: the budget disagreement it existed to adjudicate was already resolved by two retractions, so it now emits the first machine-generated position instead of re-litigating them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
davisbuilds
added a commit
that referenced
this pull request
Aug 3, 2026
) Follows Task 0 (#56) and spec revision 11 (#57). Suite **350 → 391**. ## What landed Eight profile definitions (one file each, so a duplicate `name` is a detectable condition rather than a silently last-wins key), `harness-equivalences.yaml`, `scripts/profiles/definitions.py`, and `profiles/README.md`. Overlays are deliberately **3–4 members**, not categories. `core` (8) plus any one overlay lands at 11–12, inside the ~13-skill figure a 200k Claude Code window allows. An overlay of nine would not be deployable at all. | Overlay | Members (⚓ = SC-02 anchor) | |---|---| | engineering | ⚓`create-cli` ⚓`secure-code` `api-design` `repo-hardening` | | research | ⚓`deep-research` ⚓`research-architect` `fetchmd` | | design | ⚓`design-critique` ⚓`web-design-guidelines` `frontend-design` | | knowledge | ⚓`obsidian-markdown` ⚓`session-retro` `compound-docs` | | shipping | ⚓`gh-commit-push-pr` ⚓`vercel-deploy` `vercel-preview-logs` | | skill-authoring | ⚓`skill-creator` ⚓`skill-standardizer` `skill-evals` | Each non-anchor member is a *sibling of an anchor along the axis the anchor sets* — `api-design` is `create-cli` at the service boundary, `repo-hardening` is `secure-code` at repo scope, `frontend-design` is the only producer among two auditors, `vercel-preview-logs` is the only skill that reads back what a deployment did. **21 of 48 catalog skills belong to no overlay**, which is correct rather than a gap. Every member carries an inline reason; every file records what was deliberately excluded. ## The mutation probe found three holes All 27 validation rules were probed by breaking each and confirming a test fails. Three did not: - **`members: core`** — plausible YAML, neither the sentinel nor a list, and it would have been iterated character by character into four one-letter "members". - An unrecognised key was silently accepted. - **`equivalence_identity` without its `sorted()` passed everything**, because every test reached identity through the loader, which sorts first. Order-independence was being provided by the wrong function and the guarantee was untested. All three closed. That third one is the same shape as Task 0's `sent := loaded` hole — a property tested only through a caller that already satisfies it. Also built in, and worth stating because it is invisible otherwise: **the non-`core` member count is checked before the anchors.** Every anchor is itself a non-`core` skill, so an overlay holding its anchors can never fail the count — with the checks reversed, that rule could not fire on any input. ## Review changes **Withdrew the `gpt-imagen` ↔ Codex `imagegen` declaration.** It was the only entry resting on a capability judgement across *differing* names, and the judgement is unverified: dojo's pins `gpt-image-2` and exposes inpainting, masking, and batch output, none of which the bundled description claims. The asymmetry decides it — a wrong declaration silently removes a selected skill, an undeclared collision is *reported* — and it buys nothing today since `gpt-imagen` is in no overlay. Kept as a candidate with the comparison it needs. Two declarations remain, both same-name same-capability: `skill-creator`, `skill-installer`. ## Three things Task 1 falsified in its own inputs 1. **`review-agent` is not a Codex-listed entry.** It exists at `~/.codex/skills/.system/review-agent/` and appears in neither the Task 0 capture nor a live re-probe. Spec and plan both named it as bundled. Only listed entries can displace a member — this is the filesystem-is-not-the-listing rule catching this contract's own prose. 2. **The plan's equivalence seed contradicted its own rule.** It named six Codex skills to seed from, while the next sentence forbids declaring a pair whose dojo member does not exist — and three of the six have no dojo counterpart. Rewritten so the rule governs, with "observed in a listing" made an explicit second condition. 3. **"8 rejection cases" was stale** (27 implemented), and `__init__.py` was listed as `Create` though it shipped in Task 0. ## Verification - `pytest tests/ -q` → **391 passed** - CI-shaped run (`cd /tmp`, `HOME=/tmp/fake-home`, `PATH=/usr/bin:/bin`) → identical - `validate_spec.py` PASS, `validate_plan.py` PASS - Every rejection test asserts the error *message* names the offending profile and member; no test compares against a hardcoded catalog total — `full` resolves against `len(skills.json)` computed at test time 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only — no code changes. Follows Task 0 (#56).
The change
SC-03 required the fits-proof against every pair whose limit could be established, which silently promoted Claude Code at 200k to a gate. The operator runs only 1M sessions.
Demand does not move with the window — only the budget does — so the existing measurements rescore with no re-probing needed:
viraldojoCorroborated by the 1M capture already in the fixtures, which emits no budget warning at all.
Deployability now turns on whether a pair is declared in use. Established-but-undeclared pairs are measured and reported, never gating — so a session that does land at 200k is told it is non-conformant rather than discovering bare skill names on its own. Declaring a pair is a maintainer edit, so the gate cannot widen because someone selected a different model.
What this does to the argument
Nothing the operator runs is in breach except
viralat 95% on Codex.That is the third consecutive revision where better information reduced urgency: revision 7 found the measurements 1.78× overstated, revision 10 found two of three Codex breaches were ordinary defects, this one finds the remaining harness conformant in practice. The spec says so plainly rather than preserving a threat that no longer exists.
What survives is now the whole case, and it is governance rather than firefighting:
Calibration recorded for later: at 200k, bundled entries are exempt from stripping and take 3,774 of the 8,000 budget at full length, leaving ~4,226 against a mean 313 chars/entry — about 13 dojo skills, which is
coreplus roughly one overlay, matching what SC-03's fits-proof already specifies.Plan changes from Task 0
render.rsfound Codex's third tier (whole-skill omission, which does warn in-prompt) and that its two truncations are different mechanisms — the 1,024-char pre-cap appends"...", budget clipping appends nothing.probe_codex's primitives, not write second implementations. Two are not what a reasonable reading of the source suggests, and Task 0 got both wrong first.deployable: true|false.Verification
validate_spec.pyPASS,validate_plan.pyPASSpytest tests/ -q→ 350 passed🤖 Generated with Claude Code