Tasks 2–5: resolution, budget policies, observation, and evidence - #59
Merged
Conversation
Composition is set union over core plus the named overlays, lexically ordered, with five distinct rejection codes so a caller can tell a correct refusal from an accidental one. The work here is the identity split spec revision 9 introduced. Profile identity hashes the normalized selection and the reviewed definition bodies and nothing else -- resolved membership is deliberately not an input, because including it would make one reviewed selection hash differently on two harnesses and SC-11 would then read every legitimate suppression as drift. Realization identity binds the six fields that decide what landed, so editing the equivalence declaration cannot replay as an idempotent repeat. Suppression follows the declaration, never a name match. An undeclared collision is reported and the member survives: a duplicated skill is visible, a missing one is not. resolve_for_harness takes the listed bundled entries rather than assuming them, since Task 0 found an entry present on disk and in no listing, which displaces nothing. 23 tests. All 720 permutations of the six overlays resolve to one member tuple and one identity. Mutation-probed: emptying definition bodies from the hash, making it order-dependent, and letting a collision suppress each fail a test, while a no-op control still passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three policy files, two deployable. Codex budgets in tokens, Claude Code in characters, and neither is converted into the other's unit at any point -- converting on the Claude path would introduce an error the harness itself never makes. The Codex primitives are imported from probe_codex rather than reimplemented, since two of them are not what a reasonable reading of render.rs suggests and a second implementation is a second thing that can disagree. claude-code-200k ships as declared-but-not-deployable per spec revision 11: its limit is exactly as authoritative as the 1M policy's, but nobody runs that pair, so it is scored and reported and never gates. Promotion is a visible edit to one field rather than a consequence of someone selecting a different model. The detector covers all five shapes. Two are invisible in the output -- Codex clips mid-word with no marker, and Claude Code drops descriptions entirely, so there is nothing for a prefix comparison to compare against and it must be caught by absence. One real defect, found by the SC-03 fit proof rather than by review: absence of an observation is not an observed absence. A hypothetical composition, scored before any target exists, carries no listed_description at all, and the detector was reading that as description-removed -- inventing a degradation out of the absence of a measurement, which is the inverse of the rule this package exists to enforce. The two cases are now distinguished by key presence and pinned by a test. Also corrected in the tests: a boundary fixture derived from the real limits landed on 8,898 rather than 8,900 through double integer flooring, testing a neighbourhood of the boundary instead of the boundary. The limit is now chosen so each basis point is exact, which is what the plan asked for. Mutation-probed: unobserved-as-removed, degradation not gating, Codex charged in characters, and an undeclared pair gating each fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The probe decides membership; the filesystem only attaches content identity, topology, and source descriptions to entries the probe already listed. An unclassified listing is refused rather than observed, since unknown origins would otherwise collapse into one bucket silently. Shadowing is read from the harness policy, not branched on a harness name. The two deployable harnesses behave oppositely -- Codex lists both copies of a duplicated name and charges for both, Claude Code deduplicates -- so hard-coding either is right for one and silently wrong for the other. Flipping the flag flips the outcome, and a test asserts both directions. Two real defects, both found by comparing computed demand against the probe's own charged figure rather than by review: The frontmatter parser was a regex and captured only the first line, so the five skills using a folded YAML scalar returned one or two characters instead of three to four hundred. Four are overlay members, including both engineering anchors, so that overlay's cost was understated by roughly 1,200 characters. Now parsed as YAML. The gap closed from 190 tokens to 6. Foreign, bundled, and plugin entries have no readable source, and were being charged as though they had no description at all -- a 20% understatement on a live target, in the direction that makes an over-budget catalog look safe. They now fall back to the listed description with the cost basis recorded, so evidence can say which figures are floors. One piece of dead code removed: this module re-classified plugin entries that probe_codex.classify had already labelled. A mutation swapping the Codex cache needle for the Claude one passed all 14 tests, which proved the line was dead rather than defensive. A second classifier is a second thing that can disagree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report is byte-identical over unchanged inputs: every date comes from the policy record rather than the clock, every list is lexically ordered, and wall-clock and hostname live in a non-normative envelope that to_json never emits. Each determinism assertion is paired with a perturbation, since byte-identity alone is satisfied by a function returning a constant. A suppressed member is distinguishable from one the profile never selected. That distinction is the entire reason suppression is declared rather than achieved by editing membership -- SC-11 requires every cross-harness difference to be attributable, and a report listing only what landed cannot support it. unprofiled is a first-class state, not an edge case: no profile can be applied until Task 12, so every real target is unprofiled through the whole of phase 1. It carries full observation, demand, limit and headroom, and exits 2. A partial report can never be conformant, enforced in __post_init__ rather than only in a test. One real defect: git status --porcelain collapses untracked directories to `?? skills/`, so a wholly-new selected skill never matched its own path prefix and a dirty target reported clean -- the failure direction that lets unreviewed content look deployable. Fixed with -uall. One mutation escaped and was closed: sort_keys=False passed every test, because the payload is built in a fixed order so two runs agree regardless. The guarantee only bites when the payload is assembled in a data-dependent order, so the new test constructs identical content in two key orders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ARCHITECTURE gains profiles/ and scripts/profiles/ in the directory map; the map claimed to be complete and had no entry for either. ROADMAP's In Progress row said "implementation planning is next", which stopped being true six tasks ago. It now records what shipped, that it is all read-only, what remains in phase 1, that phase 2 is an open scope decision, and the current measured position. Skill count 49 -> 48. OPERATIONS documents how to measure a session's listing cost, with the three things that make a number from those probes trustworthy: demand comes from untruncated source rather than the rendering, Claude Code's budget scales with the model so a figure without one is meaningless, and the filesystem is not the listing. This is the most useful capability the work has produced and it was reachable only by full path. The harness-adapter backlog item carried three superseded measurements: a 3.4x overage that does not reproduce, a "~1% budget" described as unverified when it is now a confirmed vendor constant, and token figures that charged the whole instructions block rather than only the skill lines. Rewritten with current evidence, and the wrong figures are kept explicitly so the same corrections are not made a fourth time. AGENTS.md's documentation map updated for both changed docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 124e107567
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rty scope All four findings verified against the artifact before fixing. The first is serious. P1 -- conformance was declared from budget alone. An under-budget observation containing a single foreign entry reported conformant while all eleven selected skills were absent: being cheap was mistaken for being right. SC-05 requires every selected skill present and no unselected dojo-managed one, so evidence now carries a membership block with missing and unexpected named, and both gate the verdict. An observation carrying unsupported conditions -- a Claude sent-count reconciliation failure, say -- can no longer be conformant either, since the observation itself is what is untrustworthy. This also corrects the SC-06 field set, which was missing the "missing and unexpected managed entries" the contract enumerates; 16 fields to 17. P2 -- a bundled, plugin, or foreign entry sharing a name with a canonical skill was taking dojo's description. Codex's bundled skill-creator sits beside dojo's in the fixture and their texts differ, so this scored the wrong content and fabricated a truncation signal, since listed then differs from "source" for a reason unrelated to elision. Canonical descriptions are now attached only to dojo-managed entries. P2 -- the whole-catalog-link detector inferred topology from a count. A legitimate full realization built from per-skill links reaches that count, and on a harness that does not shadow, duplicates do too. It now requires the scope root to be a symlink into the canonical tree. The test that covered this passed catalog_size=len(managed), which made the condition trivially true -- it asserted nothing, and is replaced with a positive case, a per-skill-link negative, and a foreign-target negative. P2 -- dirty-state matched all of profiles/, so editing an unrelated overlay made every composition audit-only, contradicting the function's own promise. Narrowed to the selected definitions. Mutation-probed all five changes. One escaped: reverting the description fix passed all 34 tests, so the case the reviewer named is now a test, plus one asserting it produces no false truncation signal. Suite 469 -> 475. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the measurement half of phase 1. Suite 393 → 469. Read-only throughout: four new modules and three policy files, no
Modify:to anything outside the repo, and installed globals verified byte-identical before and after.What each task delivered
Task 2 — resolution and the two identities. Profile identity hashes the normalized selection and the reviewed definition bodies and nothing else; resolved membership is deliberately not an input, because including it would make one reviewed selection hash differently on two harnesses and SC-11 would read every legitimate suppression as drift. Realization identity binds the six fields that decide what landed. Suppression follows the declaration, never a name match — an undeclared collision is reported and the member survives, since a duplicated skill is visible and a missing one is not.
Task 3 — budget policies and the degradation detector. Two units, no conversion between them.
claude-code-200kships as declared-but-not-deployable per spec revision 11: scored and reported, never gating, with promotion a visible edit to one field. All five degradation shapes, including the two that are invisible in the output.Task 4 — effective-catalog observation. The probe decides membership; the filesystem only attaches identity and topology to entries already listed. Shadowing is read from the policy, not branched on a harness name — the two harnesses behave oppositely, so hard-coding either is right for one and silently wrong for the other.
Task 5 — deterministic evidence. Byte-identical over unchanged inputs, with every determinism assertion paired with a perturbation, since byte-identity alone is satisfied by a constant.
unprofiledis first-class: through all of phase 1 every real target is unprofiled, carrying full measurement and exiting 2.Four defects found, none by review
engineeringanchors — that overlay was understated by ~1,200 charsgit status --porcelaincollapses untracked directories to?? skills/, so a wholly-new selected skill never matched its prefix and a dirty target reported clean. Needs-uallThe frontmatter one is worth dwelling on: computed demand sat 190 tokens from the probe's own charged figure, which is what pointed at it. After the fix the gap is 6 tokens on 4,132 — 0.15%. That cross-check is now a permanent test, and it is the only one here that validates the whole cost model against a number the harness produced itself.
Two mutations escaped
Both closed, both instructive:
probe_codex.classifyhad already labelled those entries. Removed rather than kept as belt-and-braces — a second classifier is a second thing that can disagree.sort_keys=Falsepassed every determinism test, because the payload is built in a fixed order so two runs agree regardless. The guarantee only bites when the payload is assembled in a data-dependent order. Now pinned by constructing identical content in two key orders.Verification
pytest tests/ -q→ 469 passed; CI-shaped run (foreignHOME, strippedPATH, foreign cwd) identicalcheck_links.py,slop_scan.py,validate_skill_contract.py --strictall pass🤖 Generated with Claude Code