Correct the measurement surface: observe what Codex sent, derive the limit by saturation - #60
Open
davisbuilds wants to merge 9 commits into
Open
Correct the measurement surface: observe what Codex sent, derive the limit by saturation#60davisbuilds wants to merge 9 commits into
davisbuilds wants to merge 9 commits into
Conversation
…not the TUI A live `codex-tui` session emitted the skill-shortening warning while the verifier reported 76% of budget and zero degradation for the same directory, model, and minute. `codex debug prompt-input` renders the `codex exec` path and does not load account-synced connector plugins. The TUI listing held 110 entries across 9 roots at 178% of budget with every description clipped to <=77 characters, unmarked, since at least 2026-07-28. The parsing, the vendor arithmetic port, and the 0.15% agreement with Codex's own charged figure were all correct about a session nobody opens. - Spec revision 12: SC-04 now requires a measurement to name its invocation surface, treats the harness's own record of what it sent as authoritative over any live probe, and requires demand outside local control to be separately attributed, with an unattributable verdict reported as unsupported. - Problem section: superseded table retained, corrected surface table added. - Plan: Task 0 partially falsified (finding 4 narrowed to surface-scoped, finding 8 promoted to the live case); Task 5A inserted to move observation to the session rollout, blocking Tasks 6-10. - OPERATIONS.md: the probe carries a do-not-quote warning until 5A lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The row still carried the 76% figure, which describes the exec surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… negatives Verifying revision 12's own projection produced two findings it did not predict. The limit is 4,000 tokens, not 5,440. Two codex-tui renders -- 110 entries and 56 -- each total exactly 4,000, entry cost plus alias table, zero difference. Codex saturates its listing budget, so two saturating renders disclose the limit to the token; the vendor catalog's 272,000 window (2% = 5,440) overstates the interactive budget by 36%. The live target was 246%, not 178%. The harness's warning has false negatives. Removing the largest connector took the target from 246% to 144%; Codex warned at the first and said nothing at the second, while still clipping 50 of 56 descriptions and removing 6,984 characters. Revision 12 had proposed consuming that warning as a check. - SC-04: an observed-by-saturation limit governs over a vendor-reported one; a vendor limit without a saturation check is provisional and cannot make a pair deployable. A warning may be consumed as positive evidence of degradation, never its absence as evidence of conformance. - Membership is now the binding constraint on Codex, not just Claude Code at 200k: the installed 31-skill set is 100% of budget with every connector removed, `full` is 140%, `core` is 47%, and all twenty core-plus-three-overlay combinations fit. Cost model calibrated to 0.00% against the live render after correcting the locator form (dojo entries render absolute, not aliased -- worth 9% and it flipped a verdict). - Task 5A gains budget-by-saturation, absolute-locator calibration, and a ban on reading warning-absence as fit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gpt-5.6-sol in the same directory produced a listing byte-identical to gpt-5.6-terra: 56 entries, 3,892 + 108 = 4,000, clipped to 207. Three renders, two models, 56-110 entries, all exactly 4,000. The interactive budget does not move with the model, contradicting the pinned vendor path where session/mod.rs passes model_info.context_window. Both models report 272,000 (2% = 5,440) and both budget 4,000, so a policy computing window x 2% is wrong by 36% on this surface. Fixed-cap vs hardcoded-200k stays unanswered: every model in this catalog reports the same window, so the two are indistinguishable here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…itself Two controlled results from 2026-08-06. `[plugins."<name>@openai-curated-remote"] enabled = false` does nothing. Tested single-variable: google-drive disabled with its cache intact, openai-developers left untouched as a control, and the next session listed all five google-drive skills unchanged. Removal in the ChatGPT web app works. This retroactively explains vercel, where a disable and an uninstall happened with no session between them. Removing google-drive and gmail freed 734 tokens; a Codex desktop update added a new openai-primary-runtime marketplace plus sites and visualize, worth 723. Net 11 tokens. 56 entries before and after, still exactly 4,000 charged, still 50 of 56 clipped. New baseline 5,468 = 137%, of which dojo is 3,121 (78%). - SC-04: a verdict is bounded by the harness build and the uncontrolled entry set, invalidated when either moves, reported stale rather than conformant, and clearable only by re-measurement -- never by a control surface's own report. - Task 5A gains set-comparison (the two changes cancelled to 11 tokens, so a totals-only check would have reported nothing happened) and a ban on inferring removal from a control surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…aturation `codex debug prompt-input` renders the `codex exec` path and omits account-synced connector plugins. Every Codex figure in this program came from it. Observation now reads `~/.codex/sessions/**/rollout-*.jsonl` -- what the harness actually sent, with the session's originator -- and reuses probe_codex.parse_block unchanged rather than adding a second parser that can disagree. The limit is established by saturation, not by the model catalog. Codex spends the listing budget to the last token, so two renders of different inputs that both saturate must total the same number. Live: 5 saturating renders across 2 entry counts agree at exactly 4,000, against the 5,440 that a 272,000 window implies. Policy limit 5440 -> 4000, limit_basis: observed. New guarantees, each pinned by the failure it repairs: - surface is a required field; only a declared surface can make a pair deployable - a provisional limit (vendor constant, unchecked) can never gate a build - surface-mismatch is reported, never reconciled - staleness compares the uncontrolled entry SET, not its size or cost - a session with no model call is absent, not empty Three defects found while building it, none by reading code: - classification ran on raw `rN/` alias locators, so the uncontrolled set came back empty -- a confident zero (caught by a test asserting it was non-empty) - uncontrolled entries keyed by bare name conflated `skill-creator`, which Codex bundles AND dojo ships; now qualified by origin - `derive_limit` pooled harness builds. A 2026-07-10 session on CLI 0.144.1 charged 6,188 without clipping while every 0.146.0 render saturates at 4,000, so the ceiling is a property of the build (found by running it live) Mutation probe: 14/14 killed. One survived first pass -- staleness compared set size rather than membership, because both fixtures happened to differ in size; fixed with a one-for-one swap case. Also gitignores raw `rollout-*.jsonl` so a capture cannot be committed by accident; the derived fixtures are named differently and stay tracked. Suite 475 -> 504. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both surfaced executing tokenmaxxing's detached-worker-authority-profiles plan (2026-07-26 -> 2026-08-04), a high-risk spec+plan that went through the full pipeline including a critique subagent pass. The first is acceptance criteria that can be true while the property is false. The contract has one degeneracy rule and it is scoped to a verification run's output magnitude, not to the Done When bullets. Three distinct forms got past it: an enumeration satisfiable while its invariant is violated, a partition assertion written before the other branch exists, and an assertion over a collection that is always empty. The critique subagent found four other blocking defects and missed all three, so a reviewer pass does not cover it. The second is that Assumptions Verified only knows how to cite in-repo lines. Five wrong claims about external tool behavior shipped into that plan; the clearest carried a correct file:line citation attached to a false claim about what tmux does with the argv that line builds. Each was a sub-minute probe. The proposed fix adds a Behavior Measured block and simultaneously loosens the citation requirement, since the current form is what made the wrong claim look verified. Recorded here because dojo owns write-plan; the plan-side corrections stay in tokenmaxxing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0163e0fc4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…entities
All three findings were valid. Verifying them found two more.
P1 -- enforce the declared surface before gating. `Policy.accepts_surface`
existed, a unit test covered it, and `assess` never called it, so an
exec-derived result under a TUI-only policy still scored and still gated. The
capability was built and never wired: a control nothing consults is not a
control. `assess` now takes `surface`, refuses an undeclared one outright rather
than scoring it, and `gating` consults it.
P1 -- restrict extraction to the harness-authored record. Walking every string
in every record let a conversation that merely discusses a listing supply the
measurement. Across 309 rollouts the block appears 238x as `developer` but also
2x as `user`, 2x as `assistant`, and 136x inside `compacted` summaries; session
2026-07-28T17-23-34 has the user copy FIRST. The spec's historical table used
that session's sibling recorded 14 seconds later and was correct by luck. Both
fixtures now carry a user-authored decoy placed before the harness record.
P2 -- compare qualified identities as a multiset. Codex charges every copy of a
duplicated name; the fixture lists `skill-creator` twice (Codex bundles it, dojo
ships it), so a bare-name set reported no mismatch when one copy vanished.
Found while addressing them:
- One unparseable rollout aborted a whole-history scan. 98 of 313 predate a
change in Codex's intro wording ("file path" -> "source locator"), so
parse_block correctly fails closed; skips are now collected and counted,
never silent.
- The ceiling claim was an overclaim. Sweeping 89 rollouts across 12 builds:
0.143.0/0.144.1/0.144.6 saturate at exactly 5,440 (= 2% x 272,000, the vendor
formula) and 0.144.1 also at 7,440 (= 2% x 372,000, a larger-window model).
So `2% x context_window` was CORRECT through 0.144.x and 0.145.0 changed it.
The vendor source was not wrong; it described a build that has since moved.
Corrected in the spec and the module. What survives is the general rule the
code already enforces: a ceiling belongs to one build and samples from two
builds may never be pooled.
Also scoped `declared_surfaces` to Codex only -- no per-entry-point rendering
split has been observed for Claude Code, whose probe reads captured request
bodies and is the effective surface by construction.
Mutation probe 21/21 killed. One survivor first pass: the provisional-limit test
passed for the wrong reason, refused on surface grounds before reaching the
check it names.
Suite 504 -> 509.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan, OPERATIONS, ARCHITECTURE, and ROADMAP all still carried claims the last two days invalidated. - Plan: the "first honest run" section reported the exec surface and a build-specific limit as the position; marked superseded, retained because the gap between what it predicted and what the first honest run said is the plan's own best argument for building the verifier. Task 5A's assumption that `exec` is categorically connector-free is corrected -- builds 0.140.0 and 0.144.1 DID load 11 connector entries into exec sessions. The paired 2026-08-04 observation stands, but as a fact about build 0.146.0, not about the two surfaces for all time. - Plan step 8 and OPERATIONS: the ceiling is a property of the CLI build. 0.143.0/0.144.x saturate at 5,440 (= 2% x 272,000, and 7,440 = 2% x 372,000 for a larger-window model), 0.145.0 changed it. Never pool builds. - ARCHITECTURE: the directory map now names rollout observation as the authoritative surface and live probes as cross-checks. - ROADMAP: current position is 137% of the 4,000 ceiling on build 0.146.0, with attribution by owner and the per-build ceiling rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A live
codex-tuisession printed "Skill descriptions were shortened to fit the 2% skills context budget" while this repository's verifier reported 76% of budget and zero degradation for the same directory, the same model, and the same minute.Both were internally correct.
codex debug prompt-input— the basis of every Codex figure in this program since Task 0 — renders thecodex execpath, which does not load account-synced connector plugins. The interactive TUI does. The gap was 69 of 110 entries.Verifying that finding then produced a second one: the limit was wrong too.
codex debug modelsreports a 272,000 window (2% = 5,440), but three interactive renders each saturated at exactly 4,000.Corrected position for the live target: 246% of budget, not 76%, with every description clipped mid-word and unmarked — and it had been that way since at least 2026-07-28, including on the day the previous figures were measured.
What changed
scripts/profiles/rollout_codex.py(new). Observation reads~/.codex/sessions/**/rollout-*.jsonl— what the harness actually sent, with the session'soriginator— and reusesprobe_codex.parse_blockunchanged rather than adding a second parser that can disagree.The limit is established by saturation, not by the model catalog. Codex spends the listing budget to the last token, so two renders of different inputs that both saturate must total the same number. Live: 5 saturating renders across 2 entry counts agree at exactly 4,000. Policy
5440 → 4000,limit_basis: observed.Spec revisions 12, 13, and 14. SC-04 gains: a measurement must name its invocation surface and only a declared surface can make a pair deployable; the harness's own record is authoritative over any live probe; an observed-by-saturation limit governs over a vendor-reported one, with an unchecked vendor limit provisional and unable to gate; a warning may be consumed as positive evidence of degradation but never its absence as evidence of conformance; and a verdict is bounded by the harness build and the uncontrolled entry set, reported
stalewhen either moves.The strategic consequence
Membership is now the binding constraint on Codex, which no prior revision established. Against the observed ceiling:
corecore+ any one overlaycore+ any three overlaysfullThe undocumented 31-skill curation does not fit the harness it runs on — by 407 tokens against the 90% ceiling — and stays over with every connector removed. All twenty
core-plus-three-overlay combinations fit. Cost model calibrated against the live render to 0.00%.Defects found while building this, none by reading code
rN/alias locators, so the uncontrolled set came back empty — a confident zero. Caught by a test asserting the set was non-empty against a fixture known to contain connectors.skill-creatoris listed twice — Codex bundles it and dojo ships it, the exact collision the harness-equivalence declaration records. Keying by bare name swept dojo's copy into the ungovernable set; now qualified by origin.derive_limitpooled harness builds. Found by running it live: a 2026-07-10 session on CLI 0.144.1 charged 6,188 tokens without clipping, while every 0.146.0 render saturates at 4,000. The ceiling is a property of the build.Mutation probe: 14/14 killed. One survived the first pass —
stalenesscompared set size rather than membership, because both fixtures happened to differ in size. Fixed with a one-for-one swap case.Verification
pytest tests/ -q→ 504 passed (was 475)validate_skill_contract.py --strict→ exit 0.gitignorenow blocks rawrollout-*.jsonl(~1.8 MB of verbatim conversation each); derived fixtures are named differently and stay trackedNote
docs: log two write-plan contract gaps found during a high-risk buildis another agent's cross-repo BACKLOG entry, included here at the maintainer's request.🤖 Generated with Claude Code