perf(agent): drop the exhaustive-audit nudge that fired on ordinary English - #749
Merged
Merged
Conversation
…nglish #740 added a one-shot "exhaustive audit" pass: when a request matched a bare quantifier, the loop appended a MUST-enumerate instruction and forced another model turn, backed by a matching system-prompt clause. The trigger was `\b(all|every|multiple|each|exhaustive|complete set|any other|others)\b` — plain English, not a request shape. Measured against the 89 terminal-bench 2.1 instructions it fired on 49% of them, almost all incidental: "each line of the file", "where each task is an async job", "in all years". One case inverted the instruction outright — regex-log says "If multiple dates are present in a line, the regex should match only the last one", and the nudge answered that by demanding exhaustive enumeration. Removed rather than tuned. Tightening the regex would have been fitting a clawcodex-only heuristic to a benchmark; the evidence says the mechanism should not exist: * the latest Claude Code scores 1.0 on both tasks this was built for (chess-best-move, regex-chess) with no audit machinery at all; * only 1 of its 89 trials writes more than two verification files, where clawcodex wrote 57 temporary verification scripts across the run; * clawcodex is +26% on mean trajectory steps against the same model at the same effort, and forced audit turns are part of that. What stays is the phrasing-independent half: "audit the result against every explicit requirement in the user's request … do not treat producing a plausible result as proof that the task is complete." A genuine request for every qualifying result IS an explicit requirement, so that sentence already covers the case the nudge was reaching for — without keying off the word "each". Removes: EXHAUSTIVE_AUDIT_NUDGE, requests_exhaustive_results, the _EXHAUSTIVE_REQUIREMENT pattern, the QueryState.exhaustive_audit_performed latch, and the forced-turn branch in the loop. Tests: 8842 passed. The replacement test pins the inverse — a completion claim on an exhaustive-sounding request must not buy an extra turn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 26, 2026
…ance (#750) Iteration 2 on the trajectory-step gap against the latest Claude Code (same model, same effort). Iteration 1 (#749) took the subset mean from 25.5 to 13.2 steps against CC's 7.5; this addresses the largest remaining structural difference. Measured on the same tasks: clawcodex emits more than one tool call per assistant turn in **5.7%** of steps, the latest Claude Code in **18.1%**. Every independent pair it does not batch is an extra step, so that ratio is most of what remains between the two step counts. Reading the trajectories side by side shows it plainly — on regex-log CC issued 8 calls, all Bash, batching probes ("for c in node deno perl ruby; do command -v $c; done") and creating files with heredocs; clawcodex issued 17, alternating Write then Bash and probing one command at a time. Cause is a truncated port. The reference bullet has three parts: You can call multiple tools in a single response. If you intend to call multiple tools and there are no dependencies between them, make all independent tool calls in parallel. Maximize use of parallel tool calls where possible to increase efficiency. However, if some tool calls depend on previous calls to inform dependent values, do NOT call these tools in parallel and instead call them sequentially. ... clawcodex kept only the first sentence. What was dropped is what turns a permission into a practice: the imperative to maximize, and the dependency carve-out. The carve-out matters as much as the push — without a statement of when NOT to parallelize, "call them in parallel" is risky advice a careful model will mostly decline, which is what the 5.7% looks like. Same defect class as #748 (dropped qualifiers on the file-creation bullet), and general rather than benchmark-specific: batching independent work is better agent behavior everywhere. Tests: 8843 passed. The new test pins all three clauses, since restoring only the push would leave the model without the safety half. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 26, 2026
#754) * perf(prompt): check the next action's assumptions before running it Iteration 5. First one derived from classifying trajectory steps by PURPOSE rather than by tool, which is what finally located the gap. Seven shared terminal-bench 2.1 tasks, same model (claude-opus-5), same effort (xhigh), per trial, two clawcodex runs against the latest Claude Code: clawcodex Claude Code explore:inspect 1.71 / 1.57 2.43 <- Claude Code inspects MORE explore:read 0.86 / 1.14 0.57 author:inline 2.71 / 2.29 1.14 author:deliverable 1.29 / 2.43 1.14 run:exec 2.29 / 3.57 1.14 <- the biggest single gap meta:memory 0.29 / 0.57 0.00 meta:tasks 0.14 / 0.29 0.00 TOTAL 12.14 / 15.43 8.43 Claude Code buys about 0.7 extra inspection steps and saves 1.2-2.4 execution steps. It looks at the actual data, then writes something that works first time. clawcodex writes from assumptions, runs, and pays in write-run-fix cycles -- run:exec and author:inline are each roughly double Claude Code's, while inspection is LOWER. Every tool-level metric already matches (Bash 78.8% vs 79.7%, Write 11.8% vs 11.9%, per-call density 5.2 vs 5.5 statements, heredoc 20.9% vs 23.4%, exact-repeat 2.4% vs 0.0%). Purpose-level classification is what separated "same actions, more of them" into "fewer looks, more retries". This also explains a failure. Iteration 3 added the opposite instruction -- act once you have enough information, do not re-check what the evidence supports -- and made both step count (+3.2 -> +7.2) and reward (1.000 -> 0.857) worse. It was pushing clawcodex further toward acting before looking, when under-inspection was the actual deficit. That bullet was withdrawn; this is the corrected direction, and the earlier failure is evidence for it. Scope is deliberately narrow: the assumptions the NEXT action rests on, not a survey of the environment. #749 had to remove exhaustive-audit machinery that fired on ordinary English, so the limiter is load-bearing and pinned by the test. Not claimed: a verified step reduction. Same-code replicates on this task set swing up to 11 steps per task and 3.43 in the mean, so at k=1 nothing under ~3 mean steps is resolvable. Claude Code replicates are in flight to establish a k=3 baseline; this change gets measured against it, not against a single run. * perf(prompt): restore the reference's skip conditions on the task tool The port kept "Break down and manage your work with the TaskCreate tool" and dropped the reference's gating, which reads: "Skip using this tool when: There is only a single, straightforward task; the task is trivial and tracking it provides no organizational benefit." What survived is an unconditional imperative. Measured over 14 clawcodex and 21 Claude Code trials on seven shared terminal-bench 2.1 tasks, same model (claude-opus-5) and effort (xhigh), steps per trial by purpose: clawcodex Claude Code delta run:exec 3.36 2.52 +0.83 other 1.43 0.67 +0.76 author:inline 2.50 1.90 +0.60 meta:memory 0.43 0.00 +0.43 explore:env 1.07 0.67 +0.40 meta:tasks 0.21 0.00 +0.21 explore:inspect 1.64 2.29 -0.64 author:file 2.00 2.19 -0.19 TOTAL 13.79 11.38 +2.40 Claude Code spends ZERO steps on the task tool across 21 trials and zero on memory writes. clawcodex spends 0.64 per trial on the two together -- 27% of the whole remaining gap, on bookkeeping rather than the task. This commit addresses the task-tool half, where the fix is a dropped reference qualifier rather than a judgment call. The memory half is deliberately untouched. Its "build up this memory system over time" imperative IS in the reference, and memory is a shipped feature whose quality this eval cannot see -- changing it on step-count evidence is what made iteration 3 regress reward. Also of note in the table: "Claude Code inspects more" (-0.64) survives Claude Code's replication, so the iteration-5 premise was sound even though the change built on it measured null and was dropped. Not claimed: a verified step reduction. 0.21 steps/trial is far inside the same-code noise floor (median 2, max 11 per task). This lands because the dropped qualifier is a real port defect, not because the number is measurable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
clawcodex runs +26% more trajectory steps than the latest Claude Code on the same model at the same effort. This is iteration 1 of closing that gap.
The mechanism
#740 added a one-shot "exhaustive audit": when a request matched a bare quantifier, the loop appended a MUST-enumerate instruction and forced another model turn, backed by a matching system-prompt clause.
The trigger was
\b(all|every|multiple|each|exhaustive|complete set|any other|others)\b— plain English, not a request shape. Against the 89 terminal-bench 2.1 instructions it fired on 49%, almost all incidental: "each line of the file", "where each task is an async job", "in all years". One case inverted the instruction outright —regex-logsays "If multiple dates are present in a line, the regex should match only the last one", and the nudge answered that by demanding exhaustive enumeration.Removed, not tuned
I first tightened the regex and got the trigger rate to 3%. I threw that away: tuning a clawcodex-only heuristic against a benchmark is fitting to the scoreboard. The evidence says the mechanism shouldn't exist:
chess-best-move,regex-chess) with no audit machinery at all;What stays is the phrasing-independent half: "audit the result against every explicit requirement … do not treat producing a plausible result as proof that the task is complete." A genuine request for every qualifying result is an explicit requirement, so that already covers the case the nudge reached for — without keying off the word "each".
Removes
EXHAUSTIVE_AUDIT_NUDGE,requests_exhaustive_results, the_EXHAUSTIVE_REQUIREMENTpattern, theQueryState.exhaustive_audit_performedlatch, and the forced-turn branch.Measured on 7 tb2.1 tasks (same model, effort, dataset)
Comparing only trials that ran clean in both iterations (a crash truncates step counts):
Step gap vs CC: +18.0 → +5.8 (−68%). Distinct verification files per trial: 2.50 → 0.75 (CC 0.25).
Reward went 0.857 → 1.000 with 0 errors (from 3), so the steps came off without losing correctness.
Caveat: 4 comparable tasks, one run each — directionally strong (3 of 4 improved substantially) but noisy at this n. The two iter0 crashes were the known OAuth-rotation issue, not this change.
Tests: 8842 passed. The replacement test pins the inverse — a completion claim on an exhaustive-sounding request must not buy an extra turn.
🤖 Generated with Claude Code