fix(agent): verify exhaustive result requests - #740
Merged
Conversation
ericleepi314
added a commit
that referenced
this pull request
Jul 26, 2026
…nglish (#749) #740 added a one-shot "exhaustive audit" pass: when a request matched a bare quantifier, the loop appended a MUST-enumerate instruction and forced another model turn, backed by a matching system-prompt clause. The trigger was `\b(all|every|multiple|each|exhaustive|complete set|any other|others)\b` — plain English, not a request shape. Measured against the 89 terminal-bench 2.1 instructions it fired on 49% of them, almost all incidental: "each line of the file", "where each task is an async job", "in all years". One case inverted the instruction outright — regex-log says "If multiple dates are present in a line, the regex should match only the last one", and the nudge answered that by demanding exhaustive enumeration. Removed rather than tuned. Tightening the regex would have been fitting a clawcodex-only heuristic to a benchmark; the evidence says the mechanism should not exist: * the latest Claude Code scores 1.0 on both tasks this was built for (chess-best-move, regex-chess) with no audit machinery at all; * only 1 of its 89 trials writes more than two verification files, where clawcodex wrote 57 temporary verification scripts across the run; * clawcodex is +26% on mean trajectory steps against the same model at the same effort, and forced audit turns are part of that. What stays is the phrasing-independent half: "audit the result against every explicit requirement in the user's request … do not treat producing a plausible result as proof that the task is complete." A genuine request for every qualifying result IS an explicit requirement, so that sentence already covers the case the nudge was reaching for — without keying off the word "each". Removes: EXHAUSTIVE_AUDIT_NUDGE, requests_exhaustive_results, the _EXHAUSTIVE_REQUIREMENT pattern, the QueryState.exhaustive_audit_performed latch, and the forced-turn branch in the loop. Tests: 8842 passed. The replacement test pins the inverse — a completion claim on an exhaustive-sounding request must not buy an extra turn. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
On terminal-bench 2.1
chess-best-move, clawcodex founde2e4and stopped despite the request to return all winning moves. Claude Code mechanically enumerated candidates and also foundg2g4. A prose-only follow-up audit repeated the same error, so the runtime guard now requires tool-backed validation for mechanically enumerable state spaces.Verification
pytest -q tests/test_query_loop_wires_round3.py tests/parity/test_query_state_parity.py tests/parity/test_error_recovery_flow.py tests/test_system_prompt_full.py tests/test_query_extended_thinking.py tests/test_effort.py— 125 passedterminal-bench/chess-best-move, Claude Opus 4.8, effort high — reward improved from 0.0 to 1.0eval/harbor/jobs/chess-rerun-tool-backed-audit