brief-to-battle: retract the hold_the_gate failures, generalize the harness

[lxsolutions]

Retraction

PRs #92 and #93 published a finding that Claude has a "systematic reasoning gap on inverted/defense briefs." That finding is withdrawn. It measured a defect in our harness.

agents/claude_battle.py ended its prompt with a hardcoded goal sentence:

Your scenario must end with: the main gate destroyed or open (not blocking), the side gate intact and closed (blocking).

That is the fortress_battle objective. hold_the_gate is its exact inverse — and every brief got the same sentence. The harness was instructing the model to open the gate the brief said to defend, then scoring the model's compliance as a tactical failure.

The "not variance, systematic" conclusion was really "not variance, constant": both runs matched the injected instruction, which is precisely why they looked so consistent. The consistency we treated as evidence of a model gap was evidence of a harness bug.

The corrected results

With the goal derived per-brief, the same brief and the same model pass on the first attempt — 66.5s, 4/4 dimensions:

[0, gate_main, lock]      [0, gate_side, open]
[0, gate_main, close]     [0, gate_side, unlock]
[2..20, gate_main, attack 12 / repair 12 alternating]

It locks the defended gate so stray open events are absorbed, opens the escort route, and sustains the permitted repair cadence for all 20 ticks. The tactics were never the problem.

The second, quieter limit

The wrapper also rejected any scenario that was not exactly two entities. The brief format already supported arbitrary entity maps, so this cap was invisible unless you read the wrapper — the corpus could only ever contain gate pairs.

Generalized to N, plus a brief that exercises it: three_gates — the sally port opens, the main gate and the postern must both be standing and shut at tick 20. Two holds and one open, with a third entity as distractor pressure. Claude Code passes it first attempt, 107.0s, 4/4. The reference baseline is now 3/3.

The neutral example event matters too: the old template showed the side gate opening, which is the answer to two briefs and the wrong answer to the third.

Changes

• goal_clause() derives the goal from each brief's expect_navigation; the template states no outcome
• render_block() builds world/replay/initial sections from the brief's entity list
• WrapperIsBriefNeutral — fails if the template states any outcome, or if a derived goal contradicts a brief. Verified: fails against the old wrapper, passes against the fix
• SCOREBOARD.md — the two entries struck through and retracted in place rather than deleted; retracted evidence should stay auditable
• Corrected + three_gates scorecards committed

Why this matters more than the bug

This project's thesis is that AI output is untrusted input that must be compiled, measured, and proven. The instrument doing the proving was not held to that standard, and it published a false finding about someone else's model as a result. A benchmark whose failures might be its own misfiring costs more credibility than it buys.

Both defects were the same species: an assumption baked into the wrapper that the brief format was already general enough to avoid. The follow-up worth doing is harness self-tests as a category, not a one-off.

🤖 Generated with Claude Code