Turn-count deltas are only valid within a model — reports must pin same-model baselines

[tmatup]

Problem

Turn counts are routinely read as an arm-quality signal ("arm B costs 2× the turns of arm A"), but visible turns are strongly model-dependent, because models differ in how much work they batch into one tool call. A turn delta computed across two different models can be mostly model style, not arm cost — and nothing in the reporting surfaces marks that.

Exhibit (measured; same task, same arm, same skill/stack era, all weighted-1.0 greens)

skill-flow-ipe-drive-to-slack, v1 arm:

• gpt-5.6-terra greens: 20–28 tool calls (e.g. nightly run 2026-08-07_04-44-12: 22 visible turns). Terra chains whole phases into one call — solution init && flow init && registry pull && node add && node add is one Bash tool call.
• claude-sonnet-5 green: 50 tool calls (run 2026-07-28_04-16-10). Sonnet issues one probe per call.

Same arm, same task, same bar — a 2.3× turn difference from the model alone.

Consequence (a real decision almost taken on noise)

A recent SDK-vs-v1 turn comparison for three tasks pinned v1 baselines from terra runs while the SDK runs were sonnet-5. Headline: drive-to-slack "+69 turns for SDK". Held to a same-model v1 baseline, the real gap was +40 — roughly 40% of the headline was the batching confound. The remainder was real and got fixed, but the inflated number is what triaged the effort; a differently-shaped confound could just as easily mask a real regression.

Related known layer inconsistency

Turns are already non-comparable at the adapter layer: src/coder_eval/agents/codex_agent.py emits assistant_turn_count=1 while the Claude adapter increments per AssistantMessage. The visible-turns contract (docs/features/2026-05-22-visible-turns.md) normalizes to executed tool calls + one persisted final response, which fixes the adapter mismatch — but no normalization can remove the batching-style difference above. The only sound rule is: turn deltas hold the model constant.

Fix contract

1. Comparison/trends surfaces: wherever a turn metric is shown as a delta or trend across runs/arms, group by model_used; a cross-model turn delta is either suppressed or explicitly badged as cross-model (cost and wall-clock remain fine to compare across models).
2. Docs: amend the visible-turns feature doc with a comparability clause — visible turns are comparable only within a model; cross-model turn comparisons are not a supported reading of the metric.
3. Run rows: any row/table that displays turns also displays model_used, so a human diffing two rows can see the confound without opening task.json.

Acceptance

• No report or dashboard emits an arm-vs-arm or before-vs-after turn delta computed across different model_used values without a visible cross-model marker.
• The visible-turns doc states the same-model rule.
• The drive-to-slack exhibit above (terra 22 vs sonnet-5 50, same arm) renders as two model-grouped rows, not one misleading trend line.

🤖 Generated with Claude Code
Co-Authored-By: Claude