Close the gap between sota-v2 evidence and framing: scoring, observation asymmetry, scaffolds, power, and site claims

[nedcut]

Status reconciliation — 2026-08-01

Current main: a0fdec5493eaf5f702e71c519910e03f39e727f5 (through PR #104).

Completed since this issue was filed

• Priority-0 non-finite input, decision-window walk-away, and compact/raw artifact-integrity defects were repaired in PR Fix P0 integrity defects and open the sota-v3 contract #85 and further hardened in PR Harden SOTA-v3 pre-spend readiness #104.
• The same-view scaffold-view reference and official-seed measurement landed in PRs Open the sota-v3 lane: score components, strict default, same-view reference #88 and Measure scaffold-view gap on official seeds 11–18 (#93) #98.
• Matched-seed Holm-corrected model tiering and narrower public claim framing landed in PRs Add matched-seed model tiering for published panels #90, Fix public-site claim framing for scores, oracle, and tokens #95, and State the phase-one claim as a decomposition, not a hedge #101.
• The SOTA-v3 lane now has a fail-closed, zero-spend preflight/rehearsal and release-gating package through PRs Add fail-closed sota-v3 preflight #102–Harden SOTA-v3 pre-spend readiness #104.

Still open

• Model-row score-component sensitivity and an outcome-only secondary view.
• Reasoning/session/full-observation companion lanes and a model memo ablation.
• Remaining construct-validity work, including decorative morale and opponent asymmetry/realism.
• Held-out private-panel evidence, raw-trace release completeness, and external reproduction.

This issue remains open as the scientific-validity and interpretation backlog. The original checklist below is retained as the audit record; unchecked boxes should be read against this reconciliation rather than assumed to describe the current implementation.

---

Summary

An independent review of the sota-v2 phase-one release found that the methodology, disclosure, and reproducibility engineering are strong, while the headline comparison is heavily shaped by a hand-designed composite score, a white-box reference policy, a fresh-spawn/memo-only scaffold, and an eight-seed public panel. That does not make the result invalid; it narrows what the result measures.

The durable, defensible finding today is: under the frozen native-minimum-reasoning, fresh-spawn/memo-only, hand-scored public protocol, all eight eligible model-plus-scaffold systems trailed the white-box pick-trader reference on all eight seeds. The items below either fix concrete correctness/integrity defects or help the benchmark support broader claims without outrunning its evidence.

---

Priority 0 — Correctness and evidence-integrity blockers

These are new findings reproduced against local HEAD 1591b1d without modifying the checkout.

0A. Non-finite salaries bypass contract and cap validation

_sign_free_agent converts salary with float(...) and then uses ordinary comparisons. Python accepts NaN through json.loads; every comparison against it is false, so a NaN offer bypasses the positive-salary, hard-cap, and reservation-price checks and is accepted. A direct probe signed 24 free agents in one round at salary: NaN, grew the user roster from 23 to 47, and left payroll/cap room as NaN.

Evidence: gm_bench/simulator.py:512-551; permissive parsing begins in examples/gm_agent_common.py:231-237.

• Reject non-finite numeric action inputs with math.isfinite, especially salary and query thresholds.
• Add adversarial tests for NaN, Infinity, and -Infinity at the adapter/parser and simulator boundaries.
• Serialize publication JSON with non-finite values forbidden so poisoned state cannot become an artifact.

0B. Negotiation walk-aways reset per interaction round, not per decision window

The prompt promises that a counterparty stops negotiating after two declines “until your next decision window.” However, League.apply_actions() resets window_walkaways on every call (simulator.py:201-205), while run_decision_point() calls it once per interaction round (runner.py:176-202). A probe produced two declines, then a third decline after the next call instead of the expected “broken off negotiations” result; the counter reset from 2 to 1.

An agent can keep the window alive with an accepted query and resume hidden-value probing on the next round, defeating the stated binary-search guard.

• Reset walk-away state once at decision-window entry, not inside each round's apply_actions.
• Add an end-to-end multi-round regression covering both free-agent and team counterparties.

0C. A tampered compact artifact passes both strict validation and panel analysis

_validate_episode_panel() checks seed/repeat coverage and season counts, but it does not validate finite score fields or recompute summaries, normalized values, or paired statistics from episode rows (official.py:623-656). The publication analyzer checks that publication.raw_artifact_sha256 merely looks like a 64-character hash; it does not bind the compact artifact to the raw artifact (scripts/analyze_publication_panel.py:319-363).

In an in-memory probe, setting every Muse candidate episode score and its displayed aggregates to 9999 still yielded:

• validate_leaderboard_payload(..., SOTA_V2_POLICY).ok == True
• one eligible panel-analysis row
• computed mean lift 9587.381125
• the old declared raw-artifact hash left untouched

The tagged release manifest/checksums are good external evidence, but the routine validator/analyzer/site path does not enforce that link.

• Validate every compact numeric field as finite.
• Recompute candidate summaries, normalized blocks, and paired statistics from compact episode rows and reject mismatches.
• When raw evidence or a release manifest is available, verify publication.raw_artifact_sha256 and compact-to-raw linkage rather than checking hash syntax only.
• Add a tampered-artifact regression that must fail the strict validator, analyzer, and site builder.

Priority 1 — Scoring validity

Problem. The composite is hand-designed and encodes a defensible but non-neutral preference for sustainable asset accumulation. Published compact rows do not retain score components, so weight sensitivity cannot be recomputed on model rows and readers cannot see how much of a result comes from wins/titles versus terminal assets, cap room, strength, and depth.

A prior draft of this issue used an invalid championship-rate comparison. The correct denominators strengthen the need for careful reporting but do not show GLM outperforming pick-trader on titles:

Agent	Score	Titles	Titles / 5-season episode	Title rate / season
pick-trader	411.619	14 / 8 episodes	1.750	35.0%
GLM 5.2	217.539	11 / 24 episodes	0.458	9.2%
Muse Spark 1.1	231.851	8 / 24 episodes	0.333	6.7%

So the current evidence does not establish a “won more, scored less” contradiction. The real limitation is that the composite's value judgments cannot yet be stress-tested on published model end states.

Proposed work:

• Persist per-episode score components in result artifacts (a new contract-lane change) so weight sensitivity can run on model rows.
• Report an outcome-only secondary view (championships, playoff rounds, wins) alongside the composite; label it descriptive rather than silently redefining the primary endpoint.
• Document explicitly that the primary composite rewards sustainable asset accumulation over win-now mortgaging.
• Run a predeclared sensitivity sweep over model-row components and report whether the “all models trail pick-trader” conclusion survives.
• Normalize championships consistently by episodes or total seasons everywhere; never compare candidate repeat totals directly with a deterministic baseline's single-repeat totals.

Priority 2 — Quantify the white-box/scaffold asymmetry

Problem. Every baseline is authored with direct knowledge of the environment and score. The published oracle adds only 19.5 points over pick-trader, but OracleAgent explicitly calls itself a partial hidden-information reference and does not optimize globally (gm_bench/oracle.py:49-62). Therefore 19.5 points is the gain achieved by this particular partial oracle, not proof that pick-trader is near-optimal or that the environment has only 19.5 points of headroom.

The observation paths also differ materially. Scripted agents receive the full simulator observation. The compact model scaffold keeps the full 23-player user roster but preselects at most 16 of 36 initial free agents, 16 of 60 draft prospects, and 12 trade-market entries (examples/gm_agent_common.py:112-181), while also injecting a host-computed legal fallback lineup (:184-227). This is a legitimate model-plus-scaffold condition, but not a controlled “same information, different decision-maker” comparison.

Proposed work:

• Replace “near-optimal” / “near-ceiling” language with “strongest tested white-box reference” unless a real optimization bound is built.
• Build a stronger search/planning oracle or a provable upper bound before interpreting the oracle gap as total strategic headroom.
• Add a same-view reference that operates only on the exact compact payload shown to models, including the same candidate truncation.
• Conversely, run a model companion lane over the untruncated full observation to estimate the cost of scaffold compaction.
• Add a weight-blind baseline track, tuned against a held-out scoring variant, to estimate how much of pick-trader's edge comes from exact score knowledge.
• Report the partial-oracle gap (19.5) next to the MDD (62), with the explicit caveat that the oracle is not an upper bound.

Priority 3 — Separate scaffold effects from model capability

Problem. The frozen lane pins a 4,096-token output ceiling, uses native-minimum reasoning (disabled where optional and lowest supported effort where mandatory), and fresh-spawns the agent between decision points so only a 2,000-character memo carries state. That is a valid, reproducible condition, but it measures model-plus-scaffold behavior under constrained continuity; it should not be generalized to unconstrained model planning capability without companion conditions.

Additional scaffold confounds:

• Soft failure substitution: non-strict fallback_actions still drafts the best public asset and sets a legal lineup while tagging model_error. Failures are counted (decision_failure_rate / sota-v2 ≤2% gate) but still move the roster, biasing scores upward for flaky adapters unless GM_AGENT_STRICT=1.
• "Same lane" ≠ equal effective compute: usage still spans ~7.6k–10.8k tokens/decision across eligible rows; score tracks tokens/decision nearly monotonically (README already notes this).
• Memo / fresh-spawn cost falls almost entirely on models: scripted strategic memo ablation is 0 points because it reconstructs policy from the observation (docs/scoring_calibration.md).

Proposed work:

• Add a reasoning-enabled companion lane (clearly separate from the frozen headline lane) so the reasoning-off result can be read as a condition, not a verdict on capability.
• Promote the --session (full-context) condition to a first-class reported lane; the delta between memo-only and full-context is itself the measurement the benchmark is designed to produce.
• Report an output-token-budget sweep result for at least one model so readers can see how sensitive the score is to the 4,096 ceiling.
• Make GM_AGENT_STRICT=1 (noop-on-failure) the default for any publication lane, or publish both soft-fallback and strict scores so fallback contribution is measurable.
• Show tokens/decision (and preferably input vs output) beside every model row on every public surface that sorts by score — already required by policy; enforce visually so bare-score reading is hard.

---

Priority 4 — Statistical power / the "leaderboard" framing

Problem. With 8 seeds the study cannot separate any two models (all lifts in one tier; pick-trader→oracle band is below the MDD). The 12/16/24-seed MDD rows are extrapolations from resampled 8-seed residuals, not measured. For GPT-5.6 Luna, within-seed run-to-run stddev (35.8) exceeds across-seed stddev (24.8) — model sampling noise is larger than seed variation, absorbed by only 3 repeats. This sits in tension with the new sports-wire "box score" site and publishable_ranking: true.

Holm-adjusted exact sign-flip results for every eligible model vs pick-trader are 0.078125 with holm_reject_at_0_05: false (results/analysis/publication-panel-analysis.json). Unadjusted (p = 0.0078125) (lose on every seed) is real; family-wise significance at α=0.05 is not. "None beat" is an observed-mean claim, not a corrected inference claim.

publishable_ranking: true means "the evidence table may be shown," but the name invites readers to treat the sorted table as a resolved ranking — conflicting with the overlapping-tier / no-ordinal-winner messaging.

Proposed work:

• Reconcile the site presentation with the no-ranking result: the box-score/leaderboard layout implies an ordinal ranking the analysis explicitly withdraws. Add a persistent "why there is no Add repo hygiene: CI, .gitignore, LICENSE, and dev deps #1" callout, or render models as an unranked tier rather than a sorted ladder.
• Rename or gloss publishable_ranking in user-facing copy (e.g. "table publishable / ranking unresolved") so the flag cannot be read as "ranking is valid."
• Increase repeats where within-seed > across-seed stddev (Luna, GLM), or report an explicit note per-row when model noise dominates seed noise.
• Expand the seed panel (even to 12–16) for any future lane that wants to make any model-vs-model claim; label the current 8-seed MDD rows clearly as extrapolation on the site, not just in the doc.
• Put Holm-adjusted vs unadjusted status next to the hero/mean-gap claim wherever "none beat pick-trader" appears.

---

Priority 5 — Validate memo, the mechanism the benchmark exists to test

Problem. memo is sold as what makes "multi-season coherence observable," but it has exactly zero effect for the deterministic reference (per calibration), and there is no ablation isolating whether models benefit from it. The one capability distinguishing this benchmark from a sequence of independent greedy decisions is asserted, not demonstrated.

Proposed work:

• Add a memo-ablation for model rows: run at least one model with memo disabled vs enabled and report the score delta. If models gain nothing, the "long-horizon coherence" claim needs to be dropped or reframed.

---

Priority 6 — Environment realism vs marketing claims

Problem. Several simulator surfaces look like GM realism but do not feed decisions or score, while marketing language overstates opponent sophistication.

• morale is updated from standings (_update_morale_from_standings) and shown in public_dict, but is never consumed by strength, development, or scoring — observation noise / decorative state.
• On-ice sample is tiny (REGULAR_SEASON_GAMES_PER_PAIR = 3 → ~33 games/team). Wins enter score at 0.42 each; seasons are high-variance relative to the weight they carry.
• Opponent FA/draft/trade policies are greedy / formulaic / limited one-for-ones — not learning multi-agent GMs. Site copy that implies "eleven AI rivals" (or similar) overstates the competitive pressure models actually face.
• Protocol competence remains a large measurement surface even after the strategy/protocol split: best published mean_strategy_score ~237 vs pick-trader ~412. Illegal-action and failed-query rates show dense-schema literacy is still part of what is being measured (prompt scaffolding already summarizes score terms).

Proposed work:

• Either wire morale into strength/development/score, or stop emitting it in observations and marketing copy.
• Soften or precise opponent language on the site ("scripted opponent offices," not implied autonomous GMs).
• Optionally offer a higher-games / lower-noise season profile as a companion lane if win terms are meant to discriminate skill rather than luck.
• Keep strategy vs protocol split prominent; add a one-line "what this measures" note that JSON/API literacy is part of the task under the current scaffold.

---

Priority 7 — Site / framing consistency

Scope note. The two Hero findings below are from the current local cursor/sports-wire-redesign-0350 worktree and are not present in committed HEAD 1591b1d. Track them here as pre-merge review findings, not as defects in the tagged phase-one release.

Problem. The findings blog is careful; the redesign's first viewport is not.

• Redesign Hero: "Eight frontier models. None beat a scripted GM" — observed means only; omits Holm non-rejection and observation/scaffold conditions.
• Redesign Hero subcopy uses {modelCount} for "against N scripted heuristics" — that is the model count, not the baseline count (web/src/components/Hero.tsx). Happens to be 8/8 today; still the wrong variable.
• Public seeds 11–18 are committed in-repo; observations include "seed"; league gen and even true_potential are regenerable (gm_bench/oracle.py). Docs correctly call public rows reproducibility artifacts, not contamination-resistant SOTA — the site still leads with them without that caveat in the first viewport.
• Researcher degrees of freedom are logged (docs/PUBLISH_READINESS.md decision log: panel reshuffles, cap policy flips, route substitutions, evidence resets). Good that it exists; public surfaces should point at it so "frozen" is not read as "never forked."

Proposed work:

• Rewrite the hero claim to match the findings blog's narrow durable conclusion (conditions + observed means + no ordinal ranking).
• Fix the baseline-count copy bug (use actual baseline panel length).
• Surface public-panel contamination / adaptation risk on the site, not only in the spec (also listed under Smaller items below).
• Link the publish-readiness decision log from the results / about surface.

---

Smaller items

• Keep the predeclared telemetry-completeness headline gate: excluding Grok and Mistral from the strict lane is conservative and avoids post-hoc imputation. Show them prominently in a separate diagnostic table, and only introduce an imputed-telemetry category prospectively under a frozen rule.
• On any leaderboard-styled surface, guard against the free Tencent HY3 ($0.00) > paid Claude/GPT rows being read as a quality ranking — the no-ranking stance needs to be visually enforced, not just stated in prose.
• Note the public-seed-panel + committed-baselines adaptation risk on the site, not only in the spec.
• Historical proof this class of bug matters: archive the v1 scout prospect_id vs player_id failure mode next to any "models can't GM" claim (up to 1,124 silent scout failures; baselines unscathed) so readers see how protocol bugs can wreck rankings unevenly.
• Website protocol snippets should match production envelopes ({"actions":[...],"usage":...}) where that is what adapters actually emit.

---

What NOT to change

The self-correcting process is the project's biggest asset and should be preserved: pre-registration, Holm correction across the full model family, the strategy/protocol score split, contract/scoring fingerprints, seed-panel hashing, the source-fingerprinted baseline cache, and the exploit canary + regression test. The v1→v2 scout-contract retraction is exemplary. These are why the benchmark's own decomposition was able to show the model deficit is real strategy (pre-penalty strategy_score ~235 vs 411), not JSON-discipline noise — keep that instrument.

Also keep: failed_queries visibility, API vs coding-harness lane separation, usage/cost/latency as first-class comparison fields, and the refusal to silently rerun poor eligible rows for a better number.

Filed from an independent review of the sota-v2-phase-one-2026-07-19 release; augmented with a second independent code/methodology pass (observation truncation, soft fallbacks, public-potential scoring, site framing, environment realism).

[nedcut]

2026-07-24 status

Priority 0 is complete and merged in #85 as 1e5cd44.

• non-finite numeric action inputs are rejected and publication JSON forbids non-finite values;
• negotiation walk-aways now persist across the whole decision window;
• compact artifacts are recomputed and compact-to-raw hashes are verified when raw evidence is supplied;
• the published sota-v2 contract remains literal, frozen historical evidence;
• corrected behavior begins at sota-v3;
• the old v2 registry/lane cannot unlock paid runs on current HEAD;
• the phase-one site still rebuilds all 8 v2 rows from frozen v2 evidence;
• the findings blog and living publication checklist now carry the post-release disclosure.

Verification on the exact PR head: 483 tests across Python 3.11/3.12/3.13, Ruff, web build/data validation, contract canaries, CodeQL, and both Semgrep jobs passed. All 8 committed v2 rows passed historical validation, and all 8 compact-to-raw hash links passed against the retained raw panel directory.

Next focus should be the claim-strength work, not another paid model panel:

1. persist score components and add the outcome-only/sensitivity view;
2. quantify compact-view and white-box-reference asymmetry;
3. tighten no-ranking/power framing across the site;
4. run memo/session/reasoning companion conditions only after their lane and stopping rules are pre-registered;
5. obtain a documented independent clean-clone reproduction.

[nedcut]

2026-07-24 — P4/P7 site-framing remainder after the results-first redesign (#87)

Reviewed the current site against P4 (statistical power / leaderboard framing) and P7 (site
consistency) at ffaabd7 (#87), which replaced the components the original P7 findings were written
against. Line references below are against that commit; web/src/components/Hero.tsx no longer
exists.

Closed by #87 — no further action

• P7.1 hero claim. "Eight frontier models. None beat a scripted GM" is gone. The first
  viewport now reads "Performance against the baseline panel," with the Holm non-rejection stated
  inline: "Descriptive intervals; the predeclared Holm-adjusted family test does not reject at
  0.05" (web/src/components/ResultsExplorer.tsx:499).
• P7.2 baseline-count copy bug. Moot — Hero.tsx was removed. The surviving modelCount
  use ("Above scripted bar: 0/8") is genuinely a model count.
• P7.4 decision log link. docs/PUBLISH_READINESS.md is linked from the footer
  (web/src/components/Footer.tsx).
• P4 primary comparison. The lift view is a forest plot with 95% intervals, captioned "All
  eight rows overlap in one descriptive tier" (ResultsExplorer.tsx:36-176). This is substantially
  what P4 asked for.

Non-issue

• publishable_ranking appears only as a gating condition (web/src/benchmarkData.ts:78) and is
  never rendered as user-facing copy. The P4 rename/gloss item applies to the JSON payload and docs,
  not the site.

Still open — P4/P7 remainder

• The Analysis chart is a bare ranked ladder with no uncertainty.
  web/src/components/Analysis.tsx:93-124 renders point estimates only, sorted descending, with
  class names analysis-ranking / ranking-row / rank-line. The interval-line / interval-cap
  marks that carry uncertainty in the forest plot are absent. Net effect: the site establishes "one
  overlapping tier, no ordinal winner" in section one, then presents a clean 1-through-8 ladder in
  section two under the headline "Where the model systems lose ground." A reader scrolling normally
  encounters the ranking last, uncaveated. This is the original P4 defect relocated rather than
  resolved. Fix: render intervals on this chart, or drop it to an unranked/tiered presentation.

• oracle ceiling overstates the oracle. Analysis.tsx:87 labels the oracle line
  "oracle ceiling {value}". OracleAgent describes itself as a partial hidden-information
  reference that does not optimize globally (gm_bench/oracle.py:49-62). P2 asks to drop
  "near-optimal" / "near-ceiling" language unless a real optimization bound is built; the chart label
  asserts an upper bound the code explicitly disclaims. Fix: relabel as "partial oracle reference"
  (or similar) and carry the P2 caveat.

• tokens_per_decision is in the data and rendered nowhere. The field is present in
  web/src/data/leaderboard.json and typed at web/src/types.ts:88, but no component reads it
  (verified by grep across web/src/components/*.tsx). P3 requires tokens/decision beside every
  model row on every public surface that sorts by score — "already required by policy; enforce
  visually." Both score-sortable surfaces omit it: the results table
  (ResultsExplorer.tsx:398-443) and the Analysis ladder. This matters more than a normal missing
  column: the README already notes score tracks tokens/decision nearly monotonically, so the omission
  is precisely what enables the bare-score reading the policy exists to prevent.

• No contamination / adaptation caveat on any public surface. Seeds 11–18 are committed
  in-repo and observations include "seed". Grep across all site components found no mention. The
  nearest copy — "No browser automation, no memorized rosters"
  (web/src/components/HowItWorks.tsx:96) — arguably implies a contamination-resistance the public
  panel does not have. (This is the "Smaller items" entry about noting adaptation risk on the site,
  not only in the spec.)

• The Holm caveat does not travel. It appears once, as a <small> in the results hero. P4
  asked for a persistent callout; neither the Analysis section nor the results table carries it.

Note on enforcement

The first and third items share a root cause: uncertainty (ci95) and effort context
(tokens_per_decision) both exist and are correct in leaderboard.json, and get dropped at specific
render sites. Payload-level data validation cannot catch this — the omission happens in JSX. If this
should be enforced rather than re-audited after each redesign, the check needs to assert on rendered
output (e.g. any component displaying mean_score must also display ci95 and
tokens_per_decision), not on the payload.

Filed while the pre-run sota-v3 contract work (persisted score components, strict-by-default
publication lanes, same-view reference) is in progress on a separate branch; these site items are
independent of that work and touch no Python.