Deepen the sota-v3 simulator with ZenGM-derived mechanics (contracts, pick slots, in-season dynamics)

[nedcut]

Follow-on to #84. GM-Bench's decision loop was modelled on ZenGM Hockey; this issue closes the gaps that a read of the ZenGM engine (GameSim.hockey, developSeason.hockey.ts, ValueChangeCalculator, player/value.ts) makes visible in ours.

Why now, and only now

Every item below moves contract_fingerprint. Today that costs nothing: sota-v3 is open, no paid panel has been bought, and the seven checkpoints under data/model_checkpoints/ are already invalidated by #88. The moment a paid sota-v3 panel runs, all of this becomes unaffordable — a docstring edit to a _CONTRACT_SOURCES file is enough to invalidate the rows. This is the window.

Pre-change reference (fingerprint 8a4eb422d548317a, seeds 11-18 x 5 seasons):

agent	mean_score
pick-trader	411.62
strategic	402.02
scaffold-view	401.95
shrewd	371.77
value	354.62
win-now	275.83
conservative	139.03
rebuild	138.75
exploit	137.13
random	96.72

The diagnosis

The benchmark contests exactly one valuation: hidden true_potential. Difficulty in a management benchmark comes from assets whose value is disputed between the agent's model and the environment's model, and we currently have a single such asset. Everything else is priced identically for both sides, which is a large part of why a 120-line scripted pick-trader beats every frontier model we ran.

Each item below converts an unpriced asset into a contested one.

P1 - Contract economics (revives #24, closed with #62 unmerged)

#62 was closed 2026-07-16 purely to protect the sota-v2 publication lane, not on quality. Its concepts should be reimplemented on current main.

• Dead cap on release. simulator.py:_release currently sets salary = 0.0 with no charge. Any bad contract is erasable at will above the 18-player floor, so the cap_room score term measures no skill. ZenGM keeps released salary on the books.
• Incumbent extension rights. Expiring players drop straight into the shared pool with no advantage to the team that developed them. In ZenGM, re-sign-vs-let-walk is the central GM decision, and it is the one decision GM-Bench never asks.
• Market inflation + term premium so the 1-5 years dial has economic content.

#62 found a real exploit here — sign 1 year, immediately extend 5 at the loyalty discount — and tuned the discount 8% -> 3% to keep INC5 above FA1. Both must be preserved.

P2 - Standings-linked pick values

pick_value is 12.0 * 0.8^n regardless of whose pick it is, so a contender's pick and a cellar-dweller's are interchangeable to both the trade market and score_components.future_pick_assets. ZenGM's ValueChangeCalculator estimates the actual slot from projected records and regresses toward the mean with distance. This is the most likely single explanation for pick-trader's dominance.

P3 - In-season dynamics

simulate_season computes ratings once (simulator.py:330) and draws all 33 games from frozen numbers, so deadline moves cannot affect the standings they were made for. Recompute at the midseason break so the trade deadline is causally connected to the second leg.

P4 - morale is dead

Generated, updated by _update_morale_from_standings, published in Player.public_dict — and read by nothing. It is a pure token tax on agents inside a 4,096-token lane. Either wire it into free-agent/extension willingness (ZenGM's mood system: loyalty / money / playing time / winning) or drop it from the observation.

Constraints

• Validity canaries in validate-contract encode baseline orderings (strategic > shrewd > value > exploit). If a change inverts one, that is a finding to reason about, not a threshold to loosen.
• The exploit canary must not gain ground; a dead-cap rule that made hoarding cheap would be a regression.
• Frozen sota-v2 evidence under results/, releases/, docs/blog/, web/src/data/ stays untouched. SOTA_V2_CONTRACT (558e8f35ea1d66b9) must not move.
• docs/scoring_calibration.md and golden scores get updated with each change, not after all of them.

[nedcut]

P2 design note: picks need an origin, not just a count

Recording this before implementation, because the fix is a data-model change and not a formula tweak.

What the code does today

Team.draft_picks is dict[int, int] — season -> count. There is no record of whose slot a pick exercises. Three consequences, all visible in simulator.py:

1. _draft_order() (line 1026) sorts owners by their own wins, and run_opponent_draft exercises every pick an owner holds at that owner's slot. So a pick acquired from the worst team in the league is exercised at the acquirer's slot. Buying a cellar-dweller's first-rounder buys you nothing but an extra late pick.
2. pick_value(current, season) is 12.0 * 0.8^n — identical for every team's pick. The trade market (simulator.py:626), the observation estimate (line 122), and score_components.future_pick_assets (scoring.py:156) all read that one number.
3. Because value is count-based and slot-free, accumulating picks is close to pure score. The pick-hoard canary is only held in check by the roster cost of acquiring them, not by the picks themselves being priced.

Together these are the most plausible mechanical explanation for pick-trader (411.62) topping the panel: it buys an asset the objective pays for at a fixed rate and the market sells at a fixed rate, so the only skill required is noticing the arbitrage exists.

Proposed change

Ownership. draft_picks: dict[int, int] -> dict[int, list[int]], season -> list of origin team ids. Initial grant becomes {year: [team.id] for year in 1..7}. _transfer_pick, _picks_owned, _validate_pick_seasons, _assign_prospect and _draft all move from counting to list operations.

Draft order. Becomes a list of (origin_slot, owning_team) pairs: sort origin teams worst-record-first, then each slot is exercised by whoever currently holds that origin's pick. This is both more correct and what makes the whole change worth doing — acquiring a bad team's pick now actually gets you an early selection.

Valuation. Follows ZenGM's ValueChangeCalculator, which estimates a pick's slot from projected records and regresses toward the mean as the pick gets further out:

slot_est(origin, S) = mean_slot + (current_slot(origin) - mean_slot) * REGRESSION ** (S - current_season)
value            = PICK_BASE_VALUE * PICK_VALUE_DISCOUNT ** (S - current) * slot_multiplier(slot_est)

with mean_slot = (num_teams + 1) / 2 = 6.5 and slot_multiplier linear in distance from mean. Starting points to tune: REGRESSION = 0.6, slot sensitivity 0.08 (slot 1 ~ 1.44x, slot 12 ~ 0.56x). Regression toward the mean matters — without it a three-years-out pick from today's worst team would be priced as a guaranteed first overall, which is exactly the overconfidence ZenGM's decay is there to prevent.

Two things this must not break

• The objective and the market must stay priced identically. scoring.py and _trade must call the same function with the same arguments. The current design already gets this right, and the docstring at scoring.py:150-153 explains why; slot-awareness must not split them.
• The future_pick_assets horizon defaulting. scoring.py:156 defaults absent seasons to one implicit pick so far-future picks cannot mint score. With origin lists, "absent" needs an explicit rule: a season not in the dict means the team still holds its own pick. Getting this wrong re-opens exactly the mint the current comment guards against.

Observation cost

Per-pick origin plus an estimated value is more tokens than a bare count, inside a 4,096-token lane. Publish it as a compact list ({"season": 4, "from": 7, "est": 14.2}) and check the effect on scaffold-view, which is the instrument for exactly this kind of view-cost question.

[nedcut]

Finding: the validity canaries cannot fail, and P1 exposed it

First P1 implementation landed on feat/contract-economics (dead cap 50%/remaining season, 4% inflation on salaries and cap, 2%/yr term premium, 3% loyalty discount). It is mechanically correct, validate-contract passes every ordering, 537 tests green — and it destroys the benchmark's ability to rank competent policies.

Paired per-seed comparison, same 8 seeds (11-18) x 5 seasons, final_score, paired t over 8 episodes:

pair	BEFORE diff / t	AFTER diff / t
pick-trader - strategic	9.59 / 1.52	7.82 / 0.38
strategic - shrewd	30.26 / 2.46	10.45 / 0.37
shrewd - value	17.15 / 0.88	1.31 / 0.05
pick-trader - value	57.00 / 2.99	19.58 / 0.66

value stddev also went 33.67 -> 61.11.

The headline separation was significant (crit t ~ 2.36 at 7 df) and is now indistinguishable from zero. Every one of the top-five policies now sits inside a 34-point band against per-seed stddevs of 38-61.

The canary gap

validate-contract reported all of this as green:

ok: pick-trader over strategic mean_score margin=7.818 min=0.0
ok: strategic over shrewd      mean_score margin=10.452 min=0.0
ok: shrewd over value          mean_score margin=1.31  min=0.0

A min=0.0 threshold on a mean margin cannot fail as long as the sign is right, and the sign of a noise-dominated difference is a coin flip that happens to be landing our way. shrewd > value at margin 1.310 with a paired t of 0.05 is not an invariant; it is a coin flip we are publishing as a guarantee.

This is the more important half of the finding. The scoring regression is one bad calibration; the canary design has been laundering noise as a contract since before this change, and would do it again for P2/P3/P4.

Why the mechanic misfired

Working hypothesis, being ablated now: dead cap at 50% of salary for every remaining guaranteed season is a large, high-variance, effectively unavoidable tax rather than a decision. Active policies release more, so they are taxed more — which compresses skilled play toward unskilled and inflates variance. The doubling of value's stddev points the same way.

A mechanic that works here has to add a contested valuation ("is escaping this contract worth a known charge?"), not a lottery. That likely means bounding the horizon (min(remaining, 2)) and publishing the exact per-player charge in the observation so it can be priced before the release, not discovered after.

Consequence for the plan

The canary significance gate moves ahead of P2/P3/P4. Landing further simulator changes against thresholds that cannot fail would mean discovering each regression the same way — by hand, after the fact, if at all.

[nedcut]

The 8-seed panel was already underpowered, before P1 touched anything

Follow-up to the canary finding. I ran a power calculation on the pre-change engine (main @ 3ea4864) — seeds needed for 80% power at alpha=0.05, two-sided, paired by seed:

pair	mean diff	paired sd	seeds needed	preset has
pick-trader - value	57.00	53.92	8	8
strategic - shrewd	30.26	34.73	11	8
pick-trader - strategic	9.59	17.91	28	8
shrewd - value	17.15	55.32	82	8

PRESETS["leaderboard"] runs 8 seeds. Only the single widest gap in the panel was ever adequately powered — and it landed exactly at the threshold (observed t = 2.99, crit 2.36). Three of the four orderings validate-contract asserts as invariants have never been supported by the panel that tests them. shrewd > value would need roughly ten times the seeds.

So the canary gap is not something P1 introduced. P1 only removed the margin of luck that was hiding it.

This may reach further than the canaries

The phase-one headline is that all eight model systems form one overlapping tier supporting no ordinal "best model" claim. That is stated honestly in the README and I am not disputing it. But the table above raises a question worth separating: how much of that overlap is the models being genuinely close, and how much is n=8 being unable to resolve differences of this size at this variance?

Those have different remedies. If it is the models, the benchmark is working and reporting a true null. If it is the panel, then a wider seed panel would resolve a ranking that exists and is currently invisible — and the "one tier" finding is partly a statement about our instrument rather than about the field.

The distinction is testable without buying a single new model row: the same power calculation run against the published per-seed model scores in results/ says which regime we are in. That analysis reads frozen evidence only and rewrites nothing.

Cost note

Raising the leaderboard seed count is not free — panel cost scales linearly with seeds, and this decision belongs to whoever is paying for the sota-v3 panel. Flagging it rather than acting on it. Scripted baselines can be re-run at any width for nothing, so the canaries can get their significance gate at a wider panel width regardless of what the paid lane ends up using.

[nedcut]

The phase-one "one overlapping tier" claim is too conservative — the frozen evidence supports a partial order

This is a re-analysis of frozen sota-v2 evidence only. Nothing was re-run, no artifact was modified, and SOTA_V2_CONTRACT is untouched. Freezing the study means not rewriting or rerunning it; reading it correctly is still allowed.

What the published claim says

Every eligible model trailed the transparent pick-trader heuristic (411.619), and all eight model uncertainty intervals form one overlapping tier — so the evidence supports no ordinal "best model" claim.

What the per-seed data says

The eight models ran the same 8 seeds (public-leaderboard, sha c68416e4...), 3 repeats each. Averaging repeats within seed and comparing models pairwise on shared seeds:

correction	pairs separable / 28
uncorrected, t > 2.365 (7 df)	20
Bonferroni, t > 3.90	11
sign test p < 0.05	8
clean 8-0 or 0-8 seed sweeps	8

Eleven pairs survive full Bonferroni correction, and eight of those are 8-of-8 seed sweeps — one model beating another on every single seed, by margins up to 102 points:

z-ai/glm-5.2            > minimax/minimax-m3        +87.66   t 5.94   8/8 seeds
meta/muse-spark-1.1     > minimax/minimax-m3       +101.97   t 5.19   8/8 seeds
z-ai/glm-5.2            > anthropic/claude-sonnet-5 +75.40   t 5.24   8/8 seeds
meta/muse-spark-1.1     > anthropic/claude-sonnet-5 +89.71   t 5.01   8/8 seeds
meta/muse-spark-1.1     > openai/gpt-5.6-luna       +57.93   t 4.44   8/8 seeds
...

Why the published reading understated this

The claim compares each model's own marginal interval and observes that they overlap. But seed difficulty is a large shared variance component — some leagues are simply harder to manage from — and every model faced the identical panel. Comparing marginal intervals throws that pairing away and asks a much weaker question. Non-overlapping intervals imply significance; overlapping intervals do not imply non-significance, and with paired designs the gap between those two readings is exactly this large.

The repo already knows this. paired.sign_flip_p_value and paired.significant_at_95 are computed in every artifact for candidate-vs-baseline. The paired method was simply never turned on model-vs-model.

What is actually supported

A partial order, roughly three tiers under the frozen 4,096-token lane:

• {muse-spark-1.1, glm-5.2, gemini-3.5-flash, hy3:free} — not separable from each other
• {qwen3.7-plus, gpt-5.6-luna}
• {claude-sonnet-5, minimax-m3}

Two things to keep saying, because both remain true:

1. No single best model is demonstrable. The top four are not separable from one another, so "highest observed mean" still does not license "best". The original caution was right about the top of the table — it was wrong to extend it to the whole table.
2. Every model still trails pick-trader (411.619). The tiering does not change the headline result.

And the lane caveat stands in full: this is a statement about eight systems under one 4,096-token cap at native-minimum reasoning, not about the models in general.

Why this matters beyond the wording

It changes what GM-Bench is. "Cannot rank models" and "ranks models into three tiers and shows all of them losing to a 120-line heuristic" are very different claims about an instrument, and the second one is both stronger and better supported by evidence already paid for.

Follow-up work, none of it requiring a new panel row:

• add pairwise paired model-vs-model comparison to scripts/ and surface tiers on the site
• revisit the Analysis view's sorted ladder with no uncertainty (already noted in Close the gap between sota-v2 evidence and framing: scoring, observation asymmetry, scaffolds, power, and site claims #84 P4/P7) — it can now show tier bands instead
• fold the same paired machinery into the validity canaries per the finding above, so baselines and models are judged by one standard

[nedcut]

Correction to the tiering comment above

I wrote that the evidence supports "roughly three tiers". That was based on uncorrected pairwise tests. With a proper multiple-comparison correction across all 28 pairs it is two overlapping tiers, not three. Correcting my own number before it gets quoted:

correction	pairs separated / 28	tiers
none	20	3 (overstated — 28 tests at alpha=.05 expects ~1.4 false positives)
Holm	8	2, overlapping
Bonferroni	7	2, overlapping

Holm is the right default: it strictly dominates Bonferroni (rejects at least as much, controls the same family-wise error rate), and 8 >= 7 above confirms the implementation behaves.

What the evidence actually supports

model                          mean   tier
meta/muse-spark-1.1          231.85   A
z-ai/glm-5.2                 217.54   A
google/gemini-3.5-flash      215.62   A
tencent/hy3:free             195.84   A
qwen/qwen3.7-plus            175.52   AB
openai/gpt-5.6-luna          173.93   AB
anthropic/claude-sonnet-5    142.14   B
minimax/minimax-m3           129.88   B

Every separated pair is a 7/8 or 8/8 seed sweep, so what survives correction is not marginal:

z-ai/glm-5.2            > minimax/minimax-m3        +87.66  t 5.94  p_adj .016  8/8
tencent/hy3:free        > minimax/minimax-m3        +65.96  t 5.33  p_adj .029  8/8
z-ai/glm-5.2            > anthropic/claude-sonnet-5 +75.40  t 5.24  p_adj .031  8/8
meta/muse-spark-1.1     > minimax/minimax-m3       +101.97  t 5.19  p_adj .032  8/8
google/gemini-3.5-flash > anthropic/claude-sonnet-5 +73.48  t 5.12  p_adj .033  7/8
google/gemini-3.5-flash > minimax/minimax-m3        +85.74  t 5.05  p_adj .034  7/8
meta/muse-spark-1.1     > anthropic/claude-sonnet-5 +89.71  t 5.01  p_adj .034  8/8
tencent/hy3:free        > anthropic/claude-sonnet-5 +53.70  t 4.73  p_adj .045  8/8

The substantive correction to the published claim stands, in weaker form than I first stated: "one overlapping tier" is not supported — a top group is cleanly separated from a bottom group, on eight pairs, under full correction, with 7-8 of 8 seeds each. But it is two groups with genuine overlap in the middle, and qwen3.7-plus / gpt-5.6-luna belong to both.

Both original cautions survive intact and should keep being stated:

1. No single best model is demonstrable. The top four do not separate from each other at all.
2. Every model still trails pick-trader (411.619).

Tooling and the revised wording are on analysis/paired-model-tiers. The script refuses to compare rows across differing contract fingerprints or seed panels, which matters now that sota-v3 is diverging from the frozen sota-v2 rows this analysis reads.