retire benchmark v2-v6 behind a data-layer version floor

[Peyton-Spencer]

Summary

Benchmark versions below 7 are retired. This makes them unscoreable by construction rather than by configuration: the bound moves into the schema, where no application path can be on the wrong side of it.

v2-v5 were already unreachable through three forward-only guards (benchmark_rollout.py, admin_benchmark_rollout.py, and the benchmark_rollout_forward CHECK). v6 was unreachable only by configuration, and two paths could still reach it:

1. The source-backfill resume path. _issue_source_backfill_ticket looked up an existing unexpired from_version lease and re-issued it above its own retired-era gate — and request_job deliberately resurrects the activated v7 rollout to feed that lane, whose from_version is 6. So a v6 lease renewed itself with no rollout open and no flag set, and the score at the end was validated against ticket.bench_version rather than the active version.
2. allow_retired_era_backfill. An MCP-exposed runtime setting whose own docstring advertised that flipping it restored retired-era admission "without a deploy". One Backroom write re-opened v6.

Both were verified present on main before this change. Production at the time of writing: active v7, no rollout open, queue_policy_settings_revisions empty (the setting had never been written), and zero issued sub-v7 tickets — the newest sub-v7 deadline passed 2026-07-26.

Merge ordering — Backroom has already landed; this closes the pair

ditto-assistant/backroom#103 merged as 632f8cc on 2026-07-28 19:26 UTC, ahead of this PR. That was the safe half of the pair, and the ordering constraint is now discharged rather than pending. Recorded here because both bodies were written while it was still an open draft and said "must land simultaneously".

The constraint was real but asymmetric, and it landed on the harmless side:

• Platform without Backroom — the dangerous direction, and the one that did not happen. The platform model is extra="forbid" and Backroom sends the carryover block whole on every write, so a Backroom still carrying allow_retired_era_backfill would have made set_queue_policy_settings return 422 for every field, not just this one. Operator queue-policy settings would have broken fleet-wide.
• Backroom without platform — what actually happened, and it is inert. Verified rather than assumed, in both directions: on the write path main's allow_retired_era_backfill: bool = False carries a default, so a payload omitting it validates and the platform fills it in — extra="forbid" rejects unknown keys, not absent defaulted ones. On the read path Backroom's prevGenCarryoverSchema is a plain z.object, not .strict(), so Zod strips the field the platform still returns. No 422, no read failure, nothing operator-visible in the gap.

So the live state right now is a Backroom that has stopped sending a switch the platform still has and still ships False. Merging this PR removes the switch itself and closes the pair.

Remaining order:

1. this PR (Backroom half already in)
2. then the Ollama container / legacy model-relay / sandbox-exposure removals in dittobench-api and ditto-subnet (v2-v6-only infrastructure, safe to delete only once nothing can score a sub-v7 benchmark)

Step 2 is already in flight and blocked on this PR — those removals are being prepared separately and must not merge before the floor lands, since until it does a sub-v7 benchmark is still theoretically scoreable and the infrastructure they delete is what would score it.

The migration

alembic/versions/2026_07_28_enforce_bench_version_floor.py (d4b8e6c1a205)

• scores, confirmation_scores → bench_version >= 7; benchmark_rollouts → desired_version >= 7. All NOT VALID: Postgres skips the existing-row scan but enforces on every INSERT and UPDATE, so the 1,685 historical sub-v7 scores rows and 106 confirmation_scores rows stay present, readable and byte-identical. This closes the era to new writes; it is not a purge. Skipping the scan is also what keeps the migration metadata-only.
• validator_tickets → a trigger, not a CHECK, because a CHECK sees only the resulting row and never the transition. It would have refused the issued -> expired drain and stranded every in-flight v6 lease in issued with no way to close it.

The trigger predicate — please do not "simplify" this

The obvious predicate is wrong, and it was my first attempt:

-- REJECTED. Do not restore.
NEW.status = 'issued' AND OLD.status <> 'issued'

issue_ticket's reuse branch (tickets.py:672-687) writes status = ISSUED over a row that is already ISSUED and simply pushes deadline out. That is exactly how the v6 lease renewed itself in production — the status never changed, so there was no transition for a status-watching guard to see. That predicate would have reproduced the original hole inside the schema, while looking like a fix.

The shipped predicate keys on the lease moving, not on the status changing:

NEW.status = 'issued'
  AND (OLD.status <> 'issued'
       OR NEW.issued_at IS DISTINCT FROM OLD.issued_at
       OR NEW.deadline > OLD.deadline)

Still permitted, so an in-flight lease can drain: issued -> expired/scored, the overdue sweep's retry_after, force_expire_lease (moves the deadline backwards), and failure_reason/first_reported_at touches. Regression-tested both ways in ditto/tests/db/test_bench_version_floor.py.

Application

• allow_retired_era_backfill removed entirely. extra="forbid" means a stale writer that still sends the key is rejected, not silently ignored.
• _issue_source_backfill_ticket gates on the floor and the active version, above the resume lookup. The lane's legitimate use — keeping a source-version validator busy while a rollout is open, where from_version is the active version — is preserved.
• Retired-era score submission → 410 Gone + code 4002.
• New MIN_SCOREABLE_BENCH_VERSION = 7.
• upsert_score now requires an explicit bench_version (the = 2 default is gone); Score.bench_version model default 2 → 7; the simulator no longer seeds v2.
• active_bench_version's no-activation-on-record fallback is now the floor, not DEFAULT_BENCH_VERSION. Worth calling out because it is the one behavioural change here that a reader would not predict from "close v2–v6", and it accounts for every remaining test failure this branch had. A ledger with no durable authority decision — a fresh deployment, or one restored without its rollout history — used to answer 2. request_job takes that at face value, cuts a v2 lease, and the ticket trigger refuses the insert: an unhandled IntegrityError surfacing as a 500 on the job-claim path where a 204 belongs. DEFAULT_BENCH_VERSION stays 2 — it is a statement about where this subnet began, frozen for the same reason LEGACY_BENCH_VERSION is — it simply stopped being a usable answer to "what era are we scoring".

🐛 Live user-facing bug: the public dataset reveal served the wrong dataset

Independently shippable — if this PR stalls in review, cherry-pick fix(public): reveal the dataset the agent actually ran first. Four files: endpoints/public.py, datapipeline/client.py, and the two test modules that stub or exercise the generator (datapipeline/test_client.py, endpoints/test_public.py). Both stubs had a fetch_dataset that predates the required bench_version, so the commit does not stand up without them.

Verified by applying the commit to a clean main worktree and running the affected tests — 178 passed. An earlier draft of this section claimed independence on the strength of git apply --check alone; applying cleanly and passing are different claims, and only the second one is worth making.

GET /api/v1/public/agent/{id}/dataset reveals the benchmark dataset a finalized submission actually ran. public.py:4299 called generator.fetch_dataset(seed, run_size) and omitted bench_version, which defaulted to 2.

Symptom, in plain terms: every miner who opened the dataset reveal for a finalized agent was shown the v2 dataset, no matter which benchmark their agent actually ran. It never errored and never looked wrong, because a v2 dataset is a perfectly well-formed artifact — it just is not that agent's. Anyone who used the reveal to check their scoring inputs was reading the wrong file.

Pre-existing and unrelated to the floor; the floor is only what made it visible, by turning a silently-wrong default into an unreachable one. Fixed by deriving the era from the agent's own scores (the authority on which benchmark finalized it), falling back to the active era rather than assuming.

⚠️ Second-order effect — believed intended, flagged for owner confirmation

Worth a deliberate yes/no before merge rather than discovering it later.

Every rollout targeting v7 has from_version = 6. Because the previous-generation carryover lane operates on the source era, and 6 is now beneath the floor, that lane is inert for the v6→v7 transition specifically. It reopens normally at 7→8, when the source era will be 7.

This reads as correct — carryover exists to hand the new era a populated leaderboard, and it cannot do that with scores the ledger will not accept. But it was not the goal of this change, so it should be an explicit decision. If anyone disagrees, now is the time.

Related and worth stating plainly: _CONTRACTS covers v2–v7 and the floor is 7, so exactly one era is currently both leasable and writable. Shipping the v8 contract (ditto-platform#513) is a hard prerequisite for the next rollout — start_rollout has no legal target until it exists.

Additional admission paths found and closed

Beyond the two above, an audit turned up three more:

1. replace_validator_score_after_infrastructure_failure had no era check of any kind. Independent of both known paths, and reachable straight from Backroom: it re-leases a ticket by UPDATE, so it never touched the insert path, and unlike reinstatement_gate next door it never asked what era it was operating in. It would hand a v6 agent another 90-minute lease on request. Now refused by the trigger, plus a 409 so an operator gets a reason rather than a 500.
2. activate_next_score_retest gated only on what the validator advertises it can run. That is a capability check, not a policy floor — it kept retired eras out only incidentally, because no modern validator advertises v6. Now closes the request as unserviceable.
3. start_rollout had no floor guard — an operator naming a retired target got a raw CheckViolationError as a 500, with a constraint name as the only explanation, where the forward-only guard beside it answers 409. Now a 409 naming both versions; and available_target_versions no longer offers retired targets, so the console cannot suggest what the start path will refuse.
4. Latent sub-v7 defaults (upsert_score(bench_version=2), Score.bench_version default 2, simulator DEFAULT_BENCH_VERSION = 2) — each now a guaranteed constraint violation rather than a silent wrong write.

In-flight leases: terminal, and structurally unable to loop

Left alone rather than force-closed. A live lease reaches a terminal state either by the validator reporting the 410 through fail_job (scoring_error — consumes the attempt, mints no grant) or by the overdue sweep expiring it at its own deadline, ≤90 minutes out.

410 rather than 409 is deliberate: a conflict invites a retry, and this must never be handed back as fail_job(reason="infrastructure"). Infrastructure is the no-fault class — it mints a compensating grant, raises the attempt cap and re-leases — so a condition that never clears would re-lease forever. That is the ditto-subnet#279 shape, where twelve dead mnemo* leases burned ~4.5 validator-hours apiece because a terminal fault was reported as retryable infrastructure.

The belt-and-braces is what makes this safe by construction rather than by policy: even a misclassified infrastructure report cannot loop, because the reissue it asks for must put a sub-v7 ticket back into issued, and the trigger refuses it. The lease dies either way. Correct classification is the fast path; the schema is the guarantee.

Silent-reset hazard on stored revisions

prev_gen_carryover is stored whole and the model is extra="forbid", so removing a field is not free — revisions already in the table still carry it. settings_from_row fails open, so this would not have crashed; it would have done something quieter and worse: silently reverting the operator's entire policy (cohort sizes, lane cycle, owner limits) to defaults behind one log line, because one dead key failed validation for the whole document.

Retired keys are now stripped before validation. Read-side only — the write path stays strict, so a stale client that still sends the key is rejected rather than quietly ignored. Forgiving a stored document is not the same as accepting a new instruction.

Reversibility

Fully reversible. downgrade() drops four constraints, the trigger and its function; nothing is dropped, rewritten or backfilled on the way up, so nothing needs restoring. Verified against a local database: downgrade leaves all rows intact and v6 writes work again afterwards; re-upgrade is clean.

Test plan

Verified against a real Postgres, seeded with sub-v7 rows at the previous head so the migration meets production's grandfathered state:

• Migration applies and rolls back cleanly; pg_constraint.convalidated = false proves it never scanned.
• Sub-v7 score INSERT rejected at the DB layer for every retired version (2-6); v7 unaffected.
• UPDATE of a historical v6 score rejected — the retired ledger is frozen, not just closed.
• Existing v2-v6 rows still readable, queryable and unchanged after migration.
• In-flight v6 lease drains (issued -> expired) and can still record failure_reason; renewal (refreshed issued_at + deadline pushed) rejected; re-lease of a drained ticket rejected; new v6 ticket rejected.
• Retired-era submission classified 410/4002 and never 5xx.
• PrevGenCarryoverSettings(allow_retired_era_backfill=True) raises ValidationError.
• A stored revision carrying the retired key still decodes with the operator's other settings intact.
• ruff check, ruff format --check, mypy ditto/ clean.
• pytest — 2541 passed, 0 failed, 95s, rebased onto main @ db7cb53.

Suite status, stated plainly

The bar is that main is green, and both sides were measured on the same tree, not inferred:

main @ db7cb53	2521 passed, 0 failed, 92s
this branch, rebased onto it	2541 passed, 0 failed, 95s

The twenty-test difference is this branch's own — the migration and floor regression suite in ditto/tests/db/test_bench_version_floor.py, plus the cases added while migrating the fixture corpus. The same delta held at the previous base (eaf697b: 2489 green on main, 2509 on the branch), so nothing here is carried by main moving.

A note so nobody reconstructs a number from the review thread: an earlier count of "~38 failures" on this branch came from a run that hung and never printed a summary, so it was a partial tally of a run that never finished — not a measurement. The suite did not regress and recover; it was never measured until the hang was fixed.

The hang is fixed, in its own commit (test(harness): make the pre-test reset fail loudly instead of deadlocking). It is not a v6 problem and is cherry-pickable on its own: a test that fails while holding an open transaction leaves its connection checked out, engine.dispose() cannot force-close it, and the next test's TRUNCATE blocks on its locks forever. On main it never fires because nothing fails. Measured on a file with 17 failing tests: previously hung indefinitely, now completes in 6 seconds and reports all 17.

The last 26, and what they actually were

They were expected to be "the retired era is now the subject" — tests asserting a v2–v6 rollout could be started, activated or selected, which the floor turns into a 409. Read one by one, none of them was that. Every one of the 26 failed on the active_bench_version fallback described under Application above: a fixture that seeds an inherited v2 world and never records an activation used to get v2 for free from DEFAULT_BENCH_VERSION, and now gets the floor. Not one 409 among them. Recorded because a reviewer skimming the diff would otherwise assume the first reading was right.

Area	Count	What they turned out to need
db/queries/test_benchmark_rollout.py	17	16 the inherited era genuinely in force; 1 the floor-era read
endpoints/test_admin_benchmark_rollout.py	5	4 the inherited era in force; 1 a floor-era read
endpoints/test_public.py (TestBenchConfig, leaderboard cache)	3	2 floor-era reads; 1 an active era below the open rollout's target
db/queries/test_prev_gen_carryover.py	1	the source era in force, so the pre-adoption answer is not the post-adoption one
Total	26	4 floor-era reads, 22 an era genuinely in force

These counts correct an earlier draft of this section, which had test_benchmark_rollout.py at 16 and so totalled 25 against a measured 26. They are read off the run's own short test summary, not recounted by hand.

Two shapes, both following the same rule — where the era is incidental, move to the floor; where the era is the subject, keep it retired and state the real behaviour:

Floor-era reads (4 tests). Empty-database status and discovery reads. The era was never their subject — "a status read starts nothing", "discovery is authenticated and read-only", "config shape and defaults". They now assert the floor. One of them changes shape rather than value and is worth a look: available_target_versions on a fresh ledger is now [], because a target must be both above the active version and at or above the floor, and with v7 the newest shipped contract those two conditions meet at exactly one era — the one already in force. That is the _CONTRACTS/floor coincidence noted above, made visible. It becomes [8] when ditto-assistant/ditto-subnet#463 lands.

A genuinely-active inherited era (22 tests). These exercise a transition, and a transition needs a source era below its target. A rollout must move forward, the newest shipped contract is v7, the floor is 7 — so the source era of any rollout that can still be built is beneath the floor, necessarily. There is no renumbering that keeps the transition and clears the floor; the fixtures already seeded their v2/v6 ledgers through retired_era_writes_allowed for exactly this reason. What was missing was the activation that made that era authoritative, so active_bench_version had no durable decision to read and answered the floor — collapsing source and target onto one era and quietly voiding the assertion. supersede_open_rollout is the clearest case: it refuses a rollout that already owns active authority, so the supersede under test 409'd before it ran.

The fix is one shared helper, ditto/tests/legacy_era.py::grandfather_active_era — a grandfathered pre-floor activated rollout row. That is a state the database legitimately holds: the floor is NOT VALID precisely so production keeps its activation history, and it is the shape production was in before the v7 transition. Not a fiction, and not a weakening — it restores bite that had been lost, e.g. test_open_v7_rollout_keeps_active_v6_harness_authoritative was asserting 7 == 7 and is once again asserting what its name says.

Its docstring, and every call site, says to re-point at a 7 → 8 transition once ditto-assistant/ditto-subnet#463 ships — the point at which a forward target above the floor exists and these fixtures stop needing to reach back into retired history.

Two tests changed shape enough to name explicitly:

• test_control_degrades_the_slow_section_instead_of_hanging had become vacuous: with no candidate target the qualification loop never runs, so the deliberately-hanging stub was never called and the test passed while proving nothing. It now seeds the v6 authority that leaves exactly one candidate (v7) to hang on.
• test_config_shape_and_defaults' harness assertions were v2-era values. The harness block is derived from the era being reported, so on the floor era they are the proxy-routed model and a pinned reasoning effort.

Known fixture hazard: _seed_agent(screened_image=True)

A default this branch introduced, and the class of bug worth watching for in the rest of the diff: it changed what several tests exercise without failing any of them. Every contract from v3 up sets requires_screened_image, so once the corpus moved off v2 an agent seeded without one is silently dropped from the candidate set. Defaulting it on is right. But it silently moved a never-disclose test off the source-tarball path whose storage key it asserts and onto the image path — still green, still passing, no longer testing its docstring.

Four call sites now pass screened_image=False explicitly, each with a comment saying why the absence is the point (TestArtifact ×2, the v7 screened-only lease tests ×2). Flagged rather than swept because the general hazard — a fixture default that changes a test's subject without breaking it — is not something a green suite can catch.

Pre-existing flake, not from this branch

TestUnmatchableWorkClaimIsLoud's two caplog tests: whichever runs first against a freshly created test database fails its "…produced no warning" assertion. Reproduced deterministically, both trees, by make test-db-clean followed by the class alone:

• unmodified main @ eaf697b → test_retest_reported_on_the_wrong_slot_warns_and_is_accepted fails
• this branch → test_running_benchmark_with_no_active_slot_warns_and_is_accepted fails

Same failure, different member of the pair, depending only on which one the scheduler hands out first. It is a logging-configuration ordering artifact, not a benchmark-version one, and it never fires in a full-suite run on either tree — something earlier configures logging first. Not among the 26 and not introduced here; recorded so it is not attributed to this PR if someone runs that file on its own.

Migration head, re-verified at submission time

This repo uses Alembic revision hashes, not numeric prefixes, so the Goose-style version-prefix collision does not apply. Checked the analogous failure instead — alembic heads reports exactly one head (d4b8e6c1a205), down_revision (45ef71514f21) is main's head, no duplicate revision ids, no branch point.

That check was re-run after rebasing, and it mattered once already: the branch originally chained from c7a4f1e2b903, which #555 has since merged with f4b7d2c91ae5 into 45ef71514f21. Left alone, this PR's green migration check would have been stale — GitHub does not re-run checks when the base moves — and merging it would have produced exactly the two-head outage #555 repaired.

main then moved again, to db7cb53 (#556, #557, #558). This branch is rebased onto it and re-verified: those three commits add no Alembic revision, so down_revision still names main's head and needed no re-pointing. scripts/check_migration_order.py — the guard #556 added for precisely this failure mode — reports ok (1 new migration(s); origin/main head 45ef71514f21; merged head d4b8e6c1a205). The full suite above was run after that rebase, not before it.

If main moves again before this merges, re-verify rather than trusting the check on the page. #556 also shipped scripts/recheck_open_pr_migrations.sh for exactly that.

🤖 Generated with Claude Code