Skip to content
This repository was archived by the owner on Aug 11, 2026. It is now read-only.

perf: stop hydrating score telemetry in the allocator floor read - #674

Closed
Peyton-Spencer wants to merge 1 commit into
fix/emission-catchup-between-wavesfrom
perf/allocator-floor-skip-details
Closed

perf: stop hydrating score telemetry in the allocator floor read#674
Peyton-Spencer wants to merge 1 commit into
fix/emission-catchup-between-wavesfrom
perf/allocator-floor-skip-details

Conversation

@Peyton-Spencer

Copy link
Copy Markdown
Contributor

Summary

Backport of ditto-assistant/ditto-subnet#559. Production is served from this repository, so the change has to live here to take effect.

Based on fix/emission-catchup-between-waves (#673) rather than main, because prod is currently running that branch — this keeps the deployed commit moving forward instead of reverting it. Review the top commit only; #673 covers the one below it.

get_score_priority_floor_rows passed include_fingerprints=False but not include_details=False, so every eligible ledger row arrived carrying its full per-case audit blob (~22KB of JSONB across ~1.2k rows on bench v8) which the allocator immediately discarded.

Why now — live production evidence

  • ditto-api pinned at 100% CPU on its single worker
  • POST /validator/job: 7,476 × 204 against 24 × 200 in one log window — 99.7% of validator polls return no work
  • slowest observed /validator/job: 40,181ms (subnet#388 filed p95 at 11.6s)
  • the continual-retest lane is not gated — 34 × 200 issued against 2 × 409. It is starved of turns on the same saturated event loop, which is what the uneven shared-seed counts on the leaderboard actually reflect.

Safety

Nothing downstream reads the field, verified before changing it:

  • rank_submissions and resolve_ranking_scores never touch telemetry
  • get_score_priority_floors returns bare floats
  • the only caller that keeps a floor row (_public_submission_pipeline) reads row.agent_id
  • no .row.details access exists anywhere in the tree

list_eligible_ledger already documents include_details=False as correct for "queue-floor, cleanup, and efficiency-cohort consumers" — this consumer was simply missed.

Validation

  • uv run pytest ditto/tests/db/queries/ -q628 passed, 4 skipped
  • uv run pytest ditto/tests/api_server/endpoints/test_validator.py -q220 passed
  • ruff check / ruff format --check / mypy → clean

The new test seeds rows that do carry details, so details is None distinguishes never selected from nothing to select.

Scope

This does not close subnet#388. The batched idle-slot claim contract (POST /validator/jobs), the shared per-batch AllocatorPrioritySnapshot, and the validator-side rolling-upgrade negotiation are all still unimplemented — that is the change that removes the per-slot recomputation, and it spans both this repo and the validator in the monorepo.

get_score_priority_floor_rows passed include_fingerprints=False but not
include_details=False, so every eligible ledger row arrived carrying its
full per-case audit blob — roughly 22KB of JSONB across ~1.2k rows on
bench v8 — which the allocator then discarded.

This read is on the hot path. Production right now: ditto-api pinned at
100% CPU on its single worker, 7,476 POST /validator/job answered 204
against 24 answered 200 in one log window, and a slowest observed
/validator/job of 40,181ms. Detoasting and decoding audit payloads no
consumer reads is user CPU spent on the one event loop everything else
queues behind — including the continual-retest lane, which is issuing
tickets fine (34 × 200 against 2 × 409) and is simply starved of turns.

Nothing downstream reads the field: neither rank_submissions nor
resolve_ranking_scores looks at telemetry, get_score_priority_floors
returns bare floats, and the one caller that keeps a floor ROW
(_public_submission_pipeline) reads only row.agent_id to name the holder.
list_eligible_ledger already documents include_details=False as the right
call for exactly this consumer class.

The test seeds rows that DO carry details, so the assertion distinguishes
"never selected" from "nothing to select".

Ported from ditto-assistant/ditto-subnet#559. Production is still served
from this repository, so the change has to live here to take effect.
Refs ditto-assistant/ditto-subnet#388 — the cheap half of that issue's
root cause; the batched claim contract is still to do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Peyton-Spencer

Copy link
Copy Markdown
Contributor Author

Closing unmerged: superseded by the monorepo cutover.

Production is now served from ditto-assistant/ditto-subnet (/opt/ditto-subnet/apps/platform), so this repository no longer deploys anywhere. This PR's content already landed in the monorepo and is live:

Merging here would land the fix on a branch nothing ships, which is worse than closing: it makes this repo look maintained and invites a deploy from it. See ditto-subnet#566 for the cutover follow-ups.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant