This repository was archived by the owner on Aug 11, 2026. It is now read-only.
perf: stop hydrating score telemetry in the allocator floor read - #674
Closed
Peyton-Spencer wants to merge 1 commit into
Closed
perf: stop hydrating score telemetry in the allocator floor read#674Peyton-Spencer wants to merge 1 commit into
Peyton-Spencer wants to merge 1 commit into
Conversation
get_score_priority_floor_rows passed include_fingerprints=False but not include_details=False, so every eligible ledger row arrived carrying its full per-case audit blob — roughly 22KB of JSONB across ~1.2k rows on bench v8 — which the allocator then discarded. This read is on the hot path. Production right now: ditto-api pinned at 100% CPU on its single worker, 7,476 POST /validator/job answered 204 against 24 answered 200 in one log window, and a slowest observed /validator/job of 40,181ms. Detoasting and decoding audit payloads no consumer reads is user CPU spent on the one event loop everything else queues behind — including the continual-retest lane, which is issuing tickets fine (34 × 200 against 2 × 409) and is simply starved of turns. Nothing downstream reads the field: neither rank_submissions nor resolve_ranking_scores looks at telemetry, get_score_priority_floors returns bare floats, and the one caller that keeps a floor ROW (_public_submission_pipeline) reads only row.agent_id to name the holder. list_eligible_ledger already documents include_details=False as the right call for exactly this consumer class. The test seeds rows that DO carry details, so the assertion distinguishes "never selected" from "nothing to select". Ported from ditto-assistant/ditto-subnet#559. Production is still served from this repository, so the change has to live here to take effect. Refs ditto-assistant/ditto-subnet#388 — the cheap half of that issue's root cause; the batched claim contract is still to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 6, 2026
Contributor
Author
|
Closing unmerged: superseded by the monorepo cutover. Production is now served from
Merging here would land the fix on a branch nothing ships, which is worse than closing: it makes this repo look maintained and invites a deploy from it. See ditto-subnet#566 for the cutover follow-ups. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of ditto-assistant/ditto-subnet#559. Production is served from this repository, so the change has to live here to take effect.
Based on
fix/emission-catchup-between-waves(#673) rather thanmain, because prod is currently running that branch — this keeps the deployed commit moving forward instead of reverting it. Review the top commit only; #673 covers the one below it.get_score_priority_floor_rowspassedinclude_fingerprints=Falsebut notinclude_details=False, so every eligible ledger row arrived carrying its full per-case audit blob (~22KB of JSONB across ~1.2k rows on bench v8) which the allocator immediately discarded.Why now — live production evidence
ditto-apipinned at 100% CPU on its single workerPOST /validator/job: 7,476 × 204 against 24 × 200 in one log window — 99.7% of validator polls return no work/validator/job: 40,181ms (subnet#388 filed p95 at 11.6s)Safety
Nothing downstream reads the field, verified before changing it:
rank_submissionsandresolve_ranking_scoresnever touch telemetryget_score_priority_floorsreturns bare floats_public_submission_pipeline) readsrow.agent_id.row.detailsaccess exists anywhere in the treelist_eligible_ledgeralready documentsinclude_details=Falseas correct for "queue-floor, cleanup, and efficiency-cohort consumers" — this consumer was simply missed.Validation
uv run pytest ditto/tests/db/queries/ -q→ 628 passed, 4 skippeduv run pytest ditto/tests/api_server/endpoints/test_validator.py -q→ 220 passedruff check/ruff format --check/mypy→ cleanThe new test seeds rows that do carry
details, sodetails is Nonedistinguishes never selected from nothing to select.Scope
This does not close subnet#388. The batched idle-slot claim contract (
POST /validator/jobs), the shared per-batchAllocatorPrioritySnapshot, and the validator-side rolling-upgrade negotiation are all still unimplemented — that is the change that removes the per-slot recomputation, and it spans both this repo and the validator in the monorepo.