fix(validator): keep agent-attributable inference declines out of no-fault - #288
Conversation
|
Companion: ditto-assistant/dittobench-api#120, which emits both new codes. Merge order across repos: ditto-subnet #286 → ditto-subnet #288 → (with) dittobench-api #120. This PR must not land materially after #120 in the fleet: #120 emits |
|
Correction to the ordering note above. The earlier comment said this must not land materially after dittobench-api#120. That constraint is gone. It existed because saturation shipped a new no-fault code that old validators would have turned into Remaining order is intra-repo only: #286 → #288. No cross-repo constraint against #120. |
…fault
dittobench-api split the relay's single finalize failure code apart. Its
grant-denial counter collapsed nine unrelated conditions -- several entirely
the harness's doing -- into validator_infrastructure/model_relay_unavailable/
retryable:true, which mints a retry grant, RAISES the attempt cap, and
re-leases. An agent that reliably spent its own allowance re-leased itself
forever.
Two new codes reach this side, and they go in opposite directions:
- inference_allowance_exhausted (sandbox_failure/retryable:false) is the
agent's: it spent the request-count or token allowance its own ticket
granted, or sent one request too large to reserve. Deliberately NOT added
to _SANDBOX_INFRASTRUCTURE_CODES, so it maps to scoring_error.
- inference_lane_saturated IS added. The scorer's call spent its whole
backpressure budget on a full platform lane. Split out only so a saturated
rail is legible without container logs; a miner can neither provision the
lane nor see its contention, so it keeps its grant.
Guards: a module-level assert, a set-property test, and a poll-path test that
pins the agent code to scoring_error under BOTH the envelope the scorer sends
and the no-fault envelope it must never be smuggled under.
Stacked on #286 (screened_image_unavailable), which edits the same set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drops the `inference_lane_saturated` allowlist entry. Adding a NEW no-fault
code only helps validators that have already rolled: this module returns None
for anything outside the set, and None becomes fail_job("scoring_error"), so
the code would have charged miners for a saturated platform lane on every
validator predating the release. The fleet lags by design.
dittobench-api #120 now keeps `model_relay_unavailable` for saturation and
carries the cause in the failure's diagnostics, which this classifier does not
consult -- so it stays no-fault on every fleet version, including ones that
will never be upgraded, and needs nothing here.
The agent code has no such hazard and is unaffected: an old validator sees
sandbox_failure/retryable=false and its terminal default is already the
intended outcome. Only the no-fault direction needed the care, which is now
recorded next to the set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
566955b to
314c057
Compare
Summary
Companion to dittobench-api #120, which splits the relay's single finalize failure code apart. That code collapsed nine unrelated conditions — several entirely the harness's doing — into
validator_infrastructure/model_relay_unavailable/retryable: true, the no-fault class that mints a retry grant, raises the attempt cap, and re-leases. An agent that reliably spent its own inference allowance re-leased itself forever.One new code reaches this side, and it must not be treated as no-fault:
_SANDBOX_INFRASTRUCTURE_CODES?inference_allowance_exhaustedsandbox_failure/retryable: falsescoring_errorIt means the harness spent the request-count or token allowance its own ticket granted, or sent one request too large to reserve (platform decline codes 4102 / 4104 / 4109). The lease was alive and the platform healthy throughout — this is the agent's.
Why there is no second code
An earlier revision of this PR added
inference_lane_saturatedto the set, for the other half of #120: the scorer's call spending its entire bounded backpressure budget on a platform lane that stayed full. That half is still classified as no-fault — a miner can neither provision the lane nor observe its contention, and charging it to the agent would be the mirror image of the misclassification #120 fixes — but it must not be a new code, and so it needs nothing here.This module returns
Nonefor any code outside the set, andNonebecomesfail_job("scoring_error"). A brand-new no-fault code therefore charges the miner on every validator that predates the release that adds it, and the fleet lags scorer releases by design — it is on v0.36.0 / v0.34.1 today. That is the same bug #120 exists to fix, arriving through the rollout instead of through the classifier.So #120 keeps
model_relay_unavailable(in every deployed validator's allowlist since the code existed) and carries the cause in the failure's diagnostics, which this classifier never consults and therefore cannot change the class anywhere.The asymmetry is worth naming, because it decides what is safe to ship in each direction:
sandbox_failure/retryable: falseand its terminal default is already the intended outcome.It is recorded in a comment next to the set, since that set is exactly where a future edit would be tempted to "finish the job".
Consequence: #288 and #120 have no ordering hazard in either direction. Neither repo depends on the other having shipped.
Why the agent code must stay out
Every code in
_SANDBOX_INFRASTRUCTURE_CODESis no-fault. Addinginference_allowance_exhaustedwould mean an agent that reliably exhausts its own allowance re-leases itself without bound — the mnemox loop, arriving through the inference lane instead of the sandbox.Excluded three independent ways:
kind: sandbox_failure, and_sandbox_infrastructure_failure_coderequiresvalidator_infrastructurebefore it looks at the code at all.retryable: false, which that function also requires to beTrue.A module-level
assertfails at import if the two sets ever intersect.Tests
make lint typecheck testgreen — ruff clean, mypy clean on 112 source files, full suite passing.test_agent_attributable_inference_failures_stay_the_agents— pins the agent code toscoring_errorthrough the poll path under both the envelope the scorer sends and the no-fault envelope it must never be smuggled under, so the guarantee holds even if a future scorer sends the wrong one.test_agent_inference_codes_are_never_no_fault— the set property, stated directly. A future edit adding the agent code to the infrastructure set (the natural mistake, since the neighbouringinference_lane_saturatedis in it) fails here rather than in production as an unbounded re-lease loop. Also asserts the complement: if someone later movesinference_lane_saturatedout, that is a decision about who pays for a saturated rail and they have to edit this line to make it.inference_lane_saturatedis not in the set andmodel_relay_unavailableis — so the rollout reasoning above is pinned, not just commented.test_unknown_sandbox_failure_code_is_not_retryablealready covers unknown codes and is unchanged._SANDBOX_INFRASTRUCTURE_CODEScollision — how it was handled#286 (
peyton/screened-image-acquisition-not-agent-fault, the companion to dittobench-api #118) edits the same set to addscreened_image_unavailable. This PR is stacked on #286's branch — base ispeyton/screened-image-acquisition-not-agent-fault, notmain— so the two additions compose instead of conflicting, and this diff shows only its own change.#286 was left completely untouched: its base is still
main, it was not adopted into a GitHub Stack object, and no git operation was run against its branch. That branch is currently checked out in another active worktree, and stack operations against a branch held by a live worktree are exactly how restacks break.Merge order: #286 first, then this. If #286 is abandoned, this needs a rebase onto
main; the two hunks are adjacent but not overlapping, so it is a clean re-apply. There is no cross-repo ordering constraint against #120 (see above).Also in flight and deliberately untouched: #287 (
fix/idle-slots-abandon-sweep), the heartbeat v17 lease work,validator_lease_audit, the fleet-health dashboard, and the scores read path.🤖 Generated with Claude Code