Skip to content

fix(validator): keep agent-attributable inference declines out of no-fault - #288

Merged
Peyton-Spencer merged 2 commits into
peyton/screened-image-acquisition-not-agent-faultfrom
peyton/relay-decline-agent-codes
Jul 28, 2026
Merged

fix(validator): keep agent-attributable inference declines out of no-fault#288
Peyton-Spencer merged 2 commits into
peyton/screened-image-acquisition-not-agent-faultfrom
peyton/relay-decline-agent-codes

Conversation

@Peyton-Spencer

@Peyton-Spencer Peyton-Spencer commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Companion to dittobench-api #120, which splits the relay's single finalize failure code apart. That code collapsed nine unrelated conditions — several entirely the harness's doing — into validator_infrastructure / model_relay_unavailable / retryable: true, the no-fault class that mints a retry grant, raises the attempt cap, and re-leases. An agent that reliably spent its own inference allowance re-leased itself forever.

One new code reaches this side, and it must not be treated as no-fault:

Code Envelope In _SANDBOX_INFRASTRUCTURE_CODES? Maps to
inference_allowance_exhausted sandbox_failure / retryable: false No — deliberately scoring_error

It means the harness spent the request-count or token allowance its own ticket granted, or sent one request too large to reserve (platform decline codes 4102 / 4104 / 4109). The lease was alive and the platform healthy throughout — this is the agent's.

Why there is no second code

An earlier revision of this PR added inference_lane_saturated to the set, for the other half of #120: the scorer's call spending its entire bounded backpressure budget on a platform lane that stayed full. That half is still classified as no-fault — a miner can neither provision the lane nor observe its contention, and charging it to the agent would be the mirror image of the misclassification #120 fixes — but it must not be a new code, and so it needs nothing here.

This module returns None for any code outside the set, and None becomes fail_job("scoring_error"). A brand-new no-fault code therefore charges the miner on every validator that predates the release that adds it, and the fleet lags scorer releases by design — it is on v0.36.0 / v0.34.1 today. That is the same bug #120 exists to fix, arriving through the rollout instead of through the classifier.

So #120 keeps model_relay_unavailable (in every deployed validator's allowlist since the code existed) and carries the cause in the failure's diagnostics, which this classifier never consults and therefore cannot change the class anywhere.

The asymmetry is worth naming, because it decides what is safe to ship in each direction:

  • New agent codes are skew-safe. An old validator sees sandbox_failure / retryable: false and its terminal default is already the intended outcome.
  • New no-fault codes are not. They need the validator side shipped and the fleet rolled first.

It is recorded in a comment next to the set, since that set is exactly where a future edit would be tempted to "finish the job".

Consequence: #288 and #120 have no ordering hazard in either direction. Neither repo depends on the other having shipped.

Why the agent code must stay out

Every code in _SANDBOX_INFRASTRUCTURE_CODES is no-fault. Adding inference_allowance_exhausted would mean an agent that reliably exhausts its own allowance re-leases itself without bound — the mnemox loop, arriving through the inference lane instead of the sandbox.

Excluded three independent ways:

  1. It is not in the set.
  2. The scorer sends kind: sandbox_failure, and _sandbox_infrastructure_failure_code requires validator_infrastructure before it looks at the code at all.
  3. It sends retryable: false, which that function also requires to be True.

A module-level assert fails at import if the two sets ever intersect.

Tests

make lint typecheck test green — ruff clean, mypy clean on 112 source files, full suite passing.

  • test_agent_attributable_inference_failures_stay_the_agents — pins the agent code to scoring_error through the poll path under both the envelope the scorer sends and the no-fault envelope it must never be smuggled under, so the guarantee holds even if a future scorer sends the wrong one.
  • test_agent_inference_codes_are_never_no_fault — the set property, stated directly. A future edit adding the agent code to the infrastructure set (the natural mistake, since the neighbouring inference_lane_saturated is in it) fails here rather than in production as an unbounded re-lease loop. Also asserts the complement: if someone later moves inference_lane_saturated out, that is a decision about who pays for a saturated rail and they have to edit this line to make it.
  • The same test also asserts the complement — inference_lane_saturated is not in the set and model_relay_unavailable is — so the rollout reasoning above is pinned, not just commented.
  • test_unknown_sandbox_failure_code_is_not_retryable already covers unknown codes and is unchanged.

_SANDBOX_INFRASTRUCTURE_CODES collision — how it was handled

#286 (peyton/screened-image-acquisition-not-agent-fault, the companion to dittobench-api #118) edits the same set to add screened_image_unavailable. This PR is stacked on #286's branch — base is peyton/screened-image-acquisition-not-agent-fault, not main — so the two additions compose instead of conflicting, and this diff shows only its own change.

#286 was left completely untouched: its base is still main, it was not adopted into a GitHub Stack object, and no git operation was run against its branch. That branch is currently checked out in another active worktree, and stack operations against a branch held by a live worktree are exactly how restacks break.

Merge order: #286 first, then this. If #286 is abandoned, this needs a rebase onto main; the two hunks are adjacent but not overlapping, so it is a clean re-apply. There is no cross-repo ordering constraint against #120 (see above).

Also in flight and deliberately untouched: #287 (fix/idle-slots-abandon-sweep), the heartbeat v17 lease work, validator_lease_audit, the fleet-health dashboard, and the scores read path.

🤖 Generated with Claude Code

@Peyton-Spencer

Copy link
Copy Markdown
Contributor Author

Companion: ditto-assistant/dittobench-api#120, which emits both new codes. Merge order across repos: ditto-subnet #286 → ditto-subnet #288 → (with) dittobench-api #120. This PR must not land materially after #120 in the fleet: #120 emits inference_lane_saturated, and a validator that does not recognise it would treat a saturated platform lane as terminal rather than no-fault.

@Peyton-Spencer

Copy link
Copy Markdown
Contributor Author

Correction to the ordering note above. The earlier comment said this must not land materially after dittobench-api#120. That constraint is gone.

It existed because saturation shipped a new no-fault code that old validators would have turned into scoring_error. #120 now keeps model_relay_unavailable for saturation and puts the cause in diagnostics, so no validator needs to know anything new to keep it no-fault. The inference_lane_saturated allowlist entry has been dropped from this PR.

Remaining order is intra-repo only: #286#288. No cross-repo constraint against #120.

Peyton-Spencer and others added 2 commits July 28, 2026 00:35
…fault

dittobench-api split the relay's single finalize failure code apart. Its
grant-denial counter collapsed nine unrelated conditions -- several entirely
the harness's doing -- into validator_infrastructure/model_relay_unavailable/
retryable:true, which mints a retry grant, RAISES the attempt cap, and
re-leases. An agent that reliably spent its own allowance re-leased itself
forever.

Two new codes reach this side, and they go in opposite directions:

  - inference_allowance_exhausted (sandbox_failure/retryable:false) is the
    agent's: it spent the request-count or token allowance its own ticket
    granted, or sent one request too large to reserve. Deliberately NOT added
    to _SANDBOX_INFRASTRUCTURE_CODES, so it maps to scoring_error.
  - inference_lane_saturated IS added. The scorer's call spent its whole
    backpressure budget on a full platform lane. Split out only so a saturated
    rail is legible without container logs; a miner can neither provision the
    lane nor see its contention, so it keeps its grant.

Guards: a module-level assert, a set-property test, and a poll-path test that
pins the agent code to scoring_error under BOTH the envelope the scorer sends
and the no-fault envelope it must never be smuggled under.

Stacked on #286 (screened_image_unavailable), which edits the same set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drops the `inference_lane_saturated` allowlist entry. Adding a NEW no-fault
code only helps validators that have already rolled: this module returns None
for anything outside the set, and None becomes fail_job("scoring_error"), so
the code would have charged miners for a saturated platform lane on every
validator predating the release. The fleet lags by design.

dittobench-api #120 now keeps `model_relay_unavailable` for saturation and
carries the cause in the failure's diagnostics, which this classifier does not
consult -- so it stays no-fault on every fleet version, including ones that
will never be upgraded, and needs nothing here.

The agent code has no such hazard and is unaffected: an old validator sees
sandbox_failure/retryable=false and its terminal default is already the
intended outcome. Only the no-fault direction needed the care, which is now
recorded next to the set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Peyton-Spencer
Peyton-Spencer force-pushed the peyton/relay-decline-agent-codes branch from 566955b to 314c057 Compare July 28, 2026 04:35
@Peyton-Spencer
Peyton-Spencer marked this pull request as ready for review July 28, 2026 04:35
@Peyton-Spencer
Peyton-Spencer merged commit 77613de into main Jul 28, 2026
4 checks passed
@Peyton-Spencer
Peyton-Spencer deleted the peyton/relay-decline-agent-codes branch July 28, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant