Skip to content
This repository was archived by the owner on Aug 11, 2026. It is now read-only.

fix: stop charging screened-image acquisition failures to the miner - #118

Merged
Peyton-Spencer merged 1 commit into
fix/sandbox-health-container-logsfrom
fix/screened-image-acquisition-not-agent-fault
Jul 28, 2026
Merged

fix: stop charging screened-image acquisition failures to the miner#118
Peyton-Spencer merged 1 commit into
fix/sandbox-health-container-logsfrom
fix/screened-image-acquisition-not-agent-fault

Conversation

@Peyton-Spencer

@Peyton-Spencer Peyton-Spencer commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on #117.

finishSandboxRun defaults every unrecognised failure to kind=sandbox_failure, code=sandbox_runtime, retryable=false. ditto-subnet's _sandbox_infrastructure_failure_code() honours a failure only when kind == "validator_infrastructure" and retryable is true, so anything else becomes DittobenchError_report_ticket_failed(job, "scoring_error") — one of the miner's finite attempts, spent, with no retry backoff.

Genuinely platform-side failures land in that bin. This PR takes out one class of them: failing to acquire the screened image the platform itself produced. The image is platform output, fetched over the platform's own URL onto this validator's disk. When that fetch fails, the miner is charged for an outage they did not cause and, critically, could not have influenced — the harness has not executed a single instruction.

THE HAZARD, and what stops it here

infrastructure is the platform's no-fault class: it mints a retry grant, raises the attempt cap, and re-leases. That is exactly the mechanism behind tonight's fleet starvation — the mnemox family had every failure classified infrastructure, so mnemox-v55 reached attempts_used: 10 against a base budget of 2 with zero scores, and 9 submissions had to be manually evicted at 20:56Z.

So this PR does not do "unclassified → validator_infrastructure". Four independent brakes:

1. The artifact is not running. Every reclassified failure occurs strictly before docker run. A self-sustaining no-fault loop requires the artifact's own behaviour in the loop — mnemox minted its grants by hanging for ~60 minutes. Nothing a harness can do influences whether a TCP connection to the object store succeeds or whether the local scratch disk has room. The feedback edge that made mnemox self-sustaining does not exist on this path.

2. Deterministic failures are excluded by rule. Only transient-by-construction conditions are marked ours: transport error (DNS/TLS/refused/timeout), object-store 5xx or 429, local filesystem fault creating or closing the temp file, and a stream truncated mid-download. A later attempt on another validator can legitimately succeed at each. Everything that will fail identically forever stays terminal, so a permanently broken image cannot re-lease without bound.

3. It is a typed sentinel, not a substring match. errors.Is(err, sandbox.ErrScreenedImageUnavailable), wrapped at the specific return sites. A harness printing screened image unavailable on stderr cannot forge itself a grant. There is a test for exactly that.

4. The consumer allowlist is closed — and this is the ship-safety backstop. ditto-subnet's _SANDBOX_INFRASTRUCTURE_CODES is a 5-member closed set (sandbox_oom, sandbox_tmpfs_exhausted, sandbox_network_unavailable, model_relay_unavailable, embedding_provider_unavailable). screened_image_unavailable is not in it. Until a separate, separately-reviewed ditto-subnet PR adds it, this classification is inert on retry policy: the worker still takes the scoring_error path exactly as today. What lands immediately is the operator value — the envelope and GET /v1/runs/{id} now name the failure instead of burying it in a generic bin. The no-fault behaviour cannot ship by accident from this PR alone.

What is reclassified

validator_infrastructure / screened_image_unavailable / retryable=true:

condition why it can differ next attempt
transport error on the fetch DNS / TLS / connection refused / timeout
object store 5xx server-side by definition
object store 429 explicitly "come back later"
temp file create/close fails scratch disk full, read-only, out of inodes
download truncated mid-stream connection reset, timeout

What is deliberately left alone

Unchanged terminal sandbox_failure / sandbox_runtime / retryable=false:

  • sha256 / size / image-id mismatch — deterministic in the platform's stored bytes; identical on every retry
  • malformed screened-image URL — deterministic platform data error
  • 4xx other than 429 (403 expired/unauthorized grant, 404 missing blob) — if the object is not there, re-leasing will not conjure it
  • archive / manifest / index / attestation rejection — deterministic
  • docker image load faileda judgement call, flagged for review. The brief named it as platform-side and it often is (daemon out of disk, storage-driver error). But it arrives after the archive has passed sha verification and structural validation, so a load failure cannot be attributed to a transient host condition with confidence, and an unloadable-but-sha-valid archive would repeat identically forever. Left terminal as the conservative cut. Worth revisiting once daemon-resource errors can be distinguished from archive rejections — but not on a night when a no-fault loop just cost us half the fleet.
  • requires ticket inference identity / inference exchange escaped ticket bounds — the brief named these as v7 raises to reclassify. Neither string exists anywhere in this repo at 3de0c62; the nearest live code is the ticket inference probe in inference_broker.go, which is already classified model_relay_unavailable by failRelayUnavailable and preserved through teardown by fix(v7): stop the deferred sandbox teardown from rebilling platform faults to the miner #107. Nothing to do; not silently skipped.
  • the generic default itself — untouched. Nothing reaches the no-fault class by falling through.

Test plan

  • go test ./cmd/... ./internal/... — green
  • real httptest object store: 500/502/503/429, unreachable host, mid-stream truncation → ErrScreenedImageUnavailable
  • 404/403/400, sha mismatch, size mismatch, size out of range, image-id mismatch, non-tar archive → not marked no-fault (the bound)
  • TestUnrecognisedFailuresCannotReachTheNoFaultClass — the guard this PR is gated on: 13 realistic non-acquisition errors (incl. nil, empty, harness crash, and the sentinel's own words as prose) all produce a nil classifier
  • a post-start failure whose harness output mentions image loading still gets the terminal default

Notes

Surface is the sandbox failure envelope and its classification only. No lease code, no heartbeat schema, no validator_lease_audit, no dashboard, no scores read path — clear of #537, #538, ditto-subnet #284, and backroom #96.

Follow-up (separate PR, deliberately not here): add screened_image_unavailable to ditto-subnet's _SANDBOX_INFRASTRUCTURE_CODES. That is the change that actually activates no-fault retry, and it deserves its own review with the mnemox history in front of it.

Failing to fetch the platform's own screened image onto this validator's
disk lands in the generic sandbox_failure/sandbox_runtime/retryable=false
bin, which ditto-subnet turns into fail_job("scoring_error") -- spending
one of the miner's finite attempts for an outage they did not cause and
could not have influenced, since the harness has not run yet.

Reclassify ONLY transient-by-construction acquisition failures as
validator_infrastructure/screened_image_unavailable. Every deterministic
failure -- sha/size/id mismatch, malformed archive, 4xx other than 429,
and docker image load -- stays terminal, and the unrecognised default is
untouched, so no unbounded no-fault re-lease loop can arise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Peyton-Spencer
Peyton-Spencer marked this pull request as ready for review July 28, 2026 03:00
@Peyton-Spencer
Peyton-Spencer merged commit 9a8ec96 into main Jul 28, 2026
6 checks passed
@Peyton-Spencer
Peyton-Spencer deleted the fix/screened-image-acquisition-not-agent-fault branch July 28, 2026 03:00
@Peyton-Spencer

Copy link
Copy Markdown
Contributor Author

The follow-up this PR names is now open as a draft: ditto-assistant/ditto-subnet#286 — it adds screened_image_unavailable to _SANDBOX_INFRASTRUCTURE_CODES, which is the change that actually activates no-fault retry.

Merge order: this PR first, then #286. Landing #286 first is harmless (no scorer emits the code yet); landing it never is the regression, since the classification stays inert on retry policy without it.

#286 re-derives the safety argument from this diff rather than restating it, and pins the bound from the consuming side: seven deterministic failures (sha256 / size / image-id mismatch, 404, malformed archive, docker image load, and ErrScreenedImageUnavailable's own words as harness prose) must all still raise DittobenchError. Since there is no shared wire package, that invariant is now asserted independently in both repos.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant