fix: stop charging screened-image acquisition failures to the miner - #118
Conversation
Failing to fetch the platform's own screened image onto this validator's
disk lands in the generic sandbox_failure/sandbox_runtime/retryable=false
bin, which ditto-subnet turns into fail_job("scoring_error") -- spending
one of the miner's finite attempts for an outage they did not cause and
could not have influenced, since the harness has not run yet.
Reclassify ONLY transient-by-construction acquisition failures as
validator_infrastructure/screened_image_unavailable. Every deterministic
failure -- sha/size/id mismatch, malformed archive, 4xx other than 429,
and docker image load -- stays terminal, and the unrecognised default is
untouched, so no unbounded no-fault re-lease loop can arise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The follow-up this PR names is now open as a draft: ditto-assistant/ditto-subnet#286 — it adds Merge order: this PR first, then #286. Landing #286 first is harmless (no scorer emits the code yet); landing it never is the regression, since the classification stays inert on retry policy without it. #286 re-derives the safety argument from this diff rather than restating it, and pins the bound from the consuming side: seven deterministic failures (sha256 / size / image-id mismatch, 404, malformed archive, |
Summary
Stacked on #117.
finishSandboxRundefaults every unrecognised failure tokind=sandbox_failure, code=sandbox_runtime, retryable=false. ditto-subnet's_sandbox_infrastructure_failure_code()honours a failure only whenkind == "validator_infrastructure"andretryable is true, so anything else becomesDittobenchError→_report_ticket_failed(job, "scoring_error")— one of the miner's finite attempts, spent, with no retry backoff.Genuinely platform-side failures land in that bin. This PR takes out one class of them: failing to acquire the screened image the platform itself produced. The image is platform output, fetched over the platform's own URL onto this validator's disk. When that fetch fails, the miner is charged for an outage they did not cause and, critically, could not have influenced — the harness has not executed a single instruction.
THE HAZARD, and what stops it here
infrastructureis the platform's no-fault class: it mints a retry grant, raises the attempt cap, and re-leases. That is exactly the mechanism behind tonight's fleet starvation — themnemoxfamily had every failure classifiedinfrastructure, somnemox-v55reachedattempts_used: 10against a base budget of 2 with zero scores, and 9 submissions had to be manually evicted at 20:56Z.So this PR does not do "unclassified →
validator_infrastructure". Four independent brakes:1. The artifact is not running. Every reclassified failure occurs strictly before
docker run. A self-sustaining no-fault loop requires the artifact's own behaviour in the loop — mnemox minted its grants by hanging for ~60 minutes. Nothing a harness can do influences whether a TCP connection to the object store succeeds or whether the local scratch disk has room. The feedback edge that made mnemox self-sustaining does not exist on this path.2. Deterministic failures are excluded by rule. Only transient-by-construction conditions are marked ours: transport error (DNS/TLS/refused/timeout), object-store 5xx or 429, local filesystem fault creating or closing the temp file, and a stream truncated mid-download. A later attempt on another validator can legitimately succeed at each. Everything that will fail identically forever stays terminal, so a permanently broken image cannot re-lease without bound.
3. It is a typed sentinel, not a substring match.
errors.Is(err, sandbox.ErrScreenedImageUnavailable), wrapped at the specificreturnsites. A harness printingscreened image unavailableon stderr cannot forge itself a grant. There is a test for exactly that.4. The consumer allowlist is closed — and this is the ship-safety backstop. ditto-subnet's
_SANDBOX_INFRASTRUCTURE_CODESis a 5-member closed set (sandbox_oom,sandbox_tmpfs_exhausted,sandbox_network_unavailable,model_relay_unavailable,embedding_provider_unavailable).screened_image_unavailableis not in it. Until a separate, separately-reviewed ditto-subnet PR adds it, this classification is inert on retry policy: the worker still takes thescoring_errorpath exactly as today. What lands immediately is the operator value — the envelope andGET /v1/runs/{id}now name the failure instead of burying it in a generic bin. The no-fault behaviour cannot ship by accident from this PR alone.What is reclassified
validator_infrastructure/screened_image_unavailable/retryable=true:What is deliberately left alone
Unchanged terminal
sandbox_failure/sandbox_runtime/retryable=false:docker image load failed— a judgement call, flagged for review. The brief named it as platform-side and it often is (daemon out of disk, storage-driver error). But it arrives after the archive has passed sha verification and structural validation, so a load failure cannot be attributed to a transient host condition with confidence, and an unloadable-but-sha-valid archive would repeat identically forever. Left terminal as the conservative cut. Worth revisiting once daemon-resource errors can be distinguished from archive rejections — but not on a night when a no-fault loop just cost us half the fleet.requires ticket inference identity/inference exchange escaped ticket bounds— the brief named these as v7 raises to reclassify. Neither string exists anywhere in this repo at3de0c62; the nearest live code is the ticket inference probe ininference_broker.go, which is already classifiedmodel_relay_unavailablebyfailRelayUnavailableand preserved through teardown by fix(v7): stop the deferred sandbox teardown from rebilling platform faults to the miner #107. Nothing to do; not silently skipped.Test plan
go test ./cmd/... ./internal/...— greenhttptestobject store: 500/502/503/429, unreachable host, mid-stream truncation →ErrScreenedImageUnavailableTestUnrecognisedFailuresCannotReachTheNoFaultClass— the guard this PR is gated on: 13 realistic non-acquisition errors (incl.nil, empty, harness crash, and the sentinel's own words as prose) all produce a nil classifierNotes
Surface is the sandbox failure envelope and its classification only. No lease code, no heartbeat schema, no
validator_lease_audit, no dashboard, no scores read path — clear of #537, #538, ditto-subnet #284, and backroom #96.Follow-up (separate PR, deliberately not here): add
screened_image_unavailableto ditto-subnet's_SANDBOX_INFRASTRUCTURE_CODES. That is the change that actually activates no-fault retry, and it deserves its own review with the mnemox history in front of it.