op4: fail closed on arch-divergent checkpoints (close the miner-eval bypass) - #99
Open
karpabot wants to merge 1 commit into
Open
op4: fail closed on arch-divergent checkpoints (close the miner-eval bypass)#99karpabot wants to merge 1 commit into
karpabot wants to merge 1 commit into
Conversation
…bypass) Exploit (Kaizen0304 2026-07-05, PR#1321): a patch adds value_embeddings to model/_v4skip.py, so the checkpoint carries value_embed/ve_lambda params the validator's canonical RalphBase lacks -> load_state_dict fails -> op4 fell back to _patched_hidden_eval, which rebuilds the model from the MINER's patched model/ code and runs it to compute the crown metric. On the HOSB-off legacy path that eval is NOT answer-blanked, so the miner's own code sees the raw held-out and the crown (val_bpb 1.2378) was never independently verified. The compute gates don't touch it — nothing about the compute is forged. - _legacy_hidden_eval: on a shape/arch mismatch, REJECT instead of falling back to the miner's eval code. Canonical-arch checkpoints load into the validator's own model + score independently (unchanged); only arch-divergent ones are refused. - Override RALPH_ALLOW_PATCHED_EVAL=1 restores the fallback (testnet/debug, or once HOSB answer-blanked eval is enforced — which safely re-enables arch innovation). - 2 tests (fail-closed refuses + never runs miner code; override restores fallback). Trades arch innovation for an airtight crown-verification guarantee while HOSB is off; HOSB enforce is the durable path that re-allows novel archs safely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The weak point (a verification bypass, not a compute forgery)
Kaizen0304 (PR #1321) added
value_embeddingstomodel/_v4skip.py. The checkpoint then carriesvalue_embed/ve_lambdaparams the validator's canonicalRalphBasedoesn't have →load_state_dictfails → op4 fell back to_patched_hidden_eval, which rebuilds the model from the miner's patched code and runs it to compute the crown metric. With HOSB off, that eval isn't answer-blanked, so the miner's own code sees the raw held-out — the crown's val_bpb 1.2378 was never independently verified. My compute gates (#96/#97/#98) don't touch it because nothing about the compute is forged.Fix: fail closed
_legacy_hidden_eval(the HOSB-off path) now rejects an arch-divergent checkpoint instead of running the miner's eval code:RALPH_ALLOW_PATCHED_EVAL=1restores the fallback for testnet/debug — or once HOSB (answer-blanked eval) is enforced, which is the durable path that re-allows arch innovation safely.Why this over enforcing HOSB now
Fail-closed is the more robust interim: it shrinks the attack surface to zero (no miner code in the eval), it's a deterministic check, and it can't be gamed. HOSB is the feature-complete long-term answer (it lets miners innovate on arch) but still runs miner code in a sandbox with answer-blanking — more surface, known failure modes (fail-closed on archs its loader can't build; raw-shard mount leak), and it's shadow-only/unhardened. Ship fail-closed now, harden HOSB next.
Trade-off: blocks legit arch changes (value_embeddings, readout-calib) until HOSB is on. Accepted during the forgery flood.
2 tests. Composes with #97 (compute) + #98 (throughput re-measure).
🤖 Generated with Claude Code