validator: on-GPU throughput re-measurement gate (durable compute-forgery fix) - #98
Open
karpabot wants to merge 1 commit into
Open
validator: on-GPU throughput re-measurement gate (durable compute-forgery fix)#98karpabot wants to merge 1 commit into
karpabot wants to merge 1 commit into
Conversation
…gery fix) The static MFU ceilings (PR#96/RalphLabsAI#97) narrow the forgery window but can't know a specific recipe's real achievable throughput. This measures it: op_compute_remeasure re-runs a would-be king's EXACT config for K steps on the validator's own GPU (synthetic data — throughput is data-independent) and rejects a declared tok/s that exceeds the measured achievable by more than a GPU-scaled margin. Nothing to game — it is the physical rate the forger lies about. It caught every real forgery this week (2-8x over measured) and passes the legit king (danielortega, 1.15x, within margin). - op_compute_remeasure: pre-crown, would-beat-king-only (cost O(king-changes)). Wired into service.score_and_decide; a failed verdict WITHHOLDS the crown (status remeasure_withheld), never dethrones the sitting king. - _remeasure_verdict (pure, unit-tested): declared <= measured * gpu_scale * margin; scale by bf16 peak (H100<->H200==1.0), margin default 1.6x absorbs memory-bandwidth + torch-version + thermal variance (danielortega's legit +15%). - _run_throughput_probe: copy canonical recipe + apply patch + synthetic shards + run train.py K steps, measure steady-state tok/s (skip step-0 compile/warmup). Env-tunable: RALPH_REMEASURE_STEPS/_TIMEOUT_S/_MARGIN. - off/shadow/enforce via RALPH_COMPUTE_REMEASURE; ships OFF. Every ambiguity fail-OPEN. - 18 tests (mode, verdict vs real numbers, GPU-scaling, fail-open, margin boundary). Ships OFF: shadow-calibrate on the box (run vs 5CqhtHE7 + known bundles, confirm the forgeries would-REJECT and legit would-PASS, tune margin), THEN flip to enforce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The durable fix for the forgery flood
Static MFU ceilings (PR #96 micro_batch, PR #97 compile-aware) narrow the window but can't know a specific recipe's real ceiling. This measures it: re-run the would-be king's exact config for ~30 steps on the validator's own GPU (synthetic data — throughput is data-independent) and reject a declared tok/s that can't be reproduced. There is no ceiling to game — it's the physical rate the forger lies about.
How it decides (
_remeasure_verdict, pure + unit-tested)declared_tps <= measured_tps × gpu_scale × margin, wheregpu_scale= declared-GPU bf16 peak / validator-GPU peak (H100↔H200 = 1.0), andmargin(default 1.6×) absorbs memory-bandwidth + torch-version + thermal variance. Verified on the real numbers:Safety / rollout
status: remeasure_withheld); it never dethrones the sitting king and never auto-blacklists. Every ambiguity (disabled / no config / probe fail / timeout) is fail-OPEN.RALPH_COMPUTE_REMEASURE∈ {off, shadow, enforce}. Rollout: shadow on the box → confirm the known forgeries logwould_REJECTand 5CqhtHE7/danielortega logwould_PASS→ tuneRALPH_REMEASURE_MARGIN→ flip to enforce.RALPH_REMEASURE_STEPS(30),_TIMEOUT_S(1500, covers max-autotune),_MARGIN(1.6).The GPU probe needs hardware, so it's validated in shadow on the box; the mode-parsing, verdict math, GPU-scaling, and fail-open are unit-tested (18 tests). Composes with PR #97 (interim compile-aware ceiling holds the line meanwhile).
🤖 Generated with Claude Code