integrity: micro-batch-aware MFU ceiling (closes the #1217 compute-forgery class) - #96
Merged
Merged
Conversation
…ass) The flat MAX_PLAUSIBLE_MFU cap is blind to micro_batch — the loophole a forged-down wall_clock exploited. The per-forward matmul M-dim is micro_batch*seq_len; small M underutilizes the tensor cores, so achievable MFU drops sharply. #1217 declared 50% MFU at micro_batch=8/seq=512 (M=4096, 324k tok/s), a ~4.5x wall under-declaration; a real H100 there tops out ~11% (measured, reproduced 3x on a rented H100). The flat 70% cap passed it. - max_plausible_mfu(micro_batch, seq_len): ceiling rises with effective batch (M>=65536->0.70, 32768->0.60, 16384->0.50, 8192->0.42, 4096->0.33, else 0.25), set ~3x above empirically-achieved MFU so honest/optimized runs never trip it. - check_compute_plausibility uses min(max_mfu, ceiling); unknown config -> flat cap (backward-compatible, never false-rejects). - halves the small-batch forgery window (~6x -> ~3x); the airtight fix is on-GPU throughput re-measurement (a pre-crown would-beat-king gate) — follow-up. - 8 tests: rejects #1217, accepts honest slow small-batch + the restored king (ubatch 64) + 5Fbh5xe (ubatch 128), large-batch forgery still caught by flat cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Live king #1217 (5FTfrwU3) was a compute forgery that passed every gate. It declared 2.62B tokens in 8,074s at
micro_batch=8= 324k tok/s / 50% MFU — under the flat 70% cap. I rented an H100 and ran its exact recipe+config:micro_batch=8tops out at 41.7k tok/s (reproduced 3×; ~70k even scaled to SXM). Its real wall was ~7-10h, over the 5h budget — thewall_clock_swas forged ~4× down.The forgery is invisible to existing gates because uniform wall-scaling preserves the log's tps-identity + jitter (log-integrity gate) and the aggregate MFU gate is blind to
micro_batch: the per-forward matmul M-dim ismicro_batch*seq_len, and small batches physically can't sustain high throughput.What
check_compute_plausibilitynow uses a micro-batch-aware ceiling instead of the flat 70%:Ceilings are ~3× the empirically-achieved MFU (H100, seq 512: M=4096→11%, 32768→21%, 65536→27%), so honest and even well-optimized runs never trip them; unknown config falls back to the flat cap (backward-compatible). The restored king 5CqhtHE7 (ubatch-64, 21%) and 5Fbh5xe (ubatch-128, 27%) pass.
Scope
This halves the small-batch forgery window (~6× → ~3× under-declaration). The airtight fix is on-GPU throughput re-measurement (a pre-crown, would-beat-king gate that re-runs the config and compares declared vs measured tok/s) — a follow-up that composes with the PR #95 re-derivation machinery.
8 new tests; full suite green (bar 3 pre-existing env-only failures).
🤖 Generated with Claude Code