integrity: compile-aware MFU ceiling (stops the uncompiled-Muon forgery flood) - #97
Open
karpabot wants to merge 1 commit into
Open
integrity: compile-aware MFU ceiling (stops the uncompiled-Muon forgery flood)#97karpabot wants to merge 1 commit into
karpabot wants to merge 1 commit into
Conversation
…ry flood) The micro_batch ceiling (PR#96) still let uncompiled Muon runs declare torch.compile-grade throughput: andreastanm 239k (37% MFU) and Kaizen 356k (54.9% MFU) both sailed under the 70% cap because the cap is blind to whether the recipe actually compiles. Measured on a real H200 (ubatch-128/seq-512): Muon UNCOMPILED = 86,436 tok/s (~13% MFU); torch.compile'd = 207,700 (~32%). - UNCOMPILED_MFU_CEILING=0.35: an uncompiled recipe declaring MFU above this is a forged-down wall_clock (honest uncompiled tops out ~27%, 5Fbh5xe). - max_plausible_mfu(...recipe_compiles): compiled keeps the micro_batch ceiling (up to 70%); uncompiled is capped at 0.35. - _recipe_uses_compile: config.compile truthy AND patch.diff actually contains torch.compile — a bare config flag is a no-op (hasattr-dropped) and can't be trusted to earn the higher ceiling. - op1 threads it into check_compute_plausibility. Verified on the real bundles: Kaizen(356k,uncompiled) + andreastanm(239k, uncompiled) REJECTED; danielortega(239k,COMPILED) + 5Fbh5xe(174k,uncompiled,27%) PASS. 8 new tests. Interim while the on-GPU re-measurement gate is built. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The forgery flood (2026-07-04/05) exploited a gap PR #96 left open: the micro_batch MFU ceiling (70% at ubatch-128) is blind to whether the recipe actually torch.compiles. Uncompiled Muon runs declared compile-grade throughput and sailed under the cap — andreastanm 239k (37% MFU) and Kaizen 356k (54.9%), both uncompiled.
I measured the ground truth on a real H200 (ubatch-128/seq-512): Muon uncompiled = 86,436 tok/s (~13% MFU), torch.compile'd = 207,700 (~32%) — a 2.4× gap. So an uncompiled recipe claiming >35% MFU is a forged-down
wall_clock_s.What
UNCOMPILED_MFU_CEILING = 0.35— uncompiled recipes are capped here (honest uncompiled tops out ~27%, verified on 5Fbh5xe); compiled recipes keep the micro_batch ceiling (up to 70%)._recipe_uses_compile— a recipe counts as compiling only ifconfig.compileis truthy ANDpatch.diffactually containstorch.compile. A bare config flag is a no-op (dropped byhasattron the canonicalTrainConfig) and can't be trusted to earn the higher ceiling.recipe_compilesintocheck_compute_plausibility.Verified on the real bundles
8 new tests; full suite green. This is the interim fix to re-enable the validator now; the durable fix (on-GPU throughput re-measurement) is the follow-up.
🤖 Generated with Claude Code