Resolve the MFU peak by serving precision and attribute the device count - #1
Merged
Merged
Conversation
Hand ported from the private agent, not mirrored: this tree names the package mfu and has no GPUType, so the hardware mismatch caveat names the raw model string here. The MFU denominator had two independent errors pushing in opposite directions, so correcting either alone produced a more plausible looking number that was still wrong. Precision. Achieved TFLOPS is a precision independent 2N per token count, but the peak table held one dense BF16 figure per SKU, so FP8 serving reported double the real MFU and FP4 quadruple. The "MFU > 100%" guard only fires when the inflation crosses 100, so the harmful case of an FP8 workload at a true 20% reported as an actionable 40% passed it. The table is now keyed by (SKU, precision) and holds dense figures only, and serving precision is read off the engine command line for vLLM, SGLang and TGI. Weight only schemes such as AWQ and GPTQ dequantize before the GEMM and keep the BF16 peak, since mapping them to a 4 bit peak would understate MFU by about 4x. The KV cache dtype is storage, not compute, and does not move the divisor. An undetected precision falls back to BF16 and says so. Devices. The peak was multiplied by a host wide GPU enumeration, which asserts the scanned engine owns every device on the box. A replica narrower than its host was measured against hardware it cannot touch, which deflates MFU, inflates the headroom fraction, and recommends consolidating capacity that was never idle. The count now carries an attribution, and an unattributed multi GPU count withholds the headroom dollar figure rather than printing one an operator would act on. The serving process is matched to the scanned endpoint by port so one replica's width is not attributed to another's throughput. A MIG host produces no estimate unless an explicit count is given, because the enumeration reports whole boards while the engine holds a slice of one. No sparse value is stored anywhere in the table. AMD's 20.1 PFLOPS FP4 headline for MI355X is that part's dense FP4 with 2:4 sparsity applied, and on H100 the BF16 sparse and FP8 dense figures are both 1979, so a lookup reaching it via sparsity would look right on H100 and be wrong elsewhere. A table wide test rejects any row whose FP8 exceeds twice its dense BF16 or whose FP4 exceeds four times it. Also splits the V100 row, which carried the PCIe figure while matching both form factors and ran 10.4% low on SXM2 hosts, and adds the H100 PCIe row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hand-ported from the private agent, not mirrored. This tree names the
package
mfuand has noGPUType, so the hardware-mismatch caveat namesthe raw model string here.
The MFU denominator had two independent errors pushing in opposite
directions, so correcting either alone produced a more plausible looking
number that was still wrong.
Precision
Achieved TFLOPS is a precision independent 2N per token count, but the
peak table held one dense BF16 figure per SKU, so FP8 serving reported
double the real MFU and FP4 quadruple. The
MFU > 100%guard only fireswhen the inflation crosses 100, so the harmful case passed it: an FP8
workload at a true 20% reported as an actionable 40%.
The table is now keyed by
(SKU, precision)and holds dense figures only,and serving precision is read off the engine command line for vLLM, SGLang
and TGI, since no engine exports dtype or quantization on its metrics
endpoint.
Weight-only schemes (AWQ, GPTQ, marlin, bitsandbytes) dequantize before
the GEMM and keep the BF16 peak; mapping them to a 4 bit peak would
understate MFU by about 4x.
--kv-cache-dtypeis storage, not compute,and does not move the divisor. An undetected precision falls back to BF16
and says so rather than printing a bare number.
Devices
The peak was multiplied by a host-wide GPU enumeration, which asserts the
scanned engine owns every device on the box. A replica narrower than its
host was measured against hardware it cannot touch, deflating MFU,
inflating the headroom fraction, and recommending consolidation of
capacity that was never idle.
The count now carries an attribution, and an unattributed multi-GPU count
withholds the headroom dollar figure rather than printing one an operator
would act on. The serving process is matched to the scanned endpoint by
port so one replica's width is not attributed to another's throughput. A
MIG host produces no estimate unless an explicit count is given, because
the enumeration reports whole boards while the engine holds a slice of
one.
cmd/scanis wired through, so the public tool attributes devices ratherthan always assuming the host.
Sourcing, and the sparsity trap
No sparse value is stored anywhere in the table.
AMD's 20.1 PFLOPS FP4 headline for MI355X is that part's dense FP4 with
2:4 sparsity applied, and is exactly four times its own dense FP8 figure.
On H100 the BF16 sparse and FP8 dense figures are both 1979, so a lookup
reaching it via sparsity would look right on H100 and be wrong elsewhere.
A table-wide test rejects any row whose FP8 exceeds twice its dense BF16
or whose FP4 exceeds four times it, and asserts every row states its form
factor. The NVIDIA rows were checked against their datasheets and were
already dense and correct.
Also
v100split into SXM2 (125) and PCIe (112). One row carried the PCIefigure while matching both names, running 10.4% low on SXM2 hosts.
h100 pcieadded as its own row (756 / 1513).Verification
go test ./...green across the tree.Hardware validation is outstanding: no MIG host and no multi-GPU host
running a narrower replica was available. The MIG path fails loudly rather
than miscalculating, so the untested case is a refusal.