Skip to content

Resolve the MFU peak by serving precision and attribute the device count - #1

Merged
BorisMorozov merged 1 commit into
mainfrom
fix/mfu-denominator-precision-and-devices
Aug 15, 2026
Merged

Resolve the MFU peak by serving precision and attribute the device count#1
BorisMorozov merged 1 commit into
mainfrom
fix/mfu-denominator-precision-and-devices

Conversation

@BorisMorozov

Copy link
Copy Markdown
Member

Hand-ported from the private agent, not mirrored. This tree names the
package mfu and has no GPUType, so the hardware-mismatch caveat names
the raw model string here.

The MFU denominator had two independent errors pushing in opposite
directions, so correcting either alone produced a more plausible looking
number that was still wrong.

Precision

Achieved TFLOPS is a precision independent 2N per token count, but the
peak table held one dense BF16 figure per SKU, so FP8 serving reported
double the real MFU and FP4 quadruple. The MFU > 100% guard only fires
when the inflation crosses 100, so the harmful case passed it: an FP8
workload at a true 20% reported as an actionable 40%.

The table is now keyed by (SKU, precision) and holds dense figures only,
and serving precision is read off the engine command line for vLLM, SGLang
and TGI, since no engine exports dtype or quantization on its metrics
endpoint.

Weight-only schemes (AWQ, GPTQ, marlin, bitsandbytes) dequantize before
the GEMM and keep the BF16 peak; mapping them to a 4 bit peak would
understate MFU by about 4x. --kv-cache-dtype is storage, not compute,
and does not move the divisor. An undetected precision falls back to BF16
and says so rather than printing a bare number.

Devices

The peak was multiplied by a host-wide GPU enumeration, which asserts the
scanned engine owns every device on the box. A replica narrower than its
host was measured against hardware it cannot touch, deflating MFU,
inflating the headroom fraction, and recommending consolidation of
capacity that was never idle.

The count now carries an attribution, and an unattributed multi-GPU count
withholds the headroom dollar figure rather than printing one an operator
would act on. The serving process is matched to the scanned endpoint by
port so one replica's width is not attributed to another's throughput. A
MIG host produces no estimate unless an explicit count is given, because
the enumeration reports whole boards while the engine holds a slice of
one.

cmd/scan is wired through, so the public tool attributes devices rather
than always assuming the host.

Sourcing, and the sparsity trap

No sparse value is stored anywhere in the table.

AMD's 20.1 PFLOPS FP4 headline for MI355X is that part's dense FP4 with
2:4 sparsity applied, and is exactly four times its own dense FP8 figure.
On H100 the BF16 sparse and FP8 dense figures are both 1979, so a lookup
reaching it via sparsity would look right on H100 and be wrong elsewhere.

A table-wide test rejects any row whose FP8 exceeds twice its dense BF16
or whose FP4 exceeds four times it, and asserts every row states its form
factor. The NVIDIA rows were checked against their datasheets and were
already dense and correct.

Also

  • v100 split into SXM2 (125) and PCIe (112). One row carried the PCIe
    figure while matching both names, running 10.4% low on SXM2 hosts.
  • h100 pcie added as its own row (756 / 1513).

Verification

go test ./... green across the tree.

Hardware validation is outstanding: no MIG host and no multi-GPU host
running a narrower replica was available. The MIG path fails loudly rather
than miscalculating, so the untested case is a refusal.

Hand ported from the private agent, not mirrored: this tree names the
package mfu and has no GPUType, so the hardware mismatch caveat names the
raw model string here.

The MFU denominator had two independent errors pushing in opposite
directions, so correcting either alone produced a more plausible looking
number that was still wrong.

Precision. Achieved TFLOPS is a precision independent 2N per token count,
but the peak table held one dense BF16 figure per SKU, so FP8 serving
reported double the real MFU and FP4 quadruple. The "MFU > 100%" guard only
fires when the inflation crosses 100, so the harmful case of an FP8
workload at a true 20% reported as an actionable 40% passed it. The table
is now keyed by (SKU, precision) and holds dense figures only, and serving
precision is read off the engine command line for vLLM, SGLang and TGI.
Weight only schemes such as AWQ and GPTQ dequantize before the GEMM and
keep the BF16 peak, since mapping them to a 4 bit peak would understate MFU
by about 4x. The KV cache dtype is storage, not compute, and does not move
the divisor. An undetected precision falls back to BF16 and says so.

Devices. The peak was multiplied by a host wide GPU enumeration, which
asserts the scanned engine owns every device on the box. A replica narrower
than its host was measured against hardware it cannot touch, which deflates
MFU, inflates the headroom fraction, and recommends consolidating capacity
that was never idle. The count now carries an attribution, and an
unattributed multi GPU count withholds the headroom dollar figure rather
than printing one an operator would act on. The serving process is matched
to the scanned endpoint by port so one replica's width is not attributed to
another's throughput. A MIG host produces no estimate unless an explicit
count is given, because the enumeration reports whole boards while the
engine holds a slice of one.

No sparse value is stored anywhere in the table. AMD's 20.1 PFLOPS FP4
headline for MI355X is that part's dense FP4 with 2:4 sparsity applied, and
on H100 the BF16 sparse and FP8 dense figures are both 1979, so a lookup
reaching it via sparsity would look right on H100 and be wrong elsewhere. A
table wide test rejects any row whose FP8 exceeds twice its dense BF16 or
whose FP4 exceeds four times it.

Also splits the V100 row, which carried the PCIe figure while matching both
form factors and ran 10.4% low on SXM2 hosts, and adds the H100 PCIe row.
@BorisMorozov
BorisMorozov merged commit 9ff0868 into main Aug 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant