Add alternating SVD refinement to SVDQuant (svd_refine_iters) - #1095
Open
Rudin6 wants to merge 1 commit into
Open
Add alternating SVD refinement to SVDQuant (svd_refine_iters)#1095Rudin6 wants to merge 1 commit into
Rudin6 wants to merge 1 commit into
Conversation
The one-shot SVDQuant split picks the low-rank branch from the smoothed
weight's singular directions alone, so it cannot account for what the
residual quantizer will get wrong. This adds an opt-in svd_refine_iters
knob: each round refits the factors against
smoothed_weight - dequant(quantize(residual))
then re-anchors the residual on smoothed_weight, letting the low-rank
branch absorb the residual quantizer's error pattern. Defaults to 0,
which is the original one-shot behaviour.
The refinement lives in the new _refine_lowrank_split() in quantizer.py
rather than inside decompose_lowrank_residual(), so the shared lowrank.py
and packing.py stay untouched and svd_refine_iters=0 is a no-op by
construction. Verified: with the knob at its default, the exported
state_dict is bitwise identical to the pre-change output across 72
configurations (shape x precision x dtype x calibrate_precision x rank).
Two details the residual simulation has to get right, both pinned by
tests against an independent replication of the packing path:
- The residual must be re-anchored in the dtype the SVD actually used
(float64 on the "high" route, float32 otherwise, since bf16/fp16 SVD
falls back to float32). math_dtype is the raw torch_dtype on the
"low" route, and reconstructing in bf16 there drifts the next round's
refit target.
- NVFP4 residual quantization is asymmetric: the packer normalizes by
the torch_dtype group scales but pack_micro_scale() stores them as
FP8 E4M3, so the kernel dequantizes with a different value than it
normalized with. INT4 group scales are not micro-scales and stay at
torch_dtype on both sides.
Averaged over 14 PixArt-Sigma attention and feed-forward layers with real
calibration activations, 5 rounds cut weight error by 7.8-8.9% and layer
output error by 17.4-18.2% versus the one-shot split, for both INT4 and
NVFP4.
Member
|
Thanks for your contribution! by the way, can you should some test results w/ or w/o this commit ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The one-shot SVDQuant split picks the low-rank branch from the smoothed weight's singular directions alone, so it cannot account for what the residual quantizer will get wrong. This adds an opt-in svd_refine_iters knob: each round refits the factors against
then re-anchors the residual on smoothed_weight, letting the low-rank branch absorb the residual quantizer's error pattern. Defaults to 0, which is the original one-shot behaviour.
The refinement lives in the new _refine_lowrank_split() in quantizer.py rather than inside decompose_lowrank_residual(), so the shared lowrank.py and packing.py stay untouched and svd_refine_iters=0 is a no-op by construction. Verified: with the knob at its default, the exported state_dict is bitwise identical to the pre-change output across 72 configurations (shape x precision x dtype x calibrate_precision x rank).
Two details the residual simulation has to get right, both pinned by tests against an independent replication of the packing path:
Averaged over 14 PixArt-Sigma attention and feed-forward layers with real calibration activations, 5 rounds cut weight error by 7.8-8.9% and layer output error by 17.4-18.2% versus the one-shot split, for both INT4 and NVFP4.