Skip to content

Add alternating SVD refinement to SVDQuant (svd_refine_iters) - #1095

Open
Rudin6 wants to merge 1 commit into
vipshop:mainfrom
Rudin6:isvdquant
Open

Add alternating SVD refinement to SVDQuant (svd_refine_iters)#1095
Rudin6 wants to merge 1 commit into
vipshop:mainfrom
Rudin6:isvdquant

Conversation

@Rudin6

@Rudin6 Rudin6 commented Aug 13, 2026

Copy link
Copy Markdown

The one-shot SVDQuant split picks the low-rank branch from the smoothed weight's singular directions alone, so it cannot account for what the residual quantizer will get wrong. This adds an opt-in svd_refine_iters knob: each round refits the factors against

smoothed_weight - dequant(quantize(residual))

then re-anchors the residual on smoothed_weight, letting the low-rank branch absorb the residual quantizer's error pattern. Defaults to 0, which is the original one-shot behaviour.

The refinement lives in the new _refine_lowrank_split() in quantizer.py rather than inside decompose_lowrank_residual(), so the shared lowrank.py and packing.py stay untouched and svd_refine_iters=0 is a no-op by construction. Verified: with the knob at its default, the exported state_dict is bitwise identical to the pre-change output across 72 configurations (shape x precision x dtype x calibrate_precision x rank).

Two details the residual simulation has to get right, both pinned by tests against an independent replication of the packing path:

  • The residual must be re-anchored in the dtype the SVD actually used (float64 on the "high" route, float32 otherwise, since bf16/fp16 SVD falls back to float32). math_dtype is the raw torch_dtype on the "low" route, and reconstructing in bf16 there drifts the next round's refit target.
  • NVFP4 residual quantization is asymmetric: the packer normalizes by the torch_dtype group scales but pack_micro_scale() stores them as FP8 E4M3, so the kernel dequantizes with a different value than it normalized with. INT4 group scales are not micro-scales and stay at torch_dtype on both sides.

Averaged over 14 PixArt-Sigma attention and feed-forward layers with real calibration activations, 5 rounds cut weight error by 7.8-8.9% and layer output error by 17.4-18.2% versus the one-shot split, for both INT4 and NVFP4.

The one-shot SVDQuant split picks the low-rank branch from the smoothed
weight's singular directions alone, so it cannot account for what the
residual quantizer will get wrong. This adds an opt-in svd_refine_iters
knob: each round refits the factors against

    smoothed_weight - dequant(quantize(residual))

then re-anchors the residual on smoothed_weight, letting the low-rank
branch absorb the residual quantizer's error pattern. Defaults to 0,
which is the original one-shot behaviour.

The refinement lives in the new _refine_lowrank_split() in quantizer.py
rather than inside decompose_lowrank_residual(), so the shared lowrank.py
and packing.py stay untouched and svd_refine_iters=0 is a no-op by
construction. Verified: with the knob at its default, the exported
state_dict is bitwise identical to the pre-change output across 72
configurations (shape x precision x dtype x calibrate_precision x rank).

Two details the residual simulation has to get right, both pinned by
tests against an independent replication of the packing path:

  - The residual must be re-anchored in the dtype the SVD actually used
    (float64 on the "high" route, float32 otherwise, since bf16/fp16 SVD
    falls back to float32). math_dtype is the raw torch_dtype on the
    "low" route, and reconstructing in bf16 there drifts the next round's
    refit target.
  - NVFP4 residual quantization is asymmetric: the packer normalizes by
    the torch_dtype group scales but pack_micro_scale() stores them as
    FP8 E4M3, so the kernel dequantizes with a different value than it
    normalized with. INT4 group scales are not micro-scales and stay at
    torch_dtype on both sides.

Averaged over 14 PixArt-Sigma attention and feed-forward layers with real
calibration activations, 5 rounds cut weight error by 7.8-8.9% and layer
output error by 17.4-18.2% versus the one-shot split, for both INT4 and
NVFP4.
@DefTruth

Copy link
Copy Markdown
Member

Thanks for your contribution! by the way, can you should some test results w/ or w/o this commit ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants