Feature Description
Are there recommended bit options for mixed-precision quantization at an average of 4 or 5 bits? For instance, is it better to supply only two options (e.g., 4 bits and 8 bits for average 5 bits), as in your 3.5-bit experiments, rather than a wider set (e.g., 2~8 bits)? Or, at higher bit-widths, is it better not to use mixed precision at all?
I ask because in SignRoundV2 Table 2, the uniform W3G128/W4G128 rows (gray) beat mixed-precision "Ours" on all four models at an average of 3 bits, and on three of the four at an average of 4 bits. For example, on Qwen3-8B the uniform baseline scores 73.24 vs 72.21 at 3 bits and 74.11 vs 73.76 at 4 bits. Is that expected?
Motivation and Use Case
Choosing options is currently guesswork: the docs give no guidance for weight-only 4/5-bit budgets. A poor option set can make AutoScheme worse than plain uniform quantization, so users burn GPU hours discovering this themselves. Documented recommendations per average bit-width would let us pick a recipe once and trust it.
Alternatives Considered
No response
Definition of Done
No response
Additional Context
No response
Feature Description
Are there recommended bit options for mixed-precision quantization at an average of 4 or 5 bits? For instance, is it better to supply only two options (e.g., 4 bits and 8 bits for average 5 bits), as in your 3.5-bit experiments, rather than a wider set (e.g., 2~8 bits)? Or, at higher bit-widths, is it better not to use mixed precision at all?
I ask because in SignRoundV2 Table 2, the uniform W3G128/W4G128 rows (gray) beat mixed-precision "Ours" on all four models at an average of 3 bits, and on three of the four at an average of 4 bits. For example, on Qwen3-8B the uniform baseline scores 73.24 vs 72.21 at 3 bits and 74.11 vs 73.76 at 4 bits. Is that expected?
Motivation and Use Case
Choosing options is currently guesswork: the docs give no guidance for weight-only 4/5-bit budgets. A poor option set can make AutoScheme worse than plain uniform quantization, so users burn GPU hours discovering this themselves. Documented recommendations per average bit-width would let us pick a recipe once and trust it.
Alternatives Considered
No response
Definition of Done
No response
Additional Context
No response