Feature Description
Add a --dry-run flag to the CLI that estimates VRAM usage, output file size, and approximate quantization time without running the full quantization process. It loads the model config via AutoConfig.from_pretrained() (no weights) and prints a summary table, then exits.
Motivation and Use Case
Users quantizing large models (70B+) need to know resource requirements before committing GPU hours. Currently the only way to find out is to start the process and watch it fail or succeed. A lightweight estimation step would save significant time and compute costs.
This is related to #1551 (reducing quant cost) and #1584 (VRAM tracking) - both involve understanding resource requirements for quantization.
Alternatives Considered
- Manually calculating estimates from model card metadata - error-prone and tedious
- Running the full process with a tiny calibration set - still loads model weights and takes significant time/memory
Definition of Done
Additional Context
The estimation approach:
- Peak VRAM: calculated from parameter count, dtype size, calibration batch size, and sequence length
- Output size: calculated from target bit width and parameter count
- Time: rough estimate from layer count, iterations, and calibration sample count
Estimation uses AutoConfig.from_pretrained() to load only the model architecture metadata.
Feature Description
Add a
--dry-runflag to the CLI that estimates VRAM usage, output file size, and approximate quantization time without running the full quantization process. It loads the model config viaAutoConfig.from_pretrained()(no weights) and prints a summary table, then exits.Motivation and Use Case
Users quantizing large models (70B+) need to know resource requirements before committing GPU hours. Currently the only way to find out is to start the process and watch it fail or succeed. A lightweight estimation step would save significant time and compute costs.
This is related to #1551 (reducing quant cost) and #1584 (VRAM tracking) - both involve understanding resource requirements for quantization.
Alternatives Considered
Definition of Done
--dry-runCLI flag prints VRAM, output size, and time estimatesdry_run=Trueparameter inAutoRoundAdditional Context
The estimation approach:
Estimation uses
AutoConfig.from_pretrained()to load only the model architecture metadata.