[Bug]: AutoScheme ram usage is too large

[wenhuach21]

Problem Description

 CUDA_VISIBLE_DEVICES=6 python3 -m auto_round Qwen/Qwen3.5-4B/ --options "gguf:q4_k_s,gguf:q3_k_s,gguf:q2_k_s,gguf:q5_k_m,gguf:q6_k,gguf:q8_0" --target_bits 4.5 --ignore_scale_zp_bits --output_dir "./test_as_orig_45" --iters 0  --tasks leaderboard_mmlu
_pro,leaderboard_gpqa_diamond,mmlu --eval_model_dtype bf16 --format fake --eval_bs 8^C

diamond,mmlu --eval_model_dtype bf16 --format fake --eval_bs 8
2026-06-11 05:09:21 INFO main.py L652: start to quantize Qwen/Qwen3.5-4B
2026-06-11 05:09:21 INFO config.py L45: enable_opt_rtn is turned on, set --disable_opt_rtn for higher speed at the cost of accuracy.
2026-06-11 05:09:21 INFO entry.py L587: Using MLLM mode for multimodal model.
404 Client Error. (Request ID: Root=1-6a2a4301-1598689354f823af28fec646;06d37a52-0881-4b45-89c5-2710da1684da)

Entry Not Found for url: https://huggingface.co/Qwen/Qwen3.5-4B/resolve/main/model_index.json.
Fetching 2 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 6200.01it/s]
Download complete: : 0.00B [00:00, ?B/s] | 0/2 [00:00<?, ?it/s]
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 723/723 [00:00<00:00, 8863.54it/s]
[transformers] loss_type=None was set in the config but it is unrecognized. Using the default loss: ForCausalLMLoss.
2026-06-11 05:09:29 INFO base.py L419: AutoScheme on multimodal LLM: scoring the language tower only with text-only calibration (multimodal dataloader will be used as a fallback if needed).
2026-06-11 05:09:29 INFO base.py L500: AutoScheme (VLM): scoring 249 language-tower layers; 1 vision/audio-tower layers kept at their original 16-bit configuration.
2026-06-11 05:09:30 INFO gen_auto_scheme.py L200: AutoScheme option GGUF:Q4_K_S -> avg_bits=4.0000
2026-06-11 05:09:30 INFO gen_auto_scheme.py L200: AutoScheme option GGUF:Q3_K_S -> avg_bits=3.0000
2026-06-11 05:09:30 INFO gen_auto_scheme.py L200: AutoScheme option GGUF:Q5_K_M -> avg_bits=5.0000
2026-06-11 05:09:30 INFO gen_auto_scheme.py L200: AutoScheme option GGUF:Q6_K -> avg_bits=6.0000
2026-06-11 05:09:30 INFO gen_auto_scheme.py L94: Average bits range: [3.000, 6.000], target = 4.500
2026-06-11 05:09:31 INFO offload.py L542: clearing module weights to free RAM...
2026-06-11 05:09:31 INFO offload.py L549: module weights cleared
Generating AutoScheme: 0%| | 0/512 [00:00<?, ?it/s]2026-06-11 05:09:39 INFO delta_loss.py L642: AutoScheme: disabled requires_grad on 278 non-wrapper parameters (only wrapper.orig_layer.weight needs grad for scoring; saves ~one model-worth of grad buffer during backward).
2026-06-11 05:09:39 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Generating AutoScheme: 6%|█████████████▊ | 32/512 [00:22<02:55, 2.74it/s]/usr/local/lib/python3.12/dist-packages/torch/autograd/graph.py:869: UserWarning: Flash Attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:124.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
Generating AutoScheme: 25%|██████████████████████████████████████████████████████▊ | 128/512 [01:19<05:31, 1.16it/s]2026-06-11 05:10:58 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Generating AutoScheme: 50%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 256/512 [02:35<03:38, 1.17it/s]2026-06-11 05:12:14 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Generating AutoScheme: 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 384/512 [03:55<02:09, 1.02s/it]2026-06-11 05:13:34 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Generating AutoScheme: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [05:14<00:00, 1.04it/s]2026-06-11 05:18:53 INFO device.py L1838: AutoScheme complete (low_cpu_mem_usage=enabled) 'peak_ram': 89.28GB, 'peak_vram': 9.49GB
Generating AutoScheme: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [09:22<00:00, 1.10s/it]
2026-06-11 05:18:54 INFO base.py L662: 'enable_torch_compile' is set to False by default. Enabling it can reduce tuning cost by 20%, but it might throw an exception.
2026-06-11 05:18:54 INFO data_driven.py L1089: start to compute imatrix
2026-06-11 05:18:54 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
2026-06-11 05:18:59 INFO mllm.py L83: Using MLLM template: qwen3_5
2026-06-11 05:18:59 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
Quantizing model.language_model.layers.0: 0%| | 0/32 [00:00<?, ?it/s]2026-06-11 05:19:09 INFO offload.py L706: OffloadManager (compressor): tempdir = /workspace/ar_work_space/offload/compressor_tks69g0h
2026-06-11 05:19:10 INFO device.py L1840: 'peak_ram': 89.28GB, 'peak_vram': 9.49GB
Quantizing model.language_model.layers.1: 3%|██████▎ | 1/32 [00:06<03:19, 6.44s/it]2026-06-11 05:19:15 INFO device.py L1840: 'peak_ram': 89.28GB, 'peak_vram': 9.49GB
Quantizing model.language_model.layers.2: 6%|████████████▋ | 2/32 [00:11<02:52, 5.75s/it]2026-06-11 05:19:20 INFO device.py L1840: 'peak_ram': 89.28GB, 'peak_vram': 9.49GB
Quantizing model.language_model.layers.3: 9%|███████████████████ | 3/32 [00:16<02:40, 5.53s/it]2026-06-11 05:19:24 INFO device.py L1840: 'peak_ram': 89.28GB, 'peak_vram': 9.49GB

Reproduction Steps

~

Environment Information

~

Error Logs

Additional Context

No response

[wenhuach21]

#1916
90->50+G, not enough