[Feature]: AutoScheme is currently underperforming compared to Unsloth's model for gguf

[wenhuach21]

Feature Description

Based on feedback from @benjamin-marie, AutoScheme has not yet achieved better results than Unsloth's GGUF models. It would be worthwhile to run a thorough evaluation to better understand the gap and identify opportunities for improvement.

@benjamin-marie We'd appreciate you could share some of your data.

Motivation and Use Case

~

Alternatives Considered

No response

Definition of Done

No response

Additional Context

No response

[wenhuach21]

potential reasons

1 The delta loss objective may not align well with the lm-head layer, which is frequently assigned the lowest bit-width.

2 The embedding layer is not explicitly handled and is instead assigned the highest bit-width, which may disproportionately consume the bit budget and impact overall allocation efficiency.

3 is the calibration data too small?

[benjamin-marie]

I have results and models to share. I'll publish everything within 48 hours, here.

[wenhuach21]

I have results and models to share. I'll publish everything within 48 hours, here.

Thanks a lot!

[benjamin-marie]

I published the models and results:
https://huggingface.co/kaitchup/Qwen3.5-4B-ar-gguf

I only have Qwen3.5 4B.

I ran it like this (I might have used different GGUF_OPTIONS depending on the model. I was limited in CPU RAM when running this so I couldn't add more):

GGUF_OPTIONS = (
    "GGUF:Q4_K_M",
    "GGUF:Q2_K_S",
    "GGUF:Q3_K_S",
    "GGUF:Q3_K_M",
    "GGUF:Q3_K_L",
    "GGUF:Q5_K_M",
    "GGUF:Q6_K",
    "GGUF:Q8_0",
)

model_name = "Qwen/Qwen3.5-4B"
avg_bits = 4.5
scheme = AutoScheme(avg_bits=avg_bits, options=GGUF_OPTIONS, ignore_scale_zp_bits=True)


ar = AutoRound(model=model_name, scheme=scheme, enable_alg_ext=True, iters=500)

[wenhuach21]

Thanks for the information! Would you mind sharing the evaluation code as well?

1. We will add embedding-layer sensitivity analysis. Currently, we use the highest bit-width for the embedding layer as a workaround, which can have a relatively large impact on small models.

2. We will refine the lm-head logic. In the latest release, we do not tune the lm-head in order to reduce memory consumption, which may affect the bit allocation strategy.

3. We will also improve the AutoScheme implementation. For example, Q3_K_M, Q3_K_L, and Q3_K_S currently produce the same results under AutoScheme because they only define different mixed-bit recipes, which are ignored by AutoScheme.

[wenhuach21]

To facilitate debugging and comparison, I evaluated several baselines. Auto-Scheme has not been evaluated yet.

Evaluation Setup

• AutoRound version: 13.0
• GGUF models were evaluated using the fake-format backend for faster evaluation.
• Example command:

lm-eval \
  --model hf \
  --model_args pretrained="Qwen/Qwen3.5-4B" \
  --tasks leaderboard_mmlu_pro,hendrycks_math500,leaderboard_gpqa_diamond,mmlu \
  --batch_size 16

Notes

• hendrycks_math500 reports a score of 0.0 even for the BF16 model, so ignore this metric when comparing results.

	leaderboard_gpqa_diamond	leaderboard_mmlu_pro	mmlu
BF16	0.3384	0.4943	0.7439
gguf:q4km opt-rtn	0.3535	0.4825	0.7368
gguf:q4km iters 500 enable_alg_ext	0.3384	0.4889	0.7351
gguf:q2ks opt-rtn	0.2778	0.2962	0.5687
gguf:q2ks iters 500 enable_alg_ext	0.3182	0.3368	0.6307

[wenhuach21]

	leaderboard_gpqa_diamond	leaderboard_mmlu_pro	mmlu
BF16	0.3384	0.4943	0.7439
gguf:q4km opt-rtn	0.3535	0.4825	0.7368
4.5 auto-scheme gguf:q4_k_s,gguf:q3_k_s,gguf:q2_k_s,gguf:q5_k_m,gguf:q6_k,gguf:q8_0	0.3434	0.4847	0.7329
gguf:q4km iters 500 enable_alg_ext	0.3384	0.4889	0.7351
gguf:q2ks opt-rtn	0.2778	0.2962	0.5687
gguf:q2ks iters 500 enable_alg_ext	0.3182	0.3368	0.6307

[benjamin-marie]

Do these benchmarks generate any tokens?
Last time I look at lm-eval, these benchmarks just relied on logits and didn't generate any tokens. That's also why you get extremely low scores for GPQA (should be >70): the model wants to reason before answering.

I recommend running evalscope instead. It's as easy as lm-eval, but expect it to take much more time. A meaningful evaluation must generate many tokens, otherwise the effect of quantization won't be visible (unless the model is broken).

[wenhuach21]

Do these benchmarks generate any tokens? Last time I look at lm-eval, these benchmarks just relied on logits and didn't generate any tokens. That's also why you get extremely low scores for GPQA (should be >70): the model wants to reason before answering.

I recommend running evalscope instead. It's as easy as lm-eval, but expect it to take much more time. A meaningful evaluation must generate many tokens, otherwise the effect of quantization won't be visible (unless the model is broken).

Sure, make sense! Could you kindly share yours script as a reference?

[wenhuach21]

just have a try, it is indeed very slow, >100x slower than lm-eval
CUDA_VISIBLE_DEVICES=7 evalscope eval --model ./Qwen3.5-4B --datasets mmlu_pro math_500 gpqa_diamond mmlu live_code_bench --model-args '{"precision": "torch.bfloat16"}' --eval-batch-size 64

[wenhuach21]

#1916 provides a small refinement, but I'm seeing conflicting evaluation results between lm-eval and EvalScope.

With lm-eval, the change shows a clear improvement:

CUDA_VISIBLE_DEVICES=7 python3 -m auto_round Qwen/Qwen3-8B \
    --options "gguf:q2_k_s,gguf:q4_k_s" \
    --target_bits 3.5 \
    --ignore_scale_zp_bits \
    --iters 0 \
    --format fake \
    --output_dir "./test_gguf" \
    --tasks mmlu,lambada_openai

However, EvalScope shows a clear drop:

CUDA_VISIBLE_DEVICES=7 evalscope eval \
    --model ./test_gguf/Qwen3-8B-w2g16/ \
    --datasets mmlu mmlu_pro \
    --model-args '{"precision": "torch.bfloat16"}' \
    --eval-batch-size 32 \
    --limit 10

[benjamin-marie]

lm-eval was not meant to evaluate reasoning models. And these benchmarks are quite old and non-generative (no token generated -> quantization errors can't accumulate).

Since you are using the fake format, I recommend serving the model with vLLM, and running evalscope on the vllm endpoint. You can find how to do that here: https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html

"--limit 10" is also way too low to get meaningful results.
The minimum I would run:
--limit 200
--datasets mmlu_pro gpqa_diamond live_code_bench

If you are not limited in compute, drop the "limit 200".

[wenhuach21]

Thanks for the suggestion.

CUDA_VISIBLE_DEVICES=6 python3 -m auto_round Qwen/Qwen3-8B     --options "gguf:q2_k_s,gguf:q4_k_s"     --target_bits 3.5     --ignore_scale_zp_bits     --iters 0     --format fake     --output_dir "./test_gguf

CUDA_VISIBLE_DEVICES=7 vllm serve ./test_gguf/Qwen3-8B-w2g16/ --port 8001 --max-model-len 8192 --host 127.0.0.1 --served-model-name qwen_pr
CUDA_VISIBLE_DEVICES=7 evalscope eval --model qwen --api-url http://127.0.0.1:8000/v1   --api-key EMPTY   --datasets math_500 gpqa_diamoid mmlu_pro  --eval-batch-size 32
~~
gpqa_diamond seems to have a clear improvement. still under tuning.

|                      | math_500    | gpqa_diamond  | mmlu_pro |
| -------------------- | ----------- | ------------- | -------- |
| qwen3-8b bf16        | 0.79        | 0.4242        | 0.6934   |
| qwen3-8b main branch | 0.76/0.768  | 0.3434/0.3687 | 0.6419   |
| pr branch            | 0.764/0.768 | 0.3838/0.4141 | 0.6463   |