[Bug]: Llama-3.1-8B_mxfp8_fp8attn_LLMC vllm eval failed

[chensuyue]

Problem Description

Llama-3.1-8B_mxfp8_fp8attn_LLMC vllm eval failed
https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/903/artifact/evaluation_Llama-3.1-8B_mxfp8.log

Reproduction Steps

cd neural-compressor/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/llama3
bash run_quant.sh --topology=Llama-3.1-8B --dtype=mxfp8 --input_model=/mnt/disk1/models/Llama-3.1-8B-Instruct --output_model=/mnt/disk1/jenkins/saved_models/Llama-3.1-8B_mxfp8_fp8attn_LLMC --static_attention_dtype=fp8 --export_format=llm_compressor
bash run_benchmark.sh --model_path=/mnt/disk1/jenkins/saved_models/Llama-3.1-8B_mxfp8_fp8attn_LLMC/Llama-3.1-8B-Instruct-mxfp-w8g32 --tasks=hellaswag,piqa,mmlu_llama,gsm8k_llama --static_attention_dtype=fp8

Environment Information

torch 2.11.0
compressed-tensors 0.17.0
transformers 5.14.1
vllm 0.25.1
lm_eval 0.4.12
auto-round 0.14.2

Error Logs

(EngineCore pid=3241557) ERROR 07-24 16:09:20 [core.py:1231] ValueError: There is no module or parameter named 'layers.0.self_attn.q_max' in LlamaModel. The available parameters belonging to layers.0.self_attn (LlamaAttention) are: {'layers.0.self_attn.attn.q_zero_point', 'layers.0.self_attn.o_proj.weight', 'layers.0.self_attn.qkv_proj.weight', 'layers.0.self_attn.attn.q_scale', 'layers.0.self_attn.qkv_proj.weight_scale', 'layers.0.self_attn.o_proj.weight_scale', 'layers.0.self_attn.attn.k_scale', 'layers.0.self_attn.attn.k_zero_point', 'layers.0.self_attn.attn.v_scale', 'layers.0.self_attn.attn.v_zero_point'}

Additional Context

No response

[chensuyue]

Qwen3 also failed with this issue, https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/940/artifact/evaluation_Qwen3-30B-A3B-Instruct-2507_mxfp8.log

[chensuyue]

#2103

[chensuyue]

Qwen3 runs successfully with correct accuracy results, but llama3 doesn't.
https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/972/
Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,hellaswag,0.2556
Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,piqa,0.5354
Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,mmlu_llama, 0
Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,gsm8k_llama, 0
Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,average,0.19775

https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/974/
Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,hellaswag,0.2608
Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,piqa,0.5299
Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,mmlu_llama, 0
Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,gsm8k_llama, 0
Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,average,0.19767

[yiliu30]

Qwen3 runs successfully with correct accuracy results, but llama3 doesn't. https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/972/ Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,hellaswag,0.2556 Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,piqa,0.5354 Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,mmlu_llama, 0 Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,gsm8k_llama, 0 Llama-3.1-8B,mxfp8,mxfp8_fp8attn_llmc,average,0.19775

https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/974/ Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,hellaswag,0.2608 Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,piqa,0.5299 Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,mmlu_llama, 0 Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,gsm8k_llama, 0 Llama-3.3-70B,mxfp8,mxfp8_fp8attn_llmc,average,0.19767

@xin3he could you help take a look, thanks!