Tthe LLM Compressor export modification in mixed‑precision mode: the original format unifies all layers as Linear, causing vLLM inference to fail during parsing.

[1607662794]

Description

This PR fixes an issue where the LLM Compressor exports all layers as Linear type in mixed‑precision mode, regardless of their original type. This causes vLLM to fail during model parsing and prevents inference from starting.

Type of Change

Bug fix

Checklist Before Submitting

• [√] My code has been tested locally.
• [√] Documentation has been updated as needed.
• New or updated tests are included where applicable.
• The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

[wenhuach21]

@Yi4Liu @WeiweiZhang1 please have a review

[yiliu30]

Hi @1607662794, we’ve handled the mixed-precision case this way. Could you follow the same approach here as well? Thanks!

auto-round/auto_round/export/export_to_llmcompressor/export_to_fp.py

Lines 313 to 322 in 98f2cdb

	scheme_groups = {} # (bits, data_type) -> list of layer names
	for name, cfg in layer_config.items():
	layer_bits = cfg.get("bits", bits)
	layer_dt = cfg.get("data_type", data_type)
	if layer_bits > 8:
	continue
	key = (layer_bits, layer_dt)
	scheme_groups.setdefault(key, []).append(name)
	is_mixed = len(scheme_groups) > 1