feat: support dense Qwen3 weight conversion - #114
Merged
chaokunyang merged 1 commit intoAug 7, 2026
Conversation
Qwen3ForCausalLM was not registered, so it fell back to the default
converters and raised NotImplementedError on self_attn.q_norm.weight while
the reader was building its parameter metadata. The reader then never
published infer_conf and the writer waited for it indefinitely.
Dense and MoE Qwen3 share the layout the existing converters target:
canonical self_attn.{q,k,v,o}_proj with per-head self_attn.{q,k}_norm, GQA
head grouping in the fused Megatron linear_qkv, and gate/up projections
that sglang serves fused. The dense model only lacks the expert
parameters, which those converters emit nothing for, so registering the
architecture against them is enough.
Dense Qwen3 also ties its output weights, which exposed a second gap: the
lm_head alias for tied embeddings was only added for vLLM. sglang ties
them the same way while the training side still publishes lm_head, and the
reader adds the same alias on its side, so without it the transfer plan
reports lm_head.weight as missing.
chaokunyang
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Qwen3ForCausalLMis not registered inawex/models, so it falls back to thedefault converters. Those follow the bailing-style attention naming, and
SGlangToHFWeightConverter._convert_layer_norm_paramraisesNotImplementedErroronself_attn.q_norm.weight. The failure happens whilethe reader is building its parameter metadata, so the reader never publishes
infer_confand the writer waits for it indefinitely: the run stalls ratherthan reporting the unsupported name.
Dense and MoE Qwen3 share the layout the existing
qwen3_moeconvertersalready target:
self_attn.{q,k,v,o}_projwith per-headself_attn.{q,k}_normlinear_qkvMergedColumnParallelLinearwhile the train side reports them splitThe dense model only lacks the expert parameters, which those converters emit
nothing for, so registering the architecture against them is enough and the
base converters are untouched. This mirrors
ling_linear.py, which reusesling.py's converters for its own architectures.Dense Qwen3 also ties its output weights, which surfaced a second gap. Three
places add the
lm_head.weightalias for tied embeddings, and they disagree:reader/weights_reader.pymeta/train_meta_resolver.pymeta/infer_meta_resolver.pyengine_name == "vllm"The reader therefore adds the alias on sglang while the inference metadata does
not declare it, and the transfer plan reports
lm_head.weightas missing. ThisPR drops the engine gate so the third site matches the other two.
Verification
awex/testspasses in full, on top of four new cases for the denseregistration and four for the tied-embedding alias.
main: asking the registry forQwen3ForCausalLMlogsModel Qwen3ForCausalLM not found, using default strategy, returns the baseSGlangToHFWeightConverter, andself_attn.q_norm.weightraisesNotImplementedError. The new tests failwithout this change and the tied-embedding case fails for
sglangwhilepassing for
vllm.training, AWEX weight transfer) completes cleanly: no
Unsupported layer norm parameter name, no wait oninfer_conf, no missinglm_head.weight, and the"using default strategy" fallback no longer appears.
Not covered by unit tests: the Megatron-side dense path. It is exercised by the
end-to-end run above but has no test of its own yet.
Related issues
None.
Does this PR introduce any user-facing change?
No signatures change and no new configuration is required;
Qwen3ForCausalLMis picked up by the existing registry discovery.
Worth noting for review: for a tied-embedding model served by sglang, the
published inference metadata now contains an additional
lm_head.weightentry.That is not a new protocol field. The reader already materialises this alias
regardless of engine, so the change makes the declared metadata agree with what
is actually transferred; previously the two sides disagreed and the transfer
plan failed.