Add scripts/merge_lora.py: bake Wan LoRAs into fp16 before quantization (4-step distill on int4) - #48
Open
kayadibi1 wants to merge 1 commit into
Open
Add scripts/merge_lora.py: bake Wan LoRAs into fp16 before quantization (4-step distill on int4)#48kayadibi1 wants to merge 1 commit into
kayadibi1 wants to merge 1 commit into
Conversation
…antization Applying LoRA at runtime on a quantized model dequantizes the int4 layers back to fp16, undoing quantization entirely (>21 GB swap on a 48 GB machine before the first step). Merging the LoRA into the fp16 base once, then quantizing the merged weights, keeps int4 inference self-contained — with the lightx2v 4-step distill LoRAs this is the difference between 32.5 min and 8.9 min (config-matched) per 81-frame Wan 2.2 I2V render on Apple Silicon. Streaming design: base header reused verbatim, tensors patched one at a time, each up@down delta computed at write time and freed (peak RSS 5 GB measured; materializing all rank-256 deltas up front would need 57 GB fp32). Covers lora_down/lora_up (alpha/rank scaling), diff and diff_b entries, with the full PyTorch-Sequential -> converter rename table (ffn.0/2 -> ffn.fc1/fc2, text/time_embedding.0/.2 -> _0/_1, time_projection.1 -> time_projection, patch_embedding -> patch_embedding_proj). Unmatched deltas are counted and reported rather than silently dropped. Validated: 1053/1053 deltas resolve with exact shapes on both Wan 2.2 I2V A14B noise experts; merged -> int4 (--bits 4 --group-size 64) renders verified end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kayadibi1
force-pushed
the
feat/wan-merge-lora
branch
from
August 2, 2026 17:25
a12ee07 to
db03341
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Running Wan 2.2 I2V with the lightx2v 4-step distill LoRAs is the difference between 32.5 min and 8.9 min per 81-frame render on Apple Silicon (measured, config-matched: 768x512, same seed/steps/LoRAs — ComfyUI Q8 GGUF on PyTorch MPS 1947.7 s vs MLX int4 534 s, M5 Pro 48 GB). But applying LoRA at runtime on a quantized model dequantizes the int4 layers back to fp16, undoing quantization entirely — on a 48 GB machine this blew swap past 21 GB before the first denoising step. The clean path is: merge the LoRA into the fp16 base once, then quantize the merged weights. There's currently no tool for that step.
What this adds
scripts/merge_lora.py— self-contained (numpy + stdlib), streaming:up @ downdelta computed at write time and freed. Peak RSS measured: 5 GB. Materializing all rank-256 deltas up front would need 57 GB fp32.lora_down/lora_up(withalpha/rankscaling),diff(norms),diff_b(biases).Sequentialflattening (ffn.0/2 → ffn.fc1/fc2,text/time_embedding.0/.2 → _0/_1,time_projection.1 → time_projection,patch_embedding → patch_embedding_proj). This matters: a partial table "succeeds" while silently dropping deltas — an early version of this script dropped all 160 FFN deltas (the model's largest matrices) that way. Unmatched deltas are therefore counted and reported (unused_deltas=N), never silent.Validation
convert --quantize-only --bits 4 --group-size 64→ 81-frame I2V renders verified (frame 0 reproduces the input image, identity preserved through motion, no runtime LoRA path executes).Happy to move/rename the script or add a doc entry if you'd like it somewhere else.
🤖 Generated with Claude Code