From 141422cdf4c9bba5b3bddc17ebab0f68c0729f57 Mon Sep 17 00:00:00 2001 From: Zhongkai Fu Date: Sun, 16 Aug 2026 21:47:51 -0700 Subject: [PATCH 1/2] optimize wan performance --- FEATURES.md | 2 +- InferenceWeb.Tests/WanVideoOracleTests.cs | 19 +- InferenceWeb.Tests/WanVideoTests.cs | 97 ++++++++++ TensorSharp.Cli/CliUsage.cs | 6 + TensorSharp.Cli/Program.cs | 9 +- TensorSharp.GGML.Native/ggml_ops_wan.cpp | 45 ++++- .../Models/WanVideo/WanVaeBase.cs | 63 +++++-- .../Models/WanVideo/WanVideoParams.cs | 42 +++++ .../Models/WanVideo/WanVideoPipeline.cs | 171 +++++++++++++++++- .../ProtocolAdapters/WebUiAdapter.cs | 26 ++- .../RequestParsers/WanVideoParamsParser.cs | 3 + TensorSharp.Server/wwwroot/index.html | 29 ++- USAGE.md | 6 + docs/models/wan.md | 123 ++++++++++++- 14 files changed, 596 insertions(+), 45 deletions(-) diff --git a/FEATURES.md b/FEATURES.md index 860e4dc1..fb61805b 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -23,7 +23,7 @@ - **Streaming** -- token-by-token output via SSE (web) or stdout (console), with abort/stop support for in-flight generations - **Text-diffusion generation** -- DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive `Forward()`. The CLI exposes `--diffusion-steps`, `--diffusion-seed`, and `--diffusion-blocks`; the Web UI streams whole-message `replace` events for live denoising previews and batches concurrent diffusion requests through `DiffusionBatchScheduler`. - **Image editing (Qwen-Image-Edit)** -- a prompt plus an input image produces an edited image. The loaded `qwen_image` GGUF is the MMDiT diffusion transformer; TensorSharp resolves two companion GGUFs alongside it — the Qwen-Image VAE (image ↔ 16-channel latent) and the Qwen2.5-VL-7B text encoder (prompt → 3584-dim conditioning, optional vision grounding via an `mmproj`). The pipeline VAE-encodes the reference, builds text (and optional image) conditioning, runs a FlowMatch-Euler true-CFG denoise loop with reference-latent concatenation, then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured (`TSGgml_QwenImageForward`), flash-attention is on by default, and the target area is auto-clamped to the device VRAM budget. An optional Lightning distillation LoRA (`--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`, `.safetensors`) merges into the DiT weights at load time, cutting the denoise to the LoRA's step count (e.g. 4 or 8) and switching CFG to 1.0 (no negative pass). Driven from C# via `QwenImageModel.EditImage(prompt, RgbImage, QwenImageParams)`, from the CLI image-edit mode (`--image`, `--prompt`, `--cfg`, `--diffusion-steps`, `--diffusion-seed`), and from the Web UI with live denoising previews. → [Qwen-Image-Edit card](docs/models/qwenimage.md) -- **Video generation (Wan 2.1 text-to-video, Wan 2.2 text/image-to-video)** -- a prompt (plus an optional first-frame image on the Wan 2.2 models) produces an H.264 MP4. The loaded `wan` GGUF is the Wan DiT — Wan 2.1 T2V, Wan 2.2 TI2V-5B (48-channel 16×16×4 latent, 24 fps) and Wan 2.2 A14B (two 14B experts switched at a timestep boundary, second GGUF auto-resolved) are auto-detected; TensorSharp resolves the companions alongside it — the UMT5-XXL text encoder GGUF (prompt → 512×4096 conditioning, exact unigram-Viterbi SentencePiece tokenization) and the matching causal 3D video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). The FlowMatch CFG denoise (UniPC or Euler) runs the whole DiT (self-attention with 3D RoPE + flash attention, cross-attention, AdaLN time modulation — per-token-timestep for TI2V image-to-video) as ONE resident-weight ggml graph per step, CUDA-graph-captured per shape (`TSGgml_WanDitForward`); the video VAE decodes all temporal chunks in one graph with the causal feature cache carried in-graph, convs on the banded im2col+GEMM path (`TSGgml_WanVaeDecode`), and image-to-video conditioning encodes the first frame through the causal VAE encoder in one graph (`TSGgml_WanVaeEncode`). Each stage releases its VRAM before the next, so TI2V-5B 81-frame 480p image-to-video and both A14B Q4_K_M experts fit a 16 GB GPU. Generation runs on every backend: the GGML paths (`ggml_cuda`, `ggml_vulkan`, `ggml_cpu`) share the whole-graph kernels, while `--backend cuda` and `--backend cpu` run a ggml-independent direct implementation (`WanDirect*`: resident-quantized linears on TensorSharp's MMQ/dp4a/cuBLAS routing with streaming online-softmax attention kernels on CUDA, parallel SIMD GEMM/attention on CPU, and a channels-last banded-im2col causal video VAE shared by both). Numerics verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decoders 59.9 dB / >35 dB PSNR) and across backends (final-latent cosine ≥ 0.999 on identical seeds). Driven from C# via `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`, the CLI (`--prompt`, `--image`, `--video-frames`, `--fps`, `--flow-shift`, `--negative-prompt`), the server API (`/v1/videos/generations` with base64 `image`, `/api/video-generate[/stream]` with `imagePath`), and the Web UI chat (type a prompt — with an attached image for image-to-video — and get the video with live progress). → [Wan card](docs/models/wan.md) +- **Video generation (Wan 2.1 text-to-video, Wan 2.2 text/image-to-video)** -- a prompt (plus an optional first-frame image on the Wan 2.2 models) produces an H.264 MP4. The loaded `wan` GGUF is the Wan DiT — Wan 2.1 T2V, Wan 2.2 TI2V-5B (48-channel 16×16×4 latent, 24 fps) and Wan 2.2 A14B (two 14B experts switched at a timestep boundary, second GGUF auto-resolved) are auto-detected; TensorSharp resolves the companions alongside it — the UMT5-XXL text encoder GGUF (prompt → 512×4096 conditioning, exact unigram-Viterbi SentencePiece tokenization) and the matching causal 3D video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). The FlowMatch CFG denoise (UniPC or Euler) runs the whole DiT (self-attention with 3D RoPE + flash attention over F16 keys/values, cross-attention, AdaLN time modulation — per-token-timestep for TI2V image-to-video) as ONE resident-weight ggml graph per step, CUDA-graph-captured per shape (`TSGgml_WanDitForward`); the video VAE decodes all temporal chunks in one graph with the causal feature cache carried in-graph, convs on the banded im2col+GEMM path (`TSGgml_WanVaeDecode`), and image-to-video conditioning encodes the first frame through the causal VAE encoder in one graph (`TSGgml_WanVaeEncode`). Each stage releases its VRAM before the next, so TI2V-5B 81-frame 480p image-to-video and both A14B Q4_K_M experts fit a 16 GB GPU. Generation runs on every backend: the GGML paths (`ggml_cuda`, `ggml_vulkan`, `ggml_cpu`) share the whole-graph kernels, while `--backend cuda` and `--backend cpu` run a ggml-independent direct implementation (`WanDirect*`: resident-quantized linears on TensorSharp's MMQ/dp4a/cuBLAS routing with streaming online-softmax attention kernels on CUDA, parallel SIMD GEMM/attention on CPU, and a channels-last banded-im2col causal video VAE shared by both). Numerics verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decoders 59.9 dB / >35 dB PSNR) and across backends (final-latent cosine ≥ 0.999 on identical seeds). Driven from C# via `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`, the CLI (`--prompt`, `--image`, `--video-frames`, `--fps`, `--flow-shift`, `--negative-prompt`), the server API (`/v1/videos/generations` with base64 `image`, `/api/video-generate[/stream]` with `imagePath`), and the Web UI chat (type a prompt — with an attached image for image-to-video — and get the video with live progress: per-pass timings, a running ETA and a 30 s heartbeat, since one pass over a 5-second 720p latent is minutes of GPU work). → [Wan card](docs/models/wan.md) - **Hybrid SSM-Transformer** -- Nemotron-H mixes Mamba2 SSM layers, attention-only layers, and MoE FFN layers in a single model. The Mamba2 step has both a per-sequence native kernel and a batched native kernel (`TSGgml_NemotronMamba2BatchedStepF32`, NEON SIMD + GCD parallelism) used by the batched path. On GGML backends the attention layers decode through the device-side flash-attention kernel against the resident KV cache (`TS_NEMOTRON_FLASH_DECODE=0` restores the host path), so decode no longer degrades with context length. - **Hybrid Attention-Recurrent** -- Qwen 3.5/3.6-family models mix full-attention layers with GatedDeltaNet recurrent layers; the batched path keeps recurrent running state in a per-slot recurrent-state pool - **Mixture of Experts** -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (`qwen35moe` / `qwen3next` variants such as Qwen3.5-35B-A3B), and Nemotron-H MoE FFN layers diff --git a/InferenceWeb.Tests/WanVideoOracleTests.cs b/InferenceWeb.Tests/WanVideoOracleTests.cs index 258ba1cf..0bbe7389 100644 --- a/InferenceWeb.Tests/WanVideoOracleTests.cs +++ b/InferenceWeb.Tests/WanVideoOracleTests.cs @@ -17,15 +17,25 @@ using TensorSharp.Models.WanVideo; using TensorSharp.Runtime; using Xunit; +using Xunit.Abstractions; namespace InferenceWeb.Tests { public class WanVideoOracleTests { - private const string FixtureDir = @"C:\Works\models\wan\fixtures"; - private const string Vae22Path = @"C:\Works\models\wan\VAE\Wan2.2_VAE.safetensors"; - private const string Vae21Path = @"C:\Works\models\wan\wan_2.1_vae.safetensors"; - private const string Ti2vGguf = @"C:\Works\models\wan\Wan2.2-TI2V-5B-Q8_0.gguf"; + private readonly ITestOutputHelper _output; + public WanVideoOracleTests(ITestOutputHelper output) { _output = output; } + + // TS_WAN_MODEL_DIR points the whole oracle suite at a local Wan checkout + // (the fixtures live in /fixtures); without it the historical Windows + // paths apply, so an existing checkout keeps running unchanged and the + // tests still no-op wherever the files are absent. + private static readonly string ModelDir = + Environment.GetEnvironmentVariable("TS_WAN_MODEL_DIR") ?? @"C:\Works\models\wan"; + private static readonly string FixtureDir = Path.Combine(ModelDir, "fixtures"); + private static readonly string Vae22Path = Path.Combine(ModelDir, "VAE", "Wan2.2_VAE.safetensors"); + private static readonly string Vae21Path = Path.Combine(ModelDir, "wan_2.1_vae.safetensors"); + private static readonly string Ti2vGguf = Path.Combine(ModelDir, "Wan2.2-TI2V-5B-Q8_0.gguf"); private static float[] ReadF32(string name) { @@ -229,6 +239,7 @@ public void Dit5bMatchesDiffusersUniformAndMasked() var vmLat = new float[xLat.Length]; WanVideoPipeline.Unpatchify(vm, vmLat, T, Hh, Ww, 48, dit.OutTok); double cosM = Cosine(vmLat, ReadF32("dit_v_masked.bin")); + _output.WriteLine($"[wan-oracle] TI2V-5B DiT cosine vs diffusers: uniform={cosU:F6} masked={cosM:F6}"); Assert.True(cosM > 0.995, $"TI2V-5B DiT (masked t) cosine vs diffusers = {cosM:F6} (want > 0.995)"); } } diff --git a/InferenceWeb.Tests/WanVideoTests.cs b/InferenceWeb.Tests/WanVideoTests.cs index 0273e514..ce27340b 100644 --- a/InferenceWeb.Tests/WanVideoTests.cs +++ b/InferenceWeb.Tests/WanVideoTests.cs @@ -150,6 +150,103 @@ public void SnapDimRoundsToSixteen(int input, int expected) public void SnapFramesRoundsToVaeTemporalGrid(int input, int expected) => Assert.Equal(expected, WanVideoPipeline.SnapFrames(input)); + // ---- VAE decode band layout ------------------------------------------- + + [Theory] + // (latent rows, plane width px, spatial scale) — TI2V-5B (16x) and Wan 2.1 (8x) + [InlineData(52, 1088, 16)] // 1088x832, the 121-frame 720p-class I2V shape + [InlineData(44, 1280, 16)] // 1280x704, the official TI2V 720p recipe + [InlineData(80, 1920, 16)] + [InlineData(120, 1664, 8)] + public void VaeBandsCoverThePlaneWithinTheMemoryBudget(int lh, int w, int scale) + { + const long threshold = 640_000; + var starts = WanVaeBase.PlanBands(lh, w, scale, threshold, out int bandLat); + Assert.NotNull(starts); + Assert.True(starts.Count >= 2); + + // every band fits the per-band pixel budget + Assert.True((long)w * bandLat * scale <= threshold, + $"band {w}x{bandLat * scale} exceeds the {threshold} px budget"); + // the plane is fully covered, first band at the top, last flush with lh + Assert.Equal(0, starts[0]); + Assert.Equal(lh - bandLat, starts[^1]); + for (int i = 1; i < starts.Count; i++) + { + Assert.True(starts[i] > starts[i - 1], "band starts must advance"); + int overlap = starts[i - 1] + bandLat - starts[i]; + Assert.True(overlap >= WanVaeBase.OverlapLat, + $"seam {i} overlaps {overlap} rows, want >= {WanVaeBase.OverlapLat}"); + } + } + + [Fact] + public void VaeBandsDecodeFewerRowsThanTheFixedHeightWalk() + { + // 1088x832 (lh 52): the old fixed-height walk used a 24-row band at + // starts 0/16/28 — 72 rows of work for 52 rows of output. + const long threshold = 640_000; + var starts = WanVaeBase.PlanBands(52, 1088, 16, threshold, out int bandLat); + Assert.Equal(2, starts.Count); + Assert.Equal(30, bandLat); + Assert.Equal(60, starts.Count * bandLat); // was 72 + } + + [Fact] + public void VaeSkipsTilingWhenThePlaneFitsWhole() + { + // 832x480 on the TI2V VAE is 30 latent rows — under the budget, one graph. + Assert.Null(WanVaeBase.PlanBands(30, 832, 16, 640_000, out int bandLat)); + Assert.Equal(30, bandLat); + } + + // ---- guidance cache --------------------------------------------------- + + [Theory] + [InlineData(0)] + [InlineData(1)] + public void GuidanceCacheOffRunsBothPassesEveryStep(int stride) + { + for (int i = 0; i < 50; i++) + Assert.True(WanVideoPipeline.UsesUncondPass(i, 50, stride)); + } + + [Theory] + [InlineData(2, 50, 27)] // 3 warm-up + every 2nd of steps 3..49 (which includes the last) + [InlineData(3, 50, 20)] // 3 warm-up + 16 strided + the last + [InlineData(4, 50, 16)] + public void GuidanceCacheStrideSkipsTheExpectedUncondPasses(int stride, int steps, int expected) + { + int uncond = 0; + for (int i = 0; i < steps; i++) + if (WanVideoPipeline.UsesUncondPass(i, steps, stride)) uncond++; + Assert.Equal(expected, uncond); + } + + [Fact] + public void GuidanceCacheAlwaysRecomputesWarmupAndFinalStep() + { + const int steps = 50, stride = 4; + // The steps that decide structure (the first few) and the one that + // produces the final latent must never run on a stale guidance delta. + for (int i = 0; i < WanVideoPipeline.CfgCacheWarmup; i++) + Assert.True(WanVideoPipeline.UsesUncondPass(i, steps, stride)); + Assert.True(WanVideoPipeline.UsesUncondPass(steps - 1, steps, stride)); + // ... and something in the middle must actually be skipped, or the + // cache would be a no-op. + Assert.False(WanVideoPipeline.UsesUncondPass(WanVideoPipeline.CfgCacheWarmup + 1, steps, stride)); + } + + // ---- progress / ETA formatting ---------------------------------------- + + [Theory] + [InlineData(-1, "ETA unknown")] + [InlineData(43, "~43s")] + [InlineData(432, "~7m 12s")] + [InlineData(3840, "~1h 04m")] + public void FormatEtaIsHumanReadable(double seconds, string expected) + => Assert.Equal(expected, WanVideoPipeline.FormatEta(seconds)); + // ---- RoPE ------------------------------------------------------------- [Fact] diff --git a/TensorSharp.Cli/CliUsage.cs b/TensorSharp.Cli/CliUsage.cs index ee55e711..60142aa8 100644 --- a/TensorSharp.Cli/CliUsage.cs +++ b/TensorSharp.Cli/CliUsage.cs @@ -415,6 +415,12 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections = "Wan video generation: negative prompt for classifier-free guidance. Default: the official " + "Wan negative prompt.", "--negative-prompt \"static, blurry\""), + new OptionHelp("--cfg-cache-stride ", + "Wan guidance cache: run the unconditional CFG pass on one step in N and reuse the cached " + + "guidance direction in between (the first three steps and the last always recompute it). " + + "At 50 steps, 2 runs 77 of the 100 passes (1.30x faster) and 3 runs 70 (1.43x). This is an " + + "approximation — leave it off when matching a reference sample matters. Default: 0 (off).", + "--cfg-cache-stride 2"), new OptionHelp("--wan-vae ", "Wan video VAE (wan_2.1_vae.safetensors, or Wan2.2_VAE.safetensors for TI2V-5B). Default: " + "same-directory scan next to the DiT model, VAE/ subfolders included (TS_WAN_VAE env var " + diff --git a/TensorSharp.Cli/Program.cs b/TensorSharp.Cli/Program.cs index d9eba41b..12dfae86 100644 --- a/TensorSharp.Cli/Program.cs +++ b/TensorSharp.Cli/Program.cs @@ -214,6 +214,7 @@ static void MainCore(string[] args) int videoFps = 0; // 0 = model default (16) float flowShift = 0f; // 0 = auto (8.0 for 1.3B video; 3.0/5.0 otherwise) string videoSampler = null; // null = unipc (the official Wan sampler) + int cfgCacheStride = 0; // 0/1 = off: every step runs both CFG passes string negativePrompt = null; string wanVaePath = null; string wanTePath = null; @@ -246,6 +247,7 @@ static void MainCore(string[] args) case "--fps": videoFps = int.Parse(args[++i]); break; case "--flow-shift": flowShift = float.Parse(args[++i], CultureInfo.InvariantCulture); break; case "--sampler": videoSampler = args[++i]; break; + case "--cfg-cache-stride": cfgCacheStride = int.Parse(args[++i]); break; case "--negative-prompt": negativePrompt = args[++i]; break; case "--wan-vae": wanVaePath = args[++i]; break; case "--wan-te": wanTePath = args[++i]; break; @@ -578,14 +580,14 @@ static void MainCore(string[] args) Console.Error.WriteLine("Wan video generation requires --prompt \"\" (or --input prompt.txt). " + "Optionally --image first_frame.png (Wan 2.2 image-to-video), --output out.mp4, --width, " + "--height, --video-frames, --fps, --diffusion-steps, --cfg, --flow-shift, " + - "--negative-prompt, --diffusion-seed."); + "--negative-prompt, --diffusion-seed, --cfg-cache-stride."); return; } RunVideoGeneration(wanModel, prompt, outputFile ?? "wan_video.mp4", imageWidth, imageHeight, videoFrames, diffusionStepsSet ? diffusionSteps : 0, cfgScaleSet ? cfgScale : 0f, diffusionSeedSet ? diffusionSeed : -1, flowShift, videoFps, negativePrompt, - videoSampler, imagePath); + videoSampler, imagePath, cfgCacheStride); return; } @@ -1626,7 +1628,7 @@ static void RunImageEdit(TensorSharp.Models.QwenImage.QwenImageModel model, static void RunVideoGeneration(TensorSharp.Models.WanVideo.WanVideoModel model, string prompt, string outputPath, int width, int height, int frames, int steps, float cfgScale, int seed, float flowShift, int fps, string negativePrompt, - string sampler = null, string imagePath = null) + string sampler = null, string imagePath = null, int cfgCacheStride = 0) { Console.WriteLine(imagePath != null ? "=== Wan Image-to-Video ===" : "=== Wan Text-to-Video ==="); Console.WriteLine($" prompt : {prompt}"); @@ -1654,6 +1656,7 @@ static void RunVideoGeneration(TensorSharp.Models.WanVideo.WanVideoModel model, NegativePrompt = negativePrompt, Sampler = sampler, ImagePath = imagePath, + CfgCacheStride = cfgCacheStride, }; var sw = Stopwatch.StartNew(); var video = model.GenerateVideo(prompt, p); diff --git a/TensorSharp.GGML.Native/ggml_ops_wan.cpp b/TensorSharp.GGML.Native/ggml_ops_wan.cpp index 683820e4..4f161c24 100644 --- a/TensorSharp.GGML.Native/ggml_ops_wan.cpp +++ b/TensorSharp.GGML.Native/ggml_ops_wan.cpp @@ -337,6 +337,25 @@ inline bool wan_flash_enabled() return on; } +// Keys/values are handed to attention as F16. Every backend's flash-attention +// kernel is built around an F16 KV cache: ggml-metal instantiates the F32-KV +// kernel with simdgroup_float8x8 accumulators (FA_TYPES_F32) where the F16 one +// uses simdgroup_half8x8, and the F32 tiles also cost twice the bandwidth in a +// kernel that re-streams K and V once per 8-query threadgroup. Measured on an +// M5 Pro at the Wan 2.2 TI2V 720p/121-frame shape (seq 27404, 24 heads, head +// dim 128), one self-attention: 4872 ms F32 KV vs 2439 ms F16 KV — 2.0x, and +// 30 blocks of it is the bulk of a denoising step. Q stays F32 (the Metal +// kernel asserts it), and F16 K/V is what every reference implementation feeds +// its attention (PyTorch/diffusers run the whole DiT in bf16/fp16; +// stable-diffusion.cpp casts K/V to F16 before ggml_flash_attn_ext), so this +// costs no accuracy the reference pipelines do not already spend. +// TS_WAN_DIT_KV_F16=0 restores F32 keys/values. +inline bool wan_kv_f16_enabled() +{ + static const bool on = []{ const char* e = std::getenv("TS_WAN_DIT_KV_F16"); return e == nullptr || e[0] != '0'; }(); + return on; +} + // Attention over q [hd, n_q, heads], k/v [hd, n_kv, heads]. Wan DiT // self-attention is fully bidirectional, so its flash path must be unmasked. // ggml-cuda handles a non-aligned KV tail directly; padding it to 256 used to @@ -357,6 +376,8 @@ ggml_tensor* wan_attention(ggml_context* ctx, ggml_tensor* q, ggml_tensor* k, gg // Materialized reference path (backends without flash support; O(n_kv * n_q) // scores). The caller's k/v are already padded when a mask is given, and // soft_max_ext folds the scale and the (F16) additive mask in one op. + // k/v arrive F16 here (wan_heads_seq_kv); ggml_mul_mat takes an F16 src0 + // against an F32 src1, so this path needs no change for that. ggml_tensor* kq = ggml_mul_mat(ctx, k, q); // [n_kv, n_q, heads] ggml_tensor* m = mask != nullptr ? ggml_view_2d(ctx, mask, k->ne[1], n_q, mask->nb[1], 0) @@ -374,6 +395,22 @@ ggml_tensor* wan_heads_seq(ggml_context* ctx, ggml_tensor* x) return ggml_cont(ctx, ggml_permute(ctx, x, 0, 2, 1, 3)); } +// Same reshape for a key/value projection, landing in F16 when the KV cast is +// enabled. ggml_cpy into a pre-typed destination does the permute and the +// narrowing in ONE pass, so the F16 path also writes half the bytes the plain +// ggml_cont did — it is strictly cheaper than the F32 layout change it replaces. +ggml_tensor* wan_heads_seq_kv(ggml_context* ctx, ggml_tensor* x) +{ + if (!wan_kv_f16_enabled()) return wan_heads_seq(ctx, x); + ggml_tensor* p = ggml_permute(ctx, x, 0, 2, 1, 3); // [hd, seq, heads] + ggml_tensor* dst = ggml_new_tensor_3d(ctx, GGML_TYPE_F16, p->ne[0], p->ne[1], p->ne[2]); + ggml_tensor* cast = ggml_cpy(ctx, p, dst); + // A strided F32 -> F16 copy is supported on ggml-cpu / -metal / -cuda; keep the + // F32 layout change for any backend whose dup kernel rejects this combination + // rather than failing the whole graph. + return backend_supports_op(cast) ? cast : wan_heads_seq(ctx, x); +} + // --------------------------------------------------------------------------- // UMT5-XXL encoder graph // --------------------------------------------------------------------------- @@ -651,8 +688,8 @@ bool wan_dit_build_graph(ggml_context* ctx, const TSGgmlWanDitDesc* d, WanDitGra k3 = wan_rope(ctx, k3, g.cosIn, g.sinIn, hd, heads, seq); ggml_tensor* qa = wan_heads_seq(ctx, q3); // [hd, seq, heads] - ggml_tensor* ka = wan_heads_seq(ctx, k3); - ggml_tensor* va = wan_heads_seq(ctx, ggml_reshape_3d(ctx, v, hd, heads, seq)); + ggml_tensor* ka = wan_heads_seq_kv(ctx, k3); // F16 (see wan_kv_f16_enabled) + ggml_tensor* va = wan_heads_seq_kv(ctx, ggml_reshape_3d(ctx, v, hd, heads, seq)); ggml_tensor* attn = wan_attention(ctx, qa, ka, va, nullptr, dim, seq, scale); attn = wan_lin(ctx, b.sow, attn, b.sob); x = ggml_add(ctx, x, segGate(attn, eGateA, eGateAB)); @@ -664,8 +701,8 @@ bool wan_dit_build_graph(ggml_context* ctx, const TSGgmlWanDitDesc* d, WanDitGra ggml_tensor* xk = wan_rms(ctx, wan_lin(ctx, b.xkw, txt, b.xkb), b.xnk, eps); ggml_tensor* xv = wan_lin(ctx, b.xvw, txt, b.xvb); ggml_tensor* xqa = wan_heads_seq(ctx, ggml_reshape_3d(ctx, xq, hd, heads, seq)); - ggml_tensor* xka = wan_heads_seq(ctx, ggml_reshape_3d(ctx, xk, hd, heads, cl)); - ggml_tensor* xva = wan_heads_seq(ctx, ggml_reshape_3d(ctx, xv, hd, heads, cl)); + ggml_tensor* xka = wan_heads_seq_kv(ctx, ggml_reshape_3d(ctx, xk, hd, heads, cl)); + ggml_tensor* xva = wan_heads_seq_kv(ctx, ggml_reshape_3d(ctx, xv, hd, heads, cl)); // ctx_len is a multiple of the KV stride (512), so flash needs no mask here. ggml_tensor* xattn = wan_attention(ctx, xqa, xka, xva, nullptr, dim, seq, scale); x = ggml_add(ctx, x, wan_lin(ctx, b.xow, xattn, b.xob)); diff --git a/TensorSharp.Models/Models/WanVideo/WanVaeBase.cs b/TensorSharp.Models/Models/WanVideo/WanVaeBase.cs index 0e0a1a1a..42396803 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVaeBase.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVaeBase.cs @@ -83,16 +83,48 @@ protected void InitFromWeights(WanVaeWeights wts) public abstract void Dispose(); + /// Latent rows blended between adjacent decode bands. + internal const int OverlapLat = 8; + + /// + /// Horizontal band layout for a -row latent plane that is + /// px wide: the band starts, and the band height in + /// . Returns null when the plane fits the budget whole. + /// Every overlapped row is decoded twice, so the layout — not just the + /// per-band budget — decides how much redundant work tiling costs. Choosing the + /// band COUNT from the budget first and then splitting the plane evenly across it + /// beats walking a fixed band height and letting the last band land wherever the + /// stride puts it: 52 rows with a 24-row band gave starts 0/16/28 and decoded 72 + /// rows to produce 52, where two 30-row bands cover it in 60 with one seam + /// instead of two — and 30 rows still fit the same per-band budget. + /// + internal static System.Collections.Generic.List PlanBands( + int lh, int W, int spatialScale, long tilePixelThreshold, out int bandLat) + { + int maxBandLat = Math.Max(16, (int)(tilePixelThreshold / ((double)W * spatialScale))); + if (lh <= maxBandLat) { bandLat = lh; return null; } + int bands = Math.Max(2, (int)Math.Ceiling((lh - OverlapLat) / (double)(maxBandLat - OverlapLat))); + // n*bandLat >= lh + (n-1)*overlap keeps every seam at least OverlapLat wide. + bandLat = (int)Math.Ceiling((lh + (bands - 1) * (double)OverlapLat) / bands); + var starts = new System.Collections.Generic.List(bands); + for (int b = 0; b < bands; b++) + starts.Add((int)((long)b * (lh - bandLat) / (bands - 1))); + return starts; + } + /// /// Decode diffusion latents [ZDim, t, lh, lw] (planar c,t,h,w) into RGB frames. - /// Above ~0.5 MP the decode is spatially tiled into full-width horizontal bands - /// (each ~0.4 MP, the scale the whole-graph kernel is fast at) blended over an - /// 8-latent-row overlap — the diffusers AutoencoderKLWan enable_tiling approach. + /// Above the per-band pixel budget the decode is spatially tiled into full-width + /// horizontal bands (see ) blended over an 8-latent-row + /// overlap — the diffusers AutoencoderKLWan enable_tiling approach. /// A 720p plane's activation planes + cross-chunk causal caches otherwise hold /// ~12 GB device-resident, which pushes 16 GB WDDM cards into shared-memory /// paging (~20x slower). TS_WAN_VAE_TILE=0 disables tiling. /// - public RgbImage[] Decode(float[] latent, int t, int lh, int lw) + /// Optional band progress: (bandsDone, bandCount). A + /// full-resolution decode is minutes of work in a handful of native calls, + /// so the caller needs something to report while it runs. + public RgbImage[] Decode(float[] latent, int t, int lh, int lw, Action onBand = null) { int outT = 1 + (t - 1) * TemporalScale; int W = lw * SpatialScale, H = lh * SpatialScale; @@ -111,29 +143,22 @@ public RgbImage[] Decode(float[] latent, int t, int lh, int lw) } } - const int OverlapLat = 8; // latent rows blended between bands - // Band height targets ~2/3 of the tile threshold per band (so a plane - // just over the threshold still splits into at least two bands). - int bandLat = Math.Max(16, (int)Math.Round(TilePixelThreshold * 0.66 / ((double)W * SpatialScale))); - bool tile = Environment.GetEnvironmentVariable("TS_WAN_VAE_TILE") != "0" - && (long)W * H > TilePixelThreshold && lh > bandLat; + bool tile = Environment.GetEnvironmentVariable("TS_WAN_VAE_TILE") != "0"; + var starts = PlanBands(lh, W, SpatialScale, TilePixelThreshold, out int bandLat); + tile &= starts != null; var pixels = new float[(long)W * H * 3 * outT]; if (!tile) { + onBand?.Invoke(0, 1); DecodeNative(z, t, lh, lw, pixels); + onBand?.Invoke(1, 1); } else { - // Band starts: stride bandLat - OverlapLat, last band pinned to the end. - var starts = new System.Collections.Generic.List(); - for (int y = 0; ; y += bandLat - OverlapLat) - { - if (y + bandLat >= lh) { starts.Add(Math.Max(0, lh - bandLat)); break; } - starts.Add(y); - } Console.WriteLine($" [wan] VAE decode tiled into {starts.Count} horizontal bands " + - $"({W}x{bandLat * SpatialScale} px, {OverlapLat * SpatialScale} px blend)"); + $"({W}x{bandLat * SpatialScale} px, " + + $"{(bandLat - (starts[1] - starts[0])) * SpatialScale} px blend)"); var band = new float[(long)lw * bandLat * ZDim * t]; var bandPx = new float[(long)W * (bandLat * SpatialScale) * 3 * outT]; for (int bi = 0; bi < starts.Count; bi++) @@ -145,6 +170,7 @@ public RgbImage[] Decode(float[] latent, int t, int lh, int lw) Array.Copy(z, (((long)tt * ZDim + c) * lh + y0) * lw, band, ((long)tt * ZDim + c) * bandLat * lw, (long)bandLat * lw); + onBand?.Invoke(bi, starts.Count); DecodeNative(band, t, bandLat, lw, bandPx); // blend the band into the canvas: cross-fade the overlap rows int py0 = y0 * SpatialScale; @@ -173,6 +199,7 @@ public RgbImage[] Decode(float[] latent, int t, int lh, int lw) } } }); + onBand?.Invoke(bi + 1, starts.Count); } } diff --git a/TensorSharp.Models/Models/WanVideo/WanVideoParams.cs b/TensorSharp.Models/Models/WanVideo/WanVideoParams.cs index 2d1bb0a8..91aa5e2b 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVideoParams.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVideoParams.cs @@ -9,6 +9,26 @@ namespace TensorSharp.Models.WanVideo { + /// A progress report from a running Wan generation. + public sealed class WanProgress + { + /// "text-encode", "image-encode", "denoise", "vae-decode" or "done". + public string Phase { get; init; } + /// Completed denoising steps (0-based count of finished steps). + public int Step { get; init; } + /// Total denoising steps for this request. + public int TotalSteps { get; init; } + /// Human-readable detail, e.g. "cond pass" or "band 3/7". + public string Detail { get; init; } + /// Seconds since generation started. + public double ElapsedSeconds { get; init; } + /// Estimated seconds still to run, or -1 before the first pass finishes. + public double EtaSeconds { get; init; } = -1; + /// True when this report is a periodic "still working" tick rather + /// than a phase transition. + public bool Heartbeat { get; init; } + } + /// Sampling parameters for Wan video generation (text-to-video, or /// image-to-video on the Wan 2.2 models when a conditioning image is supplied). public sealed class WanVideoParams @@ -69,9 +89,31 @@ public sealed class WanVideoParams /// 1000-step timestep range. 0 = model default (0.9 for I2V, 0.875 for T2V). public float BoundaryRatio { get; set; } + /// + /// Classifier-free guidance cache stride. Every step normally costs TWO DiT + /// passes (conditional + unconditional). Writing the update as + /// v = v_cond + (cfg-1) * d with d = v_cond - v_uncond isolates + /// the guidance direction d, which changes far more slowly across the + /// schedule than v itself. With a stride of N the unconditional pass + /// runs on one step in N (plus a warm-up and the final step) and the cached + /// d covers the rest. At the 50-step TI2V recipe, stride 2 runs 77 of + /// the 100 passes (1.30x faster) and stride 3 runs 70 (1.43x). + /// 0 or 1 = off (default): every step runs both passes exactly as the + /// reference pipelines do. This is an approximation — leave it off when + /// matching a reference sample matters. + /// + public int CfgCacheStride { get; set; } + /// Per-step progress callback: (stepIndex, totalSteps). public Action OnStep { get; set; } + /// + /// Fine-grained progress. Fires on every phase transition AND on a periodic + /// heartbeat while a single DiT pass is running, so a caller can show that + /// work is advancing during the minutes one 720p/121-frame pass takes. + /// + public Action OnProgress { get; set; } + /// Resolve the conditioning image from whichever input was supplied. internal RgbImage ResolveImage() { diff --git a/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs b/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs index 92c9d683..fceb6c17 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs @@ -125,6 +125,11 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) var phase = Stopwatch.StartNew(); void Phase(string n) { Console.WriteLine($" [wan-timing] {n}: {phase.Elapsed.TotalMilliseconds:F0}ms"); phase.Restart(); } + // One DiT pass over a 720p/121-frame latent is minutes of uninterrupted + // GPU work inside a single native call. Without a heartbeat the console + // and the Web UI go silent for that whole time, which reads as a hang. + using var beat = new WanHeartbeat(p, total); + WanVariant variant = _model.Variant; bool ti2v = variant == WanVariant.TI2V; bool a14b = variant == WanVariant.A14B; @@ -207,12 +212,27 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) $"(480p: 832x480, 720p: 1280x704{(ti2v ? ", the TI2V-5B native recipe" : "")}); " + "expect soft, distorted results. Prefer generating at a supported resolution " + "(e.g. --width 480 --height 704 for portrait) and downscaling afterwards."); + // Guidance-delta cache (opt-in; see WanVideoParams.CfgCacheStride). + int cfgStride = useCfg ? Math.Max(0, p.CfgCacheStride) : 0; + int passes = steps; + if (useCfg) + for (int i = 0; i < steps; i++) + if (UsesUncondPass(i, steps, cfgStride)) passes++; Console.WriteLine($"Wan {family} {(i2v ? "I2V" : "T2V")}: {width}x{height}x{frames}f " + $"({lt}x{hLen}x{wLen} = {seq} tokens), {steps} steps, " + $"{(useEuler ? "euler" : "unipc")}, cfg {cfg}{(dualExpert ? $"/{cfg2}" : "")}, " + $"shift {shift}, seed {seed}"); + // Past a few thousand tokens the quadratic self-attention term dominates a + // pass, and a pass becomes minutes long — say so up front, because the + // request that produced it looks identical to a cheap one. + if (seq > 8000) + Console.WriteLine($" [wan] large request: {passes} DiT passes over {seq} tokens. Self-attention " + + $"costs O(tokens^2) and dominates at this size, so halving the frame count or " + + $"the frame area makes it ~4x cheaper. A per-pass timing and an ETA follow the " + + $"first pass; progress ticks every 30s (TS_WAN_HEARTBEAT_S)."); // ---- 1. text conditioning (both prompts), then free the TE's VRAM ---- + beat.Set("text-encode", 0, steps, "UMT5-XXL"); float[] ctxCond = Te.Encode(prompt); float[] ctxNeg = useCfg ? Te.Encode(negPrompt) : null; // The UMT5 weights (several GB resident) are not needed again this @@ -228,6 +248,7 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) int inTok = ti2v || !i2v ? zTok : (zc + 4 + zc) * WanDiT.PatchH * WanDiT.PatchW; if (i2v) { + beat.Set("image-encode", 0, steps, "VAE encode"); var resized = ImageIO.Resize(image, width, height); if (ti2v) { @@ -299,6 +320,12 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) } bool usedLowExpert = false; + int passCount = 0, cfgCacheHits = 0; + double passSeconds = 0; + float[] dCache = cfgStride > 1 ? new float[(long)zTok * seq] : null; + if (dCache != null) + Console.WriteLine($" [wan] guidance cache: unconditional pass every {cfgStride} steps " + + $"after a {CfgCacheWarmup}-step warm-up (approximate; cfgCacheStride=1 disables)"); for (int i = 0; i < steps; i++) { float t = timesteps[i]; @@ -337,26 +364,64 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) xIn = xFull; } + var pass = Stopwatch.StartNew(); + beat.Set("denoise", i + 1, steps, "cond pass"); float[] vCond = cur.Predict(xIn, ctxCond, t, rope, seq, seq0); + double condS = pass.Elapsed.TotalSeconds; + passCount++; + passSeconds += condS; if (i == 0 && !string.IsNullOrEmpty(dumpDir)) { DumpF32(System.IO.Path.Combine(dumpDir, "v0_cond.bin"), vCond); Console.WriteLine($" [wan-debug] ctx {Stat(ctxCond)} | x {Stat(xIn)} | v {Stat(vCond)}"); } float[] v = vCond; + double negS = 0; if (useCfg) { - float[] vNeg = cur.Predict(xIn, ctxNeg, t, rope, seq, seq0); - for (long j = 0; j < v.LongLength; j++) - v[j] = vNeg[j] + curCfg * (vCond[j] - vNeg[j]); + // v = v_cond + (cfg-1) * d, d = v_cond - v_uncond. With the guidance + // cache on, d is only recomputed every cfgStride steps (always over + // the warm-up and on the final step, where the result is most + // sensitive); in between the cached d is reused and the unconditional + // pass is skipped entirely. + bool computeNeg = dCache == null || UsesUncondPass(i, steps, cfgStride); + if (computeNeg) + { + pass.Restart(); + beat.Set("denoise", i + 1, steps, "uncond pass"); + float[] vNeg = cur.Predict(xIn, ctxNeg, t, rope, seq, seq0); + negS = pass.Elapsed.TotalSeconds; + passCount++; + passSeconds += negS; + if (dCache != null) + for (long j = 0; j < v.LongLength; j++) dCache[j] = vCond[j] - vNeg[j]; + for (long j = 0; j < v.LongLength; j++) + v[j] = vNeg[j] + curCfg * (vCond[j] - vNeg[j]); + } + else + { + cfgCacheHits++; + for (long j = 0; j < v.LongLength; j++) + v[j] = vCond[j] + (curCfg - 1f) * dCache[j]; + } } if (euler != null) euler.Step(x, v, i); else x = unipc.Step(x, v); p.OnStep?.Invoke(i + 1, steps); - Console.WriteLine($" step {i + 1}/{steps} t={t:F1} ({phase.Elapsed.TotalSeconds:F1}s)"); + + // ETA from the mean pass so far. The first pass carries the one-time + // weight upload and graph build, so it over-estimates slightly; the + // running mean corrects itself from step 2 on. + double eta = passCount > 0 ? passSeconds / passCount * (passes - passCount) : -1; + Console.WriteLine($" step {i + 1}/{steps} t={t:F1} ({phase.Elapsed.TotalSeconds:F1}s" + + (useCfg ? $": cond {condS:F1}s + uncond {negS:F1}s" : "") + + $") — {FormatEta(eta)} left"); + beat.Report("denoise", i + 1, steps, "step done", eta, heartbeat: false); phase.Restart(); } Phase("denoise-last-step"); + Console.WriteLine($" [wan] denoise: {passCount} DiT passes, {passSeconds / Math.Max(1, passCount):F1}s mean" + + (cfgCacheHits > 0 ? $" ({cfgCacheHits} unconditional passes served from the guidance cache)" : "")); // TI2V I2V: pin the conditioning frame into the final latent. if (ti2v && i2v) @@ -371,8 +436,12 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) DumpF32(System.IO.Path.Combine(dumpDir, "latent_final.bin"), latent); Console.WriteLine($" [wan-debug] final latent {Stat(latent)}"); } - RgbImage[] framesOut = Vae.Decode(latent, lt, lh, lw); + beat.Set("vae-decode", steps, steps, $"{frames} frames at {width}x{height}"); + RgbImage[] framesOut = Vae.Decode(latent, lt, lh, lw, (done, bands) => + beat.Set("vae-decode", steps, steps, + $"{frames} frames at {width}x{height}, band {Math.Min(done + 1, bands)}/{bands}")); Phase("vae-decode"); + beat.Report("done", steps, steps, "complete", 0, heartbeat: false); Console.WriteLine($" [wan-timing] total: {total.Elapsed.TotalSeconds:F1}s"); return new GeneratedVideo { Frames = framesOut, Fps = fps, Seed = seed }; @@ -407,6 +476,29 @@ private static void DumpF32(string path, float[] data) System.IO.File.WriteAllBytes(path, bytes); } + /// Steps whose guidance delta is recomputed rather than reused: the + /// first steps (where the frame's structure is + /// decided), the last step, and every -th step in + /// between. A stride of 0/1 means "every step" — the cache is off. + internal static bool UsesUncondPass(int step, int steps, int stride) + { + if (stride <= 1) return true; + if (step < CfgCacheWarmup || step == steps - 1) return true; + return (step - CfgCacheWarmup) % stride == 0; + } + + /// Leading steps that always run both passes when the guidance cache is on. + internal const int CfgCacheWarmup = 3; + + /// "1h 04m" / "7m 12s" / "43s"; "?" before the first pass finishes. + internal static string FormatEta(double seconds) + { + if (seconds < 0) return "ETA unknown"; + if (seconds >= 3600) return $"~{(int)(seconds / 3600)}h {(int)(seconds % 3600 / 60):00}m"; + if (seconds >= 60) return $"~{(int)(seconds / 60)}m {(int)(seconds % 60):00}s"; + return $"~{seconds:F0}s"; + } + private static string Stat(float[] a) { double mn = double.MaxValue, mx = double.MinValue, sum = 0, sumsq = 0; long nan = 0; @@ -488,6 +580,75 @@ public void Dispose() } } + /// + /// Periodic "still working" ticks for the phases that spend minutes inside one + /// blocking native call (a DiT pass, a VAE band). Ticks go to the console and, + /// when the caller supplied one, to — + /// the Web UI turns them into a live status line. TS_WAN_HEARTBEAT_S=0 silences + /// the ticks (the phase-transition reports still fire). + /// + internal sealed class WanHeartbeat : IDisposable + { + private readonly WanVideoParams _p; + private readonly Stopwatch _total; + private readonly System.Threading.Timer _timer; + private readonly object _lock = new(); + private string _phase = "starting", _detail = ""; + private int _step, _totalSteps; + private double _eta = -1; + private Stopwatch _since = Stopwatch.StartNew(); + + public WanHeartbeat(WanVideoParams p, Stopwatch total) + { + _p = p; + _total = total; + int seconds = 30; + string env = Environment.GetEnvironmentVariable("TS_WAN_HEARTBEAT_S"); + if (!string.IsNullOrEmpty(env) && int.TryParse(env, out int v)) seconds = v; + if (seconds <= 0) return; + var period = TimeSpan.FromSeconds(seconds); + _timer = new System.Threading.Timer(_ => Tick(), null, period, period); + } + + /// Enter a phase; the next ticks describe it. + public void Set(string phase, int step, int totalSteps, string detail) + { + lock (_lock) + { + _phase = phase; _step = step; _totalSteps = totalSteps; _detail = detail; + _since = Stopwatch.StartNew(); + } + } + + public void Report(string phase, int step, int totalSteps, string detail, double eta, bool heartbeat) + { + lock (_lock) { _eta = eta; } + _p?.OnProgress?.Invoke(new WanProgress + { + Phase = phase, Step = step, TotalSteps = totalSteps, Detail = detail, + ElapsedSeconds = _total.Elapsed.TotalSeconds, EtaSeconds = eta, Heartbeat = heartbeat, + }); + } + + private void Tick() + { + string phase, detail; int step, totalSteps; double inPhase, eta; + lock (_lock) + { + phase = _phase; detail = _detail; step = _step; totalSteps = _totalSteps; + inPhase = _since.Elapsed.TotalSeconds; eta = _eta; + } + Console.WriteLine($" [wan] …{phase}" + + (totalSteps > 0 && phase == "denoise" ? $" step {step}/{totalSteps}" : "") + + $" ({detail}) {inPhase:F0}s in this pass, " + + $"{_total.Elapsed.TotalSeconds:F0}s total" + + (eta > 0 ? $", {WanVideoPipeline.FormatEta(eta)} left" : "")); + Report(phase, step, totalSteps, detail, eta, heartbeat: true); + } + + public void Dispose() => _timer?.Dispose(); + } + /// Seeded standard-normal sampler (xoshiro-seeded Box-Muller). internal sealed class GaussianRng { diff --git a/TensorSharp.Server/ProtocolAdapters/WebUiAdapter.cs b/TensorSharp.Server/ProtocolAdapters/WebUiAdapter.cs index 1d198882..04f20cb9 100644 --- a/TensorSharp.Server/ProtocolAdapters/WebUiAdapter.cs +++ b/TensorSharp.Server/ProtocolAdapters/WebUiAdapter.cs @@ -623,6 +623,11 @@ private sealed class VideoFrame public string Codec; public double Seconds; public string Error; + // Live progress detail (see WanProgress): which phase is running, how long + // it has been running and the projected time left. A 720p/121-frame pass + // is minutes long, so without these the UI has nothing to show between steps. + public string Phase, Detail; + public double Elapsed, Eta = -1; } private TensorSharp.Models.WanVideo.WanVideoParams ParseVideoParams(JsonElement root, out string error) @@ -797,7 +802,10 @@ public async Task VideoGenerateStreamAsync(HttpContext ctx) logger.LogInformation(LogEventIds.UploadReceived, "Video generate (stream): prompt='{Prompt}' {W}x{H}x{F}", prompt, p.Width, p.Height, p.Frames); - var channel = Channel.CreateUnbounded(new UnboundedChannelOptions { SingleReader = true, SingleWriter = true }); + // SingleWriter is false: the heartbeat below fires OnProgress from a timer + // thread while the generation thread is blocked inside a DiT pass, so two + // threads publish into this channel. + var channel = Channel.CreateUnbounded(new UnboundedChannelOptions { SingleReader = true, SingleWriter = false }); var genTask = Task.Run(() => { @@ -811,6 +819,14 @@ public async Task VideoGenerateStreamAsync(HttpContext ctx) if (ct.IsCancellationRequested) throw new OperationCanceledException(ct); channel.Writer.TryWrite(new VideoFrame { Step = step, Total = total }); }; + // Heartbeats and phase transitions. These arrive from a timer + // thread mid-pass, so cancellation is only observed here — the + // OnStep hook above still owns aborting between steps. + p.OnProgress = prog => channel.Writer.TryWrite(new VideoFrame + { + Step = prog.Step, Total = prog.TotalSteps, Phase = prog.Phase, + Detail = prog.Detail, Elapsed = prog.ElapsedSeconds, Eta = prog.EtaSeconds, + }); var video = videoModel.GenerateVideo(prompt, p); string codec = TensorSharp.Models.WanVideo.VideoIO.SaveMp4(outPath, video.Frames, video.Fps); channel.Writer.TryWrite(new VideoFrame @@ -855,8 +871,12 @@ public async Task VideoGenerateStreamAsync(HttpContext ctx) } else { - await SseWriter.WriteEventAsync(ctx.Response, - new { videoGen = true, step = f.Step, total = f.Total }, ct); + await SseWriter.WriteEventAsync(ctx.Response, new + { + videoGen = true, step = f.Step, total = f.Total, + phase = f.Phase, detail = f.Detail, + elapsedSeconds = f.Elapsed, etaSeconds = f.Eta, + }, ct); } } } diff --git a/TensorSharp.Server/RequestParsers/WanVideoParamsParser.cs b/TensorSharp.Server/RequestParsers/WanVideoParamsParser.cs index 2844ad5b..cb030799 100644 --- a/TensorSharp.Server/RequestParsers/WanVideoParamsParser.cs +++ b/TensorSharp.Server/RequestParsers/WanVideoParamsParser.cs @@ -50,6 +50,9 @@ public static WanVideoParams Parse( p.NegativePrompt = np.GetString(); if (root.TryGetProperty("sampler", out var sm) && sm.ValueKind == JsonValueKind.String) p.Sampler = sm.GetString(); + // Opt-in guidance cache: run the unconditional pass on one step in N. + if (root.TryGetProperty("cfgCacheStride", out var cc) && cc.TryGetInt32(out int ccv)) + p.CfgCacheStride = ccv; // Conditioning image for Wan 2.2 image-to-video: either a previously uploaded // file ("imagePath", Web UI flow) or inline base64 ("image", API flow; a diff --git a/TensorSharp.Server/wwwroot/index.html b/TensorSharp.Server/wwwroot/index.html index 94f445e1..0f541a45 100644 --- a/TensorSharp.Server/wwwroot/index.html +++ b/TensorSharp.Server/wwwroot/index.html @@ -1281,6 +1281,14 @@

} } +// "1h 04m" / "7m 12s" / "43s"; '' when the value is missing or not yet known. +function fmtDuration(s) { + if (typeof s !== 'number' || !(s > 0)) return ''; + if (s >= 3600) return `${Math.floor(s / 3600)}h ${String(Math.floor((s % 3600) / 60)).padStart(2, '0')}m`; + if (s >= 60) return `${Math.floor(s / 60)}m ${String(Math.floor(s % 60)).padStart(2, '0')}s`; + return `${Math.round(s)}s`; +} + async function runVideoGenerate(userMsg, bubbleText, statsDiv, queueDiv) { if (queueDiv) queueDiv.style.display = 'none'; bubbleText.textContent = 'Starting video generation… (this takes a few minutes)'; @@ -1320,9 +1328,24 @@

try { data = JSON.parse(line.slice(6)); } catch { continue; } if (data.videoGen) { - if (data.total > 0) { - statsDiv.textContent = `Denoising… step ${data.step}/${data.total}`; - bubbleText.textContent = `Generating video… step ${data.step} of ${data.total} (then decoding frames)`; + // A single denoising pass at 720p/121 frames runs for minutes, so the + // server also sends heartbeats carrying the phase and a running ETA. + // Show those, otherwise the UI looks frozen between steps. + const eta = fmtDuration(data.etaSeconds); + const el = fmtDuration(data.elapsedSeconds); + if (data.phase && data.phase !== 'denoise') { + const label = data.phase === 'text-encode' ? 'Encoding prompt' + : data.phase === 'image-encode' ? 'Encoding the input image' + : data.phase === 'vae-decode' ? 'Decoding frames' + : data.phase === 'done' ? 'Finishing' : data.phase; + statsDiv.textContent = `${label}… ${el}`; + bubbleText.textContent = `${label}${data.detail ? ' (' + data.detail + ')' : ''}…`; + } else if (data.total > 0) { + statsDiv.textContent = `Denoising… step ${data.step}/${data.total}` + (eta ? ` · ${eta} left` : ''); + bubbleText.textContent = `Generating video… step ${data.step} of ${data.total}` + + (data.detail ? ` — ${data.detail}` : '') + + (eta ? ` · about ${eta} left` : '') + + (el ? ` · ${el} elapsed` : ''); } } else if (data.done) { if (data.error) { diff --git a/USAGE.md b/USAGE.md index cd2eecec..8c1ed079 100644 --- a/USAGE.md +++ b/USAGE.md @@ -275,6 +275,7 @@ quietly. Measured on gemma-4-26B-A4B (`--cpu-moe`, peak VRAM): `ggml_cuda` | `--flow-shift ` | Wan FlowMatch timestep shift (default: the model's official recipe — 5.0 for Wan 2.2, 12.0 for A14B T2V, 8.0/3.0/5.0 for Wan 2.1). | | `--sampler ` | Wan sampler: `unipc` (official Wan sampler, default) or `euler`. | | `--negative-prompt ` | Wan negative prompt (default: the official Wan negative prompt). | +| `--cfg-cache-stride ` | Wan guidance cache: run the unconditional CFG pass on one step in `N` and reuse the cached guidance direction in between (default off — every step runs both passes). `2` ≈ 1.30x faster, `3` ≈ 1.43x; approximate, so leave it off when matching a reference sample matters. | | `--wan-vae ` | Override the resolved Wan video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). Env: `TS_WAN_VAE`. | | `--wan-te ` | Override the resolved UMT5-XXL text-encoder GGUF. Env: `TS_WAN_TE`. Wan 2.2 A14B additionally resolves the second high/low-noise expert automatically (env: `TS_WAN_DIT2`). | | `--tp ` | Tensor parallelism degree — split the model across N GPUs in a single process (default: `1`). Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. See [Tensor Parallelism & Distributed Inference](#tensor-parallelism--distributed-inference). | @@ -375,6 +376,11 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/Wan2.2-TI2V-5B.gguf --backend ggml_cuda \ --video-frames 121 --fps 24 +# 121 frames at the TI2V-5B native area is 27k DiT tokens, and self-attention is +# quadratic in that, so a full 50-step run is hours on a laptop-class GPU. The +# server logs a per-pass timing plus a running ETA and heartbeats every 30 s, and +# the Web UI shows both; see docs/models/wan.md for the cost table and the knobs. + # Configure server-wide default sampling parameters # (used whenever a request does not override the value itself) dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal \ diff --git a/docs/models/wan.md b/docs/models/wan.md index 49096ae2..d5b5d8fe 100644 --- a/docs/models/wan.md +++ b/docs/models/wan.md @@ -100,6 +100,13 @@ TensorSharp.Cli --model HighNoise/Wan2.2-I2V-A14B-HighNoise-Q4_K_M.gguf \ cfg 3.5 (both experts), shift 5.0, 40 steps; A14B T2V cfg 4.0/3.0, shift 12.0; Wan 2.1 cfg 6.0, shift 8.0 (1.3B video) or 3.0/5.0, 30 steps, 16 fps. - `--negative-prompt` defaults to the official Wan negative prompt. +- `--cfg-cache-stride N` (default off) — approximate speedup. A guided step + is `v = v_cond + (cfg-1)·d` with `d = v_cond - v_uncond`; the guidance + direction `d` changes much more slowly across the schedule than `v` does, so + the unconditional pass can run on one step in `N` and the cached `d` cover the + rest. At 50 steps, `2` runs 77 of the 100 passes (1.30× faster) and `3` runs + 70 (1.43×). The first three steps and the last always recompute `d`. Leave it + off when matching a reference sample matters. Server: `"cfgCacheStride": 2`. - MP4 writing prefers `ffmpeg` on `PATH` (or `TS_FFMPEG=`, or an `ffmpeg` folder next to the executable) — near-lossless CRF 17 H.264. Without it the OS codec via OpenCV is used at its default bitrate, which @@ -207,10 +214,13 @@ encoders match `AutoencoderKLWan` at cosine > 0.999, the Wan 2.2 VAE decode at HuggingFace T5 ids exactly on English and CJK prompts. Environment knobs: `TS_WAN_DIT_CAPTURE=0` (disable the persistent captured DiT -graph), `TS_WAN_DIT_FLASH=0` (materialized attention), `TS_WAN_VAE_GEMM_MAX_MB` -(im2col budget), `TS_WAN_VAE`/`TS_WAN_TE`/`TS_WAN_DIT2` (companion paths), -`TS_FFMPEG` (ffmpeg path for MP4 export), `TS_WAN_DIT_TRACE=` -(per-stage activation stats for debugging). +graph), `TS_WAN_DIT_FLASH=0` (materialized attention), `TS_WAN_DIT_KV_F16=0` +(F32 attention keys/values — the old, ~2x slower default), `TS_WAN_HEARTBEAT_S` +(progress tick interval, default 30; `0` silences the ticks), +`TS_WAN_VAE_GEMM_MAX_MB` (im2col budget), +`TS_WAN_VAE`/`TS_WAN_TE`/`TS_WAN_DIT2` (companion paths), `TS_FFMPEG` (ffmpeg +path for MP4 export), `TS_WAN_DIT_TRACE=` (per-stage activation stats for +debugging). ## Performance @@ -227,6 +237,111 @@ The TI2V-5B model's 16×16 spatial compression gives it ~2.7× fewer DiT tokens than Wan 2.1 at the same resolution — it is both the fastest and the highest-quality option for consumer GPUs, and the only 720p-24fps one. +### Long sequences: cost is quadratic in the token count + +Token count is `latent_frames × (h/2) × (w/2)`, and DiT self-attention costs +`O(tokens²)`. Past a few thousand tokens attention — not the weight matmuls — +is where the time goes, so the frame count and the frame area both matter far +more than the step count: + +| Request (TI2V-5B) | Tokens | Attention work | +|---|---|---| +| 640×384×25f | 1 200 | 1× | +| 832×480×81f | 8 190 | 47× | +| 1088×832×121f (5 s at 24 fps, 720p class) | 27 404 | 520× | + +The full 5-second 720p recipe is a genuinely large job: 50 steps × 2 CFG passes += 100 DiT passes over 27 k tokens. The pipeline prints the token count, a +per-pass timing and a running ETA, and heartbeats every 30 s while a pass is in +flight, so a long run is visibly progressing rather than apparently hung: + +``` +Wan 2.2-TI2V I2V: 1088x832x121f (31x26x34 = 27404 tokens), 50 steps, unipc, cfg 5, shift 5, seed ... + [wan] large request: 100 DiT passes over 27404 tokens. Self-attention costs O(tokens^2) ... + step 1/50 t=999.0 (249.0s: cond 124.6s + uncond 124.4s) — ~3h 24m left + [wan] …denoise step 2/50 (cond pass) 60s in this pass, 315s total, ~3h 24m left +``` + +To make such a request cheaper, in order of effect: + +1. **Fewer frames.** 121 → 61 frames roughly quarters the attention work; the + video is 2.5 s instead of 5 s at 24 fps. +2. **Smaller frame area**, but not below Wan's training resolutions — under + ~0.3 MP the model is out of distribution and the result gets *worse*, not + just cheaper. `--width 480 --height 704` is a good portrait target. +3. **Fewer steps.** 50 is the official TI2V recipe; 30 is visibly close and + 1.7× cheaper. +4. **`--cfg-cache-stride 2` or `3`** — 1.30× / 1.43× by reusing the guidance + direction between steps (approximate; see the CLI section). + +M5 Pro (20-core GPU, 48 GB unified), `ggml_metal`, Wan2.2-TI2V-5B Q8_0, +1088×832×121f = 27 404 tokens, image-to-video, the official 50-step recipe: + +| Stage | Before | Now | +|---|---|---| +| text encode (UMT5-XXL, both prompts) | 1.9 s | 1.9 s | +| image encode (VAE) | 5.2 s | 5.2 s | +| denoise, per step (2 CFG passes) | 412.3 s | **249.0 s** | +| VAE decode, 121 frames | 863 s (3 bands × 24 latent rows) | **734 s** (2 × 30) | +| **50-step total** | ≈ 5 h 58 m | **≈ 3 h 40 m** (1.63×) | +| 50-step total, `--cfg-cache-stride 3` | — | ≈ 2 h 38 m (2.27×) | + +Two changes account for that. + +**F16 attention keys and values.** Every backend's flash-attention kernel is +built around an F16 KV cache: ggml-metal instantiates the F32 variant with +`simdgroup_float8x8` accumulators where the F16 one uses `simdgroup_half8x8`, +and the F32 tiles also cost twice the bandwidth in a kernel that re-streams K +and V once per 8-query threadgroup. The permute into flash-attention layout and +the narrowing happen in one `ggml_cpy`, so the F16 path also writes half the +bytes the old `ggml_cont` did. `TS_WAN_DIT_KV_F16=0` restores F32. + +**VAE decode band layout.** Tiling picks the band *count* from the memory budget +and then splits the plane evenly, instead of walking a fixed band height and +letting the last band land wherever the stride puts it. At 52 latent rows that +was three 24-row bands — 72 rows decoded to produce 52 — and is now two 30-row +bands (60 rows, one seam instead of two) within the same per-band budget: +863 s → 734 s, the ratio the row counts predict. + +#### Alternatives measured and rejected + +One self-attention at this shape (seq 27 404, 24 heads, head dim 128), same run: + +| KV type | time | vs F32 | +|---|---|---| +| F32 (before) | 4993 ms | 1.00× | +| **F16 (now)** | **2467 ms** | **2.02×** | +| Q8_0 | 2652 ms | 1.88× | +| Q4_0 | 2395 ms | 2.08× | + +Q8_0 keys/values are *slower* than F16 — the dequantization outweighs the +bandwidth saved — and Q4_0 buys 3% for a real precision cost, so F16 is the +sweet spot rather than a compromise. Materialized chunked attention +(`mul_mat` + `soft_max_ext`, which unlike flash attention *can* use the Metal 4 +tensor API) came in at 1.08×: at this size attention is bound by score traffic, +not by the GEMM rate. Dequantizing the DiT weights from Q8_0 to F16 moved the +matmuls by <5% (25.2 → 26.4 ms for an attention projection). The Metal 4 tensor +API stays off for TI2V-5B: it makes the weight matmuls 2.9× faster (~22 s per +pass) but forces the VAE onto the direct-conv path, which at 121 frames costs +hours — see `ApplyArchitectureNativeTunables`. + +#### Quality + +F16 keys and values change the DiT's output by less than the sampler's own +sensitivity to floating-point reassociation: + +- One DiT forward against the diffusers `WanTransformer3DModel` reference + (`WanVideoOracleTests`): cosine **0.999964** with F16 K/V and **0.999964** + with F32 K/V — the per-pass accuracy is unchanged. +- Full 25-frame generations from an identical seed: F16 vs F32 K/V differ by + 39.08 dB mean PSNR. The control — F32 K/V with flash attention *off*, i.e. the + same arithmetic in a different summation order — differs by 39.40 dB. The two + are the same magnitude, so the pixel difference is an 8-step diffusion + trajectory diverging from a rounding difference, not a loss of accuracy. + +The speedup only appears where attention dominates: at 2 310 tokens +(480×704×25f) F16 and F32 K/V both run 5.0 s per pass. + Wan 2.1 vs stable-diffusion.cpp (master-769, identical GGUFs/settings, 33-frame 480p): sampling is near parity (281 s vs 258 s) but sd.cpp's VAE decode materializes ≈8 GB of 3D im2col and oversubscribes a 16 GB card into From dba1266da5c63c9f727243db6d0af02589de9730 Mon Sep 17 00:00:00 2001 From: Zhongkai Fu Date: Tue, 18 Aug 2026 08:41:42 -0700 Subject: [PATCH 2/2] optimize wan --- DEVELOPMENT.md | 2 +- FEATURES.md | 12 +- FEATURES_zh-cn.md | 13 +- InferenceWeb.Tests/WanVideoTests.cs | 74 ++++ MODEL_DOWNLOADS.md | 70 +++- MODEL_DOWNLOADS_zh-cn.md | 97 ++++- README.md | 47 ++- README_zh-cn.md | 47 ++- TensorSharp.GGML.Native/CMakeLists.txt | 25 +- TensorSharp.GGML.Native/build-linux.sh | 24 +- TensorSharp.GGML.Native/ggml_ops_core.cpp | 168 ++++++++ TensorSharp.GGML.Native/ggml_ops_internal.h | 8 + .../ggml_ops_qwen_image.cpp | 15 +- TensorSharp.GGML.Native/ggml_ops_wan.cpp | 51 ++- .../tsg_cuda_cudnn_conv.cu | 237 ++++++++++++ TensorSharp.GGML.Native/tsg_metal_mps_conv.mm | 240 ++++++++++++ TensorSharp.Models/ModelBase.cs | 13 +- TensorSharp.Models/Models/WanVideo/WanVae.cs | 28 ++ .../Models/WanVideo/WanVideoModel.cs | 33 ++ .../Models/WanVideo/WanVideoPipeline.cs | 47 ++- TensorSharp.Models/TensorSharp.Models.csproj | 1 + USAGE.md | 179 ++++++++- USAGE_zh-cn.md | 173 ++++++++- benchmarks/WanVideoBench/Program.cs | 237 ++++++++++++ benchmarks/WanVideoBench/WanVideoBench.csproj | 16 + config/qwen-image-edit-2511.json | 12 +- docs/model_cards.md | 25 +- docs/model_cards_zh-cn.md | 25 +- docs/models/README.md | 70 ++-- docs/models/README_zh-cn.md | 70 ++-- docs/models/wan.md | 217 +++++++++-- docs/models/wan_zh-cn.md | 358 +++++++++++++++++- docs_models_README_zh-cn.md | 0 docs_models_wan_zh-cn.md | 0 website/assets/app.js | 13 +- website/assets/search-index-zh.js | 24 +- website/assets/search-index.js | 25 +- website/assets/style.css | 5 + website/backends.html | 2 +- website/backends_zh-cn.html | 2 +- website/benchmarks.html | 55 +++ website/benchmarks_zh-cn.html | 55 +++ website/cli.html | 37 +- website/cli_zh-cn.html | 37 +- website/features.html | 15 +- website/features_zh-cn.html | 15 +- website/getting-started.html | 11 +- website/getting-started_zh-cn.html | 11 +- website/glossary.html | 8 +- website/glossary_zh-cn.html | 8 +- website/index.html | 6 +- website/index_zh-cn.html | 6 +- website/models-downloads.html | 65 ++++ website/models-downloads_zh-cn.html | 65 ++++ website/models-image.html | 61 +++ website/models-image_zh-cn.html | 61 +++ website/models-text.html | 175 +++++++++ website/models-text_zh-cn.html | 175 +++++++++ website/models-video.html | 122 ++++++ website/models-video_zh-cn.html | 122 ++++++ website/models.html | 279 ++------------ website/models_zh-cn.html | 279 ++------------ website/overview.html | 7 +- website/overview_zh-cn.html | 7 +- website/server.html | 37 +- website/server_zh-cn.html | 37 +- 66 files changed, 3663 insertions(+), 798 deletions(-) create mode 100644 TensorSharp.GGML.Native/tsg_cuda_cudnn_conv.cu create mode 100644 TensorSharp.GGML.Native/tsg_metal_mps_conv.mm create mode 100644 benchmarks/WanVideoBench/Program.cs create mode 100644 benchmarks/WanVideoBench/WanVideoBench.csproj create mode 100644 docs_models_README_zh-cn.md create mode 100644 docs_models_wan_zh-cn.md create mode 100644 website/models-downloads.html create mode 100644 website/models-downloads_zh-cn.html create mode 100644 website/models-image.html create mode 100644 website/models-image_zh-cn.html create mode 100644 website/models-text.html create mode 100644 website/models-text_zh-cn.html create mode 100644 website/models-video.html create mode 100644 website/models-video_zh-cn.html diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 514a9ade..c9f9d361 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -28,7 +28,7 @@ See Microsoft's [cross-platform .NET installation overview](https://learn.micros - **`git` and network access:** the GGML/CUDA native builds clone the ggml sources from [github.com/ggml-org/ggml](https://github.com/ggml-org/ggml) into `ExternalProjects/ggml/` on first build (see `eng/fetch-ggml.sh` / `eng/fetch-ggml.ps1`). The clone tracks ggml's default branch (`master`); pin a different ref with `TENSORSHARP_GGML_GIT_REF`, or set `TENSORSHARP_GGML_NO_UPDATE=1` to skip the network update once cloned (offline rebuilds) - **macOS (Metal backend):** CMake 3.20+ and the Xcode command-line tools for building the native GGML library — it embeds its Metal kernels as source and compiles them at run time, so it needs no Metal compiler at build time. The MLX backend additionally builds `libmlxc` from `TensorSharp.Backends.MLX/Native/` via `bash TensorSharp.Backends.MLX/build-native-macos.sh`, and that build *does* compile Metal shaders, so it needs a **full Xcode plus the Metal toolchain** — the command-line tools alone are not enough. `eng/ensure-metal-toolchain.sh` provisions this automatically on first build; see [Build the native MLX library](#build-the-native-mlx-library-macos-only) - **Windows (GGML CPU / CUDA backends):** CMake 3.20+ and Visual Studio 2022 or 2026 C++ build tools; for `ggml_cuda` or `cuda`, install an NVIDIA driver plus CUDA Toolkit 12.x or another compatible CUDA toolkit with cuBLAS. With Visual Studio 2026, whose MSVC 14.5x toolset is newer than current CUDA toolkits officially accept as a host compiler, the build passes `-allow-unsupported-compiler` to `nvcc` automatically; include the "C++ CMake tools for Windows" component so the build can use the Ninja generator (the Visual Studio generator additionally needs a CUDA toolkit that ships MSBuild integration for your VS version) -- **Linux (GGML CPU / CUDA backends):** CMake 3.20+; for `ggml_cuda` or `cuda`, install an NVIDIA driver plus CUDA Toolkit 12.x or another compatible CUDA toolkit with cuBLAS +- **Linux (GGML CPU / CUDA backends):** CMake 3.20+; for `ggml_cuda` or `cuda`, install an NVIDIA driver plus CUDA Toolkit 12.x or another compatible CUDA toolkit with cuBLAS. **cuDNN is optional**: when its headers and library are found (`libcudnn9-dev-cuda-12`, or any package providing `cudnn.h` + `libcudnn`), the Wan video VAE runs its convolutions through cuDNN instead of ggml's im2col+GEMM lowering, which is ~1.75x on the decode; without it the build succeeds unchanged and the VAE keeps the ggml path. The configure step prints which one applies - **Windows (GGML Vulkan backend):** enabled automatically when the machine has a Vulkan runtime (`System32\vulkan-1.dll`, shipped by every recent GPU driver). With a [LunarG Vulkan SDK](https://vulkan.lunarg.com/) installed it is used directly; without one the build auto-provisions a portable toolchain (Vulkan-Headers, a vulkan-1 import library generated from the system loader, glslc, SPIRV-Headers) into `ExternalProjects/vulkan-toolchain/` via `eng/fetch-vulkan-toolchain.ps1`. Opt out with `build-windows.ps1 --no-vulkan` or `TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF`. A GPU driver with Vulkan 1.3 support is required at runtime - **Linux (GGML Vulkan backend):** enabled automatically when a Vulkan loader (`libvulkan.so.1`) is installed. Distro dev packages are used when present (`apt install libvulkan-dev glslc spirv-headers`); otherwise the build auto-provisions the missing pieces (Vulkan-Headers, glslc from the shaderc CI prebuilts, SPIRV-Headers) into `ExternalProjects/vulkan-toolchain/` via `eng/fetch-vulkan-toolchain.sh`. Opt out with `build-linux.sh --no-vulkan` or `TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF` - GGUF model files (e.g., from [Hugging Face](https://huggingface.co)) diff --git a/FEATURES.md b/FEATURES.md index fb61805b..20f880e6 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -5,25 +5,26 @@ - **Multi-architecture support** -- DeepSeek V4 Flash, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit (image editing), and Wan 2.1/2.2 (text- and image-to-video) -- **Multimodal inference** -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5-family / Mistral 3 / Nemotron-H Omni +- **Multimodal inference** -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5/3.6-family / Mistral 3 / Muse-Glimmer / Nemotron-H Omni. Audio input is Gemma 4 only. `--pdf` is architecture-agnostic: a born-digital PDF's text layer is inlined into the prompt for any model, and only scanned PDFs fall back to page images (which then need a vision model) - **Thinking / reasoning mode** -- structured chain-of-thought output with `` / `<|channel>thought` / `<|channel>analysis` / `to=self` tags (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer, DeepSeek V4) - **Tool calling / function calling** -- models can invoke user-defined tools; multi-turn tool-call conversations supported across all three API styles - **Quantized model support** -- loads GGUF files with Q4_K_M, Q8_0, F16, MXFP4, and other quantization formats; performs native quantized matmul without dequantizing to FP32, including memory-efficient pure C# CPU loading for large GGUFs - **GPU-accelerated** -- GGML Metal on macOS, GGML CUDA on Windows/Linux with NVIDIA GPUs, GGML Vulkan on Windows/Linux with AMD/Intel/NVIDIA GPUs, a direct CUDA/cuBLAS backend with PTX kernels, and an MLX backend for Apple Silicon (mlx-c / Metal), all with CPU fallbacks for unsupported ops - **Optimized pure C# CPU backend** -- managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, fused activations, and other inference hot paths - **Continuous batching & paged KV cache** -- vLLM-style block-paged KV pool with block-hash prefix sharing across requests, iteration-level scheduler that admits / preempts sequences mid-batch, optional SSD-backed tier for very large KV working sets, and a native fused paged-attention kernel (`TSGgml_PagedAttentionForward`) that drives `ggml_flash_attn_ext` on Metal/CUDA/Vulkan. Enabled by default in `TensorSharp.Server`; opt-out with `--no-continuous-batching`. See [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md). -- **MTP / NextN speculative decoding** -- multi-token-prediction draft heads accelerate solo (non-concurrent) decode. Qwen 3.6 ships its NextN block fused into the trunk GGUF; Gemma 4 loads a separate EAGLE-style `gemma4-assistant` draft GGUF via `--mtp-draft-model` whose draft layers attend the target's own KV cache. The draft proposes up to `--mtp-draft` tokens per step (kept while draft confidence ≥ `--mtp-pmin`) and the trunk verifies them in a single batched forward; the request's own sampler — penalties included — drives both drafting and verification, so output is identical to standard decode. Opt in with the server's `--mtp-spec` flag (off by default; `TensorSharp.Cli` has no MTP flags — set the `TS_MTP_*` env vars there). On ggml backends fused multi-token-verify / draft-step kernels make it a clear win; the pure-C# `cuda` backend runs a fully GPU-resident per-op verify/draft and is also a win. CPU / MLX stay on standard decode. Env: `TS_MTP_*` (shared) and `TS_GMTP_*` (Gemma 4 tuning). +- **MTP / NextN speculative decoding** -- multi-token-prediction draft heads accelerate solo (non-concurrent) decode. Qwen 3.6 ships its NextN block fused into the trunk GGUF; Gemma 4 loads a separate EAGLE-style `gemma4-assistant` draft GGUF via `--mtp-draft-model` whose draft layers attend the target's own KV cache. The draft proposes up to `--mtp-draft` tokens per step (kept while draft confidence ≥ `--mtp-pmin`) and the trunk verifies them in a single batched forward; the request's own sampler — penalties included — drives both drafting and verification, so output is identical to standard decode. Opt in with the server's `--mtp-spec` flag (off by default; `TensorSharp.Cli` has no MTP flags — set the `TS_MTP_*` env vars there). On ggml backends fused multi-token-verify / draft-step kernels make it a clear win; the direct `cuda` backend runs a fully GPU-resident per-op verify/draft and is also a win. CPU / GGML CPU / MLX stay on standard decode. Env: `TS_MTP_*` (shared) and `TS_GMTP_*` (Gemma 4 tuning). - **Batched / parallel inference** -- `IBatchedPagedModel.ForwardBatch` implementations for Mistral 3, Gemma 4, GPT OSS, Qwen 3, Qwen 3.5/3.6-family, and Nemotron-H all run by default and pack N sequences into a single forward pass with paged K/V scatter and per-sequence attention via the native kernel. Gemma 4, Qwen 3.5/3.6, GPT OSS, and Nemotron-H expose a per-family `TS__BATCHED=0` escape hatch (`TS_GEMMA4_BATCHED=0`, `TS_QWEN35_BATCHED=0`, `TS_GPTOSS_BATCHED=0`, `TS_NEMOTRON_BATCHED=0`) to fall back to the per-sequence KV-swap path for A/B comparison or regression isolation; Qwen 3 and Mistral 3 have no per-family switch — use the global `TS_SCHED_DISABLE_BATCHED=1`. - **Tensor parallelism & distributed inference** -- split a model across multiple GPUs (Megatron-LM column/row-parallel pattern) with `--tp N` on both `TensorSharp.Cli` and `TensorSharp.Server` (or `TENSORSHARP_TP_DEGREE`), and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Hierarchical AllReduce minimizes inter-node traffic. Runs on the direct `cuda` backend and on the GGML CUDA / Vulkan backends, where each rank owns a ggml backend, weight shards, and KV cache on its own GPU. Supports all autoregressive architectures (Qwen 3, Mistral 3, Gemma 3/4, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Muse-Glimmer — `--tp 2` max there, 2 KV heads) with architecture-specific strategies for MoE expert parallelism / expert slicing, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication. Fused per-rank graphs make `--tp 2` decode faster than a single GPU (Gemma 4 E4B 51.7 vs 37.3 tok/s) and run models that do not fit one card. Optional Redis-backed KV cache and Responses API store for shared state. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference) - **Ollama & OpenAI API compatibility** -- drop-in replacement endpoints for existing tooling - **Configurable sampling** -- temperature, top-k, top-p, min-p, repetition/presence/frequency penalties, seed, stop sequences +- **Structured outputs** -- the OpenAI `response_format` JSON schema is compiled to a grammar and enforced by grammar-constrained decoding: any token that would break the schema is removed from the distribution before sampling, so the response is structurally valid by construction rather than repaired afterwards. Supported: `type`, `enum`, `const`, `properties`, `required`, `additionalProperties`, `items`, `prefixItems`, `min/maxItems`, `anyOf`, `oneOf`, `allOf`, `$ref`/`$defs` (recursive included), `min/maxLength`, `pattern`, the date/time/date-time/uuid formats, and integer `minimum`/`maximum`. Keywords a CFG cannot express (`not`, `if`/`then`/`else`, `dependentSchemas`, `dependentRequired`, `multipleOf`, `patternProperties`) are refused up front. `TS_JSON_GRAMMAR=0` falls back to prompt-and-repair. - **Chat templates** -- auto-loaded from GGUF metadata (Jinja2), with hardcoded fallbacks per architecture - **Inference engine** -- the new `InferenceEngine` (worker-thread scheduler + paged block pool) replaces the legacy single-request FIFO queue inside `TensorSharp.Server`. The old queue object is now a compatibility shim for status/event shapes; the engine itself handles concurrency. - **Batch processing** -- JSONL input support in the console application, plus a built-in inference benchmark for prefill/decode throughput - **Streaming** -- token-by-token output via SSE (web) or stdout (console), with abort/stop support for in-flight generations - **Text-diffusion generation** -- DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive `Forward()`. The CLI exposes `--diffusion-steps`, `--diffusion-seed`, and `--diffusion-blocks`; the Web UI streams whole-message `replace` events for live denoising previews and batches concurrent diffusion requests through `DiffusionBatchScheduler`. -- **Image editing (Qwen-Image-Edit)** -- a prompt plus an input image produces an edited image. The loaded `qwen_image` GGUF is the MMDiT diffusion transformer; TensorSharp resolves two companion GGUFs alongside it — the Qwen-Image VAE (image ↔ 16-channel latent) and the Qwen2.5-VL-7B text encoder (prompt → 3584-dim conditioning, optional vision grounding via an `mmproj`). The pipeline VAE-encodes the reference, builds text (and optional image) conditioning, runs a FlowMatch-Euler true-CFG denoise loop with reference-latent concatenation, then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured (`TSGgml_QwenImageForward`), flash-attention is on by default, and the target area is auto-clamped to the device VRAM budget. An optional Lightning distillation LoRA (`--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`, `.safetensors`) merges into the DiT weights at load time, cutting the denoise to the LoRA's step count (e.g. 4 or 8) and switching CFG to 1.0 (no negative pass). Driven from C# via `QwenImageModel.EditImage(prompt, RgbImage, QwenImageParams)`, from the CLI image-edit mode (`--image`, `--prompt`, `--cfg`, `--diffusion-steps`, `--diffusion-seed`), and from the Web UI with live denoising previews. → [Qwen-Image-Edit card](docs/models/qwenimage.md) -- **Video generation (Wan 2.1 text-to-video, Wan 2.2 text/image-to-video)** -- a prompt (plus an optional first-frame image on the Wan 2.2 models) produces an H.264 MP4. The loaded `wan` GGUF is the Wan DiT — Wan 2.1 T2V, Wan 2.2 TI2V-5B (48-channel 16×16×4 latent, 24 fps) and Wan 2.2 A14B (two 14B experts switched at a timestep boundary, second GGUF auto-resolved) are auto-detected; TensorSharp resolves the companions alongside it — the UMT5-XXL text encoder GGUF (prompt → 512×4096 conditioning, exact unigram-Viterbi SentencePiece tokenization) and the matching causal 3D video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). The FlowMatch CFG denoise (UniPC or Euler) runs the whole DiT (self-attention with 3D RoPE + flash attention over F16 keys/values, cross-attention, AdaLN time modulation — per-token-timestep for TI2V image-to-video) as ONE resident-weight ggml graph per step, CUDA-graph-captured per shape (`TSGgml_WanDitForward`); the video VAE decodes all temporal chunks in one graph with the causal feature cache carried in-graph, convs on the banded im2col+GEMM path (`TSGgml_WanVaeDecode`), and image-to-video conditioning encodes the first frame through the causal VAE encoder in one graph (`TSGgml_WanVaeEncode`). Each stage releases its VRAM before the next, so TI2V-5B 81-frame 480p image-to-video and both A14B Q4_K_M experts fit a 16 GB GPU. Generation runs on every backend: the GGML paths (`ggml_cuda`, `ggml_vulkan`, `ggml_cpu`) share the whole-graph kernels, while `--backend cuda` and `--backend cpu` run a ggml-independent direct implementation (`WanDirect*`: resident-quantized linears on TensorSharp's MMQ/dp4a/cuBLAS routing with streaming online-softmax attention kernels on CUDA, parallel SIMD GEMM/attention on CPU, and a channels-last banded-im2col causal video VAE shared by both). Numerics verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decoders 59.9 dB / >35 dB PSNR) and across backends (final-latent cosine ≥ 0.999 on identical seeds). Driven from C# via `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`, the CLI (`--prompt`, `--image`, `--video-frames`, `--fps`, `--flow-shift`, `--negative-prompt`), the server API (`/v1/videos/generations` with base64 `image`, `/api/video-generate[/stream]` with `imagePath`), and the Web UI chat (type a prompt — with an attached image for image-to-video — and get the video with live progress: per-pass timings, a running ETA and a 30 s heartbeat, since one pass over a 5-second 720p latent is minutes of GPU work). → [Wan card](docs/models/wan.md) +- **Image editing (Qwen-Image-Edit)** -- a prompt plus an input image produces an edited image. The loaded `qwen_image` GGUF is the MMDiT diffusion transformer; TensorSharp resolves two companion GGUFs alongside it — the Qwen-Image VAE (image ↔ 16-channel latent) and the Qwen2.5-VL-7B text encoder (prompt → 3584-dim conditioning, optional vision grounding via an `mmproj`). The pipeline VAE-encodes the reference, builds text (and optional image) conditioning, runs a FlowMatch-Euler true-CFG denoise loop with reference-latent concatenation, then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured (`TSGgml_QwenImageForward`), flash-attention is on by default, and the target area is auto-clamped to the device VRAM budget. An optional Lightning distillation LoRA (`--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`, `.safetensors`) cuts the denoise from the base 30 steps at CFG 2.5 to the LoRA's own step count (e.g. 4 or 8, parsed from its file name) at CFG 1.0 with no negative pass -- 60 DiT forwards become 4-8. It is applied as a runtime F32 side-path next to each targeted projection (`y = W_quant*x + b + (alpha/rank)*up*(down*x)`) with the quantized base weights left untouched, **not** merged into them: the Lightning deltas are ~1e-4 RMS, far below a Q2_K quantization step, and a measured merge changed the velocity by 24% relL2 of pure requantization noise. The side-path costs ~4% extra FLOPs, is CUDA-graph-capture-safe, and requires the whole-model or fused per-block CUDA forward -- on a path that cannot host it the model throws rather than emitting noise. The whole-step denoise caches (`TS_QWEN_DIT_CACHE_MODE`: `easycache` skips 40-55% of steps, `fbc` is First-Block-Cache) stay **off** by default because they measurably soften faces on edit workloads. Measured against stable-diffusion.cpp on the project's CUDA `image_edit` scenario (Q2_K DiT + 4-step Lightning LoRA, 544x1184, identical inputs and seed): 40.44 s vs 48.16 s warm. Driven from C# via `QwenImageModel.EditImage(prompt, RgbImage, QwenImageParams)`, from the CLI image-edit mode (`--image`, `--prompt`, `--cfg`, `--diffusion-steps`, `--diffusion-seed`), and from the Web UI with live denoising previews. → [Qwen-Image-Edit card](docs/models/qwenimage.md) +- **Video generation (Wan 2.1 text-to-video, Wan 2.2 text/image-to-video)** -- a prompt (plus an optional first-frame image on the Wan 2.2 models) produces an H.264 MP4. The loaded `wan` GGUF is the Wan DiT — Wan 2.1 T2V, Wan 2.2 TI2V-5B (48-channel 16×16×4 latent, 24 fps) and Wan 2.2 A14B (two 14B experts switched at a timestep boundary, second GGUF auto-resolved) are auto-detected; TensorSharp resolves the companions alongside it — the UMT5-XXL text encoder GGUF (prompt → 512×4096 conditioning, exact unigram-Viterbi SentencePiece tokenization) and the matching causal 3D video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). The FlowMatch CFG denoise (UniPC or Euler) runs the whole DiT (self-attention with 3D RoPE + flash attention over F16 keys/values, cross-attention, AdaLN time modulation — per-token-timestep for TI2V image-to-video) as ONE resident-weight ggml graph per step, CUDA-graph-captured per shape (`TSGgml_WanDitForward`); the video VAE decodes all temporal chunks in one graph with the causal feature cache carried in-graph (`TSGgml_WanVaeDecode`) -- convs go through MPSGraph on Metal (a 736x544x81f decode: 159 s -> 80 s, 1.99x, numerics unchanged at 93.9 dB PSNR; `TS_WAN_VAE_MPS_CONV=0` restores ggml's im2col+GEMM lowering) and through a banded im2col+GEMM path elsewhere, with the im2col budget and the tiling threshold now sized from free device memory instead of a fixed 16 GB card's budget, so large-memory devices decode a 720p plane whole (565 s vs 655 s banded, peak RSS 4.85 vs 5.37 GB) while small cards still tile; and image-to-video conditioning encodes the first frame through the causal VAE encoder in one graph (`TSGgml_WanVaeEncode`). Each stage releases its VRAM before the next, so TI2V-5B 81-frame 480p image-to-video and both A14B Q4_K_M experts fit a 16 GB GPU. Generation runs on every backend: the GGML paths (`ggml_cuda`, `ggml_vulkan`, `ggml_cpu`) share the whole-graph kernels, while `--backend cuda` and `--backend cpu` run a ggml-independent direct implementation (`WanDirect*`: resident-quantized linears on TensorSharp's MMQ/dp4a/cuBLAS routing with streaming online-softmax attention kernels on CUDA, parallel SIMD GEMM/attention on CPU, and a channels-last banded-im2col causal video VAE shared by both). **Step-distilled checkpoints are auto-detected from the DiT file name** (`Turbo`, `distill`, `Lightning`, `lightx2v`, `FastWan`, `-dmd`, or an explicit `…-4steps-…`) and are by far the biggest speed lever: the official 50-step x CFG recipe costs 100 DiT passes, a 4-step distilled checkpoint costs 4, and the pipeline switches to that step count with guidance off automatically (`--diffusion-steps` / `--cfg` override). Measured on an M5 Pro at 1088x832x121f = 27 404 tokens, `ggml_metal`, Wan2.2-TI2V-5B Q8_0: the base checkpoint runs 100 passes at 120.2 s for ~3 h 30 m end to end, and the identical request on a Turbo checkpoint runs 4 passes for **17 m 30 s** -- only the `--model` path differs. On base checkpoints `--cfg-cache-stride 2` / `3` reuses the guidance direction between steps for a further 1.30x / 1.43x. Numerics verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decoders 59.9 dB / >35 dB PSNR) and across backends (final-latent cosine ≥ 0.999 on identical seeds); the F16 attention keys/values that make the DiT 2.02x faster at 27 k tokens score the same 0.999964 DiT cosine as F32. Driven from C# via `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`, the CLI (`--prompt`, `--image`, `--video-frames`, `--fps`, `--flow-shift`, `--negative-prompt`), the server API (`/v1/videos/generations` with base64 `image`, `/api/video-generate[/stream]` with `imagePath`), and the Web UI chat (type a prompt — with an attached image for image-to-video — and get the video with live progress: per-pass timings, a running ETA and a 30 s heartbeat, since one pass over a 5-second 720p latent is minutes of GPU work). → [Wan card](docs/models/wan.md) - **Hybrid SSM-Transformer** -- Nemotron-H mixes Mamba2 SSM layers, attention-only layers, and MoE FFN layers in a single model. The Mamba2 step has both a per-sequence native kernel and a batched native kernel (`TSGgml_NemotronMamba2BatchedStepF32`, NEON SIMD + GCD parallelism) used by the batched path. On GGML backends the attention layers decode through the device-side flash-attention kernel against the resident KV cache (`TS_NEMOTRON_FLASH_DECODE=0` restores the host path), so decode no longer degrades with context length. - **Hybrid Attention-Recurrent** -- Qwen 3.5/3.6-family models mix full-attention layers with GatedDeltaNet recurrent layers; the batched path keeps recurrent running state in a per-slot recurrent-state pool - **Mixture of Experts** -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (`qwen35moe` / `qwen3next` variants such as Qwen3.5-35B-A3B), and Nemotron-H MoE FFN layers @@ -31,6 +32,7 @@ - **Batched GPU MoE** -- a single fused GGML graph dispatch handles all selected experts (plus the optional shared expert and residual add) for Qwen 3.5/3.6-family and Nemotron-H decode, eliminating per-expert round-trips - **Whole-model fused decode graphs** -- Gemma 4 (dense and MoE), Qwen 3.5/3.6 and GPT OSS run an entire decode token — every layer, the MoE router and experts, the final norm and the LM head — as ONE GGML graph dispatch instead of one submission per layer, so the GPU is never left waiting on the host between layers. On CUDA/Vulkan the graph is built once with stable tensor addresses and replayed (`ggml_set_rows` KV write with the row as an I64 input, a stride-padded attention window with an F16 mask input), which is what lets ggml-cuda capture it as a CUDA graph. GPT OSS decode: 24 → 154 tok/s on an A40, and flat in context length (133 tok/s at 16K) where the per-layer path collapsed to 2.3. Disable per model with `TS_GPTOSS_MODEL_DECODE=0` / `TS_GEMMA4_FD_PERSIST=0` / `TS_QWEN35_FD_PERSIST=0`. - **KV cache codecs** -- pluggable codec interface (`IKvBlockCodec`) with a built-in TurboQuant (2-bit affine / Q4 / Q8) compressed codec for paged blocks. The CLI accepts all four `--paged-kv-quant-bits 0|2|4|8` values; the server's legacy standalone flag accepts `0|4|8`, while `TS_KV_PAGED_QUANT_BITS=2` selects the 2-bit codec directly. The 2-bit tier reaches ~10x compression on fp32 blocks for very long contexts. +- **KV cache precision** -- `--kv-cache-dtype ` (CLI and server, env `KV_CACHE_DTYPE`; default auto — the backend/model pick) trades a small numerical drift for memory. `q4_0` (~0.56 bytes/element, ~1/7 of f32) is the most aggressive tier and is aimed at the very long (128K–256K) contexts where the KV cache dominates memory; the block-quantized tiers (`q8_0`/`q4_0`) require the native GGML flash path. - **Message editing** -- edit or delete previous messages in the web chat UI and regenerate from that point - **Text/Image/Audio/Video/PDF uploads** -- the web UI accepts file uploads up to 500 MB and preserves text content in full. Born-digital PDFs have their complete text layer extracted and inlined into the prompt (cap pages explicitly with `TS_PDF_MAX_PAGES`); scanned PDFs are rendered to page images for vision-capable models. The final prompt is checked against the model's actual context window instead of an arbitrary upload budget. The CLI accepts a PDF in one-shot mode via `--pdf ` - **Per-turn observability** -- structured logs capture the full user input and the full raw assistant output (both `` reasoning and the final result) plus the KV cache hit ratio. The same cache-hit stats are surfaced through every API: `prompt_cache_hit_tokens` / `prompt_cache_hit_ratio` (Ollama), `usage.prompt_tokens_details.cached_tokens` (OpenAI), and `promptTokens` / `kvReusedTokens` / `kvReusePercent` in the Web UI SSE `done` event @@ -83,7 +85,7 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-4-E4B- | Backend | Qwen 3.6 | Gemma 4 | |---|---|---| | GGML CUDA / GGML Metal | ✅ fused multi-token-verify + draft-step kernels | ✅ fused multi-token-verify + draft-step kernels | -| Direct CUDA (`cuda`, pure C#) | ✅ GPU-resident per-op verify/draft | ✅ GPU-resident per-op verify/draft | +| Direct CUDA (`cuda`, driver-API/cuBLAS) | ✅ GPU-resident per-op verify/draft | ✅ GPU-resident per-op verify/draft | | CPU / GGML CPU / MLX | standard decode (verify can't keep up) | standard decode | Tuning: `--mtp-draft` (default `8`) bounds tokens drafted per step; `--mtp-pmin` (default `0.75`) is the minimum draft-head confidence to keep a token (drafting stops at the first low-confidence token). Gemma 4 draft-path A/B switches are the `TS_GMTP_*` env vars (see the **MTP / speculative-decoding tunables** table under [Web Application](USAGE.md#web-application)). Per-architecture mechanics are in the [Qwen 3.5/3.6 card](docs/models/qwen35.md) and the [Gemma 4 card](docs/models/gemma4.md). diff --git a/FEATURES_zh-cn.md b/FEATURES_zh-cn.md index a9814241..d3383657 100644 --- a/FEATURES_zh-cn.md +++ b/FEATURES_zh-cn.md @@ -5,31 +5,34 @@ - **多架构支持** —— DeepSeek V4 Flash、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit（图像编辑），以及 Wan 2.1/2.2（文生/图生视频） -- **多模态推理** —— 图像、视频和音频输入（Gemma 4）；图像输入（Gemma 3 / Qwen 3.5-family / Mistral 3 / Nemotron-H Omni / Muse-Glimmer） +- **多模态推理** —— 图像、视频和音频输入（Gemma 4）；图像输入（Gemma 3 / Qwen 3.5/3.6-family / Mistral 3 / Muse-Glimmer / Nemotron-H Omni）。音频输入仅 Gemma 4 支持。`--pdf` 与架构无关：原生数字 PDF 的文本层会被内联进任意模型的提示词，只有扫描件才回退为页面图像（此时需要视觉模型） - **思维链 / 推理模式** —— 通过 `` / `<|channel>thought` / `<|channel>analysis` 标签输出结构化的思维链推理（Qwen 3、Qwen 3.5/3.6-family、Gemma 4、GPT OSS、Nemotron-H、Muse-Glimmer、DeepSeek V4） - **工具调用 / 函数调用** —— 模型可调用用户定义的工具；所有三种 API 风格均支持多轮工具调用对话 - **量化模型支持** —— 加载 Q4_K_M、Q8_0、F16、MXFP4 等量化格式的 GGUF 文件；执行原生量化矩阵乘法（matmul），无需反量化到 FP32，并且纯 C# CPU 后端在加载大型 GGUF 时也会保持量化权重压缩状态 - **GPU 加速** —— 通过 GGML 支持 Apple Metal（macOS）、GGML CUDA（Windows/Linux + NVIDIA）和 GGML Vulkan（Windows/Linux + AMD/Intel/NVIDIA），并提供 Direct CUDA/cuBLAS 后端（含 PTX 内核与未覆盖算子的 CPU 回退），以及面向 Apple Silicon 的 MLX 后端（mlx-c / Metal） - **优化后的纯 C# CPU 后端** —— 为 GEMM、RMSNorm、RoPE、softmax、融合激活等推理热点路径提供托管快速路径和 SIMD 内核 - **连续批处理 & 分页 KV 缓存** —— vLLM 风格的分页 KV 块池，跨请求的块级哈希前缀共享，迭代级调度器（可在批内动态加入/抢占序列），可选的 SSD 冷层用于超大 KV 工作集，原生融合分页注意力内核（`TSGgml_PagedAttentionForward`，在 Metal/CUDA/Vulkan 上驱动 `ggml_flash_attn_ext`）。`TensorSharp.Server` 默认启用，可用 `--no-continuous-batching` 关闭。详见 [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md)。 -- **MTP / NextN 投机解码** —— 多 token 预测草稿头加速单序列（无并发）decode。Qwen 3.6 将 NextN 块内嵌在主干 GGUF 中；Gemma 4 通过 `--mtp-draft-model` 加载独立的 EAGLE 风格 `gemma4-assistant` 草稿 GGUF，其草稿层读取目标模型自身的 KV 缓存。草稿头每步最多提议 `--mtp-draft` 个 token（草稿置信度 ≥ `--mtp-pmin` 时保留），主干用一次批量前向完成验证；起草与验证均由该请求自己的采样器（含惩罚项）驱动，因此输出与标准 decode 完全一致。服务端通过 `--mtp-spec` 启用（默认关闭）；CLI 没有 MTP 参数，需设置 `TS_MTP_*` 环境变量。ggml 后端有融合的多 token 验证 / 草稿步内核，是明确收益；纯 C# `cuda` 后端运行完全驻留 GPU 的逐算子验证 / 草稿，同样有收益；CPU / MLX 保持标准 decode。环境变量：`TS_MTP_*`（通用）与 `TS_GMTP_*`（Gemma 4 调优）。 +- **MTP / NextN 投机解码** —— 多 token 预测草稿头加速单序列（无并发）decode。Qwen 3.6 将 NextN 块内嵌在主干 GGUF 中；Gemma 4 通过 `--mtp-draft-model` 加载独立的 EAGLE 风格 `gemma4-assistant` 草稿 GGUF，其草稿层读取目标模型自身的 KV 缓存。草稿头每步最多提议 `--mtp-draft` 个 token（草稿置信度 ≥ `--mtp-pmin` 时保留），主干用一次批量前向完成验证；起草与验证均由该请求自己的采样器（含惩罚项）驱动，因此输出与标准 decode 完全一致。服务端通过 `--mtp-spec` 启用（默认关闭）；CLI 没有 MTP 参数，需设置 `TS_MTP_*` 环境变量。ggml 后端有融合的多 token 验证 / 草稿步内核，是明确收益；Direct `cuda` 后端运行完全驻留 GPU 的逐算子验证 / 草稿，同样有收益；CPU / GGML CPU / MLX 保持标准 decode。环境变量：`TS_MTP_*`（通用）与 `TS_GMTP_*`（Gemma 4 调优）。 - **张量并行与分布式推理** —— 用 `--tp N`（`TensorSharp.Cli` 与 `TensorSharp.Server` 均支持，也可用 `TENSORSHARP_TP_DEGREE`）把一个模型按 Megatron-LM 列/行并行范式切分到多张 GPU 上，再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。分层 AllReduce 把跨网络流量降到最低。可运行在 Direct `cuda` 后端以及 GGML CUDA / Vulkan 后端上——后者每个 rank 在自己的 GPU 上拥有独立的 ggml 后端、权重分片与 KV 缓存。支持全部自回归架构（Qwen 3、Mistral 3、Gemma 3/4、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Muse-Glimmer——因为只有 2 个 KV 头，并行度上限为 `--tp 2`），并针对 MoE 专家并行 / 专家切分、GatedDeltaNet 按 rank V-head 归属、Mamba2 复制等异构层提供各自的策略。融合的按 rank 计算图使 `--tp 2` 的 decode 快于单卡（Gemma 4 E4B 51.7 对 37.3 tok/s），也让单卡装不下的模型得以运行。服务端还可选用 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理) - **批处理 / 并行推理** —— 已为 Mistral 3、Gemma 4、GPT OSS、Qwen 3、Qwen 3.5/3.6-family、Nemotron-H 默认启用 `IBatchedPagedModel.ForwardBatch`，能在一次前向传播中打包 N 个序列，使用 `slotMapping` 进行分页 K/V 写入，并通过原生内核做按序列注意力。Gemma 4、Qwen 3.5/3.6、GPT OSS 与 Nemotron-H 提供各自的 `TS__BATCHED=0` 兜底开关；Qwen 3 与 Mistral 3 没有家族专属开关，请用全局 `TS_SCHED_DISABLE_BATCHED=1` 强制回到按序列 KV-swap 路径。 - **兼容 Ollama 与 OpenAI API** —— 可作为现有工具链的即插即用替代端点 - **可配置采样** —— temperature、top-k、top-p、min-p、重复/存在/频率惩罚、seed、停止序列 +- **结构化输出** —— OpenAI `response_format` 中的 JSON schema 会被编译成语法，并通过语法约束解码强制执行：任何会破坏 schema 的 token 在采样前就被从分布中剔除，因此返回值天然结构合法，而不是事后修补。支持 `type`、`enum`、`const`、`properties`、`required`、`additionalProperties`、`items`、`prefixItems`、`min/maxItems`、`anyOf`、`oneOf`、`allOf`、`$ref`/`$defs`（含递归）、`min/maxLength`、`pattern`，以及 date/time/date-time/uuid 格式与整数 `minimum`/`maximum`。CFG 无法表达的关键字（`not`、`if`/`then`/`else`、`dependentSchemas`、`dependentRequired`、`multipleOf`、`patternProperties`）会在请求阶段直接拒绝。`TS_JSON_GRAMMAR=0` 回退到旧的提示 + 修补行为。 - **聊天模板** —— 从 GGUF 元数据自动加载（Jinja2），并为不同架构提供硬编码回退模板 - **推理引擎** —— `TensorSharp.Server` 中的新 `InferenceEngine`（工作线程调度器 + 分页块池）取代了旧的单请求 FIFO 队列。旧队列对象现在只是状态 / 事件形状的兼容 shim；引擎本身已经处理并发。 - **批处理** —— 控制台应用支持 JSONL 输入，并内置用于测量 prefill / decode 吞吐的推理基准 - **流式输出** —— 按 token 输出（Web 通过 SSE，控制台通过 stdout），并支持中断/停止正在生成的请求 - **文本扩散生成** —— DiffusionGemma 使用 EntropyBound 迭代去噪采样器，而不是自回归 `Forward()`。CLI 提供 `--diffusion-steps`、`--diffusion-seed` 与 `--diffusion-blocks`；Web UI 使用整条消息 `replace` 事件展示实时去噪预览，并通过 `DiffusionBatchScheduler` 批处理并发扩散请求。 -- **图像编辑（Qwen-Image-Edit）** —— 提示词加输入图像生成编辑后的图像。所加载的 `qwen_image` GGUF 是 MMDiT 扩散 Transformer；TensorSharp 在其旁解析两个伴随 GGUF——Qwen-Image VAE（图像 ↔ 16 通道潜变量）与 Qwen2.5-VL-7B 文本编码器（提示词 → 3584 维条件，可选通过 `mmproj` 做视觉接地）。流水线对参考图做 VAE 编码、构建文本（及可选图像）条件、运行带参考潜变量拼接的 FlowMatch-Euler true-CFG 去噪循环，再 VAE 解码回像素。整个 60 块 DiT 前向被 CUDA 图捕获（`TSGgml_QwenImageForward`），flash 注意力默认开启，目标面积按设备 VRAM 预算自动钳制。可选的 Lightning 蒸馏 LoRA（`--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`，`.safetensors`）会在加载时合并进 DiT 权重，将去噪步数缩减为该 LoRA 的步数（例如 4 或 8），并把 CFG 切换为 1.0（无负向分支）。可从 C# 通过 `QwenImageModel.EditImage(prompt, RgbImage, QwenImageParams)` 驱动，从 CLI 图像编辑模式（`--image`、`--prompt`、`--cfg`、`--diffusion-steps`、`--diffusion-seed`）驱动，以及从带实时去噪预览的 Web UI 驱动。→ [Qwen-Image-Edit 卡片](docs/models/qwenimage_zh-cn.md) -- **视频生成（Wan 2.1 文生视频，Wan 2.2 文/图生视频）** —— 提示词（Wan 2.2 模型可再加一张首帧图片）生成 H.264 MP4 视频。所加载的 `wan` GGUF 是 Wan DiT —— 自动识别 Wan 2.1 T2V、Wan 2.2 TI2V-5B（48 通道 16×16×4 潜空间、24 fps）与 Wan 2.2 A14B（两个 14B 专家按时间步边界切换，第二个 GGUF 自动配对）；TensorSharp 在其旁解析伴随模型——UMT5-XXL 文本编码器 GGUF（提示词 → 512×4096 条件，精确的 unigram-Viterbi SentencePiece 分词）与对应的因果 3D 视频 VAE（`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`）。FlowMatch CFG 去噪（UniPC 或 Euler）每步将整个 DiT（带 3D RoPE + flash 注意力的自注意力、交叉注意力、AdaLN 时间调制——TI2V 图生视频为逐 token 时间步）作为单个常驻权重 ggml 图运行，按形状 CUDA 图捕获（`TSGgml_WanDitForward`）；视频 VAE 在单个图内解码全部时序块（`TSGgml_WanVaeDecode`），图生视频的首帧经因果 VAE 编码器单图编码（`TSGgml_WanVaeEncode`）。各阶段在进入下一阶段前释放各自 VRAM，因此 TI2V-5B 81 帧 480p 图生视频与两个 A14B Q4_K_M 专家均可在 16 GB GPU 上运行。数值已对照 diffusers 验证（DiT 余弦 > 0.995，VAE 编码器 > 0.999，解码器 59.9 dB / >35 dB PSNR）。可从 C# 通过 `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`、CLI（`--prompt`、`--image`、`--video-frames`、`--fps`、`--flow-shift`、`--negative-prompt`）、服务器 API（`/v1/videos/generations` 支持 base64 `image`，`/api/video-generate[/stream]` 支持 `imagePath`）以及 Web UI 聊天（输入提示词——附图即为图生视频——获得带实时进度的视频）驱动。→ [Wan 卡片](docs/models/wan.md) +- **图像编辑（Qwen-Image-Edit）** —— 提示词加输入图像生成编辑后的图像。所加载的 `qwen_image` GGUF 是 MMDiT 扩散 Transformer；TensorSharp 在其旁解析两个伴随 GGUF——Qwen-Image VAE（图像 ↔ 16 通道潜变量）与 Qwen2.5-VL-7B 文本编码器（提示词 → 3584 维条件，可选通过 `mmproj` 做视觉接地）。流水线对参考图做 VAE 编码、构建文本（及可选图像）条件、运行带参考潜变量拼接的 FlowMatch-Euler true-CFG 去噪循环，再 VAE 解码回像素。整个 60 块 DiT 前向被 CUDA 图捕获（`TSGgml_QwenImageForward`），flash 注意力默认开启，目标面积按设备 VRAM 预算自动钳制。可选的 Lightning 蒸馏 LoRA（`--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`，`.safetensors`）把默认的 30 步 / CFG 2.5（共 60 次 DiT 前向）降为该 LoRA 自带的步数（例如 4 或 8，从文件名解析）且 CFG 1.0、无负向分支。它以运行期 F32 旁路的形式接在每个目标投影旁（`y = W_quant*x + b + (alpha/rank)*up*(down*x)`），量化基权重原样保留，**不会**被合并：Lightning 的增量 RMS 约 1e-4，远低于一个 Q2_K 量化步长，实测合并会让速度场产生 24% 的 relL2 变化，而那全是重量化噪声。该旁路额外开销约 4% FLOPs，可安全参与 CUDA 图捕获，并且要求整模型或融合分块的 CUDA 前向路径——若落到无法承载旁路的路径上，模型会直接报错而不是输出噪声。整步去噪缓存（`TS_QWEN_DIT_CACHE_MODE`：`easycache` 可跳过 40–55% 的步骤，`fbc` 为 First-Block-Cache）默认**关闭**，因为在编辑类任务上会明显柔化人脸细节。在本项目 CUDA `image_edit` 场景下与 stable-diffusion.cpp 对比（Q2_K DiT + 4 步 Lightning LoRA、544x1184、相同输入与种子）：热启动 40.44 秒 对 48.16 秒。可从 C# 通过 `QwenImageModel.EditImage(prompt, RgbImage, QwenImageParams)` 驱动，从 CLI 图像编辑模式（`--image`、`--prompt`、`--cfg`、`--diffusion-steps`、`--diffusion-seed`）驱动，以及从带实时去噪预览的 Web UI 驱动。→ [Qwen-Image-Edit 卡片](docs/models/qwenimage_zh-cn.md) +- **视频生成（Wan 2.1 文生视频，Wan 2.2 文/图生视频）** —— 提示词（Wan 2.2 模型可再加一张首帧图片）生成 H.264 MP4 视频。所加载的 `wan` GGUF 是 Wan DiT —— 自动识别 Wan 2.1 T2V、Wan 2.2 TI2V-5B（48 通道 16×16×4 潜空间、24 fps）与 Wan 2.2 A14B（两个 14B 专家按时间步边界切换，第二个 GGUF 自动配对）；TensorSharp 在其旁解析伴随模型——UMT5-XXL 文本编码器 GGUF（提示词 → 512×4096 条件，精确的 unigram-Viterbi SentencePiece 分词）与对应的因果 3D 视频 VAE（`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`）。FlowMatch CFG 去噪（UniPC 或 Euler）每步将整个 DiT（带 3D RoPE + flash 注意力的自注意力、交叉注意力、AdaLN 时间调制——TI2V 图生视频为逐 token 时间步）作为单个常驻权重 ggml 图运行，按形状 CUDA 图捕获（`TSGgml_WanDitForward`）；视频 VAE 在单个图内解码全部时序块（`TSGgml_WanVaeDecode`）——Metal 上卷积走 MPSGraph（736x544x81f 的一次解码从 159 秒降到 80 秒，1.99×，数值不变，PSNR 93.9 dB；`TS_WAN_VAE_MPS_CONV=0` 可恢复 ggml 的 im2col+GEMM 下降路径），其他后端走带状 im2col+GEMM；im2col 预算与分块阈值现在按设备可用显存推导，而不再固定按 16 GB 显卡的预算，因此大显存设备可整幅解码 720p 平面（565 秒 / 峰值 RSS 4.85 GB，对比分成两带的 655 秒 / 5.37 GB），小显存设备仍然分块。图生视频的首帧经因果 VAE 编码器单图编码（`TSGgml_WanVaeEncode`）。各阶段在进入下一阶段前释放各自 VRAM，因此 TI2V-5B 81 帧 480p 图生视频与两个 A14B Q4_K_M 专家均可在 16 GB GPU 上运行。**步数蒸馏检查点会按 DiT 文件名自动识别**（`Turbo`、`distill`、`Lightning`、`lightx2v`、`FastWan`、`-dmd`，或显式的 `…-4steps-…`），这是最大的提速手段：官方 50 步 × CFG 配方需要 100 次 DiT 前向，而 4 步蒸馏检查点只需 4 次，管线会自动切换到该步数并关闭引导（`--diffusion-steps` / `--cfg` 可覆盖）。在 M5 Pro、`ggml_metal`、Wan2.2-TI2V-5B Q8_0、1088×832×121f = 27 404 token 上实测：基础检查点 100 次前向、每次 120.2 秒，端到端约 3 小时 30 分；同一请求换成 Turbo 检查点只需 4 次前向，端到端 **17 分 30 秒**——只有 `--model` 路径不同。基础检查点上还可用 `--cfg-cache-stride 2` / `3` 复用引导方向，再快 1.30× / 1.43×。数值已对照 diffusers 验证（DiT 余弦 > 0.995，VAE 编码器 > 0.999，解码器 59.9 dB / >35 dB PSNR）；让 DiT 在 27k token 下快 2.02× 的 F16 注意力键值，其 DiT 余弦与 F32 同为 0.999964。可从 C# 通过 `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`、CLI（`--prompt`、`--image`、`--video-frames`、`--fps`、`--flow-shift`、`--negative-prompt`）、服务器 API（`/v1/videos/generations` 支持 base64 `image`，`/api/video-generate[/stream]` 支持 `imagePath`）以及 Web UI 聊天（输入提示词——附图即为图生视频——获得带实时进度的视频）驱动。→ [Wan 卡片](docs/models/wan.md) - **混合 SSM-Transformer** —— Nemotron-H 在单个模型中混合 Mamba2 SSM 层、纯注意力层和 MoE FFN 层；Mamba2 步现在同时提供单序列原生内核与批处理原生内核（`TSGgml_NemotronMamba2BatchedStepF32`，NEON SIMD + GCD 并行）。在 GGML 后端上，注意力层直接用设备侧 flash-attention 内核对常驻 KV 缓存做 decode（`TS_NEMOTRON_FLASH_DECODE=0` 恢复主机路径），decode 速度不再随上下文长度衰减。 - **混合注意力-递归网络** —— Qwen 3.5/3.6-family 在同一模型中混合全注意力层与 GatedDeltaNet 递归层；批处理路径下递归运行状态保存在每槽位的递归状态池中 - **专家混合（MoE）** —— 支持 Gemma 4 MoE 变体（例如 gemma-4-26B-A4B）、GPT OSS MoE（例如 gpt-oss-20b）、Qwen 3.5/3.6-family MoE（`qwen35moe` / `qwen3next` 变体，例如 Qwen3.5-35B-A3B）以及 Nemotron-H MoE FFN 层 +- **MoE 专家 CPU 卸载** —— `--n-cpu-moe N` / `--cpu-moe`（对应 llama.cpp 的 `-ncmoe` / `-cmoe`，环境变量 `TS_N_CPU_MOE`）把前 N 层的路由专家权重留在系统内存并在主机侧相乘，注意力、各处 norm、router 与常驻共享专家仍留在加速器上。在所有具备整模型融合图的架构（Qwen 3.5/3.6、Gemma 4 MoE、GPT OSS、DiffusionGemma）上，被卸载的层仍留在同一张融合图内——加速器在每个被卸载层的 router 之后暂停，主机直接从 GGUF mmap 中取出被选中的专家做乘法，再把结果交回下一段，因此 decode 时每层只有约 8 KB 激活跨总线。它同样能与张量并行组合：`--tp N` 下这些接缝会并入各 rank 的 AllReduce 分段计划（Qwen3.5-35B-A3B `--tp 2`：两卡上 17.4 GB 常驻权重降到 3.2 GB；gemma-4-26B-A4B：12.9 GB 降到 2.4 GB，输出逐字节一致）。在 16 GB 的 RTX 3080 Laptop 上实测：Qwen3.6-35B-A3B `--cpu-moe` 后显存 13.4 → 4.6 GB；gemma-4-26B-A4B 16.1 → 4.8 GB（decode 39.7 → 17.7 tok/s，若只用 `--n-cpu-moe 8` 则为 38.6 tok/s 并让出 3 GB）；gpt-oss-20b 16.2 → 2.9 GB，从而避开 WDDM 溢出悬崖，`--n-cpu-moe 12` 时把 0.3 tok/s 变成 25.4。所有架构（含 DeepSeek V4 Flash）默认都是 0：装不下的模型会在加载时直接拒绝并给出所需的 `--n-cpu-moe N`，而不是悄悄牺牲 decode 吞吐。→ [MoE CPU 卸载（英文）](USAGE.md#mixture-of-experts-cpu-offload---n-cpu-moe) - **批量 GPU MoE** —— Qwen 3.5/3.6-family 与 Nemotron-H 在 decode 时通过单次融合的 GGML 计算图调度处理所有被选中的专家（Qwen 3.5-family 还包括可选的 shared expert 与残差加法），消除每个专家的 CPU-GPU 往返 - **整模型融合 decode 计算图** —— Gemma 4（dense 与 MoE）、Qwen 3.5/3.6 与 GPT OSS 把一个 decode token 的全部计算——每一层、MoE 路由与专家、最终 norm 与 LM head——作为**一次** GGML 计算图调度提交，而不是每层提交一次，GPU 因此不会在层与层之间空等主机。在 CUDA/Vulkan 上该图只构建一次、张量地址保持稳定后反复重放（KV 写入用 `ggml_set_rows`、行号作为 I64 输入，注意力窗口按 stride 补齐、掩码作为 F16 输入），这正是 ggml-cuda 能把它捕获成 CUDA 图的前提。GPT OSS decode 在 A40 上从 24 → 154 tok/s，且随上下文长度基本持平（16K 时仍有 133 tok/s，而逐层路径已跌到 2.3）。可按模型用 `TS_GPTOSS_MODEL_DECODE=0` / `TS_GEMMA4_FD_PERSIST=0` / `TS_QWEN35_FD_PERSIST=0` 关闭。 - **KV 缓存编解码器** —— 通过 `IKvBlockCodec` 接口插件化；内置 TurboQuant（2-bit 仿射 / Q4 / Q8）分页块压缩。CLI 的 `--paged-kv-quant-bits` 接受 `0|2|4|8`；服务端旧式独立分页参数接受 `0|4|8`，也可直接用 `TS_KV_PAGED_QUANT_BITS=2` 选择 2-bit 编解码器。2-bit 档位在 fp32 块上可达约 10 倍压缩，面向超长上下文。 +- **KV 缓存精度** —— `--kv-cache-dtype `（CLI 与服务端，环境变量 `KV_CACHE_DTYPE`；默认 auto，由后端 / 模型决定）用很小的数值漂移换内存。`q4_0`（约 0.56 字节/元素，约为 f32 的 1/7）是最激进的档位，面向 KV 缓存主导内存的超长上下文（128K–256K）；块量化档位（`q8_0`/`q4_0`）需要原生 GGML flash 路径。 - **消息编辑** —— 在 Web 聊天界面中编辑或删除历史消息，并从该位置重新生成回复 - **文本/图像/音频/视频/PDF 上传** —— Web 界面支持最大 500 MB 的文件上传并完整保留文本内容；原生数字 PDF 会完整提取文本层（可通过 `TS_PDF_MAX_PAGES` 显式限制页数）。最终提示词按模型的实际上下文窗口检查，而不是使用任意的上传预算 - **每轮可观测性** —— 结构化日志会完整保留用户输入与模型原始输出（包括 `` 思维链和最终结果），并记录 KV 缓存命中率。同样的命中率指标通过所有 API 透出：Ollama 的 `prompt_cache_hit_tokens` / `prompt_cache_hit_ratio`、OpenAI 的 `usage.prompt_tokens_details.cached_tokens`，以及 Web UI SSE `done` 事件中的 `promptTokens` / `kvReusedTokens` / `kvReusePercent` @@ -82,7 +85,7 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-4-E4B- | 后端 | Qwen 3.6 | Gemma 4 | |---|---|---| | GGML CUDA / GGML Metal | ✅ 融合多 token 验证 + 草稿步内核 | ✅ 融合多 token 验证 + 草稿步内核 | -| Direct CUDA（`cuda`，纯 C#） | ✅ 完全驻留 GPU 的逐算子验证 / 草稿 | ✅ 完全驻留 GPU 的逐算子验证 / 草稿 | +| Direct CUDA（`cuda`，Driver API / cuBLAS） | ✅ 完全驻留 GPU 的逐算子验证 / 草稿 | ✅ 完全驻留 GPU 的逐算子验证 / 草稿 | | CPU / GGML CPU / MLX | 标准 decode（验证跟不上） | 标准 decode | 调优：`--mtp-draft`（默认 `8`）限制每步起草的 token 数；`--mtp-pmin`（默认 `0.75`）是保留 token 所需的最低草稿置信度（遇到第一个低置信 token 即停止起草）。Gemma 4 草稿路径 A/B 开关为 `TS_GMTP_*` 环境变量（见 [Web 应用](USAGE_zh-cn.md#web-应用) 下的 **MTP / 投机解码调优变量** 表）。各架构具体机制见 [Qwen 3.5/3.6 卡片](docs/models/qwen35_zh-cn.md) 与 [Gemma 4 卡片](docs/models/gemma4_zh-cn.md)。 diff --git a/InferenceWeb.Tests/WanVideoTests.cs b/InferenceWeb.Tests/WanVideoTests.cs index ce27340b..13e1131e 100644 --- a/InferenceWeb.Tests/WanVideoTests.cs +++ b/InferenceWeb.Tests/WanVideoTests.cs @@ -150,6 +150,80 @@ public void SnapDimRoundsToSixteen(int input, int expected) public void SnapFramesRoundsToVaeTemporalGrid(int input, int expected) => Assert.Equal(expected, WanVideoPipeline.SnapFrames(input)); + // ---- degenerate-decode guard ------------------------------------------ + + private static TensorSharp.Models.QwenImage.RgbImage MakeFrame(int w, int h, Func pixel) + { + var chw = new float[3 * w * h]; + for (int i = 0; i < chw.Length; i++) chw[i] = pixel(i); + return TensorSharp.Models.QwenImage.RgbImage.FromPlanarChw(w, h, chw); + } + + [Fact] + public void FlatDecodeIsRejected() + { + // Every frame a single colour = the Metal tensor-API VAE corruption. + var frames = new[] { MakeFrame(8, 8, _ => 0f), MakeFrame(8, 8, _ => 0f), MakeFrame(8, 8, _ => 0f) }; + var ex = Assert.Throws( + () => WanVideoPipeline.AssertFramesAreNotDegenerate(frames)); + Assert.Contains("TS_WAN_METAL_TENSOR_API=0", ex.Message); + } + + [Fact] + public void NaNDecodeIsRejected() + { + var frames = new[] { MakeFrame(8, 8, _ => float.NaN), MakeFrame(8, 8, _ => float.NaN) }; + Assert.Throws( + () => WanVideoPipeline.AssertFramesAreNotDegenerate(frames)); + } + + [Fact] + public void RealDecodeIsAccepted() + { + var frames = new[] { MakeFrame(8, 8, i => (i % 97) / 97f), MakeFrame(8, 8, i => (i % 53) / 53f) }; + WanVideoPipeline.AssertFramesAreNotDegenerate(frames); // must not throw + } + + [Fact] + public void ASingleFlatFrameIsNotEnoughToReject() + { + // A legitimately flat frame (a fade to black at the end) must not fail + // the whole video — the guard only fires when every sampled frame is flat. + var frames = new[] + { + MakeFrame(8, 8, i => (i % 97) / 97f), MakeFrame(8, 8, i => (i % 53) / 53f), + MakeFrame(8, 8, i => (i % 31) / 31f), MakeFrame(8, 8, _ => 0f), + }; + WanVideoPipeline.AssertFramesAreNotDegenerate(frames); + } + + // ---- step-distilled checkpoint detection ------------------------------- + + [Theory] + // Published distilled Wan checkpoints / LoRAs. + [InlineData("Wan2_2-TI2V-5B-Turbo-Q8_0.gguf", 4)] + [InlineData("Wan22_TI2V_5B_Turbo_lora_rank_64_fp16.safetensors", 4)] + [InlineData("Wan2_2_5B_FastWanFullAttn_lora_rank_128_bf16.safetensors", 4)] + [InlineData("Wan2.2-I2V-A14B-HighNoise-Lightning-4steps-Q4_K_M.gguf", 4)] + [InlineData("wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step.safetensors", 4)] + [InlineData("Wan2.1-T2V-14B-StepDistill-CfgDistill-Lightx2v-8steps.gguf", 8)] + // Ordinary checkpoints must NOT be mistaken for distilled ones. + [InlineData("Wan2.2-TI2V-5B-Q8_0.gguf", 0)] + [InlineData("wan2.1-t2v-14b-Q4_K_M.gguf", 0)] + [InlineData("Wan2.1-T2V-1.3B-F16.gguf", 0)] + [InlineData("", 0)] + public void DistilledStepsParsedFromCheckpointName(string fileName, int expected) + => Assert.Equal(expected, WanVideoModel.ParseDistilledSteps(fileName)); + + [Fact] + public void DistilledStepCountIsBoundedToASaneRange() + { + // A "1.3B" or a Q4_K_M shard index must not be read as a step count, and + // an absurd count falls back to the marker default rather than trusting it. + Assert.Equal(0, WanVideoModel.ParseDistilledSteps("Wan2.1-T2V-1.3B-F16.gguf")); + Assert.Equal(4, WanVideoModel.ParseDistilledSteps("wan-turbo-9999steps.gguf")); + } + // ---- VAE decode band layout ------------------------------------------- [Theory] diff --git a/MODEL_DOWNLOADS.md b/MODEL_DOWNLOADS.md index 66898c61..c30cf2c7 100644 --- a/MODEL_DOWNLOADS.md +++ b/MODEL_DOWNLOADS.md @@ -4,7 +4,7 @@ > Part of the [TensorSharp](README.md) documentation. See also the [per-model architecture cards](docs/models/README.md). -TensorSharp loads models in GGUF format. Below are verified Hugging Face repos for every supported architecture, including the multimodal-projector (mmproj) and MTP-draft companion files each family uses. Pick a quantization that fits your hardware (Q4_K_M / UD-Q4_K_XL for low memory, Q8_0 for higher quality, etc.). +TensorSharp loads models in GGUF format. Below are verified Hugging Face repos for every supported architecture, including the multimodal-projector (mmproj) and MTP-draft companion files each family uses. Pick a quantization that fits your hardware (Q4_K_M / UD-Q4_K_XL for low memory, Q8_0 for higher quality, etc.). Rows marked *optional* are the speed artifacts — step-distilled checkpoints, distillation LoRAs and speculative-decoding drafters. Nothing breaks without them, but they are usually the difference between minutes and hours, so skim them before you start a long download. | Architecture | Model | GGUF Download | |---|---|---| @@ -13,29 +13,30 @@ TensorSharp loads models in GGUF format. Below are verified Hugging Face repos f | Gemma 4 | gemma-4-26B-A4B-it (MoE, QAT) | [unsloth/gemma-4-26B-A4B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF) — mmproj `mmproj-BF16.gguf` and MTP draft `mtp-gemma-4-26B-A4B-it.gguf` in the same repo | | Gemma 4 | gemma-4-26B-A4B-it (MoE) | [ggml-org/gemma-4-26B-A4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-26B-A4B-it-GGUF) — mmproj files in the same repo | | Gemma 4 | gemma-4-31B-it | [ggml-org/gemma-4-31B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-31B-it-GGUF) — mmproj files in the same repo | -| Gemma 4 | `gemma4-assistant` MTP drafts | [AtomicChat/gemma-4-E4B-it-assistant-GGUF](https://huggingface.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF) (E4B) and [AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF) (26B-A4B) — load via the server's `--mtp-spec --mtp-draft-model`; pair each draft with its matching target size | +| Gemma 4 | `gemma4-assistant` MTP drafts (optional — speculative decoding) | [AtomicChat/gemma-4-E4B-it-assistant-GGUF](https://huggingface.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF) (E4B) and [AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF) (26B-A4B) — load via the server's `--mtp-spec --mtp-draft-model`; pair each draft with its matching target size | | Gemma 3 | gemma-3-4b-it | [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) — mmproj `mmproj-model-f16.gguf` in the same repo. The official QAT repo [google/gemma-3-4b-it-qat-q4_0-gguf](https://huggingface.co/google/gemma-3-4b-it-qat-q4_0-gguf) is gated (requires HF login + accepting Google's Gemma license) | | Qwen 3 | Qwen3-4B | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) (text only — no companion files) | | Qwen 3.5 / 3.6 family | Qwen3.5-9B | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) — mmproj `mmproj-F16.gguf` in the same repo | | Qwen 3.5 / 3.6 family | Qwen3.5-35B-A3B (MoE) | [ggml-org/Qwen3.5-35B-A3B-GGUF](https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF) — mmproj `mmproj-Qwen3.5-35B-A3B-Q8_0.gguf` in the same repo | | Qwen 3.5 / 3.6 family | Qwen3.6-35B-A3B (MoE, embedded NextN MTP) | [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) — these GGUFs retain the NextN block for the server's `--mtp-spec`; mmproj `mmproj-F16.gguf` in the same repo. The base repo [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) ships the same file names with NextN stripped — those load fine but silently fall back to standard decode | -| GPT OSS | gpt-oss-20b (MoE) | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) (`gpt-oss-20b-mxfp4.gguf`, text only) | +| GPT OSS | gpt-oss-20b (MoE) | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) — `gpt-oss-20b-MXFP4.gguf` (note the uppercase `MXFP4`), text only, no companion files | | Nemotron-H | Nemotron-H-8B-Reasoning-128K | [bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF) | | Nemotron-H | Nemotron-H-47B-Reasoning-128K | [bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF) | | Nemotron-H | Nemotron 3 Nano Omni 30B-A3B (image-capable) | [unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF) — mmproj `mmproj-BF16.gguf` (same repo) is required for image input. Audio is preprocessed only: real audio inference needs a Parakeet audio mmproj these GGUFs do not ship | | Mistral 3 | Mistral-Small-3.1-24B-Instruct-2503 | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF) — Pixtral mmproj `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf` in the same repo | -| Muse-Glimmer | Muse-Glimmer-30B (dense, image-capable) | `general.architecture` = `muse-glimmer`. Needs the `mmproj-Muse-Glimmer-30B-*.gguf` from the same repo for image input (`--mmproj`). Optional DFlash drafter `dflash-*.gguf` loaded with `--draft-model` for lossless speculative decoding | +| Muse-Glimmer | Muse-Glimmer-30B (dense, image-capable) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) — e.g. `Muse-Glimmer-30B-UD-Q4_K_XL.gguf` or `Muse-Glimmer-30B-Q8_0.gguf`; `general.architecture` = `muse-glimmer` / `muse_glimmer`. Image input requires `mmproj-Muse-Glimmer-30B-Q8_0.gguf` (same repo) passed **explicitly** with `--mmproj` — this is the one family with no mmproj auto-detection. Optional speed artifact: the DFlash block drafter `dflash-kquant.gguf` (same repo) loaded with `--draft-model` for lossless speculative decoding — pass no sampler flags, it needs plain greedy | | DeepSeek V4 | DeepSeek-V4-Flash-0731 (284B MoE) | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF) — one subdirectory per quant (`UD-Q8_K_XL/`, `UD-IQ4_XS/`, `UD-IQ1_S/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. Text only | -| DeepSeek V4 | DSpark speculative drafters | see [DSpark drafters](#dspark-drafters) below — a separate GGUF loaded with `--draft-model` for ~1.3-1.4x decode | +| DeepSeek V4 | DSpark speculative drafters (optional — speed only) | see [DSpark drafters](#dspark-drafters) below — a separate GGUF loaded with `--draft-model` for ~1.3-1.4x decode | | DiffusionGemma | diffusiongemma-26B-A4B-it | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) (`general.architecture` = `diffusion-gemma`) | | Qwen-Image-Edit | MMDiT DiT (the `--model` GGUF) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (e.g. `qwen-image-edit-2511-Q4_K_M.gguf`; `general.architecture` = `qwen_image`) | | Qwen-Image-Edit | Qwen-Image VAE (required) | `VAE/Qwen_Image-VAE.safetensors` from [QuantStack/Qwen-Image-Edit-GGUF](https://huggingface.co/QuantStack/Qwen-Image-Edit-GGUF) — place next to the DiT or point `--qwen-image-vae` / `TS_QWEN_IMAGE_VAE` at it (the `.safetensors` VAE loads directly) | | Qwen-Image-Edit | Qwen2.5-VL-7B text encoder (required) | [unsloth/Qwen2.5-VL-7B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF) — place next to the DiT or set `--qwen-image-vl` / `TS_QWEN_IMAGE_TE` | | Qwen-Image-Edit | Vision mmproj (optional) | `mmproj-BF16.gguf` from [unsloth/Qwen2.5-VL-7B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF) — image-grounded conditioning via `--qwen-image-mmproj` / `TS_QWEN_IMAGE_MMPROJ` | | Qwen-Image-Edit | Lightning LoRA (optional, 4/8-step) | [lightx2v/Qwen-Image-Edit-2511-Lightning](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) (`Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors`) — `--qwen-image-lora` / `TS_QWEN_IMAGE_LORA`; auto-switches to the LoRA's step count and CFG 1.0 | -| Wan video generation | Wan DiT (the `--model` GGUF) | Wan 2.2 text/image-to-video: [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF) (e.g. `Wan2.2-TI2V-5B-Q8_0.gguf`) or [QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF) (both HighNoise + LowNoise experts); Wan 2.1 text-to-video: [samuelchristlie/Wan2.1-T2V-1.3B-GGUF](https://huggingface.co/samuelchristlie/Wan2.1-T2V-1.3B-GGUF) or [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf); `general.architecture` = `wan`. See [docs/models/wan.md](docs/models/wan.md) | -| Wan video generation | UMT5-XXL text encoder (required) | [city96/umt5-xxl-encoder-gguf](https://huggingface.co/city96/umt5-xxl-encoder-gguf) (`umt5-xxl-encoder-Q8_0.gguf`) — place next to the DiT or set `--wan-te` / `TS_WAN_TE` | -| Wan video generation | video VAE (required) | Wan 2.1 + A14B: [`wan_2.1_vae.safetensors`](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors); TI2V-5B: [`Wan2.2_VAE.safetensors`](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF/tree/main/VAE) — place next to the DiT (a `VAE/` subfolder works) or set `--wan-vae` / `TS_WAN_VAE` | +| Wan video generation | **Step-distilled DiT (start here)** | **The single biggest speed lever — pick this unless you are reproducing a reference sample.** A distilled checkpoint runs 4 denoise passes instead of the official recipe's 100 for the same video: measured on M5 Pro / `ggml_metal` at 1088×832×121 frames, **17 m 30 s** end to end versus **3 h 30 m** on the base checkpoint, same request, no other flag changed. TI2V-5B: [hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF) — `Wan2_2-TI2V-5B-Turbo-Q8_0.gguf` (5.40 GB), also Q6_K (4.22 GB), Q5_K_M (3.82 GB), Q4_K_M (3.44 GB), down to Q2_K (1.86 GB). **Mind the `Wan2_2` underscore** — copying the base repo's `Wan2.2` spelling into `hf download` 404s. I2V-A14B: [jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF) — Lightning already merged into both experts; download `high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf` **and** `low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf` (9.66 GB each; Q8_0 15.42 GB, Q2_K 5.31 GB) under one `--local-dir` and point `--model` at either — the sibling expert is found automatically. Secondary: [Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF](https://huggingface.co/Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF) (`FastWan2.2-TI2V-5B-q8_0.gguf`, 5.41 GB). **No flag is needed**: TensorSharp reads the DiT file name for `turbo` / `distill` / `lightning` / `lightx2v` / `fastwan` / `-dmd` or an explicit `steps` (1-16), switches to that step count with guidance off, and prints `step-distilled checkpoint detected -> N steps, guidance off` on load; `--diffusion-steps` / `--cfg` override it. The Turbo and A14B distilled repos ship no VAE and no text encoder — take those from the two rows below | +| Wan video generation | Base DiT (the `--model` GGUF) | The full official recipe (50 steps × 2 CFG passes = 100 DiT passes) — use it when you need to match a reference sample; otherwise prefer the distilled row above. Wan 2.2 text/image-to-video: [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF) (`Wan2.2-TI2V-5B-Q8_0.gguf` 5.40 GB or `Wan2.2-TI2V-5B-Q4_K_M.gguf` 3.43 GB; bundles `VAE/Wan2.2_VAE.safetensors`), [QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF) or [QuantStack/Wan2.2-T2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-T2V-A14B-GGUF) (both `HighNoise/` **and** `LowNoise/` experts are required; each repo bundles `VAE/Wan2.1_VAE.safetensors`); Wan 2.1 text-to-video: [samuelchristlie/Wan2.1-T2V-1.3B-GGUF](https://huggingface.co/samuelchristlie/Wan2.1-T2V-1.3B-GGUF) (`Wan2.1-T2V-1.3B-Q8_0.gguf` / `-F16.gguf`) or [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf) (lowercase names, e.g. `wan2.1-t2v-14b-Q8_0.gguf`) — neither 2.1 repo ships a VAE or encoder. `general.architecture` = `wan` / `wan2.1` / `wan2.2`. See [docs/models/wan.md](docs/models/wan.md) | +| Wan video generation | UMT5-XXL text encoder (required, every Wan checkpoint) | [city96/umt5-xxl-encoder-gguf](https://huggingface.co/city96/umt5-xxl-encoder-gguf) — `umt5-xxl-encoder-Q8_0.gguf` (6.04 GB), or `umt5-xxl-encoder-Q5_K_M.gguf` (4.15 GB) / `umt5-xxl-encoder-Q4_K_M.gguf` (3.66 GB) for tighter memory. Turns the prompt into conditioning and is freed before the denoise starts. Place next to the DiT or set `--wan-te` / `TS_WAN_TE` | +| Wan video generation | video VAE (required) | Decodes latents to frames — **which one is decided by the DiT**, not by you: TI2V-5B needs [`Wan2.2_VAE.safetensors`](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF/tree/main/VAE) (bundled in the TI2V-5B repo), Wan 2.1 and A14B need `Wan2.1_VAE.safetensors` — bundled as `VAE/Wan2.1_VAE.safetensors` in both QuantStack A14B repos, or standalone as [`wan_2.1_vae.safetensors`](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors). The distilled repos above ship no VAE, so pair them with the matching file from here. Place next to the DiT (a `VAE/` subfolder works) or set `--wan-vae` / `TS_WAN_VAE` | ### DSpark drafters @@ -88,7 +89,7 @@ block. Those are different drafters from DSpark. These commands run from the repository root. First install the [.NET 10 SDK](DEVELOPMENT.md#install-the-net-10-sdk) for your platform and run `dotnet build TensorSharp.slnx -c Release`; a runtime-only installation cannot build the binaries used below. -The `hf download` commands need the Hugging Face CLI (`pip install -U huggingface_hub`) and drop every file into `./models`. Reminders that apply to all blocks: the CLI reads its one-shot prompt from a **file** via `--input` (`--prompt` is exclusively the Qwen-Image-Edit edit instruction), samples **greedily** by default, and generates only 100 tokens unless you raise `--max-tokens`; the server always listens on **http://localhost:5000**. Swap `--backend ggml_cuda` for the backend that fits your hardware (see [Pick a Backend](README.md#pick-a-backend)). Create a prompt file first: +The `hf download` commands need the Hugging Face CLI (`pip install -U huggingface_hub`) and drop every file into `./models`. Reminders that apply to all blocks: the CLI reads its one-shot prompt from a **file** via `--input` (`--prompt` is the Qwen-Image-Edit edit instruction and the Wan video prompt), samples **greedily** by default, and generates only 100 tokens unless you raise `--max-tokens`; the server always listens on **http://localhost:5000**. Swap `--backend ggml_cuda` for the backend that fits your hardware (see [Pick a Backend](README.md#pick-a-backend)). Create a prompt file first: ```bash echo "Give me three facts about the Moon." > prompt.txt @@ -163,10 +164,10 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B- **GPT OSS** — text, thinking (always on), tools ([ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF)) ```bash -hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models +hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models -dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda -dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda ``` **Nemotron-H** — text, thinking, tools; image on the Omni distribution ([bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF)) @@ -215,3 +216,48 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-e (In the Web UI, attach an image and type the edit instruction. The Lightning LoRA download and `--qwen-image-lora` flag are optional — they cut the denoise to 4 steps at CFG 1.0.) +**Wan video generation** — prompt (+ optional first-frame image) → H.264 MP4; needs the DiT + video VAE + UMT5-XXL text encoder ([hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF)) + +```bash +# The step-distilled Turbo DiT: 4 denoise passes instead of 100, detected from the file name. +# Note the Wan2_2 underscore in the Turbo file name; the VAE and encoder come from the base repos. +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \ + --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_cuda \ + --wan-vae models/VAE/Wan2.2_VAE.safetensors --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --prompt "a cute fluffy orange cat walking through a sunny garden with flowers" \ + --output cat.mp4 --width 832 --height 480 --video-frames 81 +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \ + --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_cuda \ + --wan-vae models/VAE/Wan2.2_VAE.safetensors --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --video-frames 121 --fps 24 +``` + +The console prints `step-distilled checkpoint detected -> 4 steps, guidance off` on load — that +line is how you confirm you are on the fast path. Swapping only the `--model` path for the base +`Wan2.2-TI2V-5B-Q8_0.gguf` runs the official 50-step + CFG recipe instead: the same 1088×832×121-frame +request measured 3 h 30 m there against 17 m 30 s here (M5 Pro, `ggml_metal`). Add `--image first_frame.png` +for image-to-video, or attach an image in the Web UI (it becomes the first frame); on the server +`--video-frames` / `--fps` are defaults that a request can override. Wan is the one family that does +not run on `--backend mlx`; use `ggml_cuda`, `ggml_metal`, `ggml_vulkan`, `ggml_cpu`, `cuda` or `cpu`. + +If all three files sit in one folder (a `VAE/` subfolder counts) the `--wan-vae` / `--wan-te` flags +can be dropped — they are resolved automatically. For the two-expert A14B models download **both** +experts under the same `--local-dir` and point `--model` at either one: + +```bash +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \ + --model models/high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf \ + --backend ggml_cuda --wan-vae models/VAE/Wan2.1_VAE.safetensors \ + --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --prompt "the ship sails into the storm, waves crashing" --image ship.jpg --output ship.mp4 +``` + diff --git a/MODEL_DOWNLOADS_zh-cn.md b/MODEL_DOWNLOADS_zh-cn.md index 62d0c5f5..a951d00a 100644 --- a/MODEL_DOWNLOADS_zh-cn.md +++ b/MODEL_DOWNLOADS_zh-cn.md @@ -4,32 +4,35 @@ > [TensorSharp](README_zh-cn.md) 文档的一部分。另见[各模型架构卡片](docs/models/README_zh-cn.md)。 -TensorSharp 使用 GGUF 格式模型文件。以下是各架构对应的已核对 Hugging Face 下载入口与伴随文件。请根据硬件条件选择合适的量化版本（Q4_K_M / UD-Q4_K_XL 适合低内存，Q8_0 适合更高质量等）。 +TensorSharp 使用 GGUF 格式模型文件。以下是各架构对应的已核对 Hugging Face 下载入口与伴随文件。请根据硬件条件选择合适的量化版本（Q4_K_M / UD-Q4_K_XL 适合低内存，Q8_0 适合更高质量等）。标注“可选”的条目是提速用的产物——步数蒸馏 checkpoint、蒸馏 LoRA、推测解码 draft 模型。不下载也能跑通，但它们往往就是“几分钟”和“几小时”的差别，动手前请先扫一眼。 | 架构 | 模型 | GGUF 下载 | |---|---|---| | Gemma 4 已验证原生规格 | gemma-4-E4B-it Q8_0 | [ggml-org/gemma-4-E4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF)；推荐公开文件为 `gemma-4-E4B-it-Q8_0.gguf`，另有低内存 Q4_K_M；同仓库投影器为 `mmproj-gemma-4-E4B-it-Q8_0.gguf` | | Gemma 4 | 12B / 26B-A4B QAT | [unsloth/gemma-4-12B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF) / [unsloth/gemma-4-26B-A4B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF)；同仓库含 `mmproj-BF16.gguf` 与匹配的 MTP draft | | Gemma 4 | 31B / 26B-A4B | [ggml-org/gemma-4-31B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-31B-it-GGUF) / [ggml-org/gemma-4-26B-A4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-26B-A4B-it-GGUF)；同仓库含 mmproj | -| Gemma 4 | E4B / 26B-A4B MTP draft | [AtomicChat E4B assistant](https://huggingface.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF) / [AtomicChat 26B assistant](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF)；仅与匹配尺寸的目标配对 | -| Gemma 3 | gemma-3-4b-it | 非 gated 的 [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF)，投影器 `mmproj-model-f16.gguf`；官方 [Google QAT 仓库](https://huggingface.co/google/gemma-3-4b-it-qat-q4_0-gguf)需要登录并接受许可证 | +| Gemma 4 | E4B / 26B-A4B MTP draft（可选，仅用于推测解码） | [AtomicChat E4B assistant](https://huggingface.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF) / [AtomicChat 26B assistant](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF)；仅与匹配尺寸的目标配对 | +| Gemma 3 | gemma-3-4b-it | 非 gated 的 [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF)，投影器 `mmproj-model-f16.gguf`；官方 [Google QAT 仓库](https://huggingface.co/google/gemma-3-4b-it-qat-q4_0-gguf)是 **gated** 仓库（需 HF 登录并接受 Google 的 Gemma 许可证），也是本文唯一一个 gated 仓库 | | Qwen 3 | Qwen3-4B | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF)，如 `Qwen3-4B-Q4_K_M.gguf` | | Qwen 3.5 | Qwen3.5-9B | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)，投影器 `mmproj-F16.gguf` | | Qwen 3.5 | Qwen3.5-35B-A3B | [ggml-org/Qwen3.5-35B-A3B-GGUF](https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF)，投影器 `mmproj-Qwen3.5-35B-A3B-Q8_0.gguf` | | Qwen 3.6 | Qwen3.6-35B-A3B（保留 NextN） | [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)，投影器 `mmproj-F16.gguf`；基础仓库会剥离 NextN 块 | -| GPT OSS | gpt-oss-20b（MoE） | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF)，文件 `gpt-oss-20b-mxfp4.gguf` | +| GPT OSS | gpt-oss-20b（MoE） | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF)，文件 `gpt-oss-20b-MXFP4.gguf`（注意 `MXFP4` 为大写）；纯文本，无伴随文件 | | Nemotron-H | Nemotron-H-8B / 47B Reasoning | [8B](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF) / [47B](https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF) | | Nemotron-H | Nemotron 3 Nano Omni 30B-A3B | [unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF)，图像输入需 `mmproj-BF16.gguf`；仓库未附真实音频推理需要的 Parakeet mmproj | | Mistral 3 | Mistral-Small-3.1-24B-Instruct | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF)，Pixtral 投影器 `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf` | +| Muse-Glimmer | Muse-Glimmer-30B（稠密，支持图像） | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)，如 `Muse-Glimmer-30B-UD-Q4_K_XL.gguf` 或 `Muse-Glimmer-30B-Q8_0.gguf`；`general.architecture` 为 `muse-glimmer` / `muse_glimmer`。图像输入需同仓库的 `mmproj-Muse-Glimmer-30B-Q8_0.gguf`，且必须**显式**用 `--mmproj` 指定——这是唯一没有 mmproj 自动探测的系列。可选提速产物：同仓库的 DFlash 分块 draft `dflash-kquant.gguf`，用 `--draft-model` 加载即可无损推测解码——不要传任何采样参数，它只在纯贪心下生效 | | DeepSeek V4 | DeepSeek-V4-Flash-0731（284B MoE） | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF)；每种量化一个子目录（`UD-Q8_K_XL/`、`UD-IQ4_XS/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。仅文本 | -| DeepSeek V4 | DSpark 推测解码 draft | 见下方 [DSpark draft 模型](#dspark-draft-模型)，用 `--draft-model` 加载，解码约 1.3-1.4 倍 | +| DeepSeek V4 | DSpark 推测解码 draft（可选，仅提速） | 见下方 [DSpark draft 模型](#dspark-draft-模型)，用 `--draft-model` 加载，解码约 1.3-1.4 倍 | | DiffusionGemma | diffusiongemma-26B-A4B-it | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF)，如 `diffusiongemma-26B-A4B-it-Q4_K_M.gguf` | | Qwen-Image-Edit | MMDiT DiT（必需） | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)，如 `qwen-image-edit-2511-Q4_K_M.gguf` | | Qwen-Image-Edit | VAE + Qwen2.5-VL（必需） | [QuantStack VAE](https://huggingface.co/QuantStack/Qwen-Image-Edit-GGUF) 中的 `VAE/Qwen_Image-VAE.safetensors` + [unsloth/Qwen2.5-VL-7B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF) | -| Qwen-Image-Edit | Lightning LoRA（可选） | [lightx2v/Qwen-Image-Edit-2511-Lightning](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning)，如 4-step `.safetensors`；通过 `--qwen-image-lora` 加载 | -| Wan 视频生成 | Wan DiT（`--model` GGUF） | Wan 2.2 文/图生视频：[QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)（如 `Wan2.2-TI2V-5B-Q8_0.gguf`）或 [QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)（HighNoise + LowNoise 两个专家）；Wan 2.1 文生视频：[samuelchristlie/Wan2.1-T2V-1.3B-GGUF](https://huggingface.co/samuelchristlie/Wan2.1-T2V-1.3B-GGUF) 或 [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)；`general.architecture` = `wan`。参见 [docs/models/wan.md](docs/models/wan.md) | -| Wan 视频生成 | UMT5-XXL 文本编码器（必需） | [city96/umt5-xxl-encoder-gguf](https://huggingface.co/city96/umt5-xxl-encoder-gguf)（`umt5-xxl-encoder-Q8_0.gguf`）—— 放在 DiT 旁或用 `--wan-te` / `TS_WAN_TE` 指定 | -| Wan 视频生成 | 视频 VAE（必需） | Wan 2.1 + A14B：[`wan_2.1_vae.safetensors`](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors)；TI2V-5B：[`Wan2.2_VAE.safetensors`](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF/tree/main/VAE) —— 放在 DiT 旁（`VAE/` 子目录亦可）或用 `--wan-vae` / `TS_WAN_VAE` 指定 | +| Qwen-Image-Edit | 视觉 mmproj（可选） | [unsloth/Qwen2.5-VL-7B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF) 中的 `mmproj-BF16.gguf`，用 `--qwen-image-mmproj` / `TS_QWEN_IMAGE_MMPROJ` 加载，可让编辑指令参考源图内容 | +| Qwen-Image-Edit | Lightning LoRA（可选，4/8 步） | [lightx2v/Qwen-Image-Edit-2511-Lightning](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning)，文件 `Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors`（0.850 GB）；用 `--qwen-image-lora` / `TS_QWEN_IMAGE_LORA` 加载，会自动按文件名里的步数把采样默认值切到该步数 + CFG 1.0（基础默认为 30 步、CFG 2.5） | +| Wan 视频生成 | **步数蒸馏 DiT（首选）** | **这是最大的提速手段——除非要复现参考样例，都应该用它。**蒸馏 checkpoint 生成同一段视频只跑 4 次去噪，而官方配方要跑 100 次：在 M5 Pro / `ggml_metal` 上以 1088×832×121 帧实测，端到端 **17 分 30 秒**，而基础 checkpoint 是 **3 小时 30 分**——同一个请求，其他参数一律不变。TI2V-5B：[hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF)，文件 `Wan2_2-TI2V-5B-Turbo-Q8_0.gguf`（5.40 GB），另有 Q6_K（4.22 GB）、Q5_K_M（3.82 GB）、Q4_K_M（3.44 GB），最小到 Q2_K（1.86 GB）。**注意文件名里是 `Wan2_2` 下划线**，照抄基础仓库的 `Wan2.2` 写法会 404。I2V-A14B：[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF)，Lightning 已合并进两个专家；需同时下载 `high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf` **和** `low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf`（各 9.66 GB；Q8_0 15.42 GB，Q2_K 5.31 GB），放在同一个 `--local-dir` 下，`--model` 指向任意一个即可，另一个专家会自动找到。备选：[Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF](https://huggingface.co/Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF)（`FastWan2.2-TI2V-5B-q8_0.gguf`，5.41 GB）。**无需任何参数**：TensorSharp 读取 DiT 文件名，命中 `turbo` / `distill` / `lightning` / `lightx2v` / `fastwan` / `-dmd` 或显式的 `steps`（1-16）即切换到该步数并关闭 guidance，加载时打印 `step-distilled checkpoint detected -> N steps, guidance off`；`--diffusion-steps` / `--cfg` 可覆盖。Turbo 与 A14B 蒸馏仓库都不含 VAE 和文本编码器，请从下面两行获取 | +| Wan 视频生成 | 基础 DiT（`--model` GGUF） | 完整官方配方（50 步 × 2 次 CFG = 100 次 DiT 前向）——需要对齐参考样例时才用，否则优先用上一行的蒸馏版本。Wan 2.2 文/图生视频：[QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)（`Wan2.2-TI2V-5B-Q8_0.gguf` 5.40 GB 或 `Wan2.2-TI2V-5B-Q4_K_M.gguf` 3.43 GB，仓库自带 `VAE/Wan2.2_VAE.safetensors`）、[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF) 或 [QuantStack/Wan2.2-T2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-T2V-A14B-GGUF)（`HighNoise/` 与 `LowNoise/` 两个专家缺一不可，两个仓库都自带 `VAE/Wan2.1_VAE.safetensors`）；Wan 2.1 文生视频：[samuelchristlie/Wan2.1-T2V-1.3B-GGUF](https://huggingface.co/samuelchristlie/Wan2.1-T2V-1.3B-GGUF)（`Wan2.1-T2V-1.3B-Q8_0.gguf` / `-F16.gguf`）或 [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)（文件名为小写，如 `wan2.1-t2v-14b-Q8_0.gguf`）——这两个 2.1 仓库都不含 VAE 和编码器。`general.architecture` 为 `wan` / `wan2.1` / `wan2.2`。参见 [docs/models/wan.md](docs/models/wan.md) | +| Wan 视频生成 | UMT5-XXL 文本编码器（必需，所有 Wan checkpoint 都要） | [city96/umt5-xxl-encoder-gguf](https://huggingface.co/city96/umt5-xxl-encoder-gguf)：`umt5-xxl-encoder-Q8_0.gguf`（6.04 GB），内存紧张可用 `umt5-xxl-encoder-Q5_K_M.gguf`（4.15 GB）/ `umt5-xxl-encoder-Q4_K_M.gguf`（3.66 GB）。负责把提示词编码成条件向量，去噪开始前即从显存释放。放在 DiT 旁或用 `--wan-te` / `TS_WAN_TE` 指定 | +| Wan 视频生成 | 视频 VAE（必需） | 把 latent 解码成画面——**用哪个由 DiT 自己决定**，不是由你选：TI2V-5B 需要 [`Wan2.2_VAE.safetensors`](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF/tree/main/VAE)（TI2V-5B 仓库自带），Wan 2.1 与 A14B 需要 `Wan2.1_VAE.safetensors`——两个 QuantStack A14B 仓库里就有 `VAE/Wan2.1_VAE.safetensors`，也可单独下载 [`wan_2.1_vae.safetensors`](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors)。上面的蒸馏仓库都不含 VAE，请从这里配一个对应的。放在 DiT 旁（`VAE/` 子目录亦可）或用 `--wan-vae` / `TS_WAN_VAE` 指定 | ### DSpark draft 模型 @@ -56,6 +59,11 @@ draft，社区亦有 GGUF 转换，但它们是另一种 draft 结构：5 层 Tr `fc` 融合（`general.architecture` 为 `dspark` 或 `dflash`，block_size 7），而非 DeepSeek V4 的三个超连接块（`mtp.*`）。TensorSharp 会明确报错而不会错误加载。这里列出以便了解上游现状： +> 这套 5 层 `fc` 融合结构**已经**在 Muse-Glimmer 上实现——见 +> [DFlash 投机解码](docs/models/muse-glimmer_zh-cn.md#3-dflash-投机解码)。 +> 下表这些 draft 没有接入，是因为它们的编码器需要目标模型暴露逐层输入残差， +> 目前只有 `MuseGlimmerModel` 做到了这一点。 + | 主干 | 官方 checkpoint（safetensors） | 社区 GGUF | |---|---|---| | Qwen3-4B | [deepseek-ai/dspark_qwen3_4b_block7](https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7) | — | @@ -70,12 +78,28 @@ Gemma 4 目前已有可用的推测解码路径：上表中的 `gemma4-assistant ### 按模型下载并运行 -以下命令从仓库根目录运行；请先按平台安装完整的 [.NET 10 SDK](DEVELOPMENT_zh-cn.md#安装-net-10-sdk)，再执行 `dotnet build TensorSharp.slnx -c Release`。仅安装 Runtime 无法构建下方使用的二进制文件。`hf` 来自 Hugging Face CLI（`pip install -U huggingface_hub`）。单次文本提示词必须通过 `--input` 文件传入，`--prompt` 仅用于 Qwen-Image-Edit。按硬件把示例的 `ggml_cuda` 换成 `ggml_metal`、`ggml_vulkan` 或 `ggml_cpu`。 +以下命令从仓库根目录运行；请先按平台安装完整的 [.NET 10 SDK](DEVELOPMENT_zh-cn.md#安装-net-10-sdk)，再执行 `dotnet build TensorSharp.slnx -c Release`。仅安装 Runtime 无法构建下方使用的二进制文件。`hf` 来自 Hugging Face CLI（`pip install -U huggingface_hub`），所有文件都会下载到 `./models`。通用提示：单次文本提示词通过 `--input` 文件传入（`--prompt` 用于 Qwen-Image-Edit 的编辑指令和 Wan 的视频提示词）；CLI 默认贪心采样，且不加 `--max-tokens` 时只生成 100 个 token；服务端固定监听 **http://localhost:5000**。按硬件把示例中的 `ggml_cuda` 换成 `ggml_metal`、`ggml_vulkan` 或 `ggml_cpu`（见 [选择后端](README_zh-cn.md#选择后端)）。 ```bash echo "列出三条关于月球的事实。" > prompt.txt ``` +**DeepSeek V4 Flash**（284B MoE，纯文本，支持 DSpark 推测解码）： + +```bash +# 约 160 GB 权重：需要多张 GPU（自动按层切分），draft 另需约 7 GB +hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models +hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \ + --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \ + --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \ + --input prompt.txt --max-tokens 200 --temperature 0 +``` + +去掉 `--draft-model` 即为普通解码。CLI 上的推测解码要求纯贪心采样（`--temperature 0`）； +`--spec-draft-conf-min` 控制每个块草拟到多深。 + **Gemma 4**（文本 + 图像/视频/音频、思维链、工具、可选 MTP）： ```bash @@ -121,9 +145,9 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B- **GPT OSS**（文本、始终思考、工具）： ```bash -hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models -dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda -dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda +hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda ``` **Nemotron-H**（文本、思维链、工具）： @@ -153,6 +177,8 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda ``` +（Web UI 会实时流式展示 DiffusionGemma 的去噪过程；兼容 API 只返回最终文本。） + **Qwen-Image-Edit**（DiT + VAE + 文本编码器；Lightning LoRA 可选）： ```bash @@ -164,3 +190,48 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-25 dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-UD-IQ2_XXS.gguf --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --backend ggml_cuda ``` +（在 Web UI 里上传图片并输入编辑指令即可。Lightning LoRA 的下载与 `--qwen-image-lora` 参数是可选的——加上后去噪降到 4 步、CFG 1.0。） + +**Wan 视频生成**（提示词 + 可选首帧图片 → H.264 MP4；需要 DiT + 视频 VAE + UMT5-XXL 文本编码器）： + +```bash +# 步数蒸馏的 Turbo DiT：只跑 4 次去噪而不是 100 次，由文件名自动识别。 +# 注意 Turbo 文件名里的 Wan2_2 下划线；VAE 和文本编码器仍需从基础仓库获取。 +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \ + --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_cuda \ + --wan-vae models/VAE/Wan2.2_VAE.safetensors --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --prompt "a cute fluffy orange cat walking through a sunny garden with flowers" \ + --output cat.mp4 --width 832 --height 480 --video-frames 81 +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \ + --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_cuda \ + --wan-vae models/VAE/Wan2.2_VAE.safetensors --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --video-frames 121 --fps 24 +``` + +加载时控制台会打印 `step-distilled checkpoint detected -> 4 steps, guidance off`——看到这行就说明走的是快路径。 +只把 `--model` 换成基础的 `Wan2.2-TI2V-5B-Q8_0.gguf`，就会按官方 50 步 + CFG 配方运行：同一个 +1088×832×121 帧的请求实测为 3 小时 30 分，而这里是 17 分 30 秒（M5 Pro，`ggml_metal`）。 +加 `--image first_frame.png` 即为图生视频，Web UI 里上传图片也一样（该图作为首帧）；服务端的 +`--video-frames` / `--fps` 只是默认值，单个请求可以覆盖。Wan 是唯一不支持 `--backend mlx` 的系列， +请使用 `ggml_cuda`、`ggml_metal`、`ggml_vulkan`、`ggml_cpu`、`cuda` 或 `cpu`。 + +三个文件放在同一个目录下时（`VAE/` 子目录也算），`--wan-vae` / `--wan-te` 可以省略，会自动解析。 +双专家的 A14B 模型需要**同时**下载两个专家到同一个 `--local-dir`，`--model` 指向其中任意一个： + +```bash +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \ + --model models/high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf \ + --backend ggml_cuda --wan-vae models/VAE/Wan2.1_VAE.safetensors \ + --wan-te models/umt5-xxl-encoder-Q8_0.gguf \ + --prompt "the ship sails into the storm, waves crashing" --image ship.jpg --output ship.mp4 +``` + diff --git a/README.md b/README.md index c14f937f..16f24c2b 100644 --- a/README.md +++ b/README.md @@ -25,10 +25,10 @@ - **⚡ Trades wins with llama.cpp — from pure .NET.** On identical GGUF files and the same GPU, TensorSharp matches or beats `llama.cpp` on the workloads that matter: Gemma 4 E4B and 2-bit Qwen 3.6 35B-A3B MoE prefill **1.28×** faster on CUDA with first tokens **1.27×** sooner (multi-turn up to **1.49×**); Gemma 4 12B decodes **1.21×** faster on Vulkan (up to **1.32×** on long context). → [Benchmarks](#benchmarks) - **🚀 Continuous batching & paged KV cache.** vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → [deep dive](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md) - **🧬 DeepSeek V4 Flash (284B MoE) with three whole-model executors.** The compressed-sparse-attention 1M-context architecture runs on a direct-CUDA engine (`--backend cuda`), the native ggml executor (`--backend ggml_cuda` / `ggml_vulkan`), *and* a 100% pure-C# CPU executor (`--backend cpu`, no native dependencies). Weights layer-split automatically across every visible GPU, so a model far larger than one card still runs; the server hosts it with per-sequence slots and continuous batching. → [DeepSeek V4 card](docs/models/deepseek4.md) -- **🔮 Speculative decoding — MTP / NextN *and* DSpark.** Multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (embedded NextN block) and Gemma 4 (separate `gemma4-assistant` draft GGUF); DeepSeek V4 adds **DSpark** block drafting (`--draft-model`), which proposes a whole block of tokens per step for **1.3–1.4× decode** (up to 2.0× on multi-turn chat). In every case the draft proposes, the trunk verifies in one batched forward, and the output matches standard decode. → [Speculative decoding](FEATURES.md#mtp--nextn-speculative-decoding) +- **🔮 Speculative decoding — MTP / NextN *and* DSpark.** Multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk — use an MTP-retaining GGUF such as [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF); the base repo ships the same file names with the block stripped) and Gemma 4 (separate `gemma4-assistant` draft GGUF, `--mtp-draft-model`); DeepSeek V4 adds **DSpark** block drafting (`--draft-model`), which proposes a whole block of tokens per step for **1.3–1.4× decode** (up to 2.0× on multi-turn chat). In every case the draft proposes, the trunk verifies in one batched forward, and the output matches standard decode. → [Speculative decoding](FEATURES.md#mtp--nextn-speculative-decoding) - **🔗 Tensor parallelism & distributed clustering.** Split a model across multiple GPUs with `--tp N` — on the direct `cuda` backend **and** on GGML CUDA / Vulkan — and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Megatron-LM column/row-parallel pattern with hierarchical AllReduce; MoE expert parallelism and per-rank GatedDeltaNet kernels on GGML. Fused per-rank execution makes `--tp 2` decode **1.39×** a single GPU on Gemma 4 E4B and **1.57×** on Muse-Glimmer 30B (which also gains **1.34×** prefill — the one model that beats a single GPU on both phases), and runs models that do not fit one card at all (Qwen 3.5-35B-A3B; Muse-Glimmer 30B Q8_0 at 28.2 GB on 24 GB cards). Optional Redis-backed KV cache and Responses API store. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference) -- **🎨 Qwen-Image-Edit image editing.** Prompt + input image → edited image, driving a 60-block MMDiT with a Qwen-Image VAE and Qwen2.5-VL-7B text encoder. CUDA-graph-captured DiT, FlowMatch-Euler true-CFG denoise, live Web UI previews, and Lightning-LoRA fast paths. Beat `stable-diffusion.cpp` **1.19×** on a warm 4-step edit. → [Qwen-Image-Edit card](docs/models/qwenimage.md) -- **🎬 Wan 2.1 / 2.2 video generation (text → video and image → video).** Prompt → H.264 MP4; on the Wan 2.2 models (TI2V-5B, I2V-A14B) an uploaded image becomes the video's first frame while the prompt drives motion, camera and scene changes. One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode+decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff — TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 min, and Wan 2.1 runs **6.0×** faster end-to-end than `stable-diffusion.cpp` on the identical workload. Numerics verified against diffusers (DiT cos > 0.995, VAE encoders cos > 0.999, decode 59.9 dB / >35 dB PSNR). CLI (`--image`), `/v1/videos/generations`, and Web UI chat with image upload. → [Wan card](docs/models/wan.md) +- **🎨 Qwen-Image-Edit image editing.** Prompt + input image → edited image, driving a 60-block MMDiT with a Qwen-Image VAE and Qwen2.5-VL-7B text encoder. CUDA-graph-captured DiT, FlowMatch-Euler true-CFG denoise, live Web UI previews, and a [Lightning distillation LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) fast path (`--qwen-image-lora`, applied as a runtime side-path over the untouched quantized weights) that takes the default 30 steps × CFG — 60 DiT forwards — down to **4**. Beat `stable-diffusion.cpp` **1.19×** on a warm 4-step edit. → [Qwen-Image-Edit card](docs/models/qwenimage.md) +- **🎬 Wan 2.1 / 2.2 video generation (text → video and image → video).** Prompt → H.264 MP4; on the Wan 2.2 models (TI2V-5B, I2V-A14B) an uploaded image becomes the video's first frame while the prompt drives motion, camera and scene changes. One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode+decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff — TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 min, and Wan 2.1 runs **6.0×** faster end-to-end than `stable-diffusion.cpp` on the identical workload. **Step-distilled checkpoints are auto-detected from the DiT file name** (`Turbo` / `distill` / `Lightning` / `lightx2v` / `FastWan` / `…-4steps-…`) and switch to that step count with guidance off — 4 DiT passes instead of the official recipe's 100, which took the same 1088×832×121-frame image-to-video from **3 h 30 m to 17 m 30 s** on an M5 Pro. It is the single biggest speed lever in the repository and needs no flag, only a different `--model` file. Numerics verified against diffusers (DiT cos > 0.995, VAE encoders cos > 0.999, decode 59.9 dB / >35 dB PSNR). CLI (`--image`), `/v1/videos/generations`, and Web UI chat with image upload. → [Wan card](docs/models/wan.md) - **🌫️ DiffusionGemma text diffusion.** Block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI flags and a Web UI denoising preview stream. → [DiffusionGemma card](docs/models/diffusiongemma.md) - **🖼️ Multimodal.** Image / video / audio (Gemma 4); image input for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer; PDF documents via CLI and Web UI. → [Multimodal](FEATURES.md#multimodal-support) - **🛠️ Tool calling & thinking mode.** Multi-turn tool calls and structured chain-of-thought across Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer (ATEM markup), and DeepSeek V4 (DSML markup). → [Features](FEATURES.md) @@ -165,8 +165,29 @@ Implemented and exercised by the test/benchmark matrix. Pick a quantization that | Muse-Glimmer | [Muse-Glimmer-30B](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) (+ mmproj) | ✅ / — / — | ✅ | ✅ | [muse-glimmer.md](docs/models/muse-glimmer.md) | | Gemma 3 | [gemma-3-4b-it](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | ✅ / — / — | — | — | [gemma3.md](docs/models/gemma3.md) | | DiffusionGemma | [diffusiongemma-26B-A4B-it](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | — / — / — | — | — | [diffusiongemma.md](docs/models/diffusiongemma.md) | -| Qwen-Image-Edit | [Qwen-Image-Edit-2511](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (MMDiT + VAE + Qwen2.5-VL) | 🖼️ image→image | — | — | [qwenimage.md](docs/models/qwenimage.md) | -| Wan 2.1 / 2.2 video | [Wan2.2-TI2V-5B](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF) (also [I2V-A14B](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF), [Wan2.1-T2V-14B](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)) + UMT5-XXL + video VAE | 🎬 text→video, image→video | — | — | [wan.md](docs/models/wan.md) | +| Qwen-Image-Edit | [Qwen-Image-Edit-2511](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (MMDiT + VAE + Qwen2.5-VL) · fast lane: [Lightning 4-step LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) | 🖼️ image→image | — | — | [qwenimage.md](docs/models/qwenimage.md) | +| Wan 2.1 / 2.2 video | [Wan2.2-TI2V-5B](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF) (also [T2V-A14B](https://huggingface.co/QuantStack/Wan2.2-T2V-A14B-GGUF), [I2V-A14B](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF), [Wan2.1-T2V-14B](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)) + UMT5-XXL + video VAE · fast lane: [TI2V-5B-Turbo](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF) (4-step, 25× fewer DiT passes) | 🎬 text→video, image→video | — | — | [wan.md](docs/models/wan.md) | + +## Make It Fast + +Several families have a *fast lane* — a different artifact to download, or one flag — that changes the cost of a run by an order of magnitude. Reach for these before tuning anything else. + +| Family | Fast artifact or flag | Measured effect | +|---|---|---| +| **Wan 2.1 / 2.2 video** | A **step-distilled DiT GGUF** — e.g. [hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF) (`Wan2_2-TI2V-5B-Turbo-Q8_0.gguf`) for TI2V-5B, or [jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF) for A14B. No flag — detected from the file name. | 100 DiT passes → **4**, guidance off. The same 1088×832×121f image-to-video: **3 h 30 m → 17 m 30 s** (M5 Pro, `ggml_metal`). | +| Wan, base checkpoints only | `--cfg-cache-stride 2` / `3` | 1.30× / 1.43× at 50 steps (approximate; pointless on a distilled checkpoint, which is already guidance-free). | +| Wan, any checkpoint | Generate at a trained resolution and downscale — 736×544 instead of 1088×832 | 121 frames, Turbo checkpoint: **6 m 19 s** instead of 17 m 30 s. Below ~0.3 MP quality falls off instead. | +| **Qwen-Image-Edit** | `--qwen-image-lora` with the [Lightning 4-step LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) (`Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors`) | Sampling defaults switch from 30 steps at CFG 2.5 (60 DiT forwards) to **4** at CFG 1.0. A warm 4-step edit beat `stable-diffusion.cpp` **1.19×**. | +| Qwen-Image-Edit | `TS_QWEN_DIT_CACHE_MODE=easycache` (off by default — quality first) | Skips 40–55% of denoise steps; measurably softens fine detail on edits, which is why it is opt-in. | +| **DeepSeek V4 Flash** | `--draft-model` with a [DSpark drafter GGUF](https://huggingface.co/sakamakismile/DeepSeek-V4-Flash-DSpark-support-ds4-GGUF) (server: add `--mtp-spec`); `cuda` / `ggml_cuda` only | Decode 26.4 → **37.1 tok/s** (1.41×) on 4×A40, 69% acceptance; up to **2.0×** on multi-turn chat. Output is unchanged — the trunk verifies every block. | +| **Muse-Glimmer** | `--draft-model` with the DFlash drafter (`dflash-kquant.gguf`, in [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)); pass **no** sampler flags | 1.3–5× decode on the CUDA hosts it was built on — 35.0 → **50.9 tok/s** greedy at a 60-token prompt on one RTX PRO 6000. On Apple Silicon plain decode is still faster today. | +| **Qwen 3.6** | An MTP-retaining GGUF — [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF), not the base repo — plus the server's `--mtp-spec` | Enables NextN speculative decode on solo sequences. The base repo ships the same file names with the block stripped and silently falls back. | +| **Gemma 4** | `--mtp-draft-model` with the matching [`gemma4-assistant` draft GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF) plus `--mtp-spec` (server) | Speculative decode on GGML backends and the direct `cuda` backend. Draft and target hidden sizes must match, or startup fails. | +| Any MoE that does not fit the card | `--n-cpu-moe N` / `--cpu-moe` | gpt-oss-20b 16.2 → 2.9 GB VRAM on a 16 GB laptop card, turning the WDDM spill cliff's 0.3 tok/s into **25.4** at `--n-cpu-moe 12`. | +| Multi-GPU | `--tp N` | Gemma 4 E4B decode **1.39×** a single GPU, Muse-Glimmer 30B **1.57×** decode / **1.34×** prefill — and it runs models that fit on no single card. | +| Every family | Pick the right backend: `ggml_cuda` on NVIDIA, `ggml_metal` on Apple Silicon, `ggml_cpu` (not `cpu`) without a GPU | Gemma 4 26B-A4B decodes 78.7 tok/s on `ggml_cuda` vs 35.3 on the direct `cuda` backend; on Apple Silicon Muse-Glimmer 30B prefills 413.6 tok/s on `ggml_metal` vs 29.0 on MLX. | + +Per-family detail, including the numbers behind every row: [Wan](docs/models/wan.md) · [Qwen-Image-Edit](docs/models/qwenimage.md) · [DeepSeek V4](docs/models/deepseek4.md) · [Muse-Glimmer](docs/models/muse-glimmer.md) · [Features](FEATURES.md). ## Supported Model Architectures @@ -175,15 +196,15 @@ Implemented and exercised by the test/benchmark matrix. Pick a quantization that | DeepSeek V4 Flash | `deepseek4` | DeepSeek-V4-Flash (284B MoE, 256 experts, compressed sparse attention, 1M context) | Text only | Yes | Yes (DSML) | Yes (DSpark block drafter, separate GGUF) | [deepseek4.md](docs/models/deepseek4.md) | | Gemma 4 | `gemma4` | gemma-4-E4B, gemma-4-31B, gemma-4-26B-A4B (MoE) | Image, Video, Audio | Yes | Yes | Yes (separate draft GGUF) | [gemma4.md](docs/models/gemma4.md) | | Gemma 3 | `gemma3` | gemma-3-4b | Image | No | No | — | [gemma3.md](docs/models/gemma3.md) | -| Qwen 3 | `qwen3` | Qwen3-4B | Text only | Yes | Yes | — | [qwen3.md](docs/models/qwen3.md) | +| Qwen 3 | `qwen3`, `qwen2`, `qwen2vl`, `qwen2_vl` | Qwen3-4B (Qwen2 / Qwen2.5-VL GGUFs also load, as text-only chat) | Text only | Yes | Yes | — | [qwen3.md](docs/models/qwen3.md) | | Qwen 3.5 / 3.6 family | `qwen35`, `qwen35moe`, `qwen3next` | Qwen3.5-9B (hybrid Attn+Recurrent), Qwen3.5/3.6-35B-A3B (MoE) | Image | Yes | Yes | Yes on Qwen 3.6 (embedded NextN) | [qwen35.md](docs/models/qwen35.md) | | GPT OSS | `gptoss`, `gpt-oss` | gpt-oss-20b (MoE) | Text only | Yes (always) | Yes | — | [gptoss.md](docs/models/gptoss.md) | | Nemotron-H | `nemotron_h`, `nemotron_h_moe` | Nemotron-H-8B/47B (Hybrid SSM-Transformer, MoE), Nemotron 3 Nano Omni | Image (Omni) | Yes | Yes | — | [nemotron.md](docs/models/nemotron.md) | | Mistral 3 | `mistral3` | Mistral-Small-3.1-24B-Instruct | Image | No | No | — | [mistral3.md](docs/models/mistral3.md) | -| Muse-Glimmer | `muse-glimmer` | Muse-Glimmer-30B (interleaved SWA + NoPE full layers, attention output gate) | Image | Yes | Yes (ATEM) | Yes (DFlash block drafter, separate GGUF) | [muse-glimmer.md](docs/models/muse-glimmer.md) | -| DiffusionGemma | `diffusion-gemma` | diffusion-gemma text-diffusion GGUFs | Text only | No | No | — | [diffusiongemma.md](docs/models/diffusiongemma.md) | -| Qwen-Image-Edit | `qwen_image` | qwen-image-edit MMDiT GGUFs (+ VAE & Qwen2.5-VL) | Image edit (image+text → image) | No | No | — | [qwenimage.md](docs/models/qwenimage.md) | -| Wan video | `wan` | Wan 2.1 T2V 1.3B/14B, Wan 2.2 TI2V-5B, Wan 2.2 A14B T2V/I2V (two experts) | Video out (text→video, image→video) | No | No | — | [wan.md](docs/models/wan.md) | +| Muse-Glimmer | `muse-glimmer`, `muse_glimmer` | Muse-Glimmer-30B (interleaved SWA + NoPE full layers, attention output gate) | Image | Yes | Yes (ATEM) | Yes (DFlash block drafter, separate GGUF) | [muse-glimmer.md](docs/models/muse-glimmer.md) | +| DiffusionGemma | `diffusion-gemma`, `diffusion_gemma` | diffusion-gemma text-diffusion GGUFs | Text only | No | No | — | [diffusiongemma.md](docs/models/diffusiongemma.md) | +| Qwen-Image-Edit | `qwen_image`, `qwen-image` | qwen-image-edit MMDiT GGUFs (+ VAE & Qwen2.5-VL) | Image edit (image+text → image) | No | No | — | [qwenimage.md](docs/models/qwenimage.md) | +| Wan video | `wan`, `wan2.1`, `wan2.2` | Wan 2.1 T2V 1.3B/14B, Wan 2.2 TI2V-5B, Wan 2.2 A14B T2V/I2V (two experts) | Video out (text→video, image→video) | No | No | — | [wan.md](docs/models/wan.md) | End-to-end per-model documentation (origin, forward graph, components, parameters, prefill/decode optimizations): [architecture cards](docs/models/README.md). @@ -229,12 +250,12 @@ New here? The sections above are all you need to get running. Everything else is | Area | Status | |---|---| -| Model families | DeepSeek V4 Flash (`deepseek4`), Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family (`qwen35`, `qwen35moe`, `qwen3next`), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), Mistral 3, Muse-Glimmer (`muse-glimmer`). Image editing via Qwen-Image-Edit (`qwen_image` MMDiT); video generation via Wan 2.1 / 2.2 (`wan`). | +| Model families | DeepSeek V4 Flash (`deepseek4`), Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family (`qwen35`, `qwen35moe`, `qwen3next`), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), Mistral 3, Muse-Glimmer (`muse-glimmer`, `muse_glimmer`). Image editing via Qwen-Image-Edit (`qwen_image`, `qwen-image` MMDiT); video generation via Wan 2.1 / 2.2 (`wan`, `wan2.1`, `wan2.2`). | | Inference hosts | CLI, interactive REPL, ASP.NET Core web UI, Ollama-style API, OpenAI Chat Completions-style API. | -| Backends | Pure C# CPU, direct CUDA/cuBLAS (`cuda`), MLX Metal (`mlx`), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct-CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU (`--tp N` / `TS_DSV4_NGPU` caps the count). | +| Backends | Pure C# CPU, direct CUDA/cuBLAS (`cuda`), MLX Metal (`mlx`), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct-CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU (`--tp N` / `TS_DSV4_NGPU` caps the count). Wan is the one family that restricts its backends: it runs on the GGML backends and on the direct `cuda` / pure-C# `cpu` ones, but not on MLX. | | Multimodal | Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, Muse-Glimmer image input; PDF documents (CLI `--pdf` + Web UI). Media *out*: Qwen-Image-Edit (image) and Wan 2.1 / 2.2 (H.264 MP4 video, text→video and image→video). | | Continuous batching | vLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (default on; opt-out `--no-continuous-batching`). DeepSeek V4 serves through its own native per-sequence slots on the same engine. | -| Speculative decoding | MTP / NextN draft heads on Qwen 3.6 (embedded) and Gemma 4 (separate draft GGUF); DSpark block drafting on DeepSeek V4 and DFlash block drafting on Muse-Glimmer (separate drafter GGUF via `--draft-model`, `cuda` / `ggml_cuda`). Verification is greedy against the target, so the emitted stream is the plain-greedy stream. Off by default; opt-in via the server's `--mtp-spec`, or by passing `--draft-model` on the CLI. | +| Speculative decoding | MTP / NextN draft heads on Qwen 3.6 (embedded) and Gemma 4 (separate draft GGUF); DSpark block drafting on DeepSeek V4 (`cuda` / `ggml_cuda` only) and DFlash block drafting on Muse-Glimmer, both loading a separate drafter GGUF via `--draft-model`. Verification is greedy against the target, so the emitted stream is the plain-greedy stream. Off by default; opt-in via the server's `--mtp-spec`, or by passing `--draft-model` on the CLI. | | Tensor parallelism | Megatron-LM column/row-parallel TP on the direct `cuda` backend and on GGML CUDA / Vulkan (`--tp N` / `TENSORSHARP_TP_DEGREE`, CLI and server); distributed multi-node TP via peer-to-peer TCP (`--tp-node-id` / `--tp-peers`), with hierarchical AllReduce and automatic host-staging fallback when CUDA P2P is unavailable. All autoregressive architectures; MoE expert parallelism and fused per-rank decode/prefill graphs for Gemma 4 and Qwen 3.5/3.6 on GGML. Optional Redis-backed KV cache and Responses API store. | | Server model scope | One explicitly hosted GGUF via `--model`; optional explicit projector via `--mmproj`; no directory scanning. | | Observability | Structured per-turn logs, queue status, and KV-cache reuse metrics across Web UI, Ollama, and OpenAI shapes. | diff --git a/README_zh-cn.md b/README_zh-cn.md index 91806e6f..6b7b0b55 100644 --- a/README_zh-cn.md +++ b/README_zh-cn.md @@ -25,10 +25,10 @@ Zhongkai Fu 所著的 **[From Tensors to Tokens: Building a Multimodal LLM Infer - **⚡ 与 llama.cpp 互有胜负——用纯 .NET 做到。** 在相同 GGUF 文件、相同 GPU 上，TensorSharp 在关键负载上追平乃至超越 `llama.cpp`：Gemma 4 E4B 与 2-bit 量化的 Qwen 3.6 35B-A3B MoE 在 CUDA 上 prefill 快 **1.28×**、首 token 早 **1.27×**（多轮最高 **1.49×**）；Gemma 4 12B 在 Vulkan 上 decode 快 **1.21×**（长上下文最高 **1.32×**）。→ [性能数据](#性能数据) - **🚀 连续批处理 & 分页 KV 缓存。** vLLM 风格的分页 KV 池，支持基于内容哈希的前缀共享与迭代级调度器，服务端默认启用。→ [深入文档](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md) - **🧬 DeepSeek V4 Flash（284B MoE），三套整模型执行器。** 这套压缩稀疏注意力、1M 上下文的架构可运行在 Direct CUDA 引擎（`--backend cuda`）、原生 ggml 执行器（`--backend ggml_cuda` / `ggml_vulkan`），以及 **100% 纯 C# 的 CPU 执行器**（`--backend cpu`，零原生依赖）上。权重会自动按层切分到所有可见 GPU，因此远大于单卡显存的模型依然跑得起来；服务端以原生 per-sequence slot + 连续批处理托管它。→ [DeepSeek V4 卡片](docs/models/deepseek4.md) -- **🔮 投机解码——MTP / NextN 与 DSpark。** 多 token 预测草稿头加速单序列 decode：Qwen 3.6（内嵌 NextN 块）与 Gemma 4（独立 `gemma4-assistant` 草稿 GGUF）；DeepSeek V4 则新增 **DSpark** 块级起草（`--draft-model`），每步提议一整块 token，decode 提速 **1.3–1.4×**（多轮对话最高 2.0×）。三者都是草稿提议、主干一次批量前向验证，输出与标准 decode 一致。→ [投机解码](FEATURES_zh-cn.md#mtp--nextn-投机解码) +- **🔮 投机解码——MTP / NextN 与 DSpark。** 多 token 预测草稿头加速单序列 decode：Qwen 3.6（NextN 块内嵌于主干——需使用保留该块的 GGUF，例如 [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)；基础仓库的同名文件已剥离该块）与 Gemma 4（独立 `gemma4-assistant` 草稿 GGUF，`--mtp-draft-model`）；DeepSeek V4 则新增 **DSpark** 块级起草（`--draft-model`），每步提议一整块 token，decode 提速 **1.3–1.4×**（多轮对话最高 2.0×）。三者都是草稿提议、主干一次批量前向验证，输出与标准 decode 一致。→ [投机解码](FEATURES_zh-cn.md#mtp--nextn-投机解码) - **🔗 张量并行与分布式集群。** 用 `--tp N` 把一个模型切分到多张 GPU 上——Direct `cuda` 后端**以及** GGML CUDA / Vulkan 后端均支持——再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。采用 Megatron-LM 列/行并行范式与分层 AllReduce；GGML 上提供 MoE 专家并行与按 rank 的 GatedDeltaNet 融合内核。融合式按 rank 执行让 Gemma 4 E4B 上 `--tp 2` 的 decode 达到单卡的 **1.39×**、Muse-Glimmer 30B 达到 **1.57×**（其 prefill 同时提升 **1.34×**——是这里唯一在两个阶段都超过单卡的模型），也让单卡装不下的模型（Qwen 3.5-35B-A3B；24 GB 卡上 28.2 GB 的 Muse-Glimmer 30B Q8_0）得以运行。可选 Redis 支撑的 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理) -- **🎨 Qwen-Image-Edit 图像编辑。** 提示词 + 输入图像 → 编辑后的图像，驱动 60 块 MMDiT，配以 Qwen-Image VAE 与 Qwen2.5-VL-7B 文本编码器。CUDA 图捕获的整 DiT、FlowMatch-Euler true-CFG 去噪、Web UI 实时预览，以及 Lightning-LoRA 快速路径。热态 4 步编辑比 `stable-diffusion.cpp` 快 **1.19×**。→ [Qwen-Image-Edit 卡片](docs/models/qwenimage_zh-cn.md) -- **🎬 Wan 2.1 / 2.2 视频生成（文本→视频、图像→视频）。** 提示词 → H.264 MP4；Wan 2.2（TI2V-5B、I2V-A14B）上上传的图像作为首帧，提示词驱动运动、镜头与场景变化。每个去噪步一张常驻权重的 ggml 图（CUDA 图捕获、flash attention），因果 3D 视频 VAE 编/解码各一张图，A14B 的两个 14B 专家在时间步边界热切换，分阶段显存交接——TI2V-5B 在 16 GB GPU 上 8 分钟内生成 81 帧 480p 图生视频，Wan 2.1 端到端比 `stable-diffusion.cpp` 快 **6.0×**。→ [Wan 卡片](docs/models/wan_zh-cn.md) +- **🎨 Qwen-Image-Edit 图像编辑。** 提示词 + 输入图像 → 编辑后的图像，驱动 60 块 MMDiT，配以 Qwen-Image VAE 与 Qwen2.5-VL-7B 文本编码器。CUDA 图捕获的整 DiT、FlowMatch-Euler true-CFG 去噪、Web UI 实时预览，以及 [Lightning 蒸馏 LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) 快速路径（`--qwen-image-lora`，以运行期旁路的形式挂在原封不动的量化权重旁），把默认的 30 步 × CFG——即 60 次 DiT 前向——降到 **4** 次。热态 4 步编辑比 `stable-diffusion.cpp` 快 **1.19×**。→ [Qwen-Image-Edit 卡片](docs/models/qwenimage_zh-cn.md) +- **🎬 Wan 2.1 / 2.2 视频生成（文本→视频、图像→视频）。** 提示词 → H.264 MP4；Wan 2.2（TI2V-5B、I2V-A14B）上上传的图像作为首帧，提示词驱动运动、镜头与场景变化。每个去噪步一张常驻权重的 ggml 图（CUDA 图捕获、flash attention），因果 3D 视频 VAE 编/解码各一张图，A14B 的两个 14B 专家在时间步边界热切换，分阶段显存交接——TI2V-5B 在 16 GB GPU 上 8 分钟内生成 81 帧 480p 图生视频，Wan 2.1 端到端比 `stable-diffusion.cpp` 快 **6.0×**。**步数蒸馏检查点会按 DiT 文件名自动识别**（`Turbo` / `distill` / `Lightning` / `lightx2v` / `FastWan` / `…-4steps-…`），并切换到该步数、关闭引导——4 次 DiT 前向而不是官方配方的 100 次，同一个 1088×832×121 帧的图生视频请求因此从 **3 小时 30 分降到 17 分 30 秒**（M5 Pro）。这是本仓库中最大的单项提速手段，不需要任何参数，只是换一个 `--model` 文件。数值已对照 diffusers 验证（DiT 余弦 > 0.995，VAE 编码器余弦 > 0.999，解码 59.9 dB / >35 dB PSNR）。支持 CLI（`--image`）、`/v1/videos/generations` 与可上传图片的 Web UI 聊天。→ [Wan 卡片](docs/models/wan_zh-cn.md) - **🌫️ DiffusionGemma 文本扩散。** 基于 Gemma-4 派生 MoE backbone 的分块 EntropyBound 去噪，提供 CLI 参数与 Web UI 实时去噪预览。→ [DiffusionGemma 卡片](docs/models/diffusiongemma_zh-cn.md) - **🖼️ 多模态。** 图像 / 视频 / 音频（Gemma 4）；图像输入（Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni、Muse-Glimmer）；CLI 与 Web UI 支持 PDF。→ [多模态](FEATURES_zh-cn.md#多模态支持) - **🛠️ 工具调用与思维链。** Qwen 3、Qwen 3.5/3.6-family、Gemma 4、GPT OSS、Nemotron-H、Muse-Glimmer（ATEM 标记）、DeepSeek V4（DSML 标记）均支持多轮工具调用与结构化思维链。→ [功能特性](FEATURES_zh-cn.md) @@ -160,8 +160,29 @@ dotnet run --project TensorSharp.Server -c Release -- --help | Gemma 3 | [gemma-3-4b-it](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | ✅ / — / — | — | — | [gemma3](docs/models/gemma3_zh-cn.md) | | DiffusionGemma | [diffusiongemma-26B-A4B-it](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | — / — / — | — | — | [diffusiongemma](docs/models/diffusiongemma_zh-cn.md) | | Muse-Glimmer | [Muse-Glimmer-30B](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)（+ mmproj） | ✅ / — / — | ✅ | ✅ | [muse-glimmer](docs/models/muse-glimmer_zh-cn.md) | -| Qwen-Image-Edit | [Qwen-Image-Edit-2511](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)（MMDiT + VAE + Qwen2.5-VL） | 🖼️ 图像→图像 | — | — | [qwenimage](docs/models/qwenimage_zh-cn.md) | -| Wan 2.1 / 2.2 视频 | [Wan2.2-TI2V-5B](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)（另有 [I2V-A14B](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)、[Wan2.1-T2V-14B](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)）+ UMT5-XXL + 视频 VAE | 🎬 文本→视频、图像→视频 | — | — | [wan](docs/models/wan_zh-cn.md) | +| Qwen-Image-Edit | [Qwen-Image-Edit-2511](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)（MMDiT + VAE + Qwen2.5-VL）· 快速路径：[Lightning 4 步 LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) | 🖼️ 图像→图像 | — | — | [qwenimage](docs/models/qwenimage_zh-cn.md) | +| Wan 2.1 / 2.2 视频 | [Wan2.2-TI2V-5B](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)（另有 [T2V-A14B](https://huggingface.co/QuantStack/Wan2.2-T2V-A14B-GGUF)、[I2V-A14B](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)、[Wan2.1-T2V-14B](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)）+ UMT5-XXL + 视频 VAE · 快速路径：[TI2V-5B-Turbo](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF)（4 步，DiT 前向次数减少 25×） | 🎬 文本→视频、图像→视频 | — | — | [wan](docs/models/wan_zh-cn.md) | + +## 让它跑得更快 + +有几个家族存在一条**快速路径**——换一个可下载的权重文件，或加一个参数——就能把一次运行的开销降低一个数量级。在做其他调优之前，先看这张表。 + +| 家族 | 快速路径的权重或参数 | 实测效果 | +|---|---|---| +| **Wan 2.1 / 2.2 视频** | **步数蒸馏的 DiT GGUF**——TI2V-5B 用 [hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF)（`Wan2_2-TI2V-5B-Turbo-Q8_0.gguf`），A14B 用 [jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF)。无需任何参数——按文件名自动识别。 | 100 次 DiT 前向 → **4** 次，且关闭引导。同一个 1088×832×121 帧的图生视频请求：**3 小时 30 分 → 17 分 30 秒**（M5 Pro，`ggml_metal`）。 | +| Wan，仅基础权重 | `--cfg-cache-stride 2` / `3` | 50 步下 1.30× / 1.43×（近似方法；蒸馏权重本身已无 guidance，用它没有意义）。 | +| Wan，任意权重 | 在训练分辨率上生成后再下采样——用 736×544 代替 1088×832 | 121 帧、Turbo 权重：**6 分 19 秒**，而非 17 分 30 秒。但低于约 0.3 MP 反而会掉质量。 | +| **Qwen-Image-Edit** | `--qwen-image-lora` 加载 [Lightning 4 步 LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning)（`Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors`） | 采样默认值从 30 步 / CFG 2.5（60 次 DiT 前向）切换为 **4** 步 / CFG 1.0。热态 4 步编辑比 `stable-diffusion.cpp` 快 **1.19×**。 | +| Qwen-Image-Edit | `TS_QWEN_DIT_CACHE_MODE=easycache`（默认关闭——质量优先） | 可跳过 40–55% 的去噪步；但会让编辑结果的细节（如人脸）明显变软，因此需显式开启。 | +| **DeepSeek V4 Flash** | `--draft-model` 加载 [DSpark 草稿 GGUF](https://huggingface.co/sakamakismile/DeepSeek-V4-Flash-DSpark-support-ds4-GGUF)（服务端再加 `--mtp-spec`）；仅 `cuda` / `ggml_cuda` | 4×A40 上 decode 从 26.4 提升到 **37.1 tok/s**（1.41×），接受率 69%；多轮对话最高 **2.0×**。输出不变——主干会逐块验证。 | +| **Muse-Glimmer** | `--draft-model` 加载 DFlash 草稿模型（`dflash-kquant.gguf`，位于 [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)）；**不要**传任何采样参数 | 在其开发所用的 CUDA 主机上 decode 提升 1.3–5×——单张 RTX PRO 6000、60 token 提示、贪心解码下由 35.0 提升到 **50.9 tok/s**。Apple Silicon 上目前普通 decode 仍更快。 | +| **Qwen 3.6** | 保留 MTP 块的 GGUF——[unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)，而非基础仓库——再加服务端的 `--mtp-spec` | 启用单序列上的 NextN 投机解码。基础仓库文件名相同但已剥离该块，会静默回退到普通 decode。 | +| **Gemma 4** | `--mtp-draft-model` 加载配套的 [`gemma4-assistant` 草稿 GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF)，并加 `--mtp-spec`（服务端） | 在 GGML 各后端与 Direct `cuda` 后端上启用投机解码。草稿与目标的 hidden size 必须一致，否则启动即失败。 | +| 显存装不下的 MoE | `--n-cpu-moe N` / `--cpu-moe` | 16 GB 笔记本显卡上 gpt-oss-20b 显存从 16.2 GB 降到 2.9 GB，把 WDDM 换页造成的 0.3 tok/s 变成 `--n-cpu-moe 12` 下的 **25.4 tok/s**。 | +| 多 GPU | `--tp N` | Gemma 4 E4B decode 达单卡的 **1.39×**，Muse-Glimmer 30B decode **1.57×** / prefill **1.34×**——并且能跑单卡装不下的模型。 | +| 所有家族 | 选对后端：NVIDIA 用 `ggml_cuda`，Apple Silicon 用 `ggml_metal`，无 GPU 用 `ggml_cpu`（而非 `cpu`） | Gemma 4 26B-A4B 在 `ggml_cuda` 上 decode 78.7 tok/s，Direct `cuda` 后端仅 35.3；Apple Silicon 上 Muse-Glimmer 30B 的 prefill 在 `ggml_metal` 上为 413.6 tok/s，MLX 上为 29.0。 | + +每一行背后的完整数据与逐家族细节：[Wan](docs/models/wan_zh-cn.md) · [Qwen-Image-Edit](docs/models/qwenimage_zh-cn.md) · [DeepSeek V4](docs/models/deepseek4_zh-cn.md) · [Muse-Glimmer](docs/models/muse-glimmer_zh-cn.md) · [功能特性](FEATURES_zh-cn.md)。 ## 支持的模型架构 @@ -170,15 +191,15 @@ dotnet run --project TensorSharp.Server -c Release -- --help | DeepSeek V4 Flash | `deepseek4` | DeepSeek-V4-Flash（284B MoE，256 专家，压缩稀疏注意力，1M 上下文） | 仅文本 | 支持 | 支持（DSML） | 支持（DSpark 块级草稿，独立 GGUF） | [deepseek4](docs/models/deepseek4_zh-cn.md) | | Gemma 4 | `gemma4` | gemma-4-E4B、gemma-4-31B、gemma-4-26B-A4B（MoE） | 图像、视频、音频 | 支持 | 支持 | 支持（独立草稿 GGUF） | [gemma4](docs/models/gemma4_zh-cn.md) | | Gemma 3 | `gemma3` | gemma-3-4b | 图像 | 不支持 | 不支持 | — | [gemma3](docs/models/gemma3_zh-cn.md) | -| Qwen 3 | `qwen3` | Qwen3-4B | 仅文本 | 支持 | 支持 | — | [qwen3](docs/models/qwen3_zh-cn.md) | +| Qwen 3 | `qwen3`、`qwen2`、`qwen2vl`、`qwen2_vl` | Qwen3-4B（Qwen2 / Qwen2.5-VL 的 GGUF 也能加载，按纯文本对话运行） | 仅文本 | 支持 | 支持 | — | [qwen3](docs/models/qwen3_zh-cn.md) | | Qwen 3.5 / 3.6 family | `qwen35`, `qwen35moe`, `qwen3next` | Qwen3.5-9B（混合 Attn+递归）、Qwen3.5/3.6-35B-A3B（MoE） | 图像 | 支持 | 支持 | Qwen 3.6 支持（内嵌 NextN） | [qwen35](docs/models/qwen35_zh-cn.md) | | GPT OSS | `gptoss`, `gpt-oss` | gpt-oss-20b（MoE） | 仅文本 | 支持（始终） | 支持 | — | [gptoss](docs/models/gptoss_zh-cn.md) | | Nemotron-H | `nemotron_h`, `nemotron_h_moe` | Nemotron-H-8B/47B（混合 SSM-Transformer，MoE）、Nemotron 3 Nano Omni | 图像（Omni） | 支持 | 支持 | — | [nemotron](docs/models/nemotron_zh-cn.md) | | Mistral 3 | `mistral3` | Mistral-Small-3.1-24B-Instruct | 图像 | 不支持 | 不支持 | — | [mistral3](docs/models/mistral3_zh-cn.md) | -| Muse-Glimmer | `muse-glimmer` | Muse-Glimmer-30B（交错滑动窗口 + NoPE 全注意力层，注意力输出门控） | 图像 | 支持 | 支持（ATEM） | 支持（DFlash 块级草稿，独立 GGUF） | [muse-glimmer](docs/models/muse-glimmer_zh-cn.md) | -| DiffusionGemma | `diffusion-gemma` | diffusion-gemma 文本扩散 GGUF | 仅文本 | 不支持 | 不支持 | — | [diffusiongemma](docs/models/diffusiongemma_zh-cn.md) | -| Qwen-Image-Edit | `qwen_image` | qwen-image-edit MMDiT GGUF（+ VAE 与 Qwen2.5-VL） | 图像编辑（图像+文本 → 图像） | 不支持 | 不支持 | — | [qwenimage](docs/models/qwenimage_zh-cn.md) | -| Wan 视频 | `wan` | Wan 2.1 T2V 1.3B/14B、Wan 2.2 TI2V-5B、Wan 2.2 A14B T2V/I2V（双专家） | 视频输出（文本→视频、图像→视频） | 不支持 | 不支持 | — | [wan](docs/models/wan_zh-cn.md) | +| Muse-Glimmer | `muse-glimmer`、`muse_glimmer` | Muse-Glimmer-30B（交错滑动窗口 + NoPE 全注意力层，注意力输出门控） | 图像 | 支持 | 支持（ATEM） | 支持（DFlash 块级草稿，独立 GGUF） | [muse-glimmer](docs/models/muse-glimmer_zh-cn.md) | +| DiffusionGemma | `diffusion-gemma`、`diffusion_gemma` | diffusion-gemma 文本扩散 GGUF | 仅文本 | 不支持 | 不支持 | — | [diffusiongemma](docs/models/diffusiongemma_zh-cn.md) | +| Qwen-Image-Edit | `qwen_image`、`qwen-image` | qwen-image-edit MMDiT GGUF（+ VAE 与 Qwen2.5-VL） | 图像编辑（图像+文本 → 图像） | 不支持 | 不支持 | — | [qwenimage](docs/models/qwenimage_zh-cn.md) | +| Wan 视频 | `wan`、`wan2.1`、`wan2.2` | Wan 2.1 T2V 1.3B/14B、Wan 2.2 TI2V-5B、Wan 2.2 A14B T2V/I2V（双专家） | 视频输出（文本→视频、图像→视频） | 不支持 | 不支持 | — | [wan](docs/models/wan_zh-cn.md) | 各架构的端到端文档（前向图、组件、参数、prefill/decode 优化）见[按模型架构卡片](docs/models/README_zh-cn.md)。 @@ -224,12 +245,12 @@ TensorSharp 在 CUDA 的 prefill / 首 token 延迟上明显领先（多轮 pref | 范围 | 状态 | |---|---| -| 模型家族 | DeepSeek V4 Flash（`deepseek4`）、Gemma 3/4、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family（`qwen35`、`qwen35moe`、`qwen3next`）、GPT OSS、Nemotron-H（含 Nemotron 3 Nano Omni）、Mistral 3、Muse-Glimmer（`muse-glimmer`）。图像编辑通过 Qwen-Image-Edit（`qwen_image` MMDiT）；视频生成通过 Wan 2.1 / 2.2（`wan`）。 | +| 模型家族 | DeepSeek V4 Flash（`deepseek4`）、Gemma 3/4、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family（`qwen35`、`qwen35moe`、`qwen3next`）、GPT OSS、Nemotron-H（含 Nemotron 3 Nano Omni）、Mistral 3、Muse-Glimmer（`muse-glimmer`、`muse_glimmer`）。图像编辑通过 Qwen-Image-Edit（`qwen_image`、`qwen-image` MMDiT）；视频生成通过 Wan 2.1 / 2.2（`wan`、`wan2.1`、`wan2.2`）。 | | 推理宿主 | CLI、交互式 REPL、ASP.NET Core Web UI、Ollama 风格 API、OpenAI Chat Completions 风格 API。 | -| 后端 | 纯 C# CPU、Direct CUDA/cuBLAS（`cuda`）、MLX Metal（`mlx`）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C# CPU——都会把权重按层切分到所有可见 GPU（`--tp N` / `TS_DSV4_NGPU` 限定卡数）。 | +| 后端 | 纯 C# CPU、Direct CUDA/cuBLAS（`cuda`）、MLX Metal（`mlx`）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C# CPU——都会把权重按层切分到所有可见 GPU（`--tp N` / `TS_DSV4_NGPU` 限定卡数）。Wan 是唯一对后端有限制的家族：它可运行于各 GGML 后端以及 Direct `cuda` / 纯 C# `cpu` 后端，但不支持 MLX。 | | 多模态 | Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni、Muse-Glimmer 图像输入；PDF（CLI `--pdf` + Web UI）。媒体*输出*：Qwen-Image-Edit（图像）与 Wan 2.1 / 2.2（H.264 MP4 视频，文本→视频与图像→视频）。 | | 连续批处理 | vLLM 风格分页 KV 缓存、基于内容哈希的前缀共享、迭代级调度器（默认启用，`--no-continuous-batching` 关闭）。DeepSeek V4 在同一引擎上通过其原生 per-sequence slot 提供服务。 | -| 投机解码 | Qwen 3.6（内嵌）与 Gemma 4（独立草稿 GGUF）的 MTP / NextN 草稿头；DeepSeek V4 的 DSpark 与 Muse-Glimmer 的 DFlash 块级起草（`--draft-model` 指定独立草稿 GGUF，`cuda` / `ggml_cuda`）。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致。默认关闭；服务端用 `--mtp-spec` 启用，CLI 直接传 `--draft-model` 即可。 | +| 投机解码 | Qwen 3.6（内嵌）与 Gemma 4（独立草稿 GGUF）的 MTP / NextN 草稿头；DeepSeek V4 的 DSpark 块级起草（仅 `cuda` / `ggml_cuda`）与 Muse-Glimmer 的 DFlash 块级起草，两者都通过 `--draft-model` 加载独立的草稿 GGUF。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致。默认关闭；服务端用 `--mtp-spec` 启用，CLI 直接传 `--draft-model` 即可。 | | 张量并行 | Direct `cuda` 后端与 GGML CUDA / Vulkan 后端上的 Megatron-LM 列/行并行 TP（`--tp N` / `TENSORSHARP_TP_DEGREE`，CLI 与服务端均支持）；通过点对点 TCP 的多节点分布式 TP（`--tp-node-id` / `--tp-peers`），采用分层 AllReduce，CUDA P2P 不可用时自动回退到主机中转。覆盖全部自回归架构；GGML 上 Gemma 4 与 Qwen 3.5/3.6 使用 MoE 专家并行与融合的按 rank decode/prefill 计算图。可选 Redis 支撑的 KV 缓存与 Responses API 存储。 | | 服务端模型范围 | 通过 `--model` 显式托管单个 GGUF；可通过 `--mmproj` 显式指定投影器；不扫描目录。 | | 可观测性 | 结构化每轮日志、队列状态，以及 Web UI / Ollama / OpenAI 中的 KV 缓存复用指标。 | diff --git a/TensorSharp.GGML.Native/CMakeLists.txt b/TensorSharp.GGML.Native/CMakeLists.txt index 1c683d39..30d2672a 100644 --- a/TensorSharp.GGML.Native/CMakeLists.txt +++ b/TensorSharp.GGML.Native/CMakeLists.txt @@ -231,6 +231,8 @@ if (APPLE) # Raises the process-wide default MSL version so ggml's Metal 4 tensor-API probe can compile # under the .NET apphost's stale recorded SDK. See the file header for the full story. list(APPEND TSG_GGMLOPS_SOURCES tsg_metal_msl_default.m) + # Wan VAE convolutions on MPSGraph (see tsg_metal_mps_conv.mm). + list(APPEND TSG_GGMLOPS_SOURCES tsg_metal_mps_conv.mm) endif() if (TENSORSHARP_GGML_NATIVE_ENABLE_CUDA) list(APPEND TSG_GGMLOPS_SOURCES ggml_ops_diffusion_sample.cu) @@ -238,6 +240,19 @@ if (TENSORSHARP_GGML_NATIVE_ENABLE_CUDA) list(APPEND TSG_GGMLOPS_SOURCES ggml_ops_dsv4_fused.cu) list(APPEND TSG_GGMLOPS_SOURCES ggml_ops_tp_probe.cu) list(APPEND TSG_GGMLOPS_SOURCES ggml_ops_host_pin.cu) + # Wan VAE convolutions on cuDNN (see tsg_cuda_cudnn_conv.cu). Optional: the + # VAE falls back to ggml's im2col+GEMM when cuDNN is not present. + find_path(TSG_CUDNN_INCLUDE_DIR cudnn.h + HINTS ${CUDAToolkit_INCLUDE_DIRS} /usr/include /usr/local/cuda/include) + find_library(TSG_CUDNN_LIBRARY NAMES cudnn + HINTS /usr/lib/x86_64-linux-gnu /usr/local/cuda/lib64) + if (TSG_CUDNN_INCLUDE_DIR AND TSG_CUDNN_LIBRARY) + message(STATUS "TensorSharp: cuDNN found (${TSG_CUDNN_LIBRARY}); Wan VAE convolutions will use it") + list(APPEND TSG_GGMLOPS_SOURCES tsg_cuda_cudnn_conv.cu) + set(TSG_HAVE_CUDNN ON) + else() + message(STATUS "TensorSharp: cuDNN not found; Wan VAE convolutions stay on ggml im2col+GEMM") + endif() endif() add_library(GgmlOps SHARED ${TSG_GGMLOPS_SOURCES}) @@ -280,6 +295,11 @@ endif() if (TENSORSHARP_GGML_NATIVE_ENABLE_CUDA) target_compile_definitions(GgmlOps PRIVATE TSG_GGML_USE_CUDA=1) + if (TSG_HAVE_CUDNN) + target_include_directories(GgmlOps PRIVATE ${TSG_CUDNN_INCLUDE_DIR}) + target_link_libraries(GgmlOps PRIVATE ${TSG_CUDNN_LIBRARY}) + target_compile_definitions(GgmlOps PRIVATE TSG_HAVE_CUDNN=1) + endif() # Fast intrinsic transcendentals (tanhf/expf) for the diffusion sampler kernel — the accurate # library tanhf is ~100x slower on-device and the sampler only needs FP-reduction-level accuracy. target_compile_options(GgmlOps PRIVATE $<$:--use_fast_math>) @@ -296,7 +316,10 @@ if (APPLE) # tsg_metal_msl_default.m talks to Metal and the Objective-C runtime directly. find_library(TSG_FOUNDATION_LIBRARY Foundation REQUIRED) find_library(TSG_METAL_FRAMEWORK Metal REQUIRED) - target_link_libraries(GgmlOps PRIVATE ${TSG_FOUNDATION_LIBRARY} ${TSG_METAL_FRAMEWORK}) + find_library(TSG_MPSGRAPH_FRAMEWORK MetalPerformanceShadersGraph REQUIRED) + find_library(TSG_MPS_FRAMEWORK MetalPerformanceShaders REQUIRED) + target_link_libraries(GgmlOps PRIVATE ${TSG_FOUNDATION_LIBRARY} ${TSG_METAL_FRAMEWORK} + ${TSG_MPSGRAPH_FRAMEWORK} ${TSG_MPS_FRAMEWORK}) endif() set_target_properties(GgmlOps PROPERTIES diff --git a/TensorSharp.GGML.Native/build-linux.sh b/TensorSharp.GGML.Native/build-linux.sh index 9ae23cf0..f90d9b15 100644 --- a/TensorSharp.GGML.Native/build-linux.sh +++ b/TensorSharp.GGML.Native/build-linux.sh @@ -43,6 +43,24 @@ has_cuda_toolkit() { # libvulkan.so over the runtime soname libvulkan.so.1 — CMake accepts a full # path in Vulkan_LIBRARY either way). Fails when no loader is installed, which # is the "this machine does not support Vulkan" signal for auto-detection. +# Is a Vulkan BUILD toolchain already present? A loader (libvulkan.so) only says +# the machine can RUN Vulkan -- every NVIDIA driver install ships one -- so it is +# the wrong signal for auto-enabling the backend: on a CUDA-only box it sent the +# build off to download and compile shaderc/SPIRV-Headers before it ever reached +# ggml-cuda. Auto-enable only when the pieces needed to compile the shaders are +# already installed; `--vulkan` still opts in explicitly and will provision them. +have_vulkan_build_toolchain() { + if [[ -n "${VULKAN_SDK:-}" && -f "${VULKAN_SDK}/include/vulkan/vulkan.h" && -x "${VULKAN_SDK}/bin/glslc" ]]; then + return 0 + fi + command -v glslc >/dev/null 2>&1 || return 1 + local dir + for dir in /usr/include /usr/local/include; do + [[ -f "${dir}/vulkan/vulkan.h" ]] && return 0 + done + return 1 +} + find_vulkan_loader_library() { local candidates=() if command -v ldconfig >/dev/null 2>&1; then @@ -268,10 +286,14 @@ elif [[ "$(read_cached_backend_setting TENSORSHARP_GGML_NATIVE_VULKAN_EXPLICIT)" fi fi if [[ -z "${ENABLE_VULKAN}" ]]; then - if [[ -n "${VULKAN_SDK:-}" ]] || find_vulkan_loader_library >/dev/null; then + if have_vulkan_build_toolchain; then ENABLE_VULKAN=ON else ENABLE_VULKAN=OFF + if find_vulkan_loader_library >/dev/null; then + echo "note: a Vulkan loader is present but no build toolchain (glslc + vulkan headers);" >&2 + echo " building without ggml-vulkan. Pass --vulkan to provision the toolchain and include it." >&2 + fi fi fi if [[ "${ENABLE_VULKAN}" == "ON" ]] && ! prepare_vulkan_toolchain; then diff --git a/TensorSharp.GGML.Native/ggml_ops_core.cpp b/TensorSharp.GGML.Native/ggml_ops_core.cpp index c91ec48d..b5907949 100644 --- a/TensorSharp.GGML.Native/ggml_ops_core.cpp +++ b/TensorSharp.GGML.Native/ggml_ops_core.cpp @@ -2267,6 +2267,174 @@ namespace tsg } } + + // True when the process-global GGML backend is ggml-metal ("MTL0", "MTL1", ...). + static bool backend_is_metal() + { + const char* name = g_backend != nullptr ? ggml_backend_name(g_backend) : nullptr; + return name != nullptr && std::strncmp(name, "MTL", 3) == 0; + } + + // Wan VAE convolutions on MPSGraph (tsg_metal_mps_conv.mm). ggml lowers conv2d + // to im2col + mul_mat, which is 74.6% of a VAE decode and moves 9x the input + // before the GEMM starts; Apple's tuned convolution runs the same shapes 6-14x + // faster and reaches the matrix units WITHOUT ggml's mul_mm kernel, whose Metal 4 + // tensor path corrupts this graph. TS_VAE_MPS_CONV=0 opts out. + #if defined(__APPLE__) + extern "C" bool tsg_mps_conv2d_available(void); + extern "C" bool tsg_mps_conv2d(const void* w, int wIsF16, int kw, int kh, int ic, int oc, + const float* x, int W, int H, int T, + int stride, int pad, + float* dst, int OW, int OH); + extern "C" void tsg_mps_conv2d_release(void); + #endif + + #if defined(TSG_HAVE_CUDNN) + // The CUDA twin (tsg_cuda_cudnn_conv.cu). On an L4 the im2col lowering is 46% of + // a VAE decode against only 24.5% for the GEMM, so cuBLAS speed does not help -- + // the materialisation itself is the cost. TS_VAE_CUDNN_CONV=0 opts out. + extern "C" bool tsg_cudnn_conv2d_available(void); + extern "C" bool tsg_cudnn_conv2d(const void* w, int wIsF16, int kw, int kh, int ic, int oc, + const void* x, int W, int H, int T, + int stride, int pad, + void* dst, int OW, int OH); + extern "C" void tsg_cudnn_conv2d_release(void); + #endif + + // True when the process-global GGML backend is ggml-cuda. + static bool backend_is_cuda() + { + const char* name = g_backend != nullptr ? ggml_backend_name(g_backend) : nullptr; + return name != nullptr && std::strncmp(name, "CUDA", 4) == 0; + } + + // True when a VAE should emit single CONV_2D nodes for a vendor + // convolution library to execute instead of ggml's im2col + mul_mat lowering. + bool fast_conv_enabled() + { + #if defined(__APPLE__) + if (backend_is_metal()) + { + static const bool on = []{ + const char* e = std::getenv("TS_VAE_MPS_CONV"); + if (e != nullptr && e[0] == '0') return false; + return tsg_mps_conv2d_available(); + }(); + return on; + } + #endif + #if defined(TSG_HAVE_CUDNN) + if (backend_is_cuda()) + { + static const bool on = []{ + const char* e = std::getenv("TS_VAE_CUDNN_CONV"); + if (e != nullptr && e[0] == '0') return false; + return tsg_cudnn_conv2d_available(); + }(); + return on; + } + #endif + return false; + } + + + // Run one CONV_2D node through the backend's convolution library (MPSGraph on + // Metal, cuDNN on CUDA). ggml's kernel is [KW,KH,IC,OC] F16 and + // the activation [W,H,IC,T] F32, which are MPS OIHW / NCHW byte for byte, so the + // operands go across as they lie. Returns false for anything unsupported, and + // the caller then lets ggml execute the node normally. + static bool run_conv_fast(ggml_tensor* node) + { + ggml_tensor* kern = node->src[0]; + ggml_tensor* act = node->src[1]; + if (kern == nullptr || act == nullptr) return false; + const bool kernF16 = kern->type == GGML_TYPE_F16; + if ((!kernF16 && kern->type != GGML_TYPE_F32) || act->type != GGML_TYPE_F32 || node->type != GGML_TYPE_F32) + return false; + if (!ggml_is_contiguous(kern) || !ggml_is_contiguous(act) || !ggml_is_contiguous(node)) + return false; + + const int32_t* op = (const int32_t*) node->op_params; + const int s0 = op[0], s1 = op[1], p0 = op[2], p1 = op[3], d0 = op[4], d1 = op[5]; + if (s0 != s1 || p0 != p1 || d0 != 1 || d1 != 1) return false; // square stride/pad only + + const int kw = (int) kern->ne[0], kh = (int) kern->ne[1]; + const int ic = (int) kern->ne[2], oc = (int) kern->ne[3]; + const int W = (int) act->ne[0], H = (int) act->ne[1], T = (int) act->ne[3]; + const int OW = (int) node->ne[0], OH = (int) node->ne[1]; + if ((int) act->ne[2] != ic || (int) node->ne[2] != oc || (int) node->ne[3] != T) return false; + + #if defined(TSG_HAVE_CUDNN) + if (backend_is_cuda()) + { + // ggml tensor data is already a device pointer here, so cuDNN reads and + // writes ggml's own buffers -- no staging, no copies. + return tsg_cudnn_conv2d(kern->data, kernF16 ? 1 : 0, kw, kh, ic, oc, + act->data, W, H, T, s0, p0, + node->data, OW, OH); + } + #endif + #if defined(__APPLE__) + if (backend_is_metal()) + { + std::vector kbuf((std::size_t) ggml_nbytes(kern)); + std::vector ah((std::size_t) ggml_nelements(act)); + std::vector oh((std::size_t) ggml_nelements(node)); + ggml_backend_tensor_get(kern, kbuf.data(), 0, kbuf.size()); + ggml_backend_tensor_get(act, ah.data(), 0, ah.size() * sizeof(float)); + if (!tsg_mps_conv2d(kbuf.data(), kernF16 ? 1 : 0, kw, kh, ic, oc, + ah.data(), W, H, T, s0, p0, + oh.data(), OW, OH)) + return false; + ggml_backend_tensor_set(node, oh.data(), 0, oh.size() * sizeof(float)); + return true; + } + #endif + (void) kw; (void) kh; (void) ic; (void) oc; (void) W; (void) H; (void) T; (void) OW; (void) OH; + return false; + } + + + // Execute a graph with every CONV_2D node handed to the platform convolution + // library (MPSGraph on Metal, cuDNN on CUDA) and the stretches between them + // left to ggml. Running node ranges through ggml_graph_view is the same + // mechanism graph_compute_profiled uses, so it is safe against gallocr's + // buffer reuse. + ggml_status graph_compute_fast_conv(ggml_cgraph* graph, const char* tag) + { + (void) tag; + const int n = ggml_graph_n_nodes(graph); + int from = 0; + for (int i = 0; i < n; i++) + { + ggml_tensor* node = ggml_graph_node(graph, i); + if (node->op != GGML_OP_CONV_2D) continue; + + if (i > from) + { + ggml_cgraph view = ggml_graph_view(graph, from, i); + const ggml_status st = ggml_backend_graph_compute(g_backend, &view); + if (st != GGML_STATUS_SUCCESS) return st; + } + ggml_backend_synchronize(g_backend); + if (!run_conv_fast(node)) + { + // Unsupported shape: let ggml run just this node. + ggml_cgraph one = ggml_graph_view(graph, i, i + 1); + const ggml_status st = ggml_backend_graph_compute(g_backend, &one); + if (st != GGML_STATUS_SUCCESS) return st; + } + from = i + 1; + } + if (from < n) + { + ggml_cgraph view = ggml_graph_view(graph, from, n); + const ggml_status st = ggml_backend_graph_compute(g_backend, &view); + if (st != GGML_STATUS_SUCCESS) return st; + } + return GGML_STATUS_SUCCESS; + } + ggml_status graph_compute_profiled(ggml_backend_t backend, ggml_cgraph* graph, const char* tag) { if (!graph_node_profile_enabled()) diff --git a/TensorSharp.GGML.Native/ggml_ops_internal.h b/TensorSharp.GGML.Native/ggml_ops_internal.h index f363a45c..51d25a63 100644 --- a/TensorSharp.GGML.Native/ggml_ops_internal.h +++ b/TensorSharp.GGML.Native/ggml_ops_internal.h @@ -936,6 +936,14 @@ namespace tsg bool graph_node_profile_enabled(); ggml_status graph_compute_profiled(ggml_backend_t backend, ggml_cgraph* graph, const char* tag); + // Platform convolution offload for the VAEs (MPSGraph on Metal, cuDNN on + // CUDA). ggml lowers conv2d to im2col + mul_mat, which dominates a VAE graph + // and materialises 9x the input for a 3x3 kernel; the vendor libraries + // convolve directly. fast_conv_enabled() says whether to emit un-lowered + // CONV_2D nodes, and graph_compute_fast_conv() executes such a graph. + bool fast_conv_enabled(); + ggml_status graph_compute_fast_conv(ggml_cgraph* graph, const char* tag); + // ------------------------------------------------------------------ // Whole-model kernel phase timer (TS_GGML_PHASE_TIMING) // ------------------------------------------------------------------ diff --git a/TensorSharp.GGML.Native/ggml_ops_qwen_image.cpp b/TensorSharp.GGML.Native/ggml_ops_qwen_image.cpp index b90fcd6a..d9617cd5 100644 --- a/TensorSharp.GGML.Native/ggml_ops_qwen_image.cpp +++ b/TensorSharp.GGML.Native/ggml_ops_qwen_image.cpp @@ -257,9 +257,13 @@ TSG_EXPORT int TSGgml_QwenVaeRun(const TSGgmlQwenVaeDesc* d) const long long oh = (x->ne[1] + 2LL * p1 - op.kh) / op.sh + 1; const long long ow = (x->ne[0] + 2LL * p0 - op.kw) / op.sw + 1; const long long im2col = static_cast(op.ic) * op.kh * op.kw * oh * ow * 2; - ggml_tensor* y = im2col <= kIm2colBudget - ? ggml_conv_2d(ctx, ker, x, op.sw, op.sh, p0, p1, 1, 1) - : ggml_conv_2d_direct(ctx, ker, x, op.sw, op.sh, p0, p1, 1, 1); + // With MPSGraph/cuDNN available the convolution is executed + // whole, so emit the un-lowered node and skip im2col entirely. + ggml_tensor* y = tsg::fast_conv_enabled() + ? ggml_conv_2d_direct(ctx, ker, x, op.sw, op.sh, p0, p1, 1, 1) + : (im2col <= kIm2colBudget + ? ggml_conv_2d(ctx, ker, x, op.sw, op.sh, p0, p1, 1, 1) + : ggml_conv_2d_direct(ctx, ker, x, op.sw, op.sh, p0, p1, 1, 1)); if (op.b >= 0) { ggml_tensor* bt = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, op.oc); @@ -355,7 +359,10 @@ TSG_EXPORT int TSGgml_QwenVaeRun(const TSGgmlQwenVaeDesc* d) ggml_backend_tensor_set(input, d->input, 0, static_cast(d->in_w) * d->in_h * d->in_c * sizeof(float)); - if (ggml_backend_graph_compute(g_backend, graph) != GGML_STATUS_SUCCESS) + const ggml_status vaeSt = tsg::fast_conv_enabled() + ? tsg::graph_compute_fast_conv(graph, "qwen-image vae") + : ggml_backend_graph_compute(g_backend, graph); + if (vaeSt != GGML_STATUS_SUCCESS) { set_last_error("QwenVaeRun: graph compute failed."); return 0; } ggml_backend_synchronize(g_backend); ggml_backend_tensor_get(out, d->output, 0, static_cast(d->out_len) * sizeof(float)); diff --git a/TensorSharp.GGML.Native/ggml_ops_wan.cpp b/TensorSharp.GGML.Native/ggml_ops_wan.cpp index 4f161c24..7e617a28 100644 --- a/TensorSharp.GGML.Native/ggml_ops_wan.cpp +++ b/TensorSharp.GGML.Native/ggml_ops_wan.cpp @@ -39,6 +39,7 @@ // (patch embedding as a matmul, causal conv3d as per-temporal-tap 2D kernels). // ============================================================================ #include "ggml_ops_internal.h" +#include "ggml-impl.h" // ggml_graph_view, for the segmented MPS conv runner #include "ggml-alloc.h" #include @@ -970,22 +971,30 @@ static long long wan_vae_gemm_budget() const char* e = std::getenv("TS_WAN_VAE_GEMM_MAX_MB"); if (e != nullptr) return std::strtoll(e, nullptr, 10) * 1024 * 1024; - // ggml-metal's Metal 4 tensor-API mul_mm intermittently misreads operands - // inside the VAE decode graph on M5 (first pass, layout-dependent: the - // 32x32-latent all-NaN decode). It is correct in isolation and for every - // LLM/DiT-style graph, and has_tensor is fixed at device init, so it - // cannot be scoped per-op. When the tensor API is active, route the VAE - // convs through ggml_conv_2d_direct (budget 0): slower than im2col+GEMM, - // but correct — and it keeps the tensor API's much larger DiT/text-encoder - // wins. TS_WAN_VAE_GEMM_MAX_MB overrides in both directions. + // ggml-metal's Metal 4 tensor-API mul_mm misreads operands inside the VAE + // decode graph on M5 (first pass, buffer-layout dependent). When the tensor + // API is active, route the VAE convs through ggml_conv_2d_direct (budget 0): + // slower than im2col+GEMM, but correct. + // + // Do NOT "verify this is fixed" with an isolated decode. Tried 2026-08-17: + // WanVideoBench decoded synthetic latents at five shapes — including the + // 32x32 layout recorded as the original all-NaN repro — and tensor-API vs + // non-tensor output agreed to 91-93 dB PSNR with no NaN anywhere. It looked + // conclusively fixed. The very next full 1088x832x121f generation, with the + // DiT loaded and released before the decode, rendered 121 uniformly BLACK + // frames. The defect follows the allocation history, so only an end-to-end + // video is evidence. if (wan_metal_tensor_api_likely()) return 0; - // Non-CUDA device backends (Vulkan) stay at the banded floor: their - // drivers reject the multi-GB single gallocr arena that an unbanded - // full-plane im2col produces (Vulkan maxMemoryAllocationSize is commonly - // 4 GB or less), and the CPU backend gains nothing from bigger scratch. + // Vulkan stays at the banded floor: its drivers reject the multi-GB single + // gallocr arena an unbanded full-plane im2col produces (maxMemoryAllocationSize + // is commonly 4 GB or less). Metal and CUDA size the budget from free device + // memory — on Metal that measured 56.4s -> 49.7s for a 1088x832x9f decode, + // because it turns many small banded GEMMs into few large ones. const char* name = g_backend != nullptr ? ggml_backend_name(g_backend) : nullptr; - if (name == nullptr || std::strncmp(name, "CUDA", 4) != 0) + const bool sizedFromMemory = name != nullptr && + (std::strncmp(name, "CUDA", 4) == 0 || std::strncmp(name, "MTL", 3) == 0); + if (!sizedFromMemory) return 384LL << 20; std::size_t freeB = 0, totalB = 0; ggml_backend_dev_t dev = ggml_backend_get_device(g_backend); @@ -1062,6 +1071,10 @@ ggml_tensor* wan_vae_conv2d(WanVaeBuild& b, ggml_tensor* wt, ggml_tensor* x, int { const long long gemmMax = b.gemmMax; ggml_context* ctx = b.ctx; + // MPS executes the convolution whole, so emit the un-lowered node: this also + // skips the horizontal banding and its leading-pad shim entirely. + if (tsg::fast_conv_enabled()) + return ggml_conv_2d_direct(ctx, wt, x, stride, stride, pad, pad, 1, 1); if (gemmMax <= 0) return ggml_conv_2d_direct(ctx, wt, x, stride, stride, pad, pad, 1, 1); @@ -1616,7 +1629,12 @@ bool wan_vae_encode(const TSGgmlWanVaeEncodeDesc* d) for (auto& u : wbind.uploads) ggml_backend_tensor_set(u.t, u.d, 0, u.b); ggml_backend_tensor_set(xIn, d->x, 0, static_cast(pw) * ph * pc * pt * sizeof(float)); - if (ggml_backend_graph_compute(g_backend, graph) != GGML_STATUS_SUCCESS) + // The encoder shares wan_vae_conv2d, so on the MPS path its graph also holds + // un-lowered CONV_2D nodes and needs the same segmented runner. + const ggml_status encSt = tsg::fast_conv_enabled() + ? tsg::graph_compute_fast_conv(graph, "wan vae") + : ggml_backend_graph_compute(g_backend, graph); + if (encSt != GGML_STATUS_SUCCESS) { set_last_error("WanVaeEncode: graph compute failed."); return false; } ggml_backend_synchronize(g_backend); ggml_backend_tensor_get(outT, d->out, 0, static_cast(d->out_len) * sizeof(float)); @@ -1863,6 +1881,11 @@ bool wan_vae_decode(const TSGgmlWanVaeDecodeDesc* d) ggml_free(sctx); } } + else if (tsg::fast_conv_enabled()) + { + if (tsg::graph_compute_fast_conv(graph, "wan vae") != GGML_STATUS_SUCCESS) + { set_last_error("WanVaeDecode: graph compute failed (MPS conv path)."); return false; } + } else if (tsg::graph_compute_profiled(g_backend, graph, "wan vae decode") != GGML_STATUS_SUCCESS) { set_last_error("WanVaeDecode: graph compute failed."); return false; } ggml_backend_synchronize(g_backend); diff --git a/TensorSharp.GGML.Native/tsg_cuda_cudnn_conv.cu b/TensorSharp.GGML.Native/tsg_cuda_cudnn_conv.cu new file mode 100644 index 00000000..f4c94889 --- /dev/null +++ b/TensorSharp.GGML.Native/tsg_cuda_cudnn_conv.cu @@ -0,0 +1,237 @@ +// Copyright (c) Zhongkai Fu. All rights reserved. +// https://github.com/zhongkaifu/TensorSharp +// +// This file is part of TensorSharp. +// +// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree. +// +// Wan VAE convolutions on cuDNN. +// +// The CUDA twin of tsg_metal_mps_conv.mm. ggml lowers conv2d to im2col + +// mul_mat, and on CUDA that lowering is even more dominant than on Metal +// because cuBLAS makes the GEMM itself cheap. A decode profile on an L4 +// (benchmarks/WanVideoBench, 640x480x9f, ggml_cuda): +// +// IM2COL 46.0% 298 nodes <- materialising 9x the input +// MUL_MAT 24.5% 310 nodes +// CONT 11.8% 593 nodes +// +// So ~70% of a VAE decode goes into the convolution and most of that is the +// lowering, not the math. cuDNN convolves directly, with no im2col tensor. +// +// Layouts line up exactly, as they do for MPS: ggml's activation [W,H,C,T] is +// NCHW and its kernel [KW,KH,IC,OC] is OIHW, byte for byte, so nothing is +// transposed. Unlike the Metal path this one is also zero-copy -- on CUDA a +// ggml tensor's `data` is already a device pointer, so cuDNN reads and writes +// ggml's buffers in place. +// +// Precision: ggml runs these convolutions with F16 kernels and F16 im2col, +// accumulating in F32. Here the (small) kernel is widened to F32 once and cached, +// and the convolution runs F32 in/out with CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION, +// so the tensor cores still do the work at no worse accuracy than before. +// +// TS_WAN_VAE_CUDNN_CONV=0 falls back to ggml's im2col+GEMM. +#if defined(TSG_GGML_USE_CUDA) && defined(TSG_HAVE_CUDNN) + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace { + +struct CudnnState +{ + cudnnHandle_t handle = nullptr; + bool probed = false; + bool usable = false; + std::mutex mu; + + void* workspace = nullptr; + std::size_t workspaceBytes = 0; + + // Scratch for widening the F16 kernel to F32. Deliberately NOT a cache keyed + // by device pointer: ggml's allocator reuses addresses across graphs, so a + // pointer-keyed cache hands back another tensor's weights on a later decode. + // Re-widening costs one small kernel launch per convolution. + void* kernelScratch = nullptr; + std::size_t kernelScratchBytes = 0; + + ~CudnnState() = default; +}; + +CudnnState& state() +{ + static CudnnState s; + return s; +} + +bool ensure_handle(CudnnState& s) +{ + if (s.probed) return s.usable; + s.probed = true; + if (const char* e = std::getenv("TS_WAN_VAE_CUDNN_CONV"); e != nullptr && e[0] == '0') + return s.usable = false; + if (cudnnCreate(&s.handle) != CUDNN_STATUS_SUCCESS) + { + s.handle = nullptr; + return s.usable = false; + } + return s.usable = true; +} + +void* ensure_workspace(CudnnState& s, std::size_t need) +{ + if (need == 0) return nullptr; + if (s.workspace != nullptr && s.workspaceBytes >= need) return s.workspace; + if (s.workspace != nullptr) cudaFree(s.workspace); + s.workspace = nullptr; + s.workspaceBytes = 0; + if (cudaMalloc(&s.workspace, need) != cudaSuccess) { s.workspace = nullptr; return nullptr; } + s.workspaceBytes = need; + return s.workspace; +} + +// Widen an F16 kernel to F32 on device, once per distinct kernel pointer. +__global__ void tsg_f16_to_f32(const __half* __restrict__ src, float* __restrict__ dst, long long n) +{ + const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) dst[i] = __half2float(src[i]); +} + +float* widen_kernel_f32(CudnnState& s, const void* f16, long long n) +{ + const std::size_t need = (std::size_t) n * sizeof(float); + if (s.kernelScratch == nullptr || s.kernelScratchBytes < need) + { + if (s.kernelScratch != nullptr) cudaFree(s.kernelScratch); + s.kernelScratch = nullptr; + s.kernelScratchBytes = 0; + if (cudaMalloc(&s.kernelScratch, need) != cudaSuccess) { s.kernelScratch = nullptr; return nullptr; } + s.kernelScratchBytes = need; + } + const int threads = 256; + const int blocks = (int) ((n + threads - 1) / threads); + tsg_f16_to_f32<<>>(static_cast(f16), static_cast(s.kernelScratch), n); + if (cudaGetLastError() != cudaSuccess || cudaDeviceSynchronize() != cudaSuccess) return nullptr; + return static_cast(s.kernelScratch); +} + +struct Desc +{ + cudnnTensorDescriptor_t x = nullptr, y = nullptr; + cudnnFilterDescriptor_t w = nullptr; + cudnnConvolutionDescriptor_t c = nullptr; + ~Desc() + { + if (x) cudnnDestroyTensorDescriptor(x); + if (y) cudnnDestroyTensorDescriptor(y); + if (w) cudnnDestroyFilterDescriptor(w); + if (c) cudnnDestroyConvolutionDescriptor(c); + } +}; + +} // namespace + +extern "C" { + +bool tsg_cudnn_conv2d_available(void) +{ + CudnnState& s = state(); + std::lock_guard lock(s.mu); + return ensure_handle(s); +} + +// One convolution, in place on ggml's device buffers. `w` is the F16 kernel +// [KW,KH,IC,OC], `x` the F32 activation [W,H,IC,T], `dst` the F32 output +// [OW,OH,OC,T] -- all device pointers in ggml's memory order. +bool tsg_cudnn_conv2d( + const void* w, int wIsF16, int kw, int kh, int ic, int oc, + const void* x, int W, int H, int T, + int stride, int pad, + void* dst, int OW, int OH) +{ + if (w == nullptr || x == nullptr || dst == nullptr) return false; + if (kw <= 0 || kh <= 0 || ic <= 0 || oc <= 0 || W <= 0 || H <= 0 || T <= 0) return false; + if (stride <= 0 || pad < 0 || OW <= 0 || OH <= 0) return false; + + CudnnState& s = state(); + std::lock_guard lock(s.mu); + if (!ensure_handle(s)) return false; + + // An F32 kernel (Qwen-Image's VAE) is handed to cuDNN as it lies; an F16 one + // (Wan's) is widened into scratch first. + const float* wf32 = wIsF16 + ? widen_kernel_f32(s, w, (long long) kw * kh * ic * oc) + : static_cast(w); + if (wf32 == nullptr) return false; + + Desc d; + if (cudnnCreateTensorDescriptor(&d.x) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnCreateTensorDescriptor(&d.y) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnCreateFilterDescriptor(&d.w) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnCreateConvolutionDescriptor(&d.c) != CUDNN_STATUS_SUCCESS) return false; + + // ggml [W,H,C,T] == NCHW(T,C,H,W); ggml [KW,KH,IC,OC] == OIHW(OC,IC,KH,KW). + if (cudnnSetTensor4dDescriptor(d.x, CUDNN_TENSOR_NCHW, CUDNN_DATA_FLOAT, T, ic, H, W) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnSetTensor4dDescriptor(d.y, CUDNN_TENSOR_NCHW, CUDNN_DATA_FLOAT, T, oc, OH, OW) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnSetFilter4dDescriptor(d.w, CUDNN_DATA_FLOAT, CUDNN_TENSOR_NCHW, oc, ic, kh, kw) != CUDNN_STATUS_SUCCESS) return false; + if (cudnnSetConvolution2dDescriptor(d.c, pad, pad, stride, stride, 1, 1, + CUDNN_CROSS_CORRELATION, CUDNN_DATA_FLOAT) != CUDNN_STATUS_SUCCESS) return false; + // Let cuDNN use the tensor cores on F32 data. + cudnnSetConvolutionMathType(d.c, CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION); + + // Confirm cuDNN agrees with the output extent ggml allocated for this node. + int n_ = 0, c_ = 0, h_ = 0, w_ = 0; + if (cudnnGetConvolution2dForwardOutputDim(d.c, d.x, d.w, &n_, &c_, &h_, &w_) != CUDNN_STATUS_SUCCESS) return false; + if (n_ != T || c_ != oc || h_ != OH || w_ != OW) return false; + + // Only the implicit-GEMM algorithms. cuDNN's heuristic will happily rank + // FFT_TILING / WINOGRAD_NONFUSED first, and those carry alignment and padding + // requirements that ggml's buffers do not always satisfy -- which is how a + // "successful" convolution turns into an illegal memory access later in the + // graph. The implicit-GEMM kernels have no such constraints and are what + // makes the win here anyway (the point is avoiding im2col, not the algorithm). + const cudnnConvolutionFwdAlgo_t safe[] = { + CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM, + CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM, + }; + const int returned = (int) (sizeof(safe) / sizeof(safe[0])); + for (int i = 0; i < returned; i++) + { + std::size_t need = 0; + if (cudnnGetConvolutionForwardWorkspaceSize(s.handle, d.x, d.w, d.c, d.y, safe[i], &need) != CUDNN_STATUS_SUCCESS) + continue; + void* ws = nullptr; + if (need > 0) + { + ws = ensure_workspace(s, need); + if (ws == nullptr) continue; // too large for the device; try the next algo + } + const float alpha = 1.0f, beta = 0.0f; + const cudnnStatus_t st = cudnnConvolutionForward( + s.handle, &alpha, d.x, x, d.w, wf32, d.c, safe[i], + ws, need, &beta, d.y, dst); + if (st != CUDNN_STATUS_SUCCESS) continue; + return cudaDeviceSynchronize() == cudaSuccess; + } + return false; +} + +void tsg_cudnn_conv2d_release(void) +{ + CudnnState& s = state(); + std::lock_guard lock(s.mu); + if (s.kernelScratch != nullptr) { cudaFree(s.kernelScratch); s.kernelScratch = nullptr; s.kernelScratchBytes = 0; } + if (s.workspace != nullptr) { cudaFree(s.workspace); s.workspace = nullptr; s.workspaceBytes = 0; } +} + +} // extern "C" + +#endif // TSG_GGML_USE_CUDA && TSG_HAVE_CUDNN diff --git a/TensorSharp.GGML.Native/tsg_metal_mps_conv.mm b/TensorSharp.GGML.Native/tsg_metal_mps_conv.mm new file mode 100644 index 00000000..f430c392 --- /dev/null +++ b/TensorSharp.GGML.Native/tsg_metal_mps_conv.mm @@ -0,0 +1,240 @@ +// Copyright (c) Zhongkai Fu. All rights reserved. +// https://github.com/zhongkaifu/TensorSharp +// +// This file is part of TensorSharp. +// +// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree. +// +// Wan VAE convolutions on MetalPerformanceShadersGraph. +// +// ggml lowers a 2D convolution to im2col + mul_mat. For the Wan video VAE that +// is the single most expensive thing in a generation: a decode profile +// (benchmarks/WanVideoBench, 640x480x9f) attributes 44.4% of the graph to +// MUL_MAT and a further 30.2% to IM2COL alone -- 74.6% in the convolution, with +// im2col moving 865 MB per node at ~57 GB/s, a fifth of the machine's +// bandwidth. The lowering is pure overhead: a 3x3 conv materialises 9x its +// input before the GEMM ever starts. +// +// Apple ships a tuned convolution in MPSGraph. Measured on an M5 Pro at the +// shapes the Wan 2.2 decoder actually runs (ggml im2col+GEMM vs MPSGraph): +// +// 512->512 k3 320x240 t9 666 ms -> 108 ms 6.2x +// 256->256 k3 640x480 t9 947 ms -> 110 ms 8.6x +// 160->160 k3 640x480 t9 491 ms -> 47 ms 10.4x +// 512->512 k1 320x240 t9 184 ms -> 14 ms 13.9x +// +// MPS reaches ~30 TFLOP/s where ggml's Metal GEMM gets ~4.9. Critically it does +// so WITHOUT ggml's mul_mm kernel -- the one whose Metal 4 tensor path corrupts +// this exact graph into black frames (see wan_vae_gemm_budget). This buys the +// matrix-unit throughput while stepping around that defect entirely. +// +// Layout costs nothing: ggml's activation [W,H,C,T] and kernel [KW,KH,IC,OC] +// are, byte for byte, MPS NCHW and OIHW. NCHW measured as fast as NHWC here, so +// no transpose is inserted on either side. +// +// Numerics are unchanged: ggml already runs these convolutions with F16 kernels +// and F16 im2col, accumulating to F32. MPS does the same (the graph casts the +// F32 activation to F16, convolves, and casts back), and the two agree to +// relL2 2.07e-4 / max rel err 4.8e-4 on random data -- F16 rounding. +// +// TS_WAN_VAE_MPS_CONV=0 falls back to ggml's im2col+GEMM. +#if defined(__APPLE__) + +#import +#import +#import + +#include +#include +#include +#include +#include +#include + +namespace { + +struct MpsState +{ + id device = nil; + id queue = nil; + bool probed = false; + bool usable = false; + std::mutex mu; + + // One compiled MPSGraph per (kw,kh,ic,oc,stride,pad) -- the shape-independent + // part. Activation extents are placeholders so a graph is reused across the + // decoder's temporal chunks and band sizes. + struct Key + { + int kw, kh, ic, oc, stride, pad, w, h, t, wf16; + bool operator<(const Key& o) const + { + return std::tie(kw, kh, ic, oc, stride, pad, w, h, t, wf16) < + std::tie(o.kw, o.kh, o.ic, o.oc, o.stride, o.pad, o.w, o.h, o.t, o.wf16); + } + }; + struct Entry + { + MPSGraph* graph; + MPSGraphTensor* x; + MPSGraphTensor* w; + MPSGraphTensor* y; + }; + std::map cache; + + // Staging buffers, grown on demand and reused: ggml's tensor storage is not + // guaranteed page-aligned, which newBufferWithBytesNoCopy requires, so the + // operands are staged. At the shapes that matter the copies are ~3% of what + // the convolution itself costs. + id bufX = nil, bufW = nil, bufY = nil; + std::size_t capX = 0, capW = 0, capY = 0; +}; + +MpsState& state() +{ + static MpsState s; + return s; +} + +bool ensure_device(MpsState& s) +{ + if (s.probed) return s.usable; + s.probed = true; + if (const char* e = std::getenv("TS_WAN_VAE_MPS_CONV"); e != nullptr && e[0] == '0') + return s.usable = false; + s.device = MTLCreateSystemDefaultDevice(); + if (s.device == nil) return s.usable = false; + s.queue = [s.device newCommandQueue]; + if (s.queue == nil) return s.usable = false; + // MPSGraph convolution2D needs macOS 12.0+ / iOS 15.0+. + if (@available(macOS 12.0, iOS 15.0, *)) { s.usable = true; } + else { s.usable = false; } + return s.usable; +} + +id ensure_buffer(MpsState& s, id& buf, std::size_t& cap, std::size_t need) +{ + if (buf != nil && cap >= need) return buf; + buf = [s.device newBufferWithLength:need options:MTLResourceStorageModeShared]; + cap = (buf != nil) ? need : 0; + return buf; +} + +} // namespace + +extern "C" { + +// True when MPSGraph convolutions can be used for the Wan VAE on this process. +bool tsg_mps_conv2d_available(void) +{ + MpsState& s = state(); + std::lock_guard lock(s.mu); + return ensure_device(s); +} + +// One convolution. Activation `x` is F32 [W,H,IC,T] and kernel `w` is F16 +// [KW,KH,IC,OC], both in ggml's memory order; `dst` receives F32 +// [OW,OH,OC,T]. Returns false when the shape is unsupported, in which case the +// caller must fall back to the ggml path. +bool tsg_mps_conv2d( + const void* w, int wIsF16, int kw, int kh, int ic, int oc, + const float* x, int W, int H, int T, + int stride, int pad, + float* dst, int OW, int OH) +{ + if (w == nullptr || x == nullptr || dst == nullptr) return false; + if (kw <= 0 || kh <= 0 || ic <= 0 || oc <= 0 || W <= 0 || H <= 0 || T <= 0) return false; + if (stride <= 0 || pad < 0 || OW <= 0 || OH <= 0) return false; + + MpsState& s = state(); + std::lock_guard lock(s.mu); + if (!ensure_device(s)) return false; + + @autoreleasepool { + const std::size_t nx = (std::size_t)W * H * ic * T * sizeof(float); + const std::size_t nw = (std::size_t)kw * kh * ic * oc * (wIsF16 ? sizeof(std::uint16_t) : sizeof(float)); + const std::size_t ny = (std::size_t)OW * OH * oc * T * sizeof(float); + + id bx = ensure_buffer(s, s.bufX, s.capX, nx); + id bw = ensure_buffer(s, s.bufW, s.capW, nw); + id by = ensure_buffer(s, s.bufY, s.capY, ny); + if (bx == nil || bw == nil || by == nil) return false; + + std::memcpy(bx.contents, x, nx); + std::memcpy(bw.contents, w, nw); + + MpsState::Key key{ kw, kh, ic, oc, stride, pad, W, H, T, wIsF16 }; + auto it = s.cache.find(key); + if (it == s.cache.end()) + { + MPSGraph* g = [MPSGraph new]; + // ggml [W,H,C,T] == NCHW(T,C,H,W); ggml [KW,KH,IC,OC] == OIHW(OC,IC,KH,KW). + MPSShape* xs = @[ @(T), @(ic), @(H), @(W) ]; + MPSShape* ws = @[ @(oc), @(ic), @(kh), @(kw) ]; + MPSGraphTensor* xt = [g placeholderWithShape:xs dataType:MPSDataTypeFloat32 name:@"x"]; + const MPSDataType wdt = wIsF16 ? MPSDataTypeFloat16 : MPSDataTypeFloat32; + MPSGraphTensor* wt = [g placeholderWithShape:ws dataType:wdt name:@"w"]; + MPSGraphConvolution2DOpDescriptor* d = + [MPSGraphConvolution2DOpDescriptor descriptorWithStrideInX:stride strideInY:stride + dilationRateInX:1 dilationRateInY:1 + groups:1 + paddingLeft:pad paddingRight:pad + paddingTop:pad paddingBottom:pad + paddingStyle:MPSGraphPaddingStyleExplicit + dataLayout:MPSGraphTensorNamedDataLayoutNCHW + weightsLayout:MPSGraphTensorNamedDataLayoutOIHW]; + if (d == nil) return false; + // Match the precision ggml would have used for this node: an F16 + // kernel means ggml lowered through an F16 im2col, so convolve in F16 + // and hand back F32; an F32 kernel (Qwen-Image's VAE) stays F32 + // end to end. Either way this is a drop-in for the node it replaces. + MPSGraphTensor* yt; + if (wIsF16) + { + MPSGraphTensor* xh = [g castTensor:xt toType:MPSDataTypeFloat16 name:@"xh"]; + MPSGraphTensor* yh = [g convolution2DWithSourceTensor:xh weightsTensor:wt descriptor:d name:nil]; + yt = [g castTensor:yh toType:MPSDataTypeFloat32 name:@"y"]; + } + else + { + yt = [g convolution2DWithSourceTensor:xt weightsTensor:wt descriptor:d name:nil]; + } + it = s.cache.emplace(key, MpsState::Entry{ g, xt, wt, yt }).first; + } + MpsState::Entry& e = it->second; + + MPSGraphTensorData* xd = [[MPSGraphTensorData alloc] initWithMTLBuffer:bx + shape:@[ @(T), @(ic), @(H), @(W) ] + dataType:MPSDataTypeFloat32]; + MPSGraphTensorData* wd = [[MPSGraphTensorData alloc] initWithMTLBuffer:bw + shape:@[ @(oc), @(ic), @(kh), @(kw) ] + dataType:(wIsF16 ? MPSDataTypeFloat16 : MPSDataTypeFloat32)]; + MPSGraphTensorData* yd = [[MPSGraphTensorData alloc] initWithMTLBuffer:by + shape:@[ @(T), @(oc), @(OH), @(OW) ] + dataType:MPSDataTypeFloat32]; + if (xd == nil || wd == nil || yd == nil) return false; + + [e.graph runWithMTLCommandQueue:s.queue + feeds:@{ e.x : xd, e.w : wd } + targetOperations:nil + resultsDictionary:@{ e.y : yd }]; + + std::memcpy(dst, by.contents, ny); + return true; + } +} + +// Drop every cached graph and staging buffer (called when the VAE hands back its +// device residency between generations). +void tsg_mps_conv2d_release(void) +{ + MpsState& s = state(); + std::lock_guard lock(s.mu); + s.cache.clear(); + s.bufX = s.bufW = s.bufY = nil; + s.capX = s.capW = s.capY = 0; +} + +} // extern "C" + +#endif // __APPLE__ diff --git a/TensorSharp.Models/ModelBase.cs b/TensorSharp.Models/ModelBase.cs index 4d8046b6..f0e6281f 100644 --- a/TensorSharp.Models/ModelBase.cs +++ b/TensorSharp.Models/ModelBase.cs @@ -6126,8 +6126,8 @@ private static void ApplyArchitectureNativeTunables(string arch, BackendType bac if (arch is not ("wan" or "wan2.1" or "wan2.2")) return; - // Wan rendered NaN - a uniformly black frame - on Metal whenever ggml - // routed mul_mm through the Metal 4 tensor API. Diagnosis (2026-08-13): + // Wan renders NaN - uniformly black frames - on Metal whenever ggml + // routes mul_mm through the Metal 4 tensor API. Diagnosis (2026-08-13): // the corruption is confined to the VAE's conv GEMMs. The tensor-API // mul_mm intermittently misreads operand columns there on the FIRST // pass over a graph (M5, macOS 26.6) - buffer-layout dependent (32x32 @@ -6140,6 +6140,13 @@ private static void ApplyArchitectureNativeTunables(string arch, BackendType bac // is a device-level property fixed at init, so the kernel choice // cannot be scoped per-op. // + // Re-verified 2026-08-17 that this is NOT yet fixed, and how easy it is + // to conclude otherwise: an isolated WanVideoBench decode of synthetic + // latents at five shapes - including the 32x32 all-NaN repro - matched + // the non-tensor decode to 91-93 dB PSNR with no NaN. The next full + // 1088x832x121f generation with the tensor API on produced 121 BLACK + // frames. Only an end-to-end video decides this. + // // With the tensor API on, the VAE stays CORRECT because // ggml_ops_wan.cpp routes its convs through ggml_conv_2d_direct on // tensor-API devices (wan_vae_gemm_budget) - slower than im2col+GEMM, @@ -6155,7 +6162,7 @@ private static void ApplyArchitectureNativeTunables(string arch, BackendType bac // models (patch_embedding oc >= 5120), disabled for smaller ones. // TS_WAN_METAL_TENSOR_API=1/0 forces either way. When upstream fixes // the tensor-API mul_mm, enable it everywhere and drop the direct-conv - // carve-out. + // carve-out - and confirm with a full video, not a decode probe. long dim = 0; if (probe.Tensors.TryGetValue("patch_embedding.weight", out var patch) && patch.Shape.Length >= 5) dim = (long)patch.Shape[4]; diff --git a/TensorSharp.Models/Models/WanVideo/WanVae.cs b/TensorSharp.Models/Models/WanVideo/WanVae.cs index 7d768058..10a95cb9 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVae.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVae.cs @@ -143,6 +143,34 @@ WanVaeConv Conv(WanVaeConvWeights c) } } + /// + /// Band tiling exists to keep a full-resolution decode inside a small card's + /// VRAM, and it is not free: the bands overlap, so the decoder runs more + /// latent rows than the plane has, plus a band buffer alongside the canvas. + /// Measured at 1088x832x121f on an M5 Pro, decoding the plane whole is both + /// FASTER and LIGHTER than two bands — 565 s vs 655 s, peak RSS 4.85 vs + /// 5.37 GB. So scale the threshold with the memory actually available + /// instead of pinning it at a 16 GB card's budget; the constant is anchored + /// so that ~16 GB free reproduces the historical 640 kpx. + /// + protected override long TilePixelThreshold => _tileThreshold ??= ResolveTileThreshold(); + + private long? _tileThreshold; + + private static long ResolveTileThreshold() + { + const long Anchor = 640_000; // px, the previous fixed value + const long AnchorFreeBytes = 16L << 30; // the card size it was chosen for + try + { + if (GgmlBasicOps.TryGetDeviceMemoryInfo(out long freeBytes, out _) && freeBytes > 0) + return Math.Max(Anchor / 4, (long)((double)freeBytes / AnchorFreeBytes * Anchor)); + } + catch (DllNotFoundException) { /* managed-only host */ } + catch (EntryPointNotFoundException) { /* older GgmlOps */ } + return Anchor; + } + private IntPtr Reg(float[] data) { long bytes = data.LongLength * sizeof(float); diff --git a/TensorSharp.Models/Models/WanVideo/WanVideoModel.cs b/TensorSharp.Models/Models/WanVideo/WanVideoModel.cs index e1a8348a..fe046546 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVideoModel.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVideoModel.cs @@ -78,6 +78,34 @@ public sealed class WanVideoModel : ModelBase internal GgufFile HighNoiseGguf { get; } internal GgufFile LowNoiseGguf { get; } + /// + /// Denoising steps a step-distilled checkpoint was trained for; 0 for an + /// ordinary one. Step distillation (Wan2.2-TI2V-5B-Turbo, Wan2.2-Lightning, + /// lightx2v, FastWan/DMD) is the difference between a 5-second 720p video + /// costing 100 DiT passes and costing 4, so the pipeline has to notice — run + /// a distilled checkpoint on the 50-step CFG recipe and it is 25x slower for + /// a worse picture, since these models are also trained to run WITHOUT + /// classifier-free guidance. + /// + public int DistilledSteps { get; } + + /// + /// Steps a distilled Wan checkpoint/LoRA advertises in its file name. + /// "…-4steps-…" / "…8step…" wins; otherwise a distillation marker implies the + /// 4 steps every published Wan 2.2 distillation recommends. + /// + internal static int ParseDistilledSteps(string fileName) + { + if (string.IsNullOrEmpty(fileName)) return 0; + string n = Path.GetFileNameWithoutExtension(fileName).ToLowerInvariant(); + bool marker = n.Contains("turbo") || n.Contains("distill") || n.Contains("lightning") + || n.Contains("lightx2v") || n.Contains("fastwan") || n.Contains("-dmd"); + var m = System.Text.RegularExpressions.Regex.Match(n, @"(\d+)\s*_?steps?\b"); + if (m.Success && int.TryParse(m.Groups[1].Value, out int s) && s > 0 && s <= 16) + return s; + return marker ? 4 : 0; + } + public WanVideoModel(string ggufPath, BackendType backend) : base(ggufPath, backend) { if (backend is not (BackendType.GgmlCuda or BackendType.GgmlCpu or BackendType.GgmlMetal @@ -147,7 +175,12 @@ public WanVideoModel(string ggufPath, BackendType backend) : base(ggufPath, back new[] { "umt5-xxl-encoder-Q8_0.gguf" }, n => (n.Contains("umt5") || n.Contains("t5xxl") || n.Contains("t5-xxl")) && n.EndsWith(".gguf")); + DistilledSteps = ParseDistilledSteps(Path.GetFileName(ggufPath)); + Console.WriteLine($"Wan video ({Variant}): DiT={Path.GetFileName(ggufPath)}"); + if (DistilledSteps > 0) + Console.WriteLine($" step-distilled checkpoint detected -> {DistilledSteps} steps, guidance off " + + $"(--diffusion-steps / --cfg override)"); if (_dit2Path != null) Console.WriteLine($" expert #2 = {_dit2Path}"); Console.WriteLine($" VAE = {_vaePath ?? ""}"); diff --git a/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs b/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs index fceb6c17..17fdced9 100644 --- a/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs +++ b/TensorSharp.Models/Models/WanVideo/WanVideoPipeline.cs @@ -170,12 +170,21 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) height = SnapDim(p.Height > 0 ? p.Height : defH, grid); } int frames = p.Frames > 0 ? SnapFrames(p.Frames) : (ti2v ? 49 : 33); - int steps = p.Steps > 0 ? p.Steps : ti2v ? 50 : a14b ? 40 : 30; + // A step-distilled checkpoint replaces the whole sampling recipe: it is + // trained to land in a handful of steps AND without classifier-free + // guidance, so it costs one pass per step instead of two. Applying the + // base recipe to one wastes 25x the work and looks worse. + int distilled = _model.DistilledSteps; + int steps = p.Steps > 0 ? p.Steps + : distilled > 0 ? distilled + : ti2v ? 50 : a14b ? 40 : 30; float cfg = p.CfgScale > 0 ? p.CfgScale + : distilled > 0 ? 1.0f : ti2v ? 5.0f : a14b ? (i2v ? 3.5f : 4.0f) : 6.0f; float cfg2 = p.CfgScale2 > 0 ? p.CfgScale2 + : distilled > 0 ? 1.0f : a14b ? (i2v ? 3.5f : 3.0f) : cfg; // FlowMatch shift. Wan 2.2 recipes: 5.0 (12.0 for A14B T2V). Wan 2.1: 8.0 @@ -441,6 +450,7 @@ private GeneratedVideo GenerateCore(string prompt, WanVideoParams p) beat.Set("vae-decode", steps, steps, $"{frames} frames at {width}x{height}, band {Math.Min(done + 1, bands)}/{bands}")); Phase("vae-decode"); + AssertFramesAreNotDegenerate(framesOut); beat.Report("done", steps, steps, "complete", 0, heartbeat: false); Console.WriteLine($" [wan-timing] total: {total.Elapsed.TotalSeconds:F1}s"); @@ -476,6 +486,41 @@ private static void DumpF32(string path, float[] data) System.IO.File.WriteAllBytes(path, bytes); } + /// + /// Fail loudly when the decode produced a uniformly flat (usually black) + /// video. A backend that miscomputes the VAE's conv GEMMs — the Metal 4 + /// tensor-API mul_mm defect is the known one — yields exactly this, and it + /// is otherwise silent: the request "succeeds" after however many minutes + /// the generation took and writes an unusable MP4. Two frames are enough to + /// catch it and the scan is microseconds next to the decode. + /// + internal static void AssertFramesAreNotDegenerate(RgbImage[] frames) + { + if (frames == null || frames.Length == 0) return; + // Spread the samples: a legitimately flat frame (a fade to or from black) + // must not condemn the video, while the corruption flattens all of them. + int n = frames.Length; + foreach (int idx in new[] { 0, n / 3, 2 * n / 3, n - 1 }) + { + var px = frames[idx].Pixels; + float min = float.MaxValue, max = float.MinValue; + long bad = 0; + foreach (float v in px) + { + if (float.IsNaN(v) || float.IsInfinity(v)) { bad++; continue; } + if (v < min) min = v; + if (v > max) max = v; + } + if (bad == 0 && max - min > 1e-3f) return; // a real picture; done + } + throw new InvalidOperationException( + "The Wan VAE decoded a uniformly flat video (every sampled frame is a single " + + "colour, usually black). That is a backend numerics failure, not a sampling " + + "problem. On Metal the known cause is the Metal 4 tensor-API mul_mm defect in " + + "the VAE's conv GEMMs: run with TS_WAN_METAL_TENSOR_API=0 (or check that " + + "GGML_METAL_TENSOR_DISABLE is not being overridden)."); + } + /// Steps whose guidance delta is recomputed rather than reused: the /// first steps (where the frame's structure is /// decided), the last step, and every -th step in diff --git a/TensorSharp.Models/TensorSharp.Models.csproj b/TensorSharp.Models/TensorSharp.Models.csproj index aec8eb54..26c85662 100644 --- a/TensorSharp.Models/TensorSharp.Models.csproj +++ b/TensorSharp.Models/TensorSharp.Models.csproj @@ -12,6 +12,7 @@ + diff --git a/USAGE.md b/USAGE.md index 8c1ed079..cf2441d6 100644 --- a/USAGE.md +++ b/USAGE.md @@ -133,7 +133,9 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model H.264 MP4). The UMT5-XXL text-encoder GGUF and # video-VAE companions are resolved next to the DiT GGUF (or set --wan-te / # --wan-vae). Wan 2.1 T2V, Wan 2.2 TI2V-5B, and Wan 2.2 A14B (both experts) -# are auto-detected. See docs/models/wan.md. +# are auto-detected, and so are step-distilled (Turbo / Lightning / FastWan) +# checkpoints -- 4 DiT passes instead of 100 for the same video. See the +# "Video generation (Wan)" section below and docs/models/wan.md. dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model \ --prompt "a lovely cat walking through a garden" --output cat.mp4 \ --width 832 --height 480 --video-frames 49 --backend ggml_cuda \ @@ -269,12 +271,14 @@ quietly. Measured on gemma-4-26B-A4B (`--cpu-moe`, peak VRAM): `ggml_cuda` | `--qwen-image-vae ` | Override the resolved Qwen-Image VAE companion (`.gguf` or `.safetensors`). | | `--qwen-image-vl ` | Override the resolved Qwen2.5-VL-7B text-encoder GGUF. | | `--qwen-image-mmproj ` | Override the resolved Qwen2.5-VL mmproj (vision grounding) GGUF. | -| `--qwen-image-lora ` | Qwen-Image-Edit Lightning distillation LoRA (`.safetensors`), merged into the DiT at load time. Auto-derives the step count (e.g. 4 or 8) and switches CFG to 1.0. Env: `TS_QWEN_IMAGE_LORA`. | +| `--qwen-image-lora ` | Qwen-Image-Edit Lightning distillation LoRA (`.safetensors`). Applied as a runtime F32 side-path next to each targeted projection (`y = W_quant·x + b + (alpha/rank)·up·(down·x)`) with the quantized base weights left untouched — **not** merged into them. Auto-derives the step count from the file name (e.g. 4 or 8), switches CFG to 1.0 and pins the timestep shift to 3, so the default 30 steps × 2 CFG passes (60 DiT forwards) become 4–8. Needs the whole-model or fused per-block CUDA forward — on a path without the side-path it throws rather than emitting noise. Env: `TS_QWEN_IMAGE_LORA`. | +| `--width ` / `--height ` | Output size for Qwen-Image-Edit and Wan video. Default: `0` — auto (Qwen-Image-Edit: the source size, VRAM-clamped; Wan: the model's native area at the input image's aspect ratio, 1280×704 for TI2V-5B and 832×480 otherwise). | | `--video-frames ` | Wan video frame count, snapped to `4k+1` (default: 33; 49 for Wan2.2-TI2V). `1` generates a still image (use `--output out.png`). | | `--fps ` | Wan video playback frame rate of the saved MP4 (default: 16; 24 for Wan2.2-TI2V). | | `--flow-shift ` | Wan FlowMatch timestep shift (default: the model's official recipe — 5.0 for Wan 2.2, 12.0 for A14B T2V, 8.0/3.0/5.0 for Wan 2.1). | | `--sampler ` | Wan sampler: `unipc` (official Wan sampler, default) or `euler`. | | `--negative-prompt ` | Wan negative prompt (default: the official Wan negative prompt). | +| _(step-distilled checkpoints)_ | Auto-detected from the DiT file name (`Turbo`, `distill`, `Lightning`, `lightx2v`, `FastWan`, `-dmd`, or an explicit `…-4steps-…` / `…8step…` for 1–16): the pipeline switches to that step count with guidance off, turning the official 50-step × CFG recipe's 100 DiT passes into 4. This is the single biggest speed lever for Wan — see **[Video generation (Wan)](#video-generation-wan)** below. `--diffusion-steps` / `--cfg` override it. | | `--cfg-cache-stride ` | Wan guidance cache: run the unconditional CFG pass on one step in `N` and reuse the cached guidance direction in between (default off — every step runs both passes). `2` ≈ 1.30x faster, `3` ≈ 1.43x; approximate, so leave it off when matching a reference sample matters. | | `--wan-vae ` | Override the resolved Wan video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). Env: `TS_WAN_VAE`. | | `--wan-te ` | Override the resolved UMT5-XXL text-encoder GGUF. Env: `TS_WAN_TE`. Wan 2.2 A14B additionally resolves the second high/low-noise expert automatically (env: `TS_WAN_DIT2`). | @@ -377,9 +381,12 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/Wan2.2-TI2 --video-frames 121 --fps 24 # 121 frames at the TI2V-5B native area is 27k DiT tokens, and self-attention is -# quadratic in that, so a full 50-step run is hours on a laptop-class GPU. The -# server logs a per-pass timing plus a running ETA and heartbeats every 30 s, and -# the Web UI shows both; see docs/models/wan.md for the cost table and the knobs. +# quadratic in that, so a full 50-step run on the BASE checkpoint is hours on a +# laptop-class GPU (measured: ~3 h 30 m on an M5 Pro). Point --model at a +# step-distilled checkpoint instead and the identical request takes 17 m 30 s -- +# nothing else changes. See "Video generation (Wan)" below. +# The server logs a per-pass timing plus a running ETA and heartbeats every 30 s, +# and the Web UI shows both; docs/models/wan.md has the full cost table. # Configure server-wide default sampling parameters # (used whenever a request does not override the value itself) @@ -445,7 +452,7 @@ Running `TensorSharp.Server` with no arguments prints the full parameter referen | `--kv-cache-dtype ` | KV cache precision for the hosted model: `f32`, `f16`, `q8_0`, or `q4_0` (quantized caches trade small numerical drift for memory; see the CLI table above for the tier trade-offs). Default: auto — the backend/model pick. Env: `KV_CACHE_DTYPE`. | | `--continuous-batching` / `--no-continuous-batching` | Enable (default) or disable iteration-level paged-batching. When enabled the server admits / preempts sequences mid-batch and packs them into one forward pass on models that implement `IBatchedPagedModel`. `--no-continuous-batching` falls back to per-sequence KV-swap for every model. Alias: `--paged-batching` / `--no-paged-batching`. | | `--prefill-chunk-size ` | Chunked-prefill granularity under contention — the maximum prefill tokens scheduled per step while other requests are running, so parallel decodes get frequent turns at the GPU (default: `1024`). Env: `TS_SCHED_PREFILL_CHUNK`. | -| `--mtp-spec` / `--no-mtp-spec` | Enable NextN/MTP speculative decoding (default off) on models that ship a multi-token-prediction draft head (Qwen 3.6's embedded NextN block, or a Gemma 4 `gemma4-assistant` draft loaded via `--mtp-draft-model`). Engages for solo (non-concurrent) sequences: the draft head proposes up to `--mtp-draft` tokens per step and the trunk verifies them in one batched forward, with the request's own sampler (penalties included) driving both drafting and verification, so output matches standard decode. Engaged automatically only where profitable (ggml backends and the pure-C# `cuda` backend); CPU / MLX serve standard decode. Env: `TS_MTP_SPEC`. | +| `--mtp-spec` / `--no-mtp-spec` | Enable NextN/MTP speculative decoding (default off) on models that ship a multi-token-prediction draft head (Qwen 3.6's embedded NextN block, or a Gemma 4 `gemma4-assistant` draft loaded via `--mtp-draft-model`). Engages for solo (non-concurrent) sequences: the draft head proposes up to `--mtp-draft` tokens per step and the trunk verifies them in one batched forward, with the request's own sampler (penalties included) driving both drafting and verification, so output matches standard decode. Engaged automatically only where profitable: Qwen 3.6 reports its embedded NextN block profitable on every backend, while Gemma 4's separate draft head engages on the ggml backends and on the direct `cuda` backend only. CPU / GGML CPU / MLX serve standard decode. Env: `TS_MTP_SPEC`. | | `--mtp-draft ` | Maximum tokens drafted per speculative step (default `8`). Env: `TS_MTP_DRAFT`. | | `--mtp-pmin ` | Minimum draft confidence in `(0, 1]` for a drafted token to be kept; drafting stops at the first low-confidence token. Default: chosen per drafter kind — `0.75` for a per-token draft head (top-1 probability over its top-10 logits), `0.35` for a block drafter, where the gate is the CUMULATIVE prefix probability and so the same number means something far stricter. Env: `TS_MTP_PMIN`. | | `--draft-model ` | Speculative-decoding draft model for architectures whose drafter ships as its own file: DeepSeek V4's DSpark support GGUF (see [DeepSeek V4](docs/models/deepseek4.md#dspark-speculative-decoding)) and Muse-Glimmer's DFlash drafter (see [Muse-Glimmer](docs/models/muse-glimmer.md#3-dflash-speculative-decoding), env `TS_MUSE_GLIMMER_DFLASH`). Either one drafts a whole block per step and the trunk verifies it in one batched forward, so greedy output is unchanged. Engages on every single-sequence CLI path — `--input`, `--multi-turn-jsonl` and `--interactive` — with `--backend cuda` or `--backend ggml_cuda`. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p or repetition penalty turns it off, because the standalone decoder verifies with argmax). `TensorSharp.Server` accepts the same flag alongside `--mtp-spec`, where verification runs the request's own sampler and so composes with any sampling settings. Env: `TS_DSV4_DSPARK`. | @@ -589,6 +596,166 @@ the rest. Either way `--stop` sequences pinned on the server stay in force under `config` (merged with the request's) and are replaced by the request under `request`. +## Video generation (Wan) + +A `wan` GGUF turns a prompt — plus an optional first-frame image on the Wan 2.2 +models — into an H.264 MP4, from `TensorSharp.Cli`, the server's three video +endpoints, and the Web UI chat. Full architecture detail is in the +[Wan card](docs/models/wan.md); this section is the operator's view: which +checkpoint to download, and which knobs actually change the wall clock. + +### Which checkpoint + +| Family | Latent | Modes | Notes | +|---|---|---|---| +| Wan 2.2 TI2V-5B | 48 ch, 16×16×4 (`Wan2.2_VAE.safetensors`) | T2V + I2V | dense 5B, 24 fps, natively 720p; ~2.7× fewer DiT tokens than Wan 2.1 at the same resolution, so it is both the fastest and the highest-quality option on consumer GPUs | +| Wan 2.2 A14B (T2V / I2V) | 16 ch (36 ch I2V input), `wan_2.1_vae.safetensors` | T2V + I2V | two 14B experts switched at a timestep boundary; **both** expert GGUFs must be present (same folder, or `HighNoise/` + `LowNoise/`) | +| Wan 2.1 T2V (1.3B / 14B) | 16 ch, `wan_2.1_vae.safetensors` | T2V | single DiT | + +Every family also needs the UMT5-XXL text encoder +(`umt5-xxl-encoder-Q8_0.gguf`) and the matching video VAE. All three companions +are resolved from the DiT's own directory, subfolders such as `VAE/`, +`HighNoise/` and `LowNoise/` included, so one `--local-dir` is enough; +`--wan-vae` / `--wan-te` (and `TS_WAN_DIT2` for the second A14B expert) +override the search. + +Wan is the one family that rejects a backend outright: it runs on `ggml_cuda`, +`ggml_vulkan`, `ggml_metal`, `ggml_cpu`, `cuda` and `cpu`, and **not** on +`--backend mlx`. `ggml_cuda` is the fastest (RTX 2000 Ada, Wan2.1-1.3B F16, +832×480×33f, 30 steps: `ggml_cuda` 12.0 s/step vs `ggml_vulkan` 17.2 and direct +`cuda` 19.3); `cpu` / `ggml_cpu` are for functional use only. + +### The fast lane: step-distilled checkpoints + +**This is the single biggest speed lever in Wan, and it costs nothing but a +different download.** The official Wan2.2-TI2V-5B recipe is 50 steps × 2 +classifier-free-guidance passes = **100 DiT passes**. A step-distilled +checkpoint (Turbo / Lightning / FastWan / DMD) is trained to run guidance-free +in a handful of steps, so the same video costs **4 DiT passes** — 1/25th of the +denoising work. + +TensorSharp detects one from the DiT **file name**: any of `turbo`, `distill`, +`lightning`, `lightx2v`, `fastwan`, `-dmd` (case-insensitive), or an explicit +`steps` / `_steps` token for 1 ≤ N ≤ 16, which wins over the markers. A +marker with no step count means 4. On load the console prints + +``` +step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override) +``` + +so you can confirm from the log that it fired. No flag is involved — a distilled +GGUF is passed as an ordinary `--model`. + +Measured on an M5 Pro (20-core GPU, 48 GB unified), `ggml_metal`, +Wan2.2-TI2V-5B Q8_0, 1088×832×121f = 27 404 tokens, image-to-video — i.e. the +full five-second 720p-class request: + +| | Base checkpoint | **Turbo checkpoint** | +|---|---|---| +| DiT passes | 100 (50 steps × CFG) | **4** (4 steps, guidance-free) | +| per pass | 120.2 s | 120.2 s | +| denoise total | 12 020 s | **481 s** | +| VAE decode, 121 frames | 563 s | 563 s | +| **end to end** | **≈ 3 h 30 m** | **17 m 30 s** | + +Only the `--model` path changes between those two columns. Once distilled, the +VAE decode is the bottleneck (~55% of the run), not the DiT. + +Getting one (TI2V-5B — note the file names spell the version with an +**underscore**, `Wan2_2`, unlike the base repo): + +```bash +pip install -U huggingface_hub +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +# the Turbo repo ships no VAE and no text encoder — take them from the base repos +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \ + --backend ggml_metal --image first_frame.png --output out.mp4 \ + --prompt "the cat runs toward the camera, cinematic tracking shot" \ + --video-frames 121 --fps 24 + +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \ + --backend ggml_metal --video-frames 121 --fps 24 +``` + +For Wan 2.2 I2V-A14B the drop-in distilled GGUFs are +[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF), +which ships the distillation already merged into both experts under `high_noise/` +and `low_noise/`; download both and point `--model` at either one. It ships no +VAE and no text encoder, so take `VAE/Wan2.1_VAE.safetensors` from +[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF) +and the UMT5-XXL encoder as above. + +> [lightx2v/Wan2.2-Lightning](https://huggingface.co/lightx2v/Wan2.2-Lightning) +> publishes LoRA `.safetensors` only, and TensorSharp has no Wan LoRA option — +> use a repo that ships the distillation already baked into the GGUF. + +### `--cfg-cache-stride` (base checkpoints only) + +A guided step is `v = v_cond + (cfg-1)·d` with `d = v_cond - v_uncond`. The +guidance direction `d` changes far more slowly across the schedule than `v` +does, so `--cfg-cache-stride N` runs the unconditional pass on one step in `N` +and reuses the cached `d` in between. At 50 steps, `2` runs 77 of the 100 passes +(**1.30×**) and `3` runs 70 (**1.43×**). The first three steps and the last +always recompute `d`. Server JSON field: `"cfgCacheStride": 2`. + +It is an approximation — leave it off when matching a reference sample matters — +and it is pointless on a step-distilled checkpoint, which already runs +guidance-free (the cache is disabled whenever cfg ≤ 1.0). + +### Making a large request cheaper, in order of effect + +1. **Use a step-distilled checkpoint.** 100 DiT passes become 4; this dwarfs + everything else. +2. **Fewer frames.** 121 → 61 roughly quarters the attention work (token count + is `latent_frames × (h/2) × (w/2)` and self-attention is `O(tokens²)`) and + halves the VAE decode. +3. **Smaller frame area** — but not below Wan's training resolutions. Under + ~0.3 MP the model is out of distribution and the video gets *worse*, not just + cheaper; the pipeline warns below that. Wan is trained at 480p (832×480) and + 720p (1280×704), so generate at a supported size and downscale afterwards. +4. **Fewer steps**, base checkpoints only — 30 instead of the official 50 is + visibly close and 1.7× cheaper. +5. **`--cfg-cache-stride 2` or `3`** — 1.30× / 1.43×, base checkpoints only. + +Resolution against wall clock on the same M5 Pro, same Turbo checkpoint and +image: + +| Output | Tokens | Denoise | VAE decode | **Total** | +|---|---|---|---|---| +| 736×544 × 81f (3.4 s, 480p class) | 8 211 | 84 s | 159 s | **4 m 09 s** | +| 736×544 × 121f (5 s, 480p class) | 12 121 | 137 s | 237 s | **6 m 19 s** | +| 1088×832 × 121f (5 s, 720p class) | 27 404 | 481 s | 563 s | **17 m 30 s** | + +480p (≈0.4 MP) is a resolution Wan is *trained* at, so the first two rows are +in-distribution rather than a degraded mode — that is the setting to reach for +when a few minutes matters. + +### Server defaults + +`--video-frames N` and `--fps N` set server-wide **defaults**, not caps, for the +Web UI and all three video endpoints; a request that supplies `frames` or `fps` +overrides each independently. With both omitted the model recipe applies: 49 +frames at 24 fps for Wan2.2-TI2V, 33 at 16 fps otherwise. Frame counts are +snapped to the VAE's `4k+1` temporal grid. Keep the model's native FPS and change +the frame count to change duration — changing only FPS changes playback speed. + +### Other Wan knobs + +These exist for A/B and debugging; all of them make things slower except where +noted. `TS_WAN_DIT_KV_F16=0` restores F32 attention keys/values (F16 is the +default and is 2.02× faster at 27 k tokens, with no measurable accuracy cost); +`TS_WAN_VAE_MPS_CONV=0` restores ggml's im2col+GEMM conv lowering on Metal +(MPSGraph is the default and took a 736×544×81f VAE decode from 159 s to 80 s); +`TS_WAN_VAE_GEMM_MAX_MB` sets the im2col budget and `TS_WAN_VAE_TILE=0` disables +tiling; `TS_WAN_DIT_CAPTURE=0` disables the persistent CUDA-graph-captured DiT +graph; `TS_WAN_DIT_FLASH=0` forces materialized attention; +`TS_WAN_HEARTBEAT_S` sets the progress tick interval (default 30 s, `0` +silences it); `TS_FFMPEG` points at the `ffmpeg` used for near-lossless CRF 17 +H.264 export. + ## Mixture-of-Experts CPU offload (`--n-cpu-moe`) Large MoE models spend almost all of their bytes on routed experts while diff --git a/USAGE_zh-cn.md b/USAGE_zh-cn.md index dd02855a..0a569e7c 100644 --- a/USAGE_zh-cn.md +++ b/USAGE_zh-cn.md @@ -117,6 +117,21 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model H.264 MP4）。UMT5-XXL 文本编码器 GGUF 与视频 VAE +# 伴随文件会在 DiT GGUF 旁解析（或用 --wan-te / --wan-vae 指定）。 +# Wan 2.1 T2V、Wan 2.2 TI2V-5B 与 Wan 2.2 A14B（两个专家）都会自动识别； +# 步数蒸馏（Turbo / Lightning / FastWan）检查点同样自动识别——同一段视频只需 +# 4 次 DiT 前向而不是 100 次。详见下文“视频生成（Wan）”与 docs/models/wan_zh-cn.md。 +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model \ + --prompt "a lovely cat walking through a garden" --output cat.mp4 \ + --width 832 --height 480 --video-frames 49 --backend ggml_cuda \ + --diffusion-seed 7 +# Wan 2.2 图生视频：--image 提供首帧，提示词控制运动、镜头与场景变化 +# （TI2V-5B 或 I2V-A14B 检查点）。 +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model \ + --prompt "the cat runs toward the camera, cinematic tracking shot" \ + --image first_frame.png --output cat_run.mp4 --backend ggml_cuda + # 思维链 / 推理模式 dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model --input prompt.txt --backend ggml_metal --think @@ -214,7 +229,17 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --test-templates ~/models | `--qwen-image-vae ` | 覆盖解析到的 Qwen-Image VAE 伴随文件（`.gguf` 或 `.safetensors`）。 | | `--qwen-image-vl ` | 覆盖解析到的 Qwen2.5-VL-7B 文本编码器 GGUF。 | | `--qwen-image-mmproj ` | 覆盖解析到的 Qwen2.5-VL mmproj（视觉接地）GGUF。 | -| `--qwen-image-lora ` | Qwen-Image-Edit 的 Lightning 蒸馏 LoRA（`.safetensors`），在加载时合并进 DiT。自动推导步数（例如 4 或 8）并把 CFG 切换为 1.0。环境变量：`TS_QWEN_IMAGE_LORA`。 | +| `--qwen-image-lora ` | Qwen-Image-Edit 的 Lightning 蒸馏 LoRA（`.safetensors`）。它以运行期 F32 旁路的形式接在每个目标投影旁（`y = W_quant·x + b + (alpha/rank)·up·(down·x)`），量化基权重原样保留——**不会**被合并进权重。步数从文件名自动推导（例如 4 或 8），并把 CFG 切换为 1.0、时间步 shift 固定为 3，于是默认的 30 步 × 2 次 CFG 前向（60 次 DiT 前向）变成 4–8 次。它需要整模型或融合逐块的 CUDA 前向路径；在没有该旁路的路径上会直接报错而不是输出噪声。环境变量：`TS_QWEN_IMAGE_LORA`。 | +| `--width ` / `--height ` | Qwen-Image-Edit 与 Wan 视频的输出尺寸。默认 `0` —— 自动（Qwen-Image-Edit：源图尺寸，按 VRAM 钳制；Wan：按输入图的宽高比取模型原生面积，TI2V-5B 为 1280×704，其余为 832×480）。 | +| `--video-frames ` | Wan 视频帧数，会对齐到 `4k+1`（默认：33；Wan2.2-TI2V 为 49）。`1` 生成一张静态图（配合 `--output out.png`）。 | +| `--fps ` | 保存的 Wan MP4 的播放帧率（默认：16；Wan2.2-TI2V 为 24）。 | +| `--flow-shift ` | Wan FlowMatch 时间步 shift（默认：模型官方配方 —— Wan 2.2 为 5.0，A14B T2V 为 12.0，Wan 2.1 为 8.0/3.0/5.0）。 | +| `--sampler ` | Wan 采样器：`unipc`（官方采样器，默认）或 `euler`。 | +| `--negative-prompt ` | Wan 负向提示词（默认：官方 Wan 负向提示词）。 | +| _（步数蒸馏检查点）_ | 按 DiT 文件名自动识别（`Turbo`、`distill`、`Lightning`、`lightx2v`、`FastWan`、`-dmd`，或显式的 `…-4steps-…` / `…8step…`，N 取 1–16）：管线切换到该步数并关闭引导，把官方 50 步 × CFG 配方的 100 次 DiT 前向变成 4 次。这是 Wan 最大的提速手段——详见下文 **[视频生成（Wan）](#视频生成wan)**。`--diffusion-steps` / `--cfg` 可覆盖它。 | +| `--cfg-cache-stride ` | Wan 引导缓存：每 `N` 步只跑一次无条件 CFG 前向，其余步复用缓存的引导方向（默认关闭——每步都跑两次前向）。`2` 约快 1.30×，`3` 约快 1.43×；属于近似，需要严格对齐参考样本时请关闭。 | +| `--wan-vae ` | 覆盖解析到的 Wan 视频 VAE（`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`）。环境变量：`TS_WAN_VAE`。 | +| `--wan-te ` | 覆盖解析到的 UMT5-XXL 文本编码器 GGUF。环境变量：`TS_WAN_TE`。Wan 2.2 A14B 还会自动解析第二个 high/low-noise 专家（环境变量 `TS_WAN_DIT2`）。 | | `--test` | 运行内置的分词器、Qwen3 聊天模板与 ollama 对比测试 | | `--test-templates ` | 对 `` 下的每个 *.gguf 校验硬编码模板与 GGUF Jinja2 模板的一致性 | | `--config ` | 从 JSON 配置文件读取参数（命令行参数会覆盖它）。支持 `${变量}` 与通过 `{ "path": ..., "urls": [...] }` 自动下载模型。可重复。见[配置文件](#配置文件cli--server)。 | @@ -309,6 +334,12 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf # 默认以 24 fps 生成 121 帧（约五秒）。 dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/Wan2.2-TI2V-5B.gguf --backend ggml_cuda \ --video-frames 121 --fps 24 +# TI2V-5B 原生面积下的 121 帧是 27k 个 DiT token，而自注意力对 token 数是平方级， +# 因此在**基础**检查点上跑完 50 步在笔记本级 GPU 上需要数小时（实测 M5 Pro 约 3 小时 +# 30 分）。把 --model 换成步数蒸馏检查点后，同一个请求只需 17 分 30 秒——其他参数 +# 一律不变。详见下文“视频生成（Wan）”。 +# 服务端会打印每次前向的耗时与滚动 ETA，并每 30 秒发一次心跳，Web UI 两者都会显示； +# 完整成本表见 docs/models/wan_zh-cn.md。 # 配置服务端默认采样参数（仅在请求未自行覆盖时生效） dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal \ @@ -370,10 +401,10 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basi | `--kv-cache-dtype ` | 托管模型的 KV 缓存精度：`f32`、`f16`、`q8_0` 或 `q4_0`（量化缓存以微小数值漂移换取内存节省；各档位的取舍见上文 CLI 参数表）。默认：自动 —— 由后端 / 模型决定。环境变量：`KV_CACHE_DTYPE`。 | | `--continuous-batching` / `--no-continuous-batching` | 启用（默认）或关闭迭代级分页批处理。启用时服务会在批内动态加入 / 抢占序列，并在实现了 `IBatchedPagedModel` 的模型上将多个序列打包到一次前向中执行。`--no-continuous-batching` 会让所有模型回退到按序列 KV 交换。别名：`--paged-batching` / `--no-paged-batching`。 | | `--prefill-chunk-size ` | 存在竞争时的分块 prefill 粒度 —— 有其他请求同时运行时，每个调度步最多处理的 prefill token 数；块越小，并行 decode 请求越容易频繁轮到 GPU（默认：`1024`）。环境变量：`TS_SCHED_PREFILL_CHUNK`。 | -| `--mtp-spec` / `--no-mtp-spec` | 在带有多 token 预测草稿头的模型上启用 NextN/MTP 投机解码（默认关闭）。草稿头可以是 Qwen 3.6 内嵌的 NextN 块，或通过 `--mtp-draft-model` 加载的 Gemma 4 `gemma4-assistant` 草稿。仅对单序列（无并发）请求生效：草稿头每步最多提议 `--mtp-draft` 个 token，主干网络用一次批量前向完成验证；起草与验证均由该请求自己的采样器（含惩罚项）驱动，输出与标准 decode 一致。仅在有收益处自动启用（ggml 后端与纯 C# `cuda` 后端）；CPU / MLX 走标准 decode。环境变量：`TS_MTP_SPEC`。 | +| `--mtp-spec` / `--no-mtp-spec` | 在带有多 token 预测草稿头的模型上启用 NextN/MTP 投机解码（默认关闭）。草稿头可以是 Qwen 3.6 内嵌的 NextN 块，或通过 `--mtp-draft-model` 加载的 Gemma 4 `gemma4-assistant` 草稿。仅对单序列（无并发）请求生效：草稿头每步最多提议 `--mtp-draft` 个 token，主干网络用一次批量前向完成验证；起草与验证均由该请求自己的采样器（含惩罚项）驱动，输出与标准 decode 一致。仅在有收益处自动启用：Qwen 3.6 的内嵌 NextN 块在所有后端上都被认为有收益，而 Gemma 4 的独立草稿头只在各 ggml 后端与 Direct `cuda` 后端上启用；CPU / GGML CPU / MLX 走标准 decode。环境变量：`TS_MTP_SPEC`。 | | `--mtp-draft ` | 每个投机步最多起草的 token 数（默认 `8`）。环境变量：`TS_MTP_DRAFT`。 | | `--mtp-pmin ` | 草稿 token 被保留所需的最低置信度，取值 `(0, 1]`；遇到第一个低置信 token 即停止起草。默认值按草稿器类型选择：逐 token 草稿头为 `0.75`（其 top-10 logits 上的 top-1 概率），块级草稿器为 `0.35`——后者的门限是**累积**前缀概率，因此同一个数字要严格得多。环境变量：`TS_MTP_PMIN`。 | -| `--draft-model ` | 草稿器以独立文件发布的架构所用的投机解码草稿模型，即 DeepSeek V4 的 DSpark 支持 GGUF（见 [DeepSeek V4](docs/models/deepseek4_zh-cn.md#dspark-投机解码)）：它每步起草一整块 token，主干用一次批量前向验证，因此贪心输出保持不变。需要与 `--mtp-spec` 一起使用，在 `cuda` 与 `ggml_cuda` 后端上对单序列请求生效。与 CLI 不同，服务端的每一行验证都用该请求自己的采样器，因此可与任意采样设置组合。环境变量：`TS_DSV4_DSPARK`。 | +| `--draft-model ` | 草稿器以独立文件发布的架构所用的投机解码草稿模型：DeepSeek V4 的 DSpark 支持 GGUF（见 [DeepSeek V4](docs/models/deepseek4_zh-cn.md#dspark-投机解码)）与 Muse-Glimmer 的 DFlash 草稿器（见 [Muse-Glimmer](docs/models/muse-glimmer_zh-cn.md)，环境变量 `TS_MUSE_GLIMMER_DFLASH`）。两者都每步起草一整块 token，主干用一次批量前向验证，因此贪心输出保持不变。需要与 `--mtp-spec` 一起使用，在 `cuda` 与 `ggml_cuda` 后端上对单序列请求生效。与 CLI 不同，服务端的每一行验证都用该请求自己的采样器，因此可与任意采样设置组合。环境变量：`TS_DSV4_DSPARK`。 | | `--spec-draft-n-max ` | 每个投机块最多起草的 token 数（默认：草稿器训练时的块大小）。 | | `--spec-draft-conf-min

` | 保留某个起草位置所需的最小累积接受概率（置信度头各位置估计值的乘积，默认 `0.35`）。 | | `--mtp-draft-model ` | 对于草稿头作为独立文件发布的架构（Gemma 4 的 `gemma4-assistant`），指定其草稿 GGUF 路径。草稿的隐藏维度必须与目标一致（例如 12B 目标配 12B 草稿，而非 26B-A4B 草稿）；草稿不匹配或不完整会在启动时立即失败并给出修复提示。Qwen 3.6 将 NextN 块内嵌在主干 GGUF 中，此参数对其无效。环境变量：`TS_MTP_DRAFT_MODEL`。 | @@ -505,6 +536,142 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basi 于服务端参数与环境变量，其余参数仍由服务端填充。无论哪种模式，服务端 `--stop` 在 `config` 下始终生效（与请求的列表合并），在 `request` 下则被请求替换。 +## 视频生成（Wan） + +一个 `wan` GGUF 可以把提示词——Wan 2.2 模型还可再加一张首帧图片——变成 H.264 MP4， +`TensorSharp.Cli`、服务端的三个视频端点以及 Web UI 聊天都能驱动。完整的架构细节见 +[Wan 卡片](docs/models/wan_zh-cn.md)；本节是运维视角：该下载哪个检查点，以及哪些开关 +真正影响墙钟时间。 + +### 选哪个检查点 + +| 家族 | 潜空间 | 模式 | 说明 | +|---|---|---|---| +| Wan 2.2 TI2V-5B | 48 通道、16×16×4（`Wan2.2_VAE.safetensors`） | 文生视频 + 图生视频 | 稠密 5B、24 fps、原生 720p；同分辨率下 DiT token 数约为 Wan 2.1 的 1/2.7，因此在消费级 GPU 上既最快也质量最好 | +| Wan 2.2 A14B（T2V / I2V） | 16 通道（I2V 输入 36 通道），`wan_2.1_vae.safetensors` | 文生视频 + 图生视频 | 两个 14B 专家在时间步边界切换；**两个**专家 GGUF 都必须就位（同一目录，或 `HighNoise/` + `LowNoise/`） | +| Wan 2.1 T2V（1.3B / 14B） | 16 通道，`wan_2.1_vae.safetensors` | 文生视频 | 单个 DiT | + +每个家族都还需要 UMT5-XXL 文本编码器（`umt5-xxl-encoder-Q8_0.gguf`）和匹配的视频 VAE。 +这三个伴随文件都会从 DiT 自身所在目录解析，包括 `VAE/`、`HighNoise/`、`LowNoise/` 这类 +子目录，因此一个 `--local-dir` 就够了；`--wan-vae` / `--wan-te`（以及第二个 A14B 专家的 +`TS_WAN_DIT2`）可以覆盖搜索结果。 + +Wan 是唯一会直接拒绝某个后端的家族：它可以运行在 `ggml_cuda`、`ggml_vulkan`、 +`ggml_metal`、`ggml_cpu`、`cuda` 与 `cpu` 上，**不支持** `--backend mlx`。`ggml_cuda` +最快（RTX 2000 Ada、Wan2.1-1.3B F16、832×480×33f、30 步：`ggml_cuda` 12.0 秒/步， +`ggml_vulkan` 17.2 秒/步，Direct `cuda` 19.3 秒/步）；`cpu` / `ggml_cpu` 仅供功能性使用。 + +### 提速主路径：步数蒸馏检查点 + +**这是 Wan 最大的提速手段，代价只是换一个下载。** Wan2.2-TI2V-5B 的官方配方是 +50 步 × 2 次无分类器引导前向 = **100 次 DiT 前向**。步数蒸馏检查点（Turbo / Lightning / +FastWan / DMD）本身就是按无引导、少步数训练的，因此同一段视频只需 **4 次 DiT 前向**， +去噪工作量降到 1/25。 + +TensorSharp 从 DiT 的**文件名**识别它：不区分大小写地匹配 `turbo`、`distill`、 +`lightning`、`lightx2v`、`fastwan`、`-dmd`，或显式的 `steps` / `_steps`（1 ≤ N ≤ 16， +显式步数优先于标记）。只有标记而没有步数时默认为 4 步。加载时控制台会打印 + +``` +step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override) +``` + +因此可以从日志确认识别是否生效。这里没有任何参数——蒸馏 GGUF 就当作普通的 `--model` 传入。 + +在 M5 Pro（20 核 GPU、48 GB 统一内存）、`ggml_metal`、Wan2.2-TI2V-5B Q8_0、 +1088×832×121f = 27 404 token、图生视频（即完整的五秒 720p 级请求）上实测： + +| | 基础检查点 | **Turbo 检查点** | +|---|---|---| +| DiT 前向次数 | 100（50 步 × CFG） | **4**（4 步，无引导） | +| 每次前向 | 120.2 秒 | 120.2 秒 | +| 去噪合计 | 12 020 秒 | **481 秒** | +| VAE 解码 121 帧 | 563 秒 | 563 秒 | +| **端到端** | **约 3 小时 30 分** | **17 分 30 秒** | + +这两列之间只有 `--model` 路径不同。一旦用上蒸馏检查点，瓶颈就变成 VAE 解码（约占整段 +运行的 55%），而不再是 DiT。 + +下载方式（TI2V-5B —— 注意文件名里的版本号用的是**下划线** `Wan2_2`，与基础仓库不同）： + +```bash +pip install -U huggingface_hub +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +# Turbo 仓库不含 VAE 与文本编码器——从基础仓库取 +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \ + --backend ggml_metal --image first_frame.png --output out.mp4 \ + --prompt "the cat runs toward the camera, cinematic tracking shot" \ + --video-frames 121 --fps 24 + +dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \ + --backend ggml_metal --video-frames 121 --fps 24 +``` + +Wan 2.2 I2V-A14B 的即插即用蒸馏 GGUF 是 +[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF)， +它已经把蒸馏合并进两个专家，分别放在 `high_noise/` 与 `low_noise/` 下；两个都下载， +`--model` 指向其中任意一个即可。该仓库同样不含 VAE 与文本编码器，因此 +`VAE/Wan2.1_VAE.safetensors` 取自 +[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)， +UMT5-XXL 编码器同上。 + +> [lightx2v/Wan2.2-Lightning](https://huggingface.co/lightx2v/Wan2.2-Lightning) +> 只发布 LoRA `.safetensors`，而 TensorSharp 没有任何 Wan LoRA 参数——请选择已经把 +> 蒸馏烘焙进 GGUF 的仓库。 + +### `--cfg-cache-stride`（仅基础检查点） + +一次带引导的去噪步是 `v = v_cond + (cfg-1)·d`，其中 `d = v_cond - v_uncond`。引导方向 `d` +在整个调度上的变化远比 `v` 缓慢，因此 `--cfg-cache-stride N` 让无条件前向每 `N` 步只跑 +一次，中间各步复用缓存的 `d`。在 50 步下，`2` 只跑 100 次前向中的 77 次（**1.30×**）， +`3` 跑 70 次（**1.43×**）。前三步与最后一步始终重新计算 `d`。服务端 JSON 字段： +`"cfgCacheStride": 2`。 + +这是一种近似——需要严格对齐参考样本时请关闭；在步数蒸馏检查点上也没有意义，因为它们本来 +就无引导运行（cfg ≤ 1.0 时缓存会被自动禁用）。 + +### 让一个大请求变便宜，按收益排序 + +1. **换用步数蒸馏检查点。** 100 次 DiT 前向变成 4 次，收益远超其他所有手段。 +2. **减少帧数。** 121 → 61 大约把注意力工作量降到 1/4（token 数是 + `latent_frames × (h/2) × (w/2)`，自注意力是 `O(tokens²)`），VAE 解码减半。 +3. **减小画面面积**——但不要低于 Wan 的训练分辨率。低于约 0.3 MP 后模型进入分布外， + 视频会变*差*而不只是变便宜，管线也会给出警告。Wan 在 480p（832×480）与 + 720p（1280×704）上训练，所以请在受支持的尺寸上生成，再事后缩小。 +4. **减少步数**，仅限基础检查点——30 步相对官方 50 步在观感上很接近，成本降到 1/1.7。 +5. **`--cfg-cache-stride 2` 或 `3`**——1.30× / 1.43×，仅限基础检查点。 + +同一台 M5 Pro、同一个 Turbo 检查点与输入图，分辨率与墙钟时间的关系： + +| 输出 | Token 数 | 去噪 | VAE 解码 | **合计** | +|---|---|---|---|---| +| 736×544 × 81f（3.4 秒，480p 级） | 8 211 | 84 秒 | 159 秒 | **4 分 09 秒** | +| 736×544 × 121f（5 秒，480p 级） | 12 121 | 137 秒 | 237 秒 | **6 分 19 秒** | +| 1088×832 × 121f（5 秒，720p 级） | 27 404 | 481 秒 | 563 秒 | **17 分 30 秒** | + +480p（约 0.4 MP）是 Wan *训练过*的分辨率，因此前两行属于分布内，而不是降级模式——当你 +只能接受几分钟时，就该选它。 + +### 服务端默认值 + +`--video-frames N` 与 `--fps N` 为 Web UI 和三个视频端点设置服务端级**默认值**，不是上限； +请求中提供的 `frames` 或 `fps` 会各自独立覆盖。两者都不提供时使用模型配方：Wan2.2-TI2V +为 49 帧 / 24 fps，其余为 33 帧 / 16 fps。帧数会对齐到 VAE 的 `4k+1` 时序网格。改变时长 +时请保持模型原生 FPS 而调整帧数——只改 FPS 只会改变播放速度。 + +### 其他 Wan 开关 + +这些开关用于 A/B 与调试，除注明外都会让速度变慢。`TS_WAN_DIT_KV_F16=0` 恢复 F32 的注意力 +键值（F16 是默认值，在 27k token 下快 2.02×，且没有可测的精度损失）； +`TS_WAN_VAE_MPS_CONV=0` 在 Metal 上恢复 ggml 的 im2col+GEMM 卷积下降路径（MPSGraph 是默认 +值，把 736×544×81f 的 VAE 解码从 159 秒降到 80 秒）；`TS_WAN_VAE_GEMM_MAX_MB` 设置 im2col +预算，`TS_WAN_VAE_TILE=0` 关闭分块；`TS_WAN_DIT_CAPTURE=0` 关闭常驻的 CUDA 图捕获 DiT 图； +`TS_WAN_DIT_FLASH=0` 强制走物化注意力；`TS_WAN_HEARTBEAT_S` 设置进度心跳间隔（默认 30 秒， +`0` 静默）；`TS_FFMPEG` 指定用于近无损 CRF 17 H.264 导出的 `ffmpeg`。 + ## 张量并行与分布式推理 TensorSharp 支持**张量并行（TP）**——按 Megatron-LM 列/行并行范式把单个模型切分 diff --git a/benchmarks/WanVideoBench/Program.cs b/benchmarks/WanVideoBench/Program.cs new file mode 100644 index 00000000..a746680c --- /dev/null +++ b/benchmarks/WanVideoBench/Program.cs @@ -0,0 +1,237 @@ +// Wan video-generation benchmark harness. +// +// A 5-second 720p video is one long DiT loop plus one long VAE decode, and the +// knobs that decide their cost (attention KV precision, the VAE conv im2col +// budget, band tiling, the Metal 4 tensor API) are process-global and read at +// backend init. Sweeping them through a whole generation costs hours per data +// point, so this harness runs each stage in isolation against synthetic inputs +// of the real shapes. +// +// Modes: +// vae-decode [t] [lh] [lw] [backend] +// Decode a synthetic latent; reports MP/s and sanity-checks that the +// frames are a real picture (the Metal tensor-API mul_mm bug decodes to a +// flat/NaN plane, which this catches). +// dit [t] [hLen] [wLen] [backend] [passes] +// Time DiT forwards at a given token grid. +using System; +using System.Diagnostics; +using System.IO; +using TensorSharp.GGML; +using TensorSharp.Models.WanVideo; +using TensorSharp.Runtime; + +internal static class Program +{ + private static int Main(string[] args) + { + if (args.Length == 0) { Usage(); return 1; } + return args[0] switch + { + "vae-decode" => VaeDecode(args), + "dit" => Dit(args), + "pipeline" => Pipeline(args), + _ => Usage(), + }; + } + + private static int Usage() + { + Console.WriteLine(""" + WanVideoBench ... + + vae-decode [t=31] [lh=52] [lw=68] [backend=metal] + Wan 2.2 VAE decode of a synthetic [z, t, lh, lw] latent. The + defaults are the 1088x832x121-frame image-to-video shape. + + dit [t=31] [hLen=26] [wLen=34] [backend=metal] [passes=2] + Wan DiT forwards over a t x hLen x wLen token grid (the same + defaults = 27404 tokens). + + Knobs (read at backend init, so set them in the environment): + TS_WAN_DIT_KV_F16=0 F32 attention keys/values + TS_WAN_VAE_GEMM_MAX_MB= VAE conv im2col budget + TS_WAN_VAE_TILE=0 disable VAE band tiling + GGML_METAL_TENSOR_ENABLE / GGML_METAL_TENSOR_DISABLE + """); + return 1; + } + + private static GgmlBackendType Backend(string name) => (name ?? "metal").ToLowerInvariant() switch + { + "cuda" => GgmlBackendType.Cuda, + "vulkan" => GgmlBackendType.Vulkan, + "cpu" => GgmlBackendType.Cpu, + _ => GgmlBackendType.Metal, + }; + + private static int Arg(string[] a, int i, int fallback) + => a.Length > i && int.TryParse(a[i], out int v) && v > 0 ? v : fallback; + + private static string Knobs() => + $"kv_f16={Environment.GetEnvironmentVariable("TS_WAN_DIT_KV_F16") ?? "1"} " + + $"gemm_mb={Environment.GetEnvironmentVariable("TS_WAN_VAE_GEMM_MAX_MB") ?? "auto"} " + + $"tile={Environment.GetEnvironmentVariable("TS_WAN_VAE_TILE") ?? "1"} " + + $"tensor={(Environment.GetEnvironmentVariable("GGML_METAL_TENSOR_DISABLE") != null ? "off" : Environment.GetEnvironmentVariable("GGML_METAL_TENSOR_ENABLE") != null ? "on" : "auto")}"; + + private static int VaeDecode(string[] args) + { + if (args.Length < 2 || !File.Exists(args[1])) + { + Console.Error.WriteLine($"vae-decode: VAE safetensors not found: {(args.Length > 1 ? args[1] : "")}"); + return 1; + } + int t = Arg(args, 2, 31), lh = Arg(args, 3, 52), lw = Arg(args, 4, 68); + GgmlBasicOps.EnsureBackendAvailable(Backend(args.Length > 5 ? args[5] : null)); + + using var vae = new WanVae(args[1]); + var rng = new Random(1234); + var latent = new float[(long)vae.ZDim * t * lh * lw]; + for (long i = 0; i < latent.LongLength; i++) latent[i] = (float)(rng.NextDouble() * 2 - 1); + + var sw = Stopwatch.StartNew(); + var frames = vae.Decode(latent, t, lh, lw); + sw.Stop(); + + double mp = (double)frames.Length * frames[0].Width * frames[0].Height / 1e6; + // A corrupt decode (the tensor-API mul_mm failure mode) comes back flat or + // NaN. "Not flat" is necessary but nowhere near sufficient — that bug was + // layout-dependent and could perturb a subset of columns — so also emit a + // digest over EVERY pixel, which a caller diffs across configurations. + float min = 1f, max = 0f; + double mean = 0, absMean = 0; + long nan = 0, n = 0; + ulong hash = 1469598103934665603; // FNV-1a over pixels quantized to 1/1024 + foreach (var f in frames) + { + foreach (float v in f.Pixels) + { + if (float.IsNaN(v) || float.IsInfinity(v)) { nan++; continue; } + if (v < min) min = v; + if (v > max) max = v; + mean += v; absMean += Math.Abs(v); n++; + hash = (hash ^ (ulong)(uint)(int)MathF.Round(v * 1024f)) * 1099511628211; + } + } + mean /= Math.Max(1, n); absMean /= Math.Max(1, n); + bool ok = max > min + 0.05f && nan == 0; + + Console.WriteLine( + $"[vae-decode] {frames[0].Width}x{frames[0].Height}x{frames.Length}f ({mp:F1} MP) " + + $"{sw.Elapsed.TotalSeconds:F1}s = {mp / sw.Elapsed.TotalSeconds:F2} MP/s | " + + $"min={min:F4} max={max:F4} mean={mean:F6} absmean={absMean:F6} nan={nan} digest={hash:x16} " + + $"{(ok ? "OK" : "*** FLAT/NaN ***")} | {Knobs()}"); + + // TS_WAN_BENCH_DUMP=: raw F32 pixels, so two configurations can be + // compared with a real PSNR instead of summary statistics. + string dump = Environment.GetEnvironmentVariable("TS_WAN_BENCH_DUMP"); + if (!string.IsNullOrEmpty(dump)) + { + using var fs = new FileStream(dump, FileMode.Create, FileAccess.Write); + using var bw = new BinaryWriter(fs); + foreach (var f in frames) + foreach (float v in f.Pixels) bw.Write(v); + Console.WriteLine($"[vae-decode] wrote {dump}"); + } + return ok ? 0 : 2; + } + + ///

+ /// DiT-then-VAE in one process, the way a real generation runs. The Metal 4 + /// tensor-API VAE corruption follows the process's ALLOCATION HISTORY, so a + /// fresh-process `vae-decode` cannot see it — it decoded clean at every shape + /// while full generations came out black. This reproduces the real ordering + /// (load DiT, forward, release its device residency, then decode) in minutes + /// instead of the ~17 a 121-frame video takes. + /// + private static int Pipeline(string[] args) + { + if (args.Length < 3 || !File.Exists(args[1]) || !File.Exists(args[2])) + { + Console.Error.WriteLine("pipeline [ditT=31] [vaeT=3] [lh=52] [lw=68]"); + return 1; + } + int ditT = Arg(args, 3, 31), vaeT = Arg(args, 4, 3), lh = Arg(args, 5, 52), lw = Arg(args, 6, 68); + GgmlBasicOps.EnsureBackendAvailable(GgmlBackendType.Metal); + + int hLen = lh / 2, wLen = lw / 2; + var sw = Stopwatch.StartNew(); + + // Image-to-video runs the VAE ENCODER before the DiT, so the encoder's + // buffers are part of the allocation history the decode inherits. Leaving + // it out is what made the first version of this repro come back clean. + { + int H = lh * 16, W = lw * 16; + using var vaeEnc = new WanVae(args[2]); + var pixels = new float[3L * 1 * H * W]; + var prng = new Random(11); + for (long i = 0; i < pixels.LongLength; i++) pixels[i] = (float)(prng.NextDouble() * 2 - 1); + float[] mu = vaeEnc.Encode(pixels, 1, H, W); + Console.WriteLine($"[pipeline] VAE encode {W}x{H}x1: {sw.Elapsed.TotalSeconds:F1}s, mu[0]={mu[0]:F4}"); + } + GgmlBasicOps.ReleaseReuseComputeBuffers(); + GgmlBasicOps.ClearHostBufferCache(); + using (var gguf = new GgufFile(args[1])) + using (var dit = new WanDiT(gguf)) + { + int seq = ditT * hLen * wLen; + var x = new float[(long)dit.InTok * seq]; + var rng = new Random(7); + for (long i = 0; i < x.LongLength; i++) x[i] = (float)(rng.NextDouble() * 2 - 1); + var ctx = new float[WanDiT.TextDim * WanTextEncoder.TextLen]; + for (long i = 0; i < ctx.LongLength; i++) ctx[i] = (float)(rng.NextDouble() * 0.2 - 0.1); + float[] v = dit.Predict(x, ctx, 999f, WanRope.Build(ditT, hLen, wLen), seq); + long nan = 0; + foreach (float f in v) if (float.IsNaN(f) || float.IsInfinity(f)) nan++; + Console.WriteLine($"[pipeline] DiT {ditT}x{hLen}x{wLen} = {seq} tokens: {sw.Elapsed.TotalSeconds:F1}s, nan={nan}"); + } + // Exactly what WanVideoPipeline does between stages. + GgmlBasicOps.ReleaseReuseComputeBuffers(); + GgmlBasicOps.ClearHostBufferCache(); + + var vargs = new[] { "vae-decode", args[2], vaeT.ToString(), lh.ToString(), lw.ToString(), "metal" }; + int rc = VaeDecode(vargs); + Console.WriteLine($"[pipeline] {(rc == 0 ? "DECODE OK" : "*** DECODE CORRUPT ***")} | total {sw.Elapsed.TotalSeconds:F1}s"); + return rc; + } + + private static int Dit(string[] args) + { + if (args.Length < 2 || !File.Exists(args[1])) + { + Console.Error.WriteLine($"dit: GGUF not found: {(args.Length > 1 ? args[1] : "")}"); + return 1; + } + int t = Arg(args, 2, 31), hLen = Arg(args, 3, 26), wLen = Arg(args, 4, 34); + int passes = Arg(args, 6, 2); + GgmlBasicOps.EnsureBackendAvailable(Backend(args.Length > 5 ? args[5] : null)); + + using var gguf = new GgufFile(args[1]); + using var dit = new WanDiT(gguf); + int seq = t * hLen * wLen; + var x = new float[(long)dit.InTok * seq]; + var rng = new Random(7); + for (long i = 0; i < x.LongLength; i++) x[i] = (float)(rng.NextDouble() * 2 - 1); + var ctx = new float[WanDiT.TextDim * WanTextEncoder.TextLen]; + for (long i = 0; i < ctx.LongLength; i++) ctx[i] = (float)(rng.NextDouble() * 0.2 - 0.1); + var rope = WanRope.Build(t, hLen, wLen); + + Console.WriteLine($"[dit] {t}x{hLen}x{wLen} = {seq} tokens, {dit.Layers} blocks, dim={dit.Dim} | {Knobs()}"); + double best = double.MaxValue, total = 0; + for (int i = 0; i < passes; i++) + { + var sw = Stopwatch.StartNew(); + float[] v = dit.Predict(x, ctx, 999f, rope, seq); + sw.Stop(); + double s = sw.Elapsed.TotalSeconds; + best = Math.Min(best, s); + if (i > 0) total += s; // pass 0 carries the weight upload + graph build + double mn = double.MaxValue, mx = double.MinValue; + long nan = 0; + foreach (float f in v) { if (float.IsNaN(f) || float.IsInfinity(f)) { nan++; continue; } if (f < mn) mn = f; if (f > mx) mx = f; } + Console.WriteLine($" pass {i + 1}/{passes}: {s:F1}s v[min={mn:E2} max={mx:E2} nan={nan}]"); + } + Console.WriteLine($"[dit] best {best:F1}s, steady-state mean {(passes > 1 ? total / (passes - 1) : best):F1}s"); + return 0; + } +} diff --git a/benchmarks/WanVideoBench/WanVideoBench.csproj b/benchmarks/WanVideoBench/WanVideoBench.csproj new file mode 100644 index 00000000..32a0a6bd --- /dev/null +++ b/benchmarks/WanVideoBench/WanVideoBench.csproj @@ -0,0 +1,16 @@ + + + Exe + net10.0 + enable + disable + false + true + Wan video-generation benchmark harness: isolated VAE decode and DiT forward timings, so the knobs that dominate a long video (attention precision, VAE conv budget, band tiling, Metal tensor API) can be swept in minutes instead of hours. + + + + + + + diff --git a/config/qwen-image-edit-2511.json b/config/qwen-image-edit-2511.json index e0c5251c..c1fc7829 100644 --- a/config/qwen-image-edit-2511.json +++ b/config/qwen-image-edit-2511.json @@ -12,18 +12,18 @@ // --cfg, --diffusion-seed, --width, --height. With the Lightning LoRA merged // the sampler auto-selects its fast few-step schedule. "variables": { - "modelRoot": "C:/Works/models" + "modelRoot": "/Users/zhongkaifu/work/models/Qwen" }, - "backend": "ggml_cuda", + "backend": "ggml_metal", // Stream DiT weights from RAM so native-resolution edits fit on VRAM-limited // cards. Drop this if you have plenty of VRAM and want maximum speed. - "offload-cpu": true, + "offload-cpu": false, - "model": "${modelRoot}/qwen-image-edit-2511-Q4_K_M.gguf", + "model": "${modelRoot}/Qwen-image-edit-2511-Q4_K_M.gguf", "qwen-image-vae": "${modelRoot}/Qwen_Image-VAE.safetensors", - "qwen-image-vl": "${modelRoot}/qwen-image-te-Qwen2.5-VL-7B-Q4_K_M.gguf", - "qwen-image-mmproj": "${modelRoot}/Qwen2.5-VL-7B-mmproj-BF16.gguf", + "qwen-image-vl": "${modelRoot}/Qwen2.5-VL-7B-Instruct-abliterated.Q4_K_M.gguf", + "qwen-image-mmproj": "${modelRoot}/Qwen2.5-VL-7B-Instruct-abliterated.mmproj-Q8_0.gguf", // Lightning step-distillation LoRA (8 steps). Swap in the 4steps file for an // even faster, slightly lower-quality edit. diff --git a/docs/model_cards.md b/docs/model_cards.md index 3dec04e8..9d12f9ce 100644 --- a/docs/model_cards.md +++ b/docs/model_cards.md @@ -23,17 +23,20 @@ The matching `mmproj` is optional for text and required for image, video, or audio input; no particular public-file checksum is asserted as the benchmark input. -| Architecture | English card | 中文卡片 | -|---|---|---| -| Gemma 3 | [models/gemma3.md](models/gemma3.md) | [models/gemma3_zh-cn.md](models/gemma3_zh-cn.md) | -| Gemma 4 | [models/gemma4.md](models/gemma4.md) | [models/gemma4_zh-cn.md](models/gemma4_zh-cn.md) | -| DiffusionGemma | [models/diffusiongemma.md](models/diffusiongemma.md) | [models/diffusiongemma_zh-cn.md](models/diffusiongemma_zh-cn.md) | -| Qwen 3 | [models/qwen3.md](models/qwen3.md) | [models/qwen3_zh-cn.md](models/qwen3_zh-cn.md) | -| Qwen 3.5 / 3.6 family | [models/qwen35.md](models/qwen35.md) | [models/qwen35_zh-cn.md](models/qwen35_zh-cn.md) | -| GPT OSS | [models/gptoss.md](models/gptoss.md) | [models/gptoss_zh-cn.md](models/gptoss_zh-cn.md) | -| Nemotron-H | [models/nemotron.md](models/nemotron.md) | [models/nemotron_zh-cn.md](models/nemotron_zh-cn.md) | -| Mistral 3 | [models/mistral3.md](models/mistral3.md) | [models/mistral3_zh-cn.md](models/mistral3_zh-cn.md) | -| Qwen-Image-Edit | [models/qwenimage.md](models/qwenimage.md) | [models/qwenimage_zh-cn.md](models/qwenimage_zh-cn.md) | +| Architecture | GGUF arch keys | What it does | English card | 中文卡片 | +|---|---|---|---|---| +| DeepSeek V4 Flash | `deepseek4` | Sparse-MoE text model with compressed attention; DSpark block speculative decoding via a separate `--draft-model` GGUF | [models/deepseek4.md](models/deepseek4.md) | [models/deepseek4_zh-cn.md](models/deepseek4_zh-cn.md) | +| Gemma 3 | `gemma3` | Dense text + image chat | [models/gemma3.md](models/gemma3.md) | [models/gemma3_zh-cn.md](models/gemma3_zh-cn.md) | +| Gemma 4 | `gemma4` | Dense and MoE text + image + video + audio chat, thinking, tools; MTP speculative decoding with a separate `gemma4-assistant` draft GGUF | [models/gemma4.md](models/gemma4.md) | [models/gemma4_zh-cn.md](models/gemma4_zh-cn.md) | +| DiffusionGemma | `diffusion-gemma`, `diffusion_gemma` | Text **diffusion** generation — an EntropyBound denoising sampler instead of autoregressive decode | [models/diffusiongemma.md](models/diffusiongemma.md) | [models/diffusiongemma_zh-cn.md](models/diffusiongemma_zh-cn.md) | +| Qwen 3 | `qwen3`, `qwen2`, `qwen2vl`, `qwen2_vl` | Dense text chat with thinking and tools (Qwen2 / Qwen2.5-VL checkpoints load as text-only chat) | [models/qwen3.md](models/qwen3.md) | [models/qwen3_zh-cn.md](models/qwen3_zh-cn.md) | +| Qwen 3.5 / 3.6 family | `qwen35`, `qwen35moe`, `qwen3next` | Hybrid full-attention + GatedDeltaNet text + image chat, dense or MoE; Qwen 3.6 embeds a NextN draft block for speculative decoding | [models/qwen35.md](models/qwen35.md) | [models/qwen35_zh-cn.md](models/qwen35_zh-cn.md) | +| GPT OSS | `gptoss`, `gpt-oss` | MXFP4 MoE text model with attention sinks and Harmony thinking/tools | [models/gptoss.md](models/gptoss.md) | [models/gptoss_zh-cn.md](models/gptoss_zh-cn.md) | +| Nemotron-H | `nemotron_h`, `nemotron_h_moe` | Hybrid Mamba2 SSM + attention + (MoE) FFN text model; the Omni checkpoints add image input | [models/nemotron.md](models/nemotron.md) | [models/nemotron_zh-cn.md](models/nemotron_zh-cn.md) | +| Mistral 3 | `mistral3` | Dense text + image chat with YaRN-corrected RoPE and the Pixtral vision encoder | [models/mistral3.md](models/mistral3.md) | [models/mistral3_zh-cn.md](models/mistral3_zh-cn.md) | +| Muse-Glimmer | `muse-glimmer`, `muse_glimmer` | Interleaved-SWA text + image chat with thinking and ATEM tools; DFlash block speculative decoding via a separate `--draft-model` GGUF | [models/muse-glimmer.md](models/muse-glimmer.md) | [models/muse-glimmer_zh-cn.md](models/muse-glimmer_zh-cn.md) | +| Qwen-Image-Edit | `qwen_image`, `qwen-image` | **Image editing** — prompt + input image → edited image, through a 60-block MMDiT diffusion loop; a Lightning LoRA cuts 60 DiT forwards to 4–8 | [models/qwenimage.md](models/qwenimage.md) | [models/qwenimage_zh-cn.md](models/qwenimage_zh-cn.md) | +| Wan video | `wan`, `wan2.1`, `wan2.2` | **Video generation** — prompt (+ optional first frame) → H.264 MP4, Wan 2.1 T2V and Wan 2.2 TI2V-5B / A14B; a step-distilled checkpoint turns the 100-DiT-pass recipe into 4 | [models/wan.md](models/wan.md) | [models/wan_zh-cn.md](models/wan_zh-cn.md) | Each card walks an engineer or researcher from "I have never heard of this model" to "I can explain the forward graph and reproduce the inference path diff --git a/docs/model_cards_zh-cn.md b/docs/model_cards_zh-cn.md index 8510e176..76ae0ecf 100644 --- a/docs/model_cards_zh-cn.md +++ b/docs/model_cards_zh-cn.md @@ -19,17 +19,20 @@ 纯文本不需要 `mmproj`；图像、视频或音频输入需要匹配的 `mmproj`。这里不声称基准输入 对应某个公开文件的特定校验和。 -| 架构 | 中文卡片 | English card | -|---|---|---| -| Gemma 3 | [models/gemma3_zh-cn.md](models/gemma3_zh-cn.md) | [models/gemma3.md](models/gemma3.md) | -| Gemma 4 | [models/gemma4_zh-cn.md](models/gemma4_zh-cn.md) | [models/gemma4.md](models/gemma4.md) | -| DiffusionGemma | [models/diffusiongemma_zh-cn.md](models/diffusiongemma_zh-cn.md) | [models/diffusiongemma.md](models/diffusiongemma.md) | -| Qwen 3 | [models/qwen3_zh-cn.md](models/qwen3_zh-cn.md) | [models/qwen3.md](models/qwen3.md) | -| Qwen 3.5 / 3.6 family | [models/qwen35_zh-cn.md](models/qwen35_zh-cn.md) | [models/qwen35.md](models/qwen35.md) | -| GPT OSS | [models/gptoss_zh-cn.md](models/gptoss_zh-cn.md) | [models/gptoss.md](models/gptoss.md) | -| Nemotron-H | [models/nemotron_zh-cn.md](models/nemotron_zh-cn.md) | [models/nemotron.md](models/nemotron.md) | -| Mistral 3 | [models/mistral3_zh-cn.md](models/mistral3_zh-cn.md) | [models/mistral3.md](models/mistral3.md) | -| Qwen-Image-Edit | [models/qwenimage_zh-cn.md](models/qwenimage_zh-cn.md) | [models/qwenimage.md](models/qwenimage.md) | +| 架构 | GGUF 架构标识 | 功能 | 中文卡片 | English card | +|---|---|---|---|---| +| DeepSeek V4 Flash | `deepseek4` | 带压缩注意力的稀疏 MoE 文本模型；通过独立 `--draft-model` GGUF 支持 DSpark 块级投机解码 | [models/deepseek4_zh-cn.md](models/deepseek4_zh-cn.md) | [models/deepseek4.md](models/deepseek4.md) | +| Gemma 3 | `gemma3` | 稠密文本 + 图像对话 | [models/gemma3_zh-cn.md](models/gemma3_zh-cn.md) | [models/gemma3.md](models/gemma3.md) | +| Gemma 4 | `gemma4` | 稠密与 MoE 的文本 + 图像 + 视频 + 音频对话，支持思维链与工具调用；通过独立 `gemma4-assistant` 草稿 GGUF 支持 MTP 投机解码 | [models/gemma4_zh-cn.md](models/gemma4_zh-cn.md) | [models/gemma4.md](models/gemma4.md) | +| DiffusionGemma | `diffusion-gemma`、`diffusion_gemma` | 文本**扩散**生成——用 EntropyBound 去噪采样器取代自回归 decode | [models/diffusiongemma_zh-cn.md](models/diffusiongemma_zh-cn.md) | [models/diffusiongemma.md](models/diffusiongemma.md) | +| Qwen 3 | `qwen3`、`qwen2`、`qwen2vl`、`qwen2_vl` | 稠密文本对话，支持思维链与工具调用（Qwen2 / Qwen2.5-VL 检查点按纯文本对话加载） | [models/qwen3_zh-cn.md](models/qwen3_zh-cn.md) | [models/qwen3.md](models/qwen3.md) | +| Qwen 3.5 / 3.6 family | `qwen35`、`qwen35moe`、`qwen3next` | 全注意力 + GatedDeltaNet 混合的文本 + 图像对话，稠密或 MoE；Qwen 3.6 内嵌 NextN 草稿块用于投机解码 | [models/qwen35_zh-cn.md](models/qwen35_zh-cn.md) | [models/qwen35.md](models/qwen35.md) | +| GPT OSS | `gptoss`、`gpt-oss` | 带 attention sinks 的 MXFP4 MoE 文本模型，使用 Harmony 思维链 / 工具格式 | [models/gptoss_zh-cn.md](models/gptoss_zh-cn.md) | [models/gptoss.md](models/gptoss.md) | +| Nemotron-H | `nemotron_h`、`nemotron_h_moe` | Mamba2 SSM + 注意力 +（MoE）FFN 混合文本模型；Omni 版本增加图像输入 | [models/nemotron_zh-cn.md](models/nemotron_zh-cn.md) | [models/nemotron.md](models/nemotron.md) | +| Mistral 3 | `mistral3` | 稠密文本 + 图像对话，YaRN 校正 RoPE 与 Pixtral 视觉编码器 | [models/mistral3_zh-cn.md](models/mistral3_zh-cn.md) | [models/mistral3.md](models/mistral3.md) | +| Muse-Glimmer | `muse-glimmer`、`muse_glimmer` | 交错滑动窗口的文本 + 图像对话，支持思维链与 ATEM 工具调用；通过独立 `--draft-model` GGUF 支持 DFlash 块级投机解码 | [models/muse-glimmer_zh-cn.md](models/muse-glimmer_zh-cn.md) | [models/muse-glimmer.md](models/muse-glimmer.md) | +| Qwen-Image-Edit | `qwen_image`、`qwen-image` | **图像编辑**——提示词 + 输入图像 → 编辑后的图像，走 60 块 MMDiT 扩散循环；Lightning LoRA 可把 60 次 DiT 前向降到 4–8 次 | [models/qwenimage_zh-cn.md](models/qwenimage_zh-cn.md) | [models/qwenimage.md](models/qwenimage.md) | +| Wan 视频 | `wan`、`wan2.1`、`wan2.2` | **视频生成**——提示词（可选首帧图）→ H.264 MP4，涵盖 Wan 2.1 T2V 与 Wan 2.2 TI2V-5B / A14B；换用步数蒸馏检查点可把 100 次 DiT 前向的官方配方降到 4 次 | [models/wan_zh-cn.md](models/wan_zh-cn.md) | [models/wan.md](models/wan.md) | 每张卡片会把工程师或研究员从“从未听说过这个模型”带到“可以解释它的前向计算图， 并能在 TensorSharp 中复现推理路径”，统一覆盖： diff --git a/docs/models/README.md b/docs/models/README.md index d4e284c8..72d3783e 100644 --- a/docs/models/README.md +++ b/docs/models/README.md @@ -65,14 +65,14 @@ a complete multimodal inference engine—use Zhongkai Fu's | Gemma 3 | [gemma3.md](gemma3.md) | [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | `Gemma3Model` | `gemma3` | Text, image | No | No | No (legacy per-seq) | Alternating SWA / global attention, GeGLU FFN, QK-norm, V-norm | | Gemma 4 | [gemma4.md](gemma4.md) | E4B Q8_0 is the verified native-GGML family/path tier; [ggml-org/gemma-4-E4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) is the recommended public artifact | `Gemma4Model` | `gemma4` (`gemma4-assistant` / `gemma4_assistant` load only as the MTP draft) | Text, image, video, audio | Yes | Yes | **Default** (toggle off with `TS_GEMMA4_BATCHED=0`) | Single-graph fused decode (all layers in one GGML dispatch), fused whole-model prefill/verify with in-kernel PLE + shared-KV handling, chunked prefill, circular SWA cache, and MoE variants. Batched path matches legacy logits within FP noise (`Gemma4BatchedForwardTests`); reaches ~1.5× legacy at batch=8 and ~1.6× at 4×800-token prompts. | | DiffusionGemma | [diffusiongemma.md](diffusiongemma.md) | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | `DiffusionGemmaModel` + `DiffusionGemmaSampler` | `diffusion-gemma`, `diffusion_gemma` | Text | No | No | Separate Web UI `DiffusionBatchScheduler`; not an autoregressive `IBatchedPagedModel` path | EntropyBound block denoising over `[prompt \| canvas]`, prompt-KV caching on GPU backends, self-conditioning, fused GGML whole-model diffusion decode and fused lm-head tail | -| Qwen-Image-Edit | [qwenimage.md](qwenimage.md) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (DiT; VAE / text-encoder companions in the card) | `QwenImageModel` (+ `QwenImagePipeline`) | `qwen_image`, `qwen-image` | Image edit (image+text → image) | No | No | None — `Forward()` throws; editing runs through `EditImage()` and edits are serialized | 60-block MMDiT diffusion (FlowMatch-Euler, true-CFG, reference-latent concat), CUDA-graph-captured whole-DiT forward, default flash attention, CFG-batching, First-Block-Cache, fused Qwen2.5-VL vision encoder, VRAM-aware area clamp | -| Qwen 3 | [qwen3.md](qwen3.md) | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | `Qwen3Model` | `qwen3` | Text | Yes | Yes | Reference port (`Qwen3Model.BatchedForward.cs`) — exercised by `Qwen3BatchedForwardTests` when a base-Qwen3 GGUF is provided | Native whole-model decode with pre-resolved weight pointers | +| Qwen-Image-Edit | [qwenimage.md](qwenimage.md) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (DiT; VAE / text-encoder companions in the card) | `QwenImageModel` (+ `QwenImagePipeline`) | `qwen_image`, `qwen-image` | Image edit (image+text → image) | No | No | None — `Forward()` throws; editing runs through `EditImage()` and edits are serialized | 60-block MMDiT diffusion (FlowMatch-Euler, true-CFG, reference-latent concat), CUDA-graph-captured whole-DiT forward (~2.9x per forward), optional Lightning distillation LoRA as a runtime side-path (`--qwen-image-lora`: 60 DiT forwards -> 4-8), default flash attention, CFG-batching, opt-in EasyCache / First-Block-Cache denoise caches, fused Qwen2.5-VL conditioning encoders and fused whole-VAE graph, VRAM-aware area clamp | +| Qwen 3 | [qwen3.md](qwen3.md) | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | `Qwen3Model` | `qwen3`, `qwen2`, `qwen2vl`, `qwen2_vl` (Qwen2 / Qwen2.5-VL checkpoints load as text-only chat) | Text | Yes | Yes | Reference port (`Qwen3Model.BatchedForward.cs`) — exercised by `Qwen3BatchedForwardTests` when a base-Qwen3 GGUF is provided | Native whole-model decode with pre-resolved weight pointers | | Qwen 3.5 / 3.6 family | [qwen35.md](qwen35.md) | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF); NextN MTP: [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) (base-repo Qwen3.6 GGUFs strip the NextN block and silently fall back to standard decode) | `Qwen35Model` | `qwen35`, `qwen35moe`, `qwen3next` | Text, image | Yes | Yes | **Default** (toggle off with `TS_QWEN35_BATCHED=0` or `--no-continuous-batching`). Per-slot recurrent-state pool + optional native GatedDeltaNet kernel (`TS_QWEN35_BATCHED_GDN_NATIVE=1`) | Hybrid FullAttention + GatedDeltaNet recurrent, fused attention layer decode, fused prefill attention, fused output-projection + FFN, fused output-projection + norm + router, batched MoE (routed + shared + residual in a single kernel), fused vision encoder blocks | | GPT OSS | [gptoss.md](gptoss.md) | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) | `GptOssModel` | `gptoss`, `gpt-oss` | Text | Yes (always) | Yes | **Default** (toggle off with `TS_GPTOSS_BATCHED=0`). Per-head attention sinks via `TSGgml_PagedAttentionForwardWithSinks` (or `TS_GPTOSS_PAGED_ATTN_MANAGED=1` for the C# fallback). 100% greedy match vs legacy in `GptOssBatchedCorrectnessTests`. | Stacked MoE prefill kernel (mul_mat_id + add_id + swiglu_oai), attention sinks, MXFP4 expert weights | | Nemotron-H | [nemotron.md](nemotron.md) | [bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF); Omni: [unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF) (+ `mmproj-BF16.gguf` for image) | `NemotronModel` | `nemotron_h`, `nemotron_h_moe` | Text, image (Omni-class) | Yes | Yes | **Default** (toggle off with `TS_NEMOTRON_BATCHED=0`). Per-slot Mamba2 conv + SSM state pool; optional native batched Mamba2 step (`TS_NEMOTRON_MAMBA2_BATCHED_NATIVE=1`). 100% greedy match vs legacy; up to 3.95× tps at batch=3 on Apple M4 Pro. | Mamba2 + attention + MoE FFN hybrid stack, batched GPU MoE, RADIO/v2_vl image encoder, Parakeet audio preprocessor (audio inference needs a Parakeet mmproj the GGUF distributions do not ship) | | Mistral 3 | [mistral3.md](mistral3.md) | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF) | `Mistral3Model` | `mistral3` | Text, image | No | No | **Default** — reference IBatchedPagedModel implementation. End-to-end validated on Ministral-3-14B; native paged-attention kernel is ~21% faster than the legacy per-seq path on long context. | YaRN-corrected RoPE with position-dependent Q scaling, fused QKV / gate_up, Pixtral vision encoder | -| Muse-Glimmer | [muse-glimmer.md](muse-glimmer.md) | Muse-Glimmer-30B (`Muse-Glimmer-30B-*.gguf` + `mmproj-Muse-Glimmer-30B-*.gguf`) | `MuseGlimmerModel` | `muse-glimmer` | Text, image | Yes | Yes | No (legacy per-seq) | Interleaved SWA with NoPE full layers, attention output gate, 4 RMSNorms/layer (post-norms at eps 1e-8), logit scale + tanh softcap, sparse-window 2D-RoPE ViT with 2x2 pixel shuffle, optional DFlash block drafter (`--draft-model`, lossless), **tensor parallelism** (`--tp 2` on GGML CUDA/Vulkan — 2 KV heads cap the degree at 2) | -| Wan video | [wan.md](wan.md) | [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF), [QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF), [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf) (+ UMT5-XXL encoder and video VAE, see the card) | `WanVideoModel` (+ `WanVideoPipeline`) | `wan` | Video out (text -> video, image -> video) | No | No | None - `Forward()` throws; generation runs through `GenerateVideo()` and is serialized | One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode and decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff (TE -> DiT -> VAE), tiled 720p decode | +| Muse-Glimmer | [muse-glimmer.md](muse-glimmer.md) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) (`Muse-Glimmer-30B-*.gguf` + `mmproj-Muse-Glimmer-30B-*.gguf`; DFlash drafter `dflash-kquant.gguf` in the same repo) | `MuseGlimmerModel` | `muse-glimmer`, `muse_glimmer` | Text, image | Yes | Yes | No (legacy per-seq) | Interleaved SWA with NoPE full layers, attention output gate, 4 RMSNorms/layer (post-norms at eps 1e-8), logit scale + tanh softcap, sparse-window 2D-RoPE ViT with 2x2 pixel shuffle, optional DFlash block drafter (`--draft-model`, lossless), **tensor parallelism** (`--tp 2` on GGML CUDA/Vulkan — 2 KV heads cap the degree at 2) | +| Wan video | [wan.md](wan.md) | Base: [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF), [QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF), [city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf). **Step-distilled (25× less denoising work, same flags):** [hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF), [jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF). (+ UMT5-XXL encoder and video VAE, see the card) | `WanVideoModel` (+ `WanVideoPipeline`) | `wan`, `wan2.1`, `wan2.2` | Video out (text -> video, image -> video) | No | No | None - `Forward()` throws; generation runs through `GenerateVideo()` and is serialized | Step-distilled checkpoints auto-detected from the DiT file name (100 DiT passes -> 4; M5 Pro 1088x832x121f: 3 h 30 m -> 17 m 30 s), one resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention over F16 keys/values -- 2.02x at 27 k tokens, per-token-timestep modulation for TI2V i2v), `--cfg-cache-stride` guidance reuse (1.30x / 1.43x on base checkpoints), causal 3D video VAE encode and decode each as a single graph with the convs on MPSGraph on Metal (VAE decode 1.99x), A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff (TE -> DiT -> VAE), memory-sized im2col budget and 720p decode tiling | ## Backend notes @@ -85,7 +85,7 @@ the actual ops to the backend that owns those allocators: | `Cpu` | `TensorSharp.Core` | Pure managed tensors with SIMD/managed quantized fast paths (RMSNorm, RoPE, softmax, fused activations, GEMM, dequant). | | `Cuda` | `TensorSharp.Backends.Cuda` | Direct CUDA Driver-API allocator and storage, cuBLAS GEMM, PTX kernels for hot ops (RMSNorm, softmax, RoPE/RoPEEx, SDPA, GQA prefill/decode, causal mask, gather/concat, activation fusions), native quantized matmul / get_rows for supported quant types, CPU fallback for ops that are not yet implemented. | | `Mlx` | `TensorSharp.Backends.MLX` | Apple Silicon `mlx-c` bridge with quantized / fused / compiled kernels, async worker dispatch, MoE expert offload, and a CPU fallback layer. Requires `libmlxc`. | -| `GgmlCpu` / `GgmlMetal` / `GgmlCuda` | `TensorSharp.Backends.GGML` + `TensorSharp.GGML.Native` | Native ggml bridge with quantized graph dispatch and platform backends. mmap-backed quantized weights are bound zero-copy through host-pointer buffers. Includes the paged-attention kernel (`TSGgml_PagedAttentionForward`, plus the GPT OSS sinks variant) that powers the batched / paged execution path. | +| `GgmlCpu` / `GgmlMetal` / `GgmlCuda` / `GgmlVulkan` | `TensorSharp.Backends.GGML` + `TensorSharp.GGML.Native` | Native ggml bridge with quantized graph dispatch and platform backends. mmap-backed quantized weights are bound zero-copy through host-pointer buffers. Includes the paged-attention kernel (`TSGgml_PagedAttentionForward`, plus the GPT OSS sinks variant) that powers the batched / paged execution path. | When a card mentions a fused GGML kernel (for example `Qwen35AttentionLayerDecode`, `Gemma4LayerPrefill`, or `MoEExpertsSwiGLUResidual`), the kernel is compiled from @@ -128,36 +128,36 @@ drafter's weights must be counted by the layer split, it is passed to ## Architecture comparison -| Feature | DeepSeek V4 | Gemma 3 | Gemma 4 | DiffusionGemma | Qwen 3 | Qwen 3.5 / 3.6 family | GPT OSS | Nemotron-H | Mistral 3 | -|---|---|---|---|---|---|---|---|---|---| -| Layer type | MoE (256 routed experts, top-6 + 1 shared) | Dense | Dense / MoE | Gemma-4-derived MoE encoder/decoder | Dense | Hybrid (Attn + Recurrent) ± MoE | MoE | Hybrid (Mamba2 + Attn + FFN, dense or MoE) | Dense | -| Attention | Raw SWA-128 + compressed CSA 4:1 / HCA 128:1 (lightning-indexer top-512 on CSA layers) | SWA + Global | SWA + Global | Region-aware prompt/canvas attention | Full GQA | Full GQA + Sigmoid Gate | Full + Sinks | Full GQA (no RoPE) | Full GQA | -| FFN activation | SwiGLU with a per-layer clamp | GeGLU | GeGLU | Dense GeGLU + top-8 MoE | SwiGLU | SwiGLU | SiLUAlphaLimit (clamped GLU) | ReLU² | SwiGLU | -| RoPE variant | Interleaved-pair + YaRN; separate raw and compress bases, inverted after attention | NeoX (dual base) | NeoX + proportional / partial | NeoX, local/global bases | NeoX | NeoX / MRoPE | NeoX + YaRN | None | GPT-J + YaRN | -| QK-norm | Q only (per-head RMS) | Yes | Yes | Yes | Yes | Yes | No | No | No | -| V-norm | No | No | Yes (unweighted) | Yes (unweighted) | No | No | No | No | No | -| Bias in projections | No (router selection bias only) | No | No | No | No | No | Yes (all linear) | No | No | -| Per-layer scaling | No (per-layer swiglu clamp and compress ratio instead) | No | Yes | Encoder / decoder scalars | No | No | No | No | No | -| Per-Layer Embedding (PLE) | No | No | Yes | No | No | No | No | No | No | -| KV sharing | Yes (one shared 512-dim K=V head for all queries) | No | Yes (tail layers) | Prompt-KV cache across denoising steps | No | No | No | No | No | -| Attention sinks | Yes | No | No | No | No | No | Yes | No | No | -| Circular KV cache | Yes (raw SWA-128 ring) | No | Yes (SWA layers) | No autoregressive KV | No | No | No | No | No | -| SSM / recurrent layers | No (4-stream hyper-connections replace the plain residual) | No | No | No | No | Yes (GatedDeltaNet) | No | Yes (Mamba2) | No | -| Shared experts | Yes | No | No | No | No | Yes (qwen35moe / qwen3next) | No | Yes (optional) | No | -| Latent bottleneck FFN | No (LoRA-factored Q / output projections instead) | No | No | No | No | No | No | Yes (optional) | No | -| Position-dependent Q scaling | No | No | No | No | No | No | No | No | Yes (with YaRN) | -| Vision | No | Yes | Yes | No | No | Yes | No | Yes (Omni) | Yes (Pixtral) | -| Audio | No | No | Yes | No | No | No | No | No — image-only Omni (Parakeet log-mel preprocessing exists, but inference needs an audio mmproj that is not shipped) | No | -| Video | No | No | Yes | No | No | No | No | No | No | -| Thinking | Yes | No | Yes | No | Yes | Yes | Yes (always) | Yes | No | -| Tool calling | Yes (DSML markup) | No | Yes | No | Yes | Yes | Yes | Yes | No | -| MTP / NextN speculative decoding | DSpark block drafter (separate GGUF via `--draft-model`) | No | Yes (separate `gemma4-assistant` draft GGUF) | No | No | Yes on Qwen 3.6 (embedded NextN block) | No | No | No | -| Fused QKV | n/a (LoRA-factored Q, single shared K=V head) | No | Yes | Yes | Yes | Mixed (full attention layers split, recurrent layers fuse a 5-way pack) | Yes | Yes | Yes | -| Fused single-graph decode | Yes (whole-model executor, one graph per ubatch, CUDA-graph replayed) | No | Yes (Gemma4ModelDecode) | Yes (DiffusionModelDecode + lm-head tail) | Yes (TransformerModelDecode, native loop) | Per-layer fused (Qwen35AttentionLayerDecode, FusedOutProjFFN, FusedOutProjNormRouter) | Per-layer | Per-layer / batched MoE | No | -| Fused single-graph prefill | Yes (same whole-model executor, chunked ubatches) | No | Yes (whole-model NativeGemma4ModelVerify + per-layer Gemma4LayerPrefill fallback) | Prompt-KV prefill cache | No | Yes (FusedPrefillAttention, FusedOutProjFFN, MoE prefill) | Yes (MoE prefill via mul_mat_id) | No | No | -| Batched GPU MoE | Yes (grouped expert kernels) | n/a | Yes for all-MoE variants (fused whole-model MoE decode/verify); mixed dense+MoE pending | Fused per-canvas MoE; concurrent requests batched by diffusion scheduler | n/a | Yes (routed + shared + residual fused) | Yes (stacked weight slabs) | Yes | n/a | -| Fused vision encoder | n/a | n/a | Standard | n/a | n/a | Yes (FusedVisionAttention + FusedVisionMLP) | n/a | Standard (RADIO ViT) | Standard (Pixtral) | -| Output parser | `DeepSeek4OutputParser` | `PassthroughOutputParser` | `Gemma4OutputParser` | `PassthroughOutputParser` | `Qwen3OutputParser` | `Qwen35OutputParser` | `HarmonyOutputParser` (always required) | `Qwen3OutputParser` | `PassthroughOutputParser` | +| Feature | DeepSeek V4 | Gemma 3 | Gemma 4 | DiffusionGemma | Qwen 3 | Qwen 3.5 / 3.6 family | GPT OSS | Nemotron-H | Mistral 3 | Muse-Glimmer | +|---|---|---|---|---|---|---|---|---|---|---| +| Layer type | MoE (256 routed experts, top-6 + 1 shared) | Dense | Dense / MoE | Gemma-4-derived MoE encoder/decoder | Dense | Hybrid (Attn + Recurrent) ± MoE | MoE | Hybrid (Mamba2 + Attn + FFN, dense or MoE) | Dense | Dense (52 layers, 32 Q / 2 KV heads) | +| Attention | Raw SWA-128 + compressed CSA 4:1 / HCA 128:1 (lightning-indexer top-512 on CSA layers) | SWA + Global | SWA + Global | Region-aware prompt/canvas attention | Full GQA | Full GQA + Sigmoid Gate | Full + Sinks | Full GQA (no RoPE) | Full GQA | Interleaved SWA-2048 + full NoPE layers (39 + 13), sigmoid attention output gate | +| FFN activation | SwiGLU with a per-layer clamp | GeGLU | GeGLU | Dense GeGLU + top-8 MoE | SwiGLU | SwiGLU | SiLUAlphaLimit (clamped GLU) | ReLU² | SwiGLU | SwiGLU | +| RoPE variant | Interleaved-pair + YaRN; separate raw and compress bases, inverted after attention | NeoX (dual base) | NeoX + proportional / partial | NeoX, local/global bases | NeoX | NeoX / MRoPE | NeoX + YaRN | None | GPT-J + YaRN | ggml NORM (interleaved pairs) on the SWA layers only; the full layers are NoPE | +| QK-norm | Q only (per-head RMS) | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes (per-head; the Q norm carries the folded qk_scale_factor) | +| V-norm | No | No | Yes (unweighted) | Yes (unweighted) | No | No | No | No | No | No | +| Bias in projections | No (router selection bias only) | No | No | No | No | No | Yes (all linear) | No | No | No | +| Per-layer scaling | No (per-layer swiglu clamp and compress ratio instead) | No | Yes | Encoder / decoder scalars | No | No | No | No | No | No (logit scale 0.19612 + tanh softcap 20.0 on the output instead) | +| Per-Layer Embedding (PLE) | No | No | Yes | No | No | No | No | No | No | No | +| KV sharing | Yes (one shared 512-dim K=V head for all queries) | No | Yes (tail layers) | Prompt-KV cache across denoising steps | No | No | No | No | No | No | +| Attention sinks | Yes | No | No | No | No | No | Yes | No | No | No | +| Circular KV cache | Yes (raw SWA-128 ring) | No | Yes (SWA layers) | No autoregressive KV | No | No | No | No | No | Yes (SWA ring on the GPU backends; `TS_MUSE_GLIMMER_SWA_RING=0` disables) | +| SSM / recurrent layers | No (4-stream hyper-connections replace the plain residual) | No | No | No | No | Yes (GatedDeltaNet) | No | Yes (Mamba2) | No | No | +| Shared experts | Yes | No | No | No | No | Yes (qwen35moe / qwen3next) | No | Yes (optional) | No | No (dense FFN) | +| Latent bottleneck FFN | No (LoRA-factored Q / output projections instead) | No | No | No | No | No | No | Yes (optional) | No | No | +| Position-dependent Q scaling | No | No | No | No | No | No | No | No | Yes (with YaRN) | No | +| Vision | No | Yes | Yes | No | No | Yes | No | Yes (Omni) | Yes (Pixtral) | Yes (sparse-window 2D-RoPE ViT with 2×2 pixel shuffle) | +| Audio | No | No | Yes | No | No | No | No | No — image-only Omni (Parakeet log-mel preprocessing exists, but inference needs an audio mmproj that is not shipped) | No | No | +| Video | No | No | Yes | No | No | No | No | No | No | No | +| Thinking | Yes | No | Yes | No | Yes | Yes | Yes (always) | Yes | No | Yes (`assistant to=self` channel) | +| Tool calling | Yes (DSML markup) | No | Yes | No | Yes | Yes | Yes | Yes | No | Yes (ATEM XML markup) | +| MTP / NextN speculative decoding | DSpark block drafter (separate GGUF via `--draft-model`) | No | Yes (separate `gemma4-assistant` draft GGUF) | No | No | Yes on Qwen 3.6 (embedded NextN block) | No | No | No | DFlash block drafter (separate 5-layer GGUF via `--draft-model`, lossless) | +| Fused QKV | n/a (LoRA-factored Q, single shared K=V head) | No | Yes | Yes | Yes | Mixed (full attention layers split, recurrent layers fuse a 5-way pack) | Yes | Yes | Yes | No | +| Fused single-graph decode | Yes (whole-model executor, one graph per ubatch, CUDA-graph replayed) | No | Yes (Gemma4ModelDecode) | Yes (DiffusionModelDecode + lm-head tail) | Yes (TransformerModelDecode, native loop) | Per-layer fused (Qwen35AttentionLayerDecode, FusedOutProjFFN, FusedOutProjNormRouter) | Per-layer | Per-layer / batched MoE | No | Yes (persistent whole-model decode graph on GGML CUDA / Vulkan / Metal / CPU) | +| Fused single-graph prefill | Yes (same whole-model executor, chunked ubatches) | No | Yes (whole-model NativeGemma4ModelVerify + per-layer Gemma4LayerPrefill fallback) | Prompt-KV prefill cache | No | Yes (FusedPrefillAttention, FusedOutProjFFN, MoE prefill) | Yes (MoE prefill via mul_mat_id) | No | No | Yes (same fused kernel, chunked with on-device causal+SWA band masks) | +| Batched GPU MoE | Yes (grouped expert kernels) | n/a | Yes for all-MoE variants (fused whole-model MoE decode/verify); mixed dense+MoE pending | Fused per-canvas MoE; concurrent requests batched by diffusion scheduler | n/a | Yes (routed + shared + residual fused) | Yes (stacked weight slabs) | Yes | n/a | n/a (dense FFN) | +| Fused vision encoder | n/a | n/a | Standard | n/a | n/a | Yes (FusedVisionAttention + FusedVisionMLP) | n/a | Standard (RADIO ViT) | Standard (Pixtral) | Yes (fused vision block + flash attention on CUDA) | +| Output parser | `DeepSeek4OutputParser` | `PassthroughOutputParser` | `Gemma4OutputParser` | `PassthroughOutputParser` | `Qwen3OutputParser` | `Qwen35OutputParser` | `HarmonyOutputParser` (always required) | `Qwen3OutputParser` | `PassthroughOutputParser` | `MuseGlimmerOutputParser` | ## Adding a new architecture diff --git a/docs/models/README_zh-cn.md b/docs/models/README_zh-cn.md index 602387c8..c27dacba 100644 --- a/docs/models/README_zh-cn.md +++ b/docs/models/README_zh-cn.md @@ -42,14 +42,14 @@ Zhongkai Fu 的 [《From Tensors to Tokens》书籍指南](../BOOK_zh-cn.md)， | Gemma 3 | [gemma3_zh-cn.md](gemma3_zh-cn.md) | [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | `Gemma3Model` | `gemma3` | 文本、图像 | 否 | 否 | 否（仅旧单序列路径） | SWA / 全局注意力交替、GeGLU FFN、QK-norm、V-norm | | Gemma 4 | [gemma4_zh-cn.md](gemma4_zh-cn.md) | E4B Q8_0 是已验证的原生 GGML 家族 / 路径层级；[ggml-org/gemma-4-E4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) 是推荐的公开文件来源 | `Gemma4Model` | `gemma4`（`gemma4-assistant` / `gemma4_assistant` 仅作为 MTP 草稿加载） | 文本、图像、视频、音频 | 是 | 是 | **默认启用**（可用 `TS_GEMMA4_BATCHED=0` 关闭） | 整模型融合 decode（一次 GGML 调度）、带内核内 PLE + 共享 KV 处理的融合整模型 prefill/verify、分块 prefill、SWA 环形缓存与 MoE 变体。批处理路径与旧路径 logits 在 FP 噪声内一致（`Gemma4BatchedForwardTests`）；batch=8 短 prompt 达 ~1.5×，4×800-token prompt 达 ~1.6×。 | | DiffusionGemma | [diffusiongemma_zh-cn.md](diffusiongemma_zh-cn.md) | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | `DiffusionGemmaModel` + `DiffusionGemmaSampler` | `diffusion-gemma`、`diffusion_gemma` | 文本 | 否 | 否 | 独立的 Web UI `DiffusionBatchScheduler`；不是自回归 `IBatchedPagedModel` 路径 | `[prompt \| canvas]` 上的 EntropyBound 分块去噪、GPU prompt-KV 缓存、self-conditioning、融合 GGML 整模型 diffusion decode 与融合 lm-head tail | -| Qwen-Image-Edit | [qwenimage_zh-cn.md](qwenimage_zh-cn.md) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)（DiT；VAE / 文本编码器配套文件见卡片） | `QwenImageModel`（+ `QwenImagePipeline`） | `qwen_image`、`qwen-image` | 图像编辑（图像+文本 → 图像） | 否 | 否 | 无——`Forward()` 抛异常；编辑通过 `EditImage()` 并串行执行 | 60 块 MMDiT 扩散（FlowMatch-Euler、true-CFG、参考潜变量拼接）、CUDA 图捕获的整 DiT 前向、默认 flash 注意力、CFG-batching、First-Block-Cache、融合 Qwen2.5-VL 视觉编码器、按 VRAM 钳制面积 | -| Qwen 3 | [qwen3_zh-cn.md](qwen3_zh-cn.md) | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | `Qwen3Model` | `qwen3` | 文本 | 是 | 是 | 参考实现（`Qwen3Model.BatchedForward.cs`）—— 当提供基础 Qwen3 GGUF 时由 `Qwen3BatchedForwardTests` 验证 | 整模型原生 decode，权重指针在加载时预解析 | +| Qwen-Image-Edit | [qwenimage_zh-cn.md](qwenimage_zh-cn.md) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)（DiT；VAE / 文本编码器配套文件见卡片） | `QwenImageModel`（+ `QwenImagePipeline`） | `qwen_image`、`qwen-image` | 图像编辑（图像+文本 → 图像） | 否 | 否 | 无——`Forward()` 抛异常；编辑通过 `EditImage()` 并串行执行 | 60 块 MMDiT 扩散（FlowMatch-Euler、true-CFG、参考潜变量拼接）、CUDA 图捕获的整 DiT 前向（单次前向约 2.9×）、可选 Lightning 蒸馏 LoRA 以运行期旁路方式接入（`--qwen-image-lora`：60 次 DiT 前向降到 4–8 次）、默认 flash 注意力、CFG-batching、可选启用的 EasyCache / First-Block-Cache 去噪缓存、融合的 Qwen2.5-VL 条件编码器与融合整 VAE 图、按 VRAM 钳制面积 | +| Qwen 3 | [qwen3_zh-cn.md](qwen3_zh-cn.md) | [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | `Qwen3Model` | `qwen3`、`qwen2`、`qwen2vl`、`qwen2_vl`（Qwen2 / Qwen2.5-VL 检查点按纯文本对话加载） | 文本 | 是 | 是 | 参考实现（`Qwen3Model.BatchedForward.cs`）—— 当提供基础 Qwen3 GGUF 时由 `Qwen3BatchedForwardTests` 验证 | 整模型原生 decode，权重指针在加载时预解析 | | Qwen 3.5 / 3.6 family | [qwen35_zh-cn.md](qwen35_zh-cn.md) | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)；NextN MTP：[unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)（基础仓库的 Qwen3.6 GGUF 去掉了 NextN 块，会静默回退到标准 decode） | `Qwen35Model` | `qwen35`、`qwen35moe`、`qwen3next` | 文本、图像 | 是 | 是 | **默认启用**（`TS_QWEN35_BATCHED=0` 或 `--no-continuous-batching` 可关闭）。带每槽位的递归状态池，可选原生 GatedDeltaNet 内核（`TS_QWEN35_BATCHED_GDN_NATIVE=1`）。 | 全注意力 + GatedDeltaNet 递归混合、融合 attention 层 decode、融合 prefill attention、融合输出投影 + FFN、融合输出投影 + norm + router、批量 MoE（routed + shared + residual 一次完成）、融合视觉编码器 | | GPT OSS | [gptoss_zh-cn.md](gptoss_zh-cn.md) | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) | `GptOssModel` | `gptoss`、`gpt-oss` | 文本 | 是（始终启用） | 是 | **默认启用**（`TS_GPTOSS_BATCHED=0` 可关闭）。通过 `TSGgml_PagedAttentionForwardWithSinks` 处理每头 attention sinks（或 `TS_GPTOSS_PAGED_ATTN_MANAGED=1` 使用 C# fallback）。在 `GptOssBatchedCorrectnessTests` 中与旧路径 100% 贪心一致。 | Stacked MoE prefill kernel（mul_mat_id + add_id + swiglu_oai）、attention sinks、MXFP4 专家权重 | | Nemotron-H | [nemotron_zh-cn.md](nemotron_zh-cn.md) | [bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF)；Omni：[unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF)（图像输入需另配 `mmproj-BF16.gguf`） | `NemotronModel` | `nemotron_h`、`nemotron_h_moe` | 文本、图像（Omni 版） | 是 | 是 | **默认启用**（`TS_NEMOTRON_BATCHED=0` 可关闭）。带每槽位 Mamba2 conv + SSM 状态池，可选原生批处理 Mamba2 步（`TS_NEMOTRON_MAMBA2_BATCHED_NATIVE=1`）。与旧路径 100% 贪心一致；Apple M4 Pro 上 batch=3 最高可达 3.95× tps。 | Mamba2 + 注意力 + MoE FFN 混合堆栈、批量 GPU MoE、RADIO/v2_vl 图像编码器、Parakeet 音频预处理器（音频推理需要 GGUF 发行版未附带的 Parakeet mmproj） | | Mistral 3 | [mistral3_zh-cn.md](mistral3_zh-cn.md) | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF) | `Mistral3Model` | `mistral3` | 文本、图像 | 否 | 否 | **默认启用** —— `IBatchedPagedModel` 的参考实现。在 Ministral-3-14B 上完成端到端验证；原生分页注意力内核在长上下文下比旧的单序列路径快 ~21%。 | YaRN 校正 RoPE 与位置相关 Q 缩放、融合 QKV / gate_up、Pixtral 视觉编码器 | -| Muse-Glimmer | [muse-glimmer_zh-cn.md](muse-glimmer_zh-cn.md) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)（`Muse-Glimmer-30B-*.gguf` + `mmproj-*.gguf`） | `MuseGlimmerModel` | `muse-glimmer` | 文本、图像 | 是 | 是 | 否（传统单序列） | 交错滑动窗口 + NoPE 全注意力层、注意力输出门控、每层 4 个 RMSNorm（post-norm eps 1e-8）、logit 缩放 + tanh 软上限、稀疏窗口 2D-RoPE ViT（2x2 像素重排）、可选 DFlash 块级草稿（`--draft-model`，无损）、**张量并行**（GGML CUDA/Vulkan 上 `--tp 2`——2 个 KV 头将并行度上限定为 2） | -| Wan 视频 | [wan_zh-cn.md](wan_zh-cn.md) | [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)、[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)、[city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)（另需 UMT5-XXL 编码器与视频 VAE，见卡片） | `WanVideoModel`（+ `WanVideoPipeline`） | `wan` | 视频输出（文本 → 视频、图像 → 视频） | 否 | 否 | 无——`Forward()` 抛异常；生成走 `GenerateVideo()` 且进程内串行 | 每个去噪步一张常驻权重的 ggml 图（CUDA 图捕获、flash attention，TI2V 图生视频带 per-token 时间步调制）、因果 3D 视频 VAE 编/解码各一张图、A14B 的两个 14B 专家在时间步边界热切换、分阶段显存交接（TE → DiT → VAE）、720p 分块解码 | +| Muse-Glimmer | [muse-glimmer_zh-cn.md](muse-glimmer_zh-cn.md) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)（`Muse-Glimmer-30B-*.gguf` + `mmproj-*.gguf`；DFlash 草稿器 `dflash-kquant.gguf` 也在同一仓库） | `MuseGlimmerModel` | `muse-glimmer`、`muse_glimmer` | 文本、图像 | 是 | 是 | 否（传统单序列） | 交错滑动窗口 + NoPE 全注意力层、注意力输出门控、每层 4 个 RMSNorm（post-norm eps 1e-8）、logit 缩放 + tanh 软上限、稀疏窗口 2D-RoPE ViT（2x2 像素重排）、可选 DFlash 块级草稿（`--draft-model`，无损）、**张量并行**（GGML CUDA/Vulkan 上 `--tp 2`——2 个 KV 头将并行度上限定为 2） | +| Wan 视频 | [wan_zh-cn.md](wan_zh-cn.md) | 基础版：[QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)、[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)、[city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf)。**步数蒸馏版（去噪工作量降到 1/25，命令完全不变）：**[hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF)、[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF)。（另需 UMT5-XXL 编码器与视频 VAE，见卡片） | `WanVideoModel`（+ `WanVideoPipeline`） | `wan`、`wan2.1`、`wan2.2` | 视频输出（文本 → 视频、图像 → 视频） | 否 | 否 | 无——`Forward()` 抛异常；生成走 `GenerateVideo()` 且进程内串行 | 按 DiT 文件名自动识别步数蒸馏检查点（100 次 DiT 前向降到 4 次；M5 Pro 1088×832×121f：3 小时 30 分 → 17 分 30 秒）、每个去噪步一张常驻权重的 ggml 图（CUDA 图捕获、F16 键值上的 flash attention——27k token 时快 2.02×，TI2V 图生视频带 per-token 时间步调制）、`--cfg-cache-stride` 引导方向复用（基础检查点上 1.30× / 1.43×）、因果 3D 视频 VAE 编/解码各一张图且 Metal 上卷积走 MPSGraph（VAE 解码 1.99×）、A14B 的两个 14B 专家在时间步边界热切换、分阶段显存交接（TE → DiT → VAE）、按设备显存推导的 im2col 预算与 720p 分块解码 | ## 后端说明 @@ -60,7 +60,7 @@ Zhongkai Fu 的 [《From Tensors to Tokens》书籍指南](../BOOK_zh-cn.md)， | `Cpu` | `TensorSharp.Core` | 纯托管张量，附带 SIMD / 托管量化快路径（RMSNorm、RoPE、softmax、融合激活、GEMM、dequant）。 | | `Cuda` | `TensorSharp.Backends.Cuda` | Direct CUDA Driver-API 分配器与存储、cuBLAS GEMM、热点算子的 PTX 内核（RMSNorm、softmax、RoPE/RoPEEx、SDPA、GQA prefill/decode、causal mask、gather/concat、融合激活）、受支持量化类型的原生 quant matmul / get_rows，未实现的算子回退到 CPU。 | | `Mlx` | `TensorSharp.Backends.MLX` | Apple Silicon `mlx-c` 桥接，含量化 / 融合 / 编译内核、异步 worker 派发、MoE 专家 offload，以及 CPU 回退层。依赖 `libmlxc`。 | -| `GgmlCpu` / `GgmlMetal` / `GgmlCuda` | `TensorSharp.Backends.GGML` + `TensorSharp.GGML.Native` | 原生 ggml 桥接，包括量化计算图调度与平台后端；mmap 量化权重通过 host 指针缓冲零拷贝绑定。还包含驱动批处理 / 分页执行路径的分页注意力内核（`TSGgml_PagedAttentionForward`，含 GPT OSS sinks 变体）。 | +| `GgmlCpu` / `GgmlMetal` / `GgmlCuda` / `GgmlVulkan` | `TensorSharp.Backends.GGML` + `TensorSharp.GGML.Native` | 原生 ggml 桥接，包括量化计算图调度与平台后端；mmap 量化权重通过 host 指针缓冲零拷贝绑定。还包含驱动批处理 / 分页执行路径的分页注意力内核（`TSGgml_PagedAttentionForward`，含 GPT OSS sinks 变体）。 | 凡是卡片中提到融合 GGML kernel（例如 `Qwen35AttentionLayerDecode`、`Gemma4LayerPrefill`、`MoEExpertsSwiGLUResidual`），其源码都在 `TensorSharp.GGML.Native/ggml_ops_*.cpp`，并通过 `TensorSharp.Backends.GGML/GgmlBasicOps.cs` 暴露给托管侧。如果某个融合路径只在 GGML CPU / Metal / CUDA 上启用而在纯托管 CPU 或 direct CUDA 上没有启用，请到原生桥侧查看。 @@ -74,36 +74,36 @@ DeepSeek V4 把一个**块级**草稿器接入了同一套核心：它的 DSpark ## 架构对比 -| 特性 | DeepSeek V4 | Gemma 3 | Gemma 4 | DiffusionGemma | Qwen 3 | Qwen 3.5 / 3.6 family | GPT OSS | Nemotron-H | Mistral 3 | -|---|---|---|---|---|---|---|---|---|---| -| 层类型 | MoE（256 个路由专家，top-6 + 1 共享） | 密集 | 密集 / MoE | Gemma-4 派生 MoE encoder/decoder | 密集 | 混合（注意力 + 递归）± MoE | MoE | 混合（Mamba2 + 注意力 + FFN，密集或 MoE） | 密集 | -| 注意力 | 原始 SWA-128 + 压缩注意力 CSA 4:1 / HCA 128:1（CSA 层用 lightning indexer 选 top-512） | SWA + 全局 | SWA + 全局 | 区分 prompt/canvas 的区域感知注意力 | 全 GQA | 全 GQA + Sigmoid Gate | 全 + Sinks | 全 GQA（无 RoPE） | 全 GQA | -| FFN 激活 | SwiGLU（每层带 clamp） | GeGLU | GeGLU | Dense GeGLU + top-8 MoE | SwiGLU | SwiGLU | SiLUAlphaLimit（带 clamp 的 GLU） | ReLU² | SwiGLU | -| RoPE 类型 | 交错成对 + YaRN；raw 与 compress 两套 base，注意力后再做逆旋转 | NeoX（双 base） | NeoX + 比例 / 部分 | NeoX，local/global base | NeoX | NeoX / MRoPE | NeoX + YaRN | 无 | GPT-J + YaRN | -| QK-norm | 仅 Q（每头 RMS） | 是 | 是 | 是 | 是 | 是 | 否 | 否 | 否 | -| V-norm | 否 | 否 | 是（无权重） | 是（无权重） | 否 | 否 | 否 | 否 | 否 | -| 投影偏置 | 无（仅路由选择偏置） | 无 | 无 | 无 | 无 | 无 | 全部都有 | 无 | 无 | -| 每层缩放 | 否（改为每层 swiglu clamp 与压缩比） | 否 | 是 | encoder / decoder 标量 | 否 | 否 | 否 | 否 | 否 | -| Per-Layer Embedding (PLE) | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 | 否 | -| KV 共享 | 是（所有 query 共用一个 512 维 K=V 头） | 否 | 是（尾部若干层） | 去噪多步间复用 prompt-KV | 否 | 否 | 否 | 否 | 否 | -| Attention sinks | 是 | 否 | 否 | 否 | 否 | 否 | 是 | 否 | 否 | -| 环形 KV cache | 是（原始 SWA-128 环） | 否 | 是（SWA 层） | 无自回归 KV | 否 | 否 | 否 | 否 | 否 | -| SSM / 递归层 | 否（用 4 路 hyper-connection 取代普通残差） | 否 | 否 | 否 | 否 | 是（GatedDeltaNet） | 否 | 是（Mamba2） | 否 | -| 共享专家 | 是 | 否 | 否 | 否 | 否 | 是（qwen35moe / qwen3next） | 否 | 是（可选） | 否 | -| Latent bottleneck FFN | 否（改为 LoRA 分解的 Q / 输出投影） | 否 | 否 | 否 | 否 | 否 | 否 | 是（可选） | 否 | -| 位置相关 Q 缩放 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | 是（与 YaRN 配合） | -| 视觉 | 否 | 是 | 是 | 否 | 否 | 是 | 否 | 是（Omni） | 是（Pixtral） | -| 音频 | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 —— Omni 仅图像（Parakeet log-mel 预处理已实现，但推理需要未随发行版提供的音频 mmproj） | 否 | -| 视频 | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 | 否 | -| 思维链 | 是 | 否 | 是 | 否 | 是 | 是 | 是（始终启用） | 是 | 否 | -| 工具调用 | 是（DSML 标记） | 否 | 是 | 否 | 是 | 是 | 是 | 是 | 否 | -| MTP / NextN 投机解码 | DSpark 块级草稿器（独立 GGUF，`--draft-model`） | 否 | 是（独立 `gemma4-assistant` 草稿 GGUF） | 否 | 否 | Qwen 3.6 支持（内嵌 NextN 块） | 否 | 否 | 否 | -| 融合 QKV | n/a（LoRA 分解的 Q，单个共享 K=V 头） | 否 | 是 | 是 | 是 | 混合（attention 层拆开，递归层融合 5 路） | 是 | 是 | 是 | -| 融合单调用 decode | 是（整模型执行器，每个 ubatch 一张图，重放 CUDA 图） | 否 | 是（Gemma4ModelDecode） | 是（DiffusionModelDecode + lm-head tail） | 是（TransformerModelDecode，原生循环） | per-layer 融合（Qwen35AttentionLayerDecode、FusedOutProjFFN、FusedOutProjNormRouter） | per-layer | per-layer / 批量 MoE | 否 | -| 融合单调用 prefill | 是（同一整模型执行器，分块 ubatch） | 否 | 是（整模型 NativeGemma4ModelVerify + 逐层 Gemma4LayerPrefill 回退） | prompt-KV prefill cache | 否 | 是（FusedPrefillAttention、FusedOutProjFFN、MoE prefill） | 是（MoE prefill via mul_mat_id） | 否 | 否 | -| 批量 GPU MoE | 是（分组专家内核） | n/a | 全 MoE 变体已支持（融合整模型 MoE decode/verify）；混合 dense+MoE 待实现 | 融合单 canvas MoE；并发请求由 diffusion scheduler 批处理 | n/a | 是（routed + shared + residual 融合） | 是（stacked weight slabs） | 是 | n/a | -| 融合视觉编码器 | n/a | n/a | 标准 | n/a | n/a | 是（FusedVisionAttention + FusedVisionMLP） | n/a | 标准（RADIO ViT） | 标准（Pixtral） | -| 输出解析器 | `DeepSeek4OutputParser` | `PassthroughOutputParser` | `Gemma4OutputParser` | `PassthroughOutputParser` | `Qwen3OutputParser` | `Qwen35OutputParser` | `HarmonyOutputParser`（始终启用） | `Qwen3OutputParser` | `PassthroughOutputParser` | +| 特性 | DeepSeek V4 | Gemma 3 | Gemma 4 | DiffusionGemma | Qwen 3 | Qwen 3.5 / 3.6 family | GPT OSS | Nemotron-H | Mistral 3 | Muse-Glimmer | +|---|---|---|---|---|---|---|---|---|---|---| +| 层类型 | MoE（256 个路由专家，top-6 + 1 共享） | 密集 | 密集 / MoE | Gemma-4 派生 MoE encoder/decoder | 密集 | 混合（注意力 + 递归）± MoE | MoE | 混合（Mamba2 + 注意力 + FFN，密集或 MoE） | 密集 | 稠密（52 层，32 Q 头 / 2 KV 头） | +| 注意力 | 原始 SWA-128 + 压缩注意力 CSA 4:1 / HCA 128:1（CSA 层用 lightning indexer 选 top-512） | SWA + 全局 | SWA + 全局 | 区分 prompt/canvas 的区域感知注意力 | 全 GQA | 全 GQA + Sigmoid Gate | 全 + Sinks | 全 GQA（无 RoPE） | 全 GQA | 交错 SWA-2048 + NoPE 全注意力层（39 + 13），带 sigmoid 注意力输出门控 | +| FFN 激活 | SwiGLU（每层带 clamp） | GeGLU | GeGLU | Dense GeGLU + top-8 MoE | SwiGLU | SwiGLU | SiLUAlphaLimit（带 clamp 的 GLU） | ReLU² | SwiGLU | SwiGLU | +| RoPE 类型 | 交错成对 + YaRN；raw 与 compress 两套 base，注意力后再做逆旋转 | NeoX（双 base） | NeoX + 比例 / 部分 | NeoX，local/global base | NeoX | NeoX / MRoPE | NeoX + YaRN | 无 | GPT-J + YaRN | ggml NORM（相邻成对交错），仅用于 SWA 层；全注意力层为 NoPE | +| QK-norm | 仅 Q（每头 RMS） | 是 | 是 | 是 | 是 | 是 | 否 | 否 | 否 | 是（每头；Q norm 权重折入了 qk_scale_factor） | +| V-norm | 否 | 否 | 是（无权重） | 是（无权重） | 否 | 否 | 否 | 否 | 否 | 否 | +| 投影偏置 | 无（仅路由选择偏置） | 无 | 无 | 无 | 无 | 无 | 全部都有 | 无 | 无 | 无 | +| 每层缩放 | 否（改为每层 swiglu clamp 与压缩比） | 否 | 是 | encoder / decoder 标量 | 否 | 否 | 否 | 否 | 否 | 否（改为输出侧 logit 缩放 0.19612 + tanh 软上限 20.0） | +| Per-Layer Embedding (PLE) | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | +| KV 共享 | 是（所有 query 共用一个 512 维 K=V 头） | 否 | 是（尾部若干层） | 去噪多步间复用 prompt-KV | 否 | 否 | 否 | 否 | 否 | 否 | +| Attention sinks | 是 | 否 | 否 | 否 | 否 | 否 | 是 | 否 | 否 | 否 | +| 环形 KV cache | 是（原始 SWA-128 环） | 否 | 是（SWA 层） | 无自回归 KV | 否 | 否 | 否 | 否 | 否 | 是（GPU 后端上的 SWA 环；`TS_MUSE_GLIMMER_SWA_RING=0` 关闭） | +| SSM / 递归层 | 否（用 4 路 hyper-connection 取代普通残差） | 否 | 否 | 否 | 否 | 是（GatedDeltaNet） | 否 | 是（Mamba2） | 否 | 否 | +| 共享专家 | 是 | 否 | 否 | 否 | 否 | 是（qwen35moe / qwen3next） | 否 | 是（可选） | 否 | 否（稠密 FFN） | +| Latent bottleneck FFN | 否（改为 LoRA 分解的 Q / 输出投影） | 否 | 否 | 否 | 否 | 否 | 否 | 是（可选） | 否 | 否 | +| 位置相关 Q 缩放 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | 是（与 YaRN 配合） | 否 | +| 视觉 | 否 | 是 | 是 | 否 | 否 | 是 | 否 | 是（Omni） | 是（Pixtral） | 是（稀疏窗口 2D-RoPE ViT，2×2 像素重排） | +| 音频 | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 —— Omni 仅图像（Parakeet log-mel 预处理已实现，但推理需要未随发行版提供的音频 mmproj） | 否 | 否 | +| 视频 | 否 | 否 | 是 | 否 | 否 | 否 | 否 | 否 | 否 | 否 | +| 思维链 | 是 | 否 | 是 | 否 | 是 | 是 | 是（始终启用） | 是 | 否 | 是（`assistant to=self` 通道） | +| 工具调用 | 是（DSML 标记） | 否 | 是 | 否 | 是 | 是 | 是 | 是 | 否 | 是（ATEM XML 标记） | +| MTP / NextN 投机解码 | DSpark 块级草稿器（独立 GGUF，`--draft-model`） | 否 | 是（独立 `gemma4-assistant` 草稿 GGUF） | 否 | 否 | Qwen 3.6 支持（内嵌 NextN 块） | 否 | 否 | 否 | DFlash 块级草稿器（独立 5 层 GGUF，`--draft-model`，无损） | +| 融合 QKV | n/a（LoRA 分解的 Q，单个共享 K=V 头） | 否 | 是 | 是 | 是 | 混合（attention 层拆开，递归层融合 5 路） | 是 | 是 | 是 | 否 | +| 融合单调用 decode | 是（整模型执行器，每个 ubatch 一张图，重放 CUDA 图） | 否 | 是（Gemma4ModelDecode） | 是（DiffusionModelDecode + lm-head tail） | 是（TransformerModelDecode，原生循环） | per-layer 融合（Qwen35AttentionLayerDecode、FusedOutProjFFN、FusedOutProjNormRouter） | per-layer | per-layer / 批量 MoE | 否 | 是（GGML CUDA / Vulkan / Metal / CPU 上的常驻整模型 decode 图） | +| 融合单调用 prefill | 是（同一整模型执行器，分块 ubatch） | 否 | 是（整模型 NativeGemma4ModelVerify + 逐层 Gemma4LayerPrefill 回退） | prompt-KV prefill cache | 否 | 是（FusedPrefillAttention、FusedOutProjFFN、MoE prefill） | 是（MoE prefill via mul_mat_id） | 否 | 否 | 是（同一融合内核，分块并在设备端生成 causal+SWA 带状掩码） | +| 批量 GPU MoE | 是（分组专家内核） | n/a | 全 MoE 变体已支持（融合整模型 MoE decode/verify）；混合 dense+MoE 待实现 | 融合单 canvas MoE；并发请求由 diffusion scheduler 批处理 | n/a | 是（routed + shared + residual 融合） | 是（stacked weight slabs） | 是 | n/a | n/a（稠密 FFN） | +| 融合视觉编码器 | n/a | n/a | 标准 | n/a | n/a | 是（FusedVisionAttention + FusedVisionMLP） | n/a | 标准（RADIO ViT） | 标准（Pixtral） | 是（CUDA 上融合视觉块 + flash attention） | +| 输出解析器 | `DeepSeek4OutputParser` | `PassthroughOutputParser` | `Gemma4OutputParser` | `PassthroughOutputParser` | `Qwen3OutputParser` | `Qwen35OutputParser` | `HarmonyOutputParser`（始终启用） | `Qwen3OutputParser` | `PassthroughOutputParser` | `MuseGlimmerOutputParser` | ## 新增模型架构 diff --git a/docs/models/wan.md b/docs/models/wan.md index d5b5d8fe..204cf461 100644 --- a/docs/models/wan.md +++ b/docs/models/wan.md @@ -100,6 +100,10 @@ TensorSharp.Cli --model HighNoise/Wan2.2-I2V-A14B-HighNoise-Q4_K_M.gguf \ cfg 3.5 (both experts), shift 5.0, 40 steps; A14B T2V cfg 4.0/3.0, shift 12.0; Wan 2.1 cfg 6.0, shift 8.0 (1.3B video) or 3.0/5.0, 30 steps, 16 fps. - `--negative-prompt` defaults to the official Wan negative prompt. +- **Step-distilled checkpoints are auto-detected** and are by far the biggest + speed lever — 100 DiT passes become 4, for the same video. No flag: it is an + ordinary `--model` GGUF. See [the fast lane](#the-fast-lane-step-distilled-checkpoints) + below. - `--cfg-cache-stride N` (default off) — approximate speedup. A guided step is `v = v_cond + (cfg-1)·d` with `d = v_cond - v_uncond`; the guidance direction `d` changes much more slowly across the schedule than `v` does, so @@ -112,6 +116,58 @@ TensorSharp.Cli --model HighNoise/Wan2.2-I2V-A14B-HighNoise-Q4_K_M.gguf \ Without it the OS codec via OpenCV is used at its default bitrate, which is visibly worse; the CLI warns when that happens. +### The fast lane: step-distilled checkpoints + +This is the difference between a 5-second 720p video costing three and a half +hours and costing seventeen minutes, and it is a one-word change to a file path. + +TensorSharp reads the DiT GGUF's **file name**. A name carrying `Turbo` / +`distill` / `Lightning` / `lightx2v` / `FastWan` / `-dmd`, or an explicit step +count (`…-4steps-…`, `…8step…`, accepted for 1–16), switches the sampling +defaults to that step count with **guidance off** — which is what those +checkpoints are trained for. On load the console confirms it: + +``` +step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override) +``` + +The official 50-step + CFG recipe costs 50 × 2 = 100 DiT passes; a 4-step +guidance-free checkpoint costs **4** — 1/25th of the denoising work. Everything +else (companion files, flags, endpoints, the Web UI) is unchanged, and +`--diffusion-steps` / `--cfg` still override the detected values. + +Drop-in GGUFs. The distilled repos ship the DiT only, so the VAE and the text +encoder still come from the base repos: + +```bash +# TI2V-5B, 4 steps — note the Wan2_2 underscore in the file name +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +TensorSharp.Cli --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_metal \ + --prompt "the cat runs toward the camera, cinematic" --image first_frame.png \ + --output cat.mp4 --video-frames 121 +``` + +```bash +# I2V-A14B, 4 steps — both experts; --model either one, the sibling is found by name +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models +``` + +[hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF) +also offers Q6_K (4.22 GB) through Q2_K (1.86 GB); +[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF) +carries the Lightning distillation already merged into both experts, Q8_0 +(15.4 GB/expert) through Q2_K (5.31 GB). +[Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF](https://huggingface.co/Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF) +is a second TI2V-5B option. Note that +[lightx2v/Wan2.2-Lightning](https://huggingface.co/lightx2v/Wan2.2-Lightning) +publishes LoRA `.safetensors` only, and TensorSharp has no Wan LoRA option — use +the pre-merged GGUFs above. + ### Getting good quality - **Resolution is the biggest lever.** Wan is trained at 480p (832×480 area) @@ -144,6 +200,25 @@ TI2V-5B, and 33 frames at 16 fps otherwise. Frame counts are snapped to `4k+1`. Keep the model's native FPS and adjust the frame count when changing duration; changing only FPS changes playback speed. +The step-distilled fast lane is a property of the `--model` GGUF, not of the +request, so it applies to the Web UI and to every endpoint with no client-side +change at all — point `--model` at a Turbo/Lightning checkpoint and nothing else +moves: + +```bash +TensorSharp.Server --model Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_metal \ + --video-frames 121 --fps 24 +``` + +This matters most from the browser, where the request is a prompt and an image +and the cost is invisible until the ETA appears. The same Web UI request — +a 1024×768 first frame, 121 frames at 24 fps, which resolves to 1088×832 and +27 404 tokens — takes **≈ 3 h 30 m** on the base `Wan2.2-TI2V-5B-Q8_0.gguf` and +**17 m 30 s** on `Wan2_2-TI2V-5B-Turbo-Q8_0.gguf` (M5 Pro, `ggml_metal`; see +[Performance](#performance)). If a run's ETA looks impossible, check the startup +log for the `step-distilled checkpoint detected` line before changing anything +else. + - **Web UI** (`http://localhost:5000`): type a prompt in the chat — attach an image to get image-to-video (the image becomes the first frame); the reply is the generated video with live denoising progress. @@ -217,13 +292,21 @@ Environment knobs: `TS_WAN_DIT_CAPTURE=0` (disable the persistent captured DiT graph), `TS_WAN_DIT_FLASH=0` (materialized attention), `TS_WAN_DIT_KV_F16=0` (F32 attention keys/values — the old, ~2x slower default), `TS_WAN_HEARTBEAT_S` (progress tick interval, default 30; `0` silences the ticks), -`TS_WAN_VAE_GEMM_MAX_MB` (im2col budget), +`TS_WAN_VAE_GEMM_MAX_MB` (im2col budget), `TS_WAN_VAE_MPS_CONV=0` +(ggml conv lowering instead of MPSGraph on Metal), `TS_WAN_VAE`/`TS_WAN_TE`/`TS_WAN_DIT2` (companion paths), `TS_FFMPEG` (ffmpeg path for MP4 export), `TS_WAN_DIT_TRACE=` (per-stage activation stats for debugging). ## Performance +**The headline: use a step-distilled checkpoint.** Every other lever on this +page is worth tens of percent; that one is worth 25×. The same 5-second 720p +image-to-video request runs in **3 h 30 m** on a base checkpoint and +**17 m 30 s** on a Turbo one, on the same machine, with the same flags — see +[the fast lane](#the-fast-lane-step-distilled-checkpoints) for where to get one +and the M5 Pro table below for the breakdown. + RTX 3080 Laptop 16 GB (Windows/WDDM), ggml_cuda: | Model / workload | Text enc | Image enc | Denoise | VAE decode | Total | @@ -264,29 +347,51 @@ Wan 2.2-TI2V I2V: 1088x832x121f (31x26x34 = 27404 tokens), 50 steps, unipc, cfg To make such a request cheaper, in order of effect: -1. **Fewer frames.** 121 → 61 frames roughly quarters the attention work; the - video is 2.5 s instead of 5 s at 24 fps. -2. **Smaller frame area**, but not below Wan's training resolutions — under +1. **Use a step-distilled checkpoint.** This dwarfs everything else: 100 DiT + passes become 4. See [the fast lane](#the-fast-lane-step-distilled-checkpoints) + — TensorSharp detects them by name and switches the recipe automatically. +2. **Fewer frames.** 121 → 61 frames roughly quarters the attention work and + halves the VAE decode; the video is 2.5 s instead of 5 s at 24 fps. +3. **Smaller frame area**, but not below Wan's training resolutions — under ~0.3 MP the model is out of distribution and the result gets *worse*, not just cheaper. `--width 480 --height 704` is a good portrait target. -3. **Fewer steps.** 50 is the official TI2V recipe; 30 is visibly close and - 1.7× cheaper. -4. **`--cfg-cache-stride 2` or `3`** — 1.30× / 1.43× by reusing the guidance - direction between steps (approximate; see the CLI section). +4. **Fewer steps** (base checkpoints only). 50 is the official TI2V recipe; + 30 is visibly close and 1.7× cheaper. +5. **`--cfg-cache-stride 2` or `3`** — 1.30× / 1.43× by reusing the guidance + direction between steps (approximate; see the CLI section). Pointless on a + distilled checkpoint, which already runs guidance-free. M5 Pro (20-core GPU, 48 GB unified), `ggml_metal`, Wan2.2-TI2V-5B Q8_0, -1088×832×121f = 27 404 tokens, image-to-video, the official 50-step recipe: +1088×832×121f = 27 404 tokens, image-to-video — the request this was all +measured against (5 s at 24 fps, 720p class): + +| | Base checkpoint, before | Base checkpoint, now | **Turbo checkpoint, now** | +|---|---|---|---| +| DiT passes | 100 (50 steps × CFG) | 100 | **4** (4 steps, guidance-free) | +| per pass | 206.2 s | **120.2 s** | **120.2 s** | +| denoise total | 20 615 s | 12 020 s | **481 s** | +| VAE decode, 121 frames | 863 s | **563 s** | **563 s** | +| **end to end** | **≈ 5 h 58 m** | ≈ 3 h 30 m | **17 m 30 s** | + +The two levers are independent: ~1.7× per pass from the attention and VAE work +below, and 25× fewer passes from running a step-distilled checkpoint. Once +distilled, the **VAE decode is the bottleneck** (~55% of the run), not the DiT. + +Frame count and frame area then set the rest, and the VAE scales linearly with +output pixels while attention scales with the square of the token count. All +measured on the same machine with the same Turbo checkpoint and image: + +| Output | Tokens | Denoise | VAE decode | **Total** | +|---|---|---|---|---| +| 736×544 × 81f (3.4 s, 480p class) | 8 211 | 84 s | 159 s | **4 m 09 s** | +| 736×544 × 121f (5 s, 480p class) | 12 121 | 137 s | 237 s | **6 m 19 s** | +| 1088×832 × 121f (5 s, 720p class) | 27 404 | 481 s | 563 s | **17 m 30 s** | -| Stage | Before | Now | -|---|---|---| -| text encode (UMT5-XXL, both prompts) | 1.9 s | 1.9 s | -| image encode (VAE) | 5.2 s | 5.2 s | -| denoise, per step (2 CFG passes) | 412.3 s | **249.0 s** | -| VAE decode, 121 frames | 863 s (3 bands × 24 latent rows) | **734 s** (2 × 30) | -| **50-step total** | ≈ 5 h 58 m | **≈ 3 h 40 m** (1.63×) | -| 50-step total, `--cfg-cache-stride 3` | — | ≈ 2 h 38 m (2.27×) | +480p (≈0.4 MP) is a resolution Wan is *trained* at, so the middle rows are +in-distribution, not a degraded mode — that is the setting to reach for when a +few minutes matters. Going below ~0.3 MP is where quality actually falls off. -Two changes account for that. +Three changes account for the per-pass part. **F16 attention keys and values.** Every backend's flash-attention kernel is built around an F16 KV cache: ggml-metal instantiates the F32 variant with @@ -300,8 +405,46 @@ bytes the old `ggml_cont` did. `TS_WAN_DIT_KV_F16=0` restores F32. and then splits the plane evenly, instead of walking a fixed band height and letting the last band land wherever the stride puts it. At 52 latent rows that was three 24-row bands — 72 rows decoded to produce 52 — and is now two 30-row -bands (60 rows, one seam instead of two) within the same per-band budget: -863 s → 734 s, the ratio the row counts predict. +bands (60 rows, one seam instead of two) within the same per-band budget. + +**VAE convolutions on MPSGraph (Metal).** ggml lowers `conv2d` to im2col + +`mul_mat`, and a decode profile (`WanVideoBench`, 640×480×9f) puts **44.4% of +the graph in MUL_MAT and a further 30.2% in IM2COL** — 74.6% in the convolution, +with im2col moving 865 MB per node at ~57 GB/s. The lowering is pure overhead: a +3×3 conv materialises 9× its input before the GEMM starts. Apple's tuned +convolution runs the same shapes far faster: + +| conv shape | ggml im2col+GEMM | MPSGraph | | +|---|---|---|---| +| 512→512 k3 320×240 t9 | 666 ms | 108 ms | 6.2× | +| 256→256 k3 640×480 t9 | 947 ms | 110 ms | 8.6× | +| 160→160 k3 640×480 t9 | 491 ms | 47 ms | 10.4× | +| 512→512 k1 320×240 t9 | 184 ms | 14 ms | 13.9× | + +MPS reaches ~30 TFLOP/s where ggml's Metal GEMM gets ~4.9 — and it does so +*without* ggml's `mul_mm`, the kernel whose Metal 4 tensor path corrupts this +exact graph. It buys matrix-unit throughput while stepping around that defect. + +Layout costs nothing: ggml's activation `[W,H,C,T]` and kernel `[KW,KH,IC,OC]` +are byte-for-byte MPS NCHW and OIHW, and NCHW measured as fast as NHWC here, so +no transpose is inserted. On this path the VAE emits un-lowered `CONV_2D` nodes, +which also skips the horizontal banding and its leading-pad shim entirely; the +decode graph is then run in segments with each `CONV_2D` handed to MPSGraph +(`ggml_graph_view`, the same mechanism the node profiler uses, so it is safe +against gallocr reuse). Measured end to end: **VAE decode 159 s → 80 s (1.99×)** +at 736×544×81f. Numerics are unchanged — ggml already convolves in F16 with F32 +accumulation and so does this, agreeing to **93.9 dB PSNR / max Δ 0.128 of 255** +over a whole decode. `TS_WAN_VAE_MPS_CONV=0` restores the ggml lowering. + +**VAE conv im2col budget, and the tiling threshold itself, sized from device +memory.** Both were pinned at a 16 GB card's budget on every non-CUDA backend. +Metal now derives them from free device memory like CUDA does (Vulkan keeps the +floor — its drivers reject multi-GB arenas). A bigger im2col budget turns many +small banded GEMMs into few large ones: 56.4 s → 49.7 s for a 1088×832×9f +decode. And tiling is not free — the bands overlap, so the decoder runs more +latent rows than the plane has, and the band buffer lives alongside the canvas. +Decoding 1088×832×121f **whole** measured both faster *and* lighter than two +bands: **565 s vs 655 s, peak RSS 4.85 vs 5.37 GB**. Small cards still tile. #### Alternatives measured and rejected @@ -320,10 +463,36 @@ sweet spot rather than a compromise. Materialized chunked attention (`mul_mat` + `soft_max_ext`, which unlike flash attention *can* use the Metal 4 tensor API) came in at 1.08×: at this size attention is bound by score traffic, not by the GEMM rate. Dequantizing the DiT weights from Q8_0 to F16 moved the -matmuls by <5% (25.2 → 26.4 ms for an attention projection). The Metal 4 tensor -API stays off for TI2V-5B: it makes the weight matmuls 2.9× faster (~22 s per -pass) but forces the VAE onto the direct-conv path, which at 121 frames costs -hours — see `ApplyArchitectureNativeTunables`. +matmuls by <5% (25.2 → 26.4 ms for an attention projection). + +#### The Metal 4 tensor API, and how not to test it + +The most tempting knob here, and it stays **off** for TI2V-5B. It measures 1.20× +on the DiT (121.4 → 100.8 s/pass) and 1.66× on the VAE decode, because it routes +`mul_mm` through Metal 4 tensor operations — but on M5 / macOS 26.6 it also +miscomputes the VAE's conv GEMMs and the video comes out uniformly **black**. + +The trap is that this does not reproduce in isolation. `WanVideoBench +vae-decode` was run at five latent shapes, including the 32×32 layout recorded +as the original all-NaN repro, dumping raw pixels with the tensor API on and +off: they agreed to **91–93 dB PSNR, max delta 0.18/255, zero NaN**. On that +evidence the workaround was removed — and the very next full 1088×832×121f +generation produced 121 black frames. The defect follows the process's +allocation history (the DiT is loaded and released before the decode), so a +fresh-process decode probe reproduces the wrong buffer layout. + +It was then re-tested a second time with a *much* more faithful harness — +`WanVideoBench pipeline`, which runs VAE-encode → DiT forward → release → VAE +decode in one process at the exact production shapes, including the full +121-frame decode. That came back clean too, and 1.50× faster (374 s vs 563 s). +The real generation still produced flat frames. The only remaining difference is +the UMT5-XXL text encoder that loads and releases before everything else, which +is the largest allocation in the pipeline. + +So: re-enabling it requires a full end-to-end video, and nothing less has ever +been sufficient. The pipeline now refuses to return a uniformly flat video and +names this cause (`AssertFramesAreNotDegenerate`), which is what caught the +second attempt automatically instead of costing another eyeballed PNG. #### Quality diff --git a/docs/models/wan_zh-cn.md b/docs/models/wan_zh-cn.md index 562cdc72..98174408 100644 --- a/docs/models/wan_zh-cn.md +++ b/docs/models/wan_zh-cn.md @@ -18,21 +18,46 @@ TensorSharp 原生运行 [Wan 2.1](https://github.com/Wan-Video/Wan2.1) 与 图生视频时，上传的图片成为视频的**首帧**，文本提示词控制动作、镜头运动、 氛围与场景变化。 -各网络放在同一目录即可自动解析（`VAE/`、`HighNoise/`、`LowNoise/` 等子目录亦可）： +各网络在每次生成中协同工作，放在同一目录即可自动解析（`VAE/`、`HighNoise/`、 +`LowNoise/` 等子目录亦可）： | 组件 | 文件 | 来源 | |---|---|---| | DiT（`--model` GGUF，`general.architecture = wan`） | 如 `Wan2.2-TI2V-5B-Q8_0.gguf` | [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF)、[QuantStack/Wan2.2-I2V-A14B-GGUF](https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF)、[city96/Wan2.1-T2V-14B-gguf](https://huggingface.co/city96/Wan2.1-T2V-14B-gguf) | -| A14B 第二专家（仅 A14B） | 对应的 `…HighNoise…`/`…LowNoise…` GGUF | 同一仓库（按文件名自动配对；`TS_WAN_DIT2` 可覆盖） | +| A14B 第二专家（仅 A14B） | 对应的 `…HighNoise…`/`…LowNoise…` GGUF | 同一仓库（在同级或兄弟目录中按文件名自动查找；`TS_WAN_DIT2` 可覆盖） | | UMT5-XXL 文本编码器 | `umt5-xxl-encoder-Q8_0.gguf` | [city96/umt5-xxl-encoder-gguf](https://huggingface.co/city96/umt5-xxl-encoder-gguf)（`--wan-te` / `TS_WAN_TE`） | | Wan 2.1 视频 VAE（2.1 + A14B） | `wan_2.1_vae.safetensors` | [Comfy-Org/Wan_2.1_ComfyUI_repackaged](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors)（`--wan-vae` / `TS_WAN_VAE`） | | Wan 2.2 视频 VAE（TI2V-5B） | `Wan2.2_VAE.safetensors` | [QuantStack/Wan2.2-TI2V-5B-GGUF](https://huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF/tree/main/VAE) 内附 | 文本编码器在去噪开始前、（图生视频时）VAE 编码器在 DiT 加载前、DiT 在 VAE 解码前 -分别释放 VRAM，因此峰值显存约为 `max(TE, DiT + 注意力, VAE)` —— TI2V-5B Q8_0 +分别释放显存，因此峰值显存约为 `max(TE, DiT + 注意力, VAE)` —— TI2V-5B Q8_0 可在 16 GB GPU 上不到 8 分钟生成 81 帧 480p 图生视频；两个 A14B Q4_K_M 专家 可在同一张卡上顺序运行。 +## 后端 + +视频生成在 TensorSharp 的每个后端上都可运行： + +| `--backend` | 路径 | 说明 | +|---|---|---| +| `ggml_cuda` | GGML 整图内核 | 最快；常驻 DiT 图 + CUDA graph 捕获 | +| `ggml_vulkan` | GGML 整图内核 | 通过 Vulkan 支持 AMD/Intel/NVIDIA；VAE 保持分带卷积路径（Vulkan 驱动拒绝多 GB 显存竞技场） | +| `ggml_cpu` | GGML 整图内核 | 纯 CPU 机器；慢但精确 | +| `cuda` | 直连 CUDA（`WanDirect*`） | 不依赖 ggml：量化权重常驻，走 TensorSharp 自有的 MMQ/dp4a/cuBLAS 路由、流式 online-softmax 注意力内核、channels-last im2col VAE | +| `cpu` | 直连纯 C#（`WanDirect*`） | 除托管运行时外无任何原生依赖；并行 SIMD GEMM/注意力内核 | + +所有后端产生数值等价的视频（互相之间以及与 diffusers 参考实现在相同种子/嵌入下 +交叉验证 —— 后端之间最终潜变量余弦相似度 ≥ 0.999，量化检查点相对 bf16 diffusers +参考约 0.994）。`cpu` / `ggml_cpu` 后端只用于功能性验证 —— 在高核心数机器上， +一段数秒的 480p 视频需要几十分钟。 + +参考耗时（RTX 2000 Ada 16 GB，Wan2.1-1.3B F16，832×480，33 帧，30 步 UniPC —— +官方 480p 配方）：`ggml_cuda` 12.0 s/步（总计 445 s）、`ggml_vulkan` 17.2 s/步 +（625 s）、直连 `cuda` 19.3 s/步（700 s）。在较短序列上直连后端与 `ggml_cuda` +持平（TI2V-5B 480×480×9 帧：两者均为 1.1 s/步）；到 14k token 的配方时，它的 +F32 注意力路径落后于 ggml 的 F16 flash attention。长序列注意力会自动走分块 +cuBLAS GEMM+softmax（`TS_WAN_DIRECT_FUSED_ATTN=0` 可强制全程使用）。 + ## CLI 文生视频（任意家族）： @@ -54,31 +79,92 @@ TensorSharp.Cli --model Wan2.2-TI2V-5B-Q8_0.gguf --backend ggml_cuda \ ```bash # A14B：--model 指向任一专家即可，另一专家自动配对 TensorSharp.Cli --model HighNoise/Wan2.2-I2V-A14B-HighNoise-Q4_K_M.gguf \ - --backend ggml_cuda --prompt "帆船驶入风暴，巨浪拍打" --image ship.jpg --output ship.mp4 + --backend ggml_cuda --prompt "帆船驶入风暴，巨浪拍打" \ + --image ship.jpg --output ship.mp4 ``` - `--image <文件>` —— 首帧条件图（PNG/JPEG/WebP/…），自动应用 EXIF 旋转。 未显式指定 `--width/--height` 时，输出分辨率按图片纵横比适配模型原生面积 （TI2V-5B：1280×704，即官方 720p 配方；其余：832×480）。 -- `--video-frames N` —— 帧数，对齐到 `4k+1`；`1` 生成静态图（配合 `--output out.png`）。 -- `--sampler unipc|euler` —— UniPC（默认）为官方采样器，数值上对照 diffusers 验证。 +- `--video-frames N` —— 帧数，对齐到 `4k+1`；`--video-frames 1` 生成静态图 + （配合 `--output out.png`）。 +- `--sampler unipc|euler` —— UniPC（默认）为官方 Wan 采样器，已与 diffusers 的 + `UniPCMultistepScheduler` 做数值对照验证。 - `--cfg` / `--flow-shift` / `--diffusion-steps` 默认为各模型官方配方：TI2V-5B cfg 5.0、shift 5.0、50 步、24 fps；A14B I2V cfg 3.5（双专家）、shift 5.0、40 步； A14B T2V cfg 4.0/3.0、shift 12.0；Wan 2.1 cfg 6.0、shift 8.0（1.3B 视频）或 3.0/5.0、30 步、16 fps。 +- `--negative-prompt` 默认使用官方 Wan 负面提示词。 +- **步数蒸馏检查点会被自动识别**，也是目前最大的提速杠杆 —— 100 次 DiT 前向 + 变成 4 次，视频内容不变。无需任何开关：它就是一个普通的 `--model` GGUF。 + 详见下方[快车道](#快车道步数蒸馏检查点)。 +- `--cfg-cache-stride N`（默认关闭）—— 近似提速。一次带引导的步进为 + `v = v_cond + (cfg-1)·d`，其中 `d = v_cond - v_uncond`；引导方向 `d` 在整个 + 调度上的变化远慢于 `v`，因此无条件前向可以每 `N` 步只跑一次，中间复用缓存的 + `d`。50 步时，`2` 只跑 100 次前向中的 77 次（快 1.30×），`3` 跑 70 次（1.43×）。 + 前三步与最后一步始终重新计算 `d`。需要与参考样本严格对齐时请关闭它。 + 服务端字段：`"cfgCacheStride": 2`。 - MP4 输出优先使用 `PATH` 上的 `ffmpeg`（或 `TS_FFMPEG=`，或可执行文件旁的 `ffmpeg` 目录）—— 近无损 CRF 17 H.264。没有 ffmpeg 时回退到系统编码器 （OpenCV），默认码率画质明显更差，CLI 会给出警告。 +### 快车道：步数蒸馏检查点 + +这就是一段 5 秒 720p 视频花三个半小时还是十七分钟的区别，而代价只是改一下 +文件路径里的一个词。 + +TensorSharp 读的是 DiT GGUF 的**文件名**。文件名中含有 `Turbo` / `distill` / +`Lightning` / `lightx2v` / `FastWan` / `-dmd`，或含有显式步数（`…-4steps-…`、 +`…8step…`，取值 1–16 有效）时，采样默认值会切换到该步数并**关闭引导（CFG）** +—— 这正是这些检查点训练时的设定。加载时控制台会确认： + +``` +step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override) +``` + +官方 50 步 + CFG 的配方需要 50 × 2 = 100 次 DiT 前向；一个 4 步、无引导的检查点 +只需 **4** 次 —— 去噪工作量只有 1/25。其余一切（配套文件、参数、端点、Web UI） +都不变，`--diffusion-steps` / `--cfg` 仍可覆盖被识别出的默认值。 + +可直接使用的 GGUF。蒸馏仓库只提供 DiT，VAE 与文本编码器仍从基础仓库获取： + +```bash +# TI2V-5B，4 步 —— 注意文件名里是 Wan2_2（下划线） +hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models +hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models +hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models + +TensorSharp.Cli --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_metal \ + --prompt "猫向镜头跑来，电影感" --image first_frame.png \ + --output cat.mp4 --video-frames 121 +``` + +```bash +# I2V-A14B，4 步 —— 两个专家都要；--model 指向任一个，另一个按文件名自动找到 +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models +hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models +``` + +[hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF](https://huggingface.co/hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF) +还提供 Q6_K（4.22 GB）直到 Q2_K（1.86 GB）； +[jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF](https://huggingface.co/jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF) +已把 Lightning 蒸馏合并进两个专家，提供 Q8_0（每专家 15.4 GB）直到 Q2_K（5.31 GB）。 +[Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF](https://huggingface.co/Green-Sky/FastWan2.2-TI2V-5B-FullAttn-GGUF) +是 TI2V-5B 的另一个选择。注意 +[lightx2v/Wan2.2-Lightning](https://huggingface.co/lightx2v/Wan2.2-Lightning) +只发布 LoRA `.safetensors`，而 TensorSharp 没有 Wan LoRA 选项 —— 请使用上面这些 +已经合并好的 GGUF。 + ### 获得好画质 - **分辨率是最大的杠杆。** Wan 的训练分辨率为 480p（832×480 面积）与 720p（1280×704）；TI2V-5B 原生是 720p 模型。远低于 480p 面积时 DiT 处于 - 分布外，无论多少步都会输出模糊、不稳定、偏色的视频——低于约 0.3 MP 时 + 分布外，无论多少步都会输出模糊、不稳定、偏色的视频 —— 低于约 0.3 MP 时 管线会打印警告。请在受支持的分辨率下生成后再缩小（例如用 `--width 480 --height 704` 而不是 320×480）。 - **把镜头描述完整。** Wan 对详细提示词（主体、动作、运镜、光线、风格）的 - 跟随远好于三四个词的短语；官方演示会先用 LLM 做"提示词扩写"。过短的 + 跟随远好于三四个词的短语；官方演示会先用 LLM 做“提示词扩写”。过短的 提示词让模型缺乏约束、容易漂移。 - 超过官方配方（50）继续加 `--diffusion-steps` 收益很小；优先解决分辨率 与提示词。 @@ -97,6 +183,23 @@ API 请求显式提供的 `frames` 或 `fps` 会分别覆盖对应的启动默 其余模型为 33 帧 / 16 fps。帧数会对齐到 `4k+1`。要调整时长，建议保持模型原生 帧率并修改帧数；仅改变 FPS 会改变播放速度。 +步数蒸馏快车道是 `--model` GGUF 自身的属性，而不是请求字段，因此它对 Web UI 和 +所有端点都生效，客户端无需做任何改动 —— 把 `--model` 指向 Turbo/Lightning +检查点，其余一切照旧： + +```bash +TensorSharp.Server --model Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --backend ggml_metal \ + --video-frames 121 --fps 24 +``` + +这一点在浏览器里最重要：那里的请求只是一句提示词加一张图片，代价在 ETA 出现 +之前完全看不见。同一个 Web UI 请求 —— 1024×768 的首帧、24 fps 下 121 帧， +解析为 1088×832、27 404 个 token —— 在基础检查点 `Wan2.2-TI2V-5B-Q8_0.gguf` +上约需 **3 小时 30 分**，在 `Wan2_2-TI2V-5B-Turbo-Q8_0.gguf` 上只需 +**17 分 30 秒**（M5 Pro，`ggml_metal`；见[性能](#性能)）。如果某次运行的 ETA +看起来离谱，先到启动日志里找 `step-distilled checkpoint detected` 那一行， +再去动别的参数。 + - **Web UI**（`http://localhost:5000`）：在聊天框输入提示词 —— 附带一张图片即为 图生视频（图片成为首帧）；回复即是带实时去噪进度的生成视频。 - **API**： @@ -107,15 +210,66 @@ curl http://localhost:5000/v1/videos/generations -H "Content-Type: application/j "image": "data:image/png;base64,....", "size": "832x480", "frames": 81, "steps": 50, "seed": 7 }' +# -> { "created": ..., "data": [ { "url": "/uploads/video-....mp4" } ], "frames": 81, ... } ``` - `image` 接受 base64（可带 `data:` 前缀），省略即为文生视频。另有 - `POST /api/video-generate`（同样的 JSON，支持 `imagePath` 引用已上传文件）与 - `POST /api/video-generate/stream`（SSE 进度，Web UI 使用）。A14B 各端点均支持 - `cfg2`（低噪专家引导系数）。 + `image` 接受 base64（可带 `data:` URL 前缀），省略即为文生视频。另有 + `POST /api/video-generate`（同样的 JSON，返回 `{ ok, url, ... }` 信封，并支持 + `imagePath` 引用已上传文件）与 `POST /api/video-generate/stream`（SSE 进度， + Web UI 使用）。A14B 在各端点均支持 `cfg2`（低噪专家引导系数）。 + +## 实现说明 + +每个网络每次调用都作为**一张**权重常驻的 ggml 图运行 +（`TensorSharp.GGML.Native/ggml_ops_wan.cpp`）： + +- `TSGgml_WanT5Encode` —— 24 层 UMT5 编码器，带逐层相对注意力偏置；输入 token， + 输出隐藏状态（条件按 Wan 的语义零填充到 512 token）。 +- `TSGgml_WanDitForward` —— 一步去噪的速度预测；在 CUDA 上按形状常驻，使 ggml 的 + CUDA graph 捕获可以消除启动开销；3D-RoPE 自注意力走 flash attention。对 + TI2V-5B 的图生视频，该图会在两个不同时间步上调制两段 token（条件帧的 token 位于 + t=0，即 diffusers 的 `expand_timesteps` 语义）—— 注意力仍在完整的联合序列上进行。 +- `TSGgml_WanVaeDecode` —— 因果 3D 视频 VAE 解码器（两代通用），在图内按时间分块 + 迭代，因果特征缓存在块之间传递；卷积在一个显存预算下以分带 im2col+GEMM 执行， + 使峰值显存有界（`TS_WAN_VAE_GEMM_MAX_MB`，默认随设备可用显存自适应，下限 + 384 MB；`0` 强制直接卷积），跨块特征缓存以 F16 存储。Wan 2.2 解码器额外包含 + 无权重的 DupUp3D 残差捷径与最后的 2×2 像素 unpatchify。超过约 0.5 MP 时，解码 + 会进一步切成全宽横向条带，并在 8 个潜变量行的重叠区上做混合（即 diffusers 的 + `enable_tiling` 方案，相对不分块解码约 59 dB）：否则一个 720p 平面的激活加上 + 因果缓存约占 12 GB 设备显存，会把 16 GB 的 WDDM 卡推入共享内存分页 + （69 帧 720p 从 33 分钟降到约 3 分钟）。`TS_WAN_VAE_TILE=0` 关闭分块。 +- `TSGgml_WanVaeEncode` —— 图生视频条件背后的因果 3D VAE 编码器（两代通用）： + 1+4k 像素帧分块、带因果缓存的步长 2 空间与时间下采样、AvgDown3D 捷径，以及 + Wan 2.2 的 2×2 像素 patchify。 + +图生视频的条件注入严格遵循参考管线：TI2V-5B 每一步都用 VAE 编码后的图片替换 +第一个潜变量帧，并把它的 token 在时间步 0 上调制；A14B 则在 16 个噪声通道之后 +拼接 4 通道首帧掩码与 VAE 编码的 `[image, zeros…]` 片段（36 通道 DiT 输入）， +并在 `t < boundary·1000`（I2V 0.9 / T2V 0.875）处从高噪专家切换到低噪专家， +切换时释放第一个专家占用的显存。 + +数值由带守卫的测试对照参考实现验证（`InferenceWeb.Tests/WanVideoOracleTests.cs`， +测试夹具由 `InferenceWeb.Tests/tools/wan22_oracle.py` 生成）：TI2V-5B 的 DiT（Q8_0） +在统一时间步与逐 token 时间步两种模式下与 diffusers 的 `WanTransformer3DModel` +余弦相似度 > 0.995；两个 VAE 编码器与 `AutoencoderKLWan` 余弦相似度 > 0.999； +Wan 2.2 VAE 解码 > 35 dB PSNR（Wan 2.1 解码：59.9 dB）；分词器在英文与中日韩 +提示词上与 HuggingFace T5 的 id 完全一致。 + +环境变量开关：`TS_WAN_DIT_CAPTURE=0`（关闭常驻捕获的 DiT 图）、 +`TS_WAN_DIT_FLASH=0`（改用材料化注意力）、`TS_WAN_DIT_KV_F16=0`（注意力 K/V 用 +F32 —— 旧的、约慢 2 倍的默认值）、`TS_WAN_HEARTBEAT_S`（进度心跳间隔，默认 30， +`0` 关闭）、`TS_WAN_VAE_GEMM_MAX_MB`（im2col 预算）、`TS_WAN_VAE_MPS_CONV=0` +（Metal 上改用 ggml 的卷积下降而非 MPSGraph）、 +`TS_WAN_VAE`/`TS_WAN_TE`/`TS_WAN_DIT2`（配套文件路径）、`TS_FFMPEG`（MP4 导出用的 +ffmpeg 路径）、`TS_WAN_DIT_TRACE=`（逐阶段激活统计，用于调试）。 ## 性能 +**一句话结论：用步数蒸馏检查点。** 本页其它每一个杠杆都只值几十个百分点， +这一个值 25 倍。同一个 5 秒 720p 图生视频请求，在同一台机器、同样的参数下， +基础检查点要 **3 小时 30 分**，Turbo 检查点只要 **17 分 30 秒** —— +去哪里下载见[快车道](#快车道步数蒸馏检查点)，拆解见下面的 M5 Pro 表格。 + RTX 3080 Laptop 16 GB（Windows/WDDM），ggml_cuda： | 模型 / 工作负载 | 文本编码 | 图像编码 | 去噪 | VAE 解码 | 总计 | @@ -127,4 +281,182 @@ RTX 3080 Laptop 16 GB（Windows/WDDM），ggml_cuda： TI2V-5B 的 16×16 空间压缩使同分辨率下 DiT token 数约为 Wan 2.1 的 1/2.7 —— 它是消费级 GPU 上速度最快、质量最高的选择，也是唯一的 720p-24fps 模型。 -数值验证、实现细节与环境变量见[英文卡片](wan.md)。 + +### 长序列：开销随 token 数平方增长 + +token 数为 `潜变量帧数 × (h/2) × (w/2)`，而 DiT 自注意力的开销是 `O(token²)`。 +超过几千 token 后，时间就花在注意力上而不是权重矩阵乘上，因此帧数和帧面积 +的影响都远大于步数： + +| 请求（TI2V-5B） | token | 注意力工作量 | +|---|---|---| +| 640×384×25 帧 | 1 200 | 1× | +| 832×480×81 帧 | 8 190 | 47× | +| 1088×832×121 帧（24 fps 下 5 秒，720p 级） | 27 404 | 520× | + +完整的 5 秒 720p 配方确实是一个大任务：50 步 × 2 次 CFG 前向 = 100 次 DiT 前向， +每次 27k token。管线会打印 token 数、每次前向的耗时和滚动 ETA，并在一次前向 +进行中每 30 秒心跳一次，让长时间运行看起来是在推进而不是卡死： + +``` +Wan 2.2-TI2V I2V: 1088x832x121f (31x26x34 = 27404 tokens), 50 steps, unipc, cfg 5, shift 5, seed ... + [wan] large request: 100 DiT passes over 27404 tokens. Self-attention costs O(tokens^2) ... + step 1/50 t=999.0 (249.0s: cond 124.6s + uncond 124.4s) — ~3h 24m left + [wan] …denoise step 2/50 (cond pass) 60s in this pass, 315s total, ~3h 24m left +``` + +要让这样的请求更便宜，按效果排序： + +1. **换用步数蒸馏检查点。** 这一项碾压其它所有手段：100 次 DiT 前向变成 4 次。 + 见[快车道](#快车道步数蒸馏检查点) —— TensorSharp 按文件名自动识别并切换配方。 +2. **减少帧数。** 121 → 61 帧大约把注意力工作量降到 1/4，VAE 解码减半； + 24 fps 下视频从 5 秒变成 2.5 秒。 +3. **减小帧面积**，但不要低于 Wan 的训练分辨率 —— 低于约 0.3 MP 时模型处于 + 分布外，结果会*更差*而不只是更便宜。`--width 480 --height 704` 是不错的 + 竖屏目标。 +4. **减少步数**（仅限基础检查点）。50 是官方 TI2V 配方；30 步观感非常接近， + 便宜 1.7×。 +5. **`--cfg-cache-stride 2` 或 `3`** —— 通过在步之间复用引导方向获得 + 1.30× / 1.43×（近似方法，见 CLI 一节）。对蒸馏检查点没有意义，它本来就 + 不用引导。 + +M5 Pro（20 核 GPU，48 GB 统一内存），`ggml_metal`，Wan2.2-TI2V-5B Q8_0， +1088×832×121 帧 = 27 404 token，图生视频 —— 下文所有优化都是针对这个请求 +测量的（24 fps 下 5 秒，720p 级）： + +| | 基础检查点，优化前 | 基础检查点，现在 | **Turbo 检查点，现在** | +|---|---|---|---| +| DiT 前向次数 | 100（50 步 × CFG） | 100 | **4**（4 步，无引导） | +| 每次前向 | 206.2 s | **120.2 s** | **120.2 s** | +| 去噪总计 | 20 615 s | 12 020 s | **481 s** | +| VAE 解码，121 帧 | 863 s | **563 s** | **563 s** | +| **端到端** | **≈ 5 小时 58 分** | ≈ 3 小时 30 分 | **17 分 30 秒** | + +这两个杠杆是独立的：单次前向约 1.7× 来自下文的注意力与 VAE 改动，前向次数 +25× 来自步数蒸馏检查点。一旦蒸馏，**瓶颈就变成 VAE 解码**（约占整轮的 55%）， +不再是 DiT。 + +之后就由帧数和帧面积决定其余部分：VAE 随输出像素线性增长，注意力随 token 数 +平方增长。以下均在同一台机器、用同一个 Turbo 检查点和同一张图片测得： + +| 输出 | token | 去噪 | VAE 解码 | **总计** | +|---|---|---|---|---| +| 736×544 × 81 帧（3.4 秒，480p 级） | 8 211 | 84 s | 159 s | **4 分 09 秒** | +| 736×544 × 121 帧（5 秒，480p 级） | 12 121 | 137 s | 237 s | **6 分 19 秒** | +| 1088×832 × 121 帧（5 秒，720p 级） | 27 404 | 481 s | 563 s | **17 分 30 秒** | + +480p（约 0.4 MP）是 Wan *训练过*的分辨率，所以中间那几行是分布内的结果， +不是降级模式 —— 当你只有几分钟时间时，就该选它。真正开始掉画质的是低于 +约 0.3 MP。 + +单次前向的提速由三处改动贡献。 + +**注意力的 K/V 用 F16。** 每个后端的 flash attention 内核都是围绕 F16 KV 缓存 +设计的：ggml-metal 的 F32 变体用 `simdgroup_float8x8` 累加器，而 F16 变体用 +`simdgroup_half8x8`；在一个每 8 个 query 线程组就要重新流式读一遍 K 和 V 的 +内核里，F32 分块还要付双倍带宽。转成 flash attention 布局的 permute 与降精度 +合并在一次 `ggml_cpy` 中完成，所以 F16 路径写出的字节数也只有旧的 `ggml_cont` +的一半。`TS_WAN_DIT_KV_F16=0` 可恢复 F32。 + +**VAE 解码的分带布局。** 分块现在从显存预算推出条带的*数量*再均分平面，而不是 +按固定条带高度往下走、让最后一条落在步长决定的任意位置。52 个潜变量行原来是 +三条 24 行的带 —— 解码 72 行才得到 52 行 —— 现在在同样的每带预算下是两条 30 行 +的带（共 60 行，接缝从两条减为一条）。 + +**Metal 上 VAE 卷积走 MPSGraph。** ggml 把 `conv2d` 下降为 im2col + `mul_mat`， +而一次解码剖析（`WanVideoBench`，640×480×9 帧）显示**图中 44.4% 在 MUL_MAT、 +另有 30.2% 在 IM2COL** —— 74.6% 都在卷积里，im2col 每个节点搬运 865 MB，速率约 +57 GB/s。这个下降过程纯粹是开销：一个 3×3 卷积在 GEMM 开始之前要先材料化出 9 倍 +于输入的数据。Apple 调优过的卷积在同样的形状上快得多： + +| 卷积形状 | ggml im2col+GEMM | MPSGraph | | +|---|---|---|---| +| 512→512 k3 320×240 t9 | 666 ms | 108 ms | 6.2× | +| 256→256 k3 640×480 t9 | 947 ms | 110 ms | 8.6× | +| 160→160 k3 640×480 t9 | 491 ms | 47 ms | 10.4× | +| 512→512 k1 320×240 t9 | 184 ms | 14 ms | 13.9× | + +MPS 能跑到约 30 TFLOP/s，而 ggml 的 Metal GEMM 只有约 4.9 —— 而且它*不*经过 +ggml 的 `mul_mm`，也就是那个 Metal 4 tensor 路径会把这张图算错的内核。它既拿到 +了矩阵单元的吞吐，又绕开了那个缺陷。 + +布局是零成本的：ggml 的激活 `[W,H,C,T]` 与卷积核 `[KW,KH,IC,OC]` 逐字节就是 +MPS 的 NCHW 与 OIHW，而且这里实测 NCHW 与 NHWC 一样快，所以不需要插入任何转置。 +在这条路径上 VAE 直接发出未下降的 `CONV_2D` 节点，也就顺带完全跳过了横向分带 +及其前置 padding 补丁；解码图随后被分段执行，每个 `CONV_2D` 交给 MPSGraph +（用 `ggml_graph_view`，与节点剖析器相同的机制，因此对 gallocr 的缓冲复用是 +安全的）。端到端实测：在 736×544×81 帧下 **VAE 解码 159 s → 80 s（1.99×）**。 +数值不变 —— ggml 本来就以 F16 卷积、F32 累加，这条路径同样如此，整轮解码一致到 +**93.9 dB PSNR / 最大偏差 0.128（满量程 255）**。`TS_WAN_VAE_MPS_CONV=0` 可恢复 +ggml 的下降路径。 + +**VAE 卷积的 im2col 预算与分块阈值本身，改由设备显存推算。** 这两者过去在所有 +非 CUDA 后端上都被钉死在一张 16 GB 卡的预算上。Metal 现在像 CUDA 一样按可用显存 +推算（Vulkan 保留下限 —— 它的驱动拒绝多 GB 的竞技场）。更大的 im2col 预算把许多 +小的分带 GEMM 变成少量大 GEMM：1088×832×9 帧的解码从 56.4 s 降到 49.7 s。而且 +分块并不免费 —— 条带彼此重叠，解码器实际跑的潜变量行数多于平面本身，而且条带 +缓冲要与整幅画布共存。把 1088×832×121 帧**整幅**解码实测既更快又更省： +**565 s vs 655 s，峰值 RSS 4.85 GB vs 5.37 GB**。小显存的卡仍然分块。 + +#### 实测后被否决的方案 + +同一轮运行中，该形状（序列 27 404、24 个头、head dim 128）下的一次自注意力： + +| KV 类型 | 耗时 | 相对 F32 | +|---|---|---| +| F32（之前） | 4993 ms | 1.00× | +| **F16（现在）** | **2467 ms** | **2.02×** | +| Q8_0 | 2652 ms | 1.88× | +| Q4_0 | 2395 ms | 2.08× | + +Q8_0 的 K/V 比 F16 *更慢* —— 反量化的开销盖过了省下的带宽 —— 而 Q4_0 只多买到 +3%，却要付出实实在在的精度代价，所以 F16 是最优点而不是折中。材料化的分块注意力 +（`mul_mat` + `soft_max_ext`，与 flash attention 不同，它*可以*用 Metal 4 tensor +API）只有 1.08×：在这个规模上注意力受限于分数矩阵的访存流量，而不是 GEMM 速率。 +把 DiT 权重从 Q8_0 反量化成 F16，矩阵乘只变化了不到 5%（一次注意力投影从 +25.2 ms 到 26.4 ms）。 + +#### Metal 4 tensor API，以及不该怎么测它 + +这是这里最诱人的一个开关，而它对 TI2V-5B 保持**关闭**。它在 DiT 上实测 1.20× +（121.4 → 100.8 s/次前向）、在 VAE 解码上 1.66×，因为它把 `mul_mm` 路由到 Metal 4 +的 tensor 操作 —— 但在 M5 / macOS 26.6 上，它同时会把 VAE 的卷积 GEMM 算错， +视频输出为一片**纯黑**。 + +陷阱在于：这个问题在隔离测试里复现不出来。`WanVideoBench vae-decode` 在五种潜变量 +形状下跑过（包括被记录为最初全 NaN 复现场景的 32×32 布局），在 tensor API 开与关 +两种情况下导出原始像素：两者一致到 **91–93 dB PSNR、最大偏差 0.18/255、零 NaN**。 +基于这个证据，规避措施被移除了 —— 而紧接着的下一次完整 1088×832×121 帧生成就 +产出了 121 张黑帧。这个缺陷跟随进程的分配历史（DiT 在解码之前被加载并释放）， +所以一个全新进程里的解码探针会复现出错误的缓冲布局。 + +随后又用一个*更*贴近真实的测试台重测了一次 —— `WanVideoBench pipeline`，它在 +同一个进程里按生产环境的确切形状依次跑 VAE 编码 → DiT 前向 → 释放 → VAE 解码， +包括完整的 121 帧解码。结果同样干净，而且快 1.50×（374 s vs 563 s）。可真实生成 +出来的仍然是平坦的画面。唯一剩下的差别是 UMT5-XXL 文本编码器 —— 它在其它一切 +之前加载并释放，是整条管线里最大的一次分配。 + +所以：重新启用它必须以一次完整的端到端视频为准，任何更弱的验证都从未足够。 +管线现在会拒绝返回一段完全平坦的视频，并指出这个成因 +（`AssertFramesAreNotDegenerate`）—— 正是它自动抓住了第二次尝试，省下了又一次 +靠肉眼看 PNG 的代价。 + +#### 质量 + +F16 的 K/V 对 DiT 输出的改变，小于采样器自身对浮点重结合顺序的敏感度： + +- 对照 diffusers `WanTransformer3DModel` 参考实现的一次 DiT 前向 + （`WanVideoOracleTests`）：F16 K/V 余弦 **0.999964**，F32 K/V 余弦 **0.999964** + —— 单次前向的精度没有变化。 +- 相同种子下的完整 25 帧生成：F16 与 F32 K/V 的平均 PSNR 差异为 39.08 dB。 + 对照组 —— F32 K/V 且*关闭* flash attention，即同样的算术但求和顺序不同 —— + 差异为 39.40 dB。两者量级相同，说明像素差异是一条 8 步扩散轨迹因舍入差异而 + 分叉的结果，而不是精度损失。 + +提速只在注意力占主导时出现：在 2 310 token（480×704×25 帧）下，F16 与 F32 K/V +每次前向都是 5.0 s。 + +Wan 2.1 对比 stable-diffusion.cpp（master-769，相同的 GGUF 与设置，33 帧 480p）： +采样阶段接近持平（281 s vs 258 s），但 sd.cpp 的 VAE 解码要材料化约 8 GB 的 3D +im2col，把一张 16 GB 的卡压进 WDDM 分页（1762 s vs **51 s**）—— TensorSharp +端到端快 6.0×。 diff --git a/docs_models_README_zh-cn.md b/docs_models_README_zh-cn.md new file mode 100644 index 00000000..e69de29b diff --git a/docs_models_wan_zh-cn.md b/docs_models_wan_zh-cn.md new file mode 100644 index 00000000..e69de29b diff --git a/website/assets/app.js b/website/assets/app.js index 4fb16c8c..ae84e647 100644 --- a/website/assets/app.js +++ b/website/assets/app.js @@ -13,7 +13,12 @@ { group: { en: "Get started", zh: "快速上手" }, items: [ { page: "getting-started", label: { en: "Getting Started", zh: "快速开始" } }, { page: "backends", label: { en: "Compute Backends", zh: "计算后端" } }, - { page: "models", label: { en: "Supported Models", zh: "支持的模型" } }, + { page: "models", label: { en: "Supported Models", zh: "支持的模型" }, children: [ + { page: "models-downloads", label: { en: "Model Downloads", zh: "模型下载" } }, + { page: "models-text", label: { en: "Text & LLM Models", zh: "文本与 LLM 模型" } }, + { page: "models-image", label: { en: "Image Generation", zh: "图像生成" } }, + { page: "models-video", label: { en: "Video Generation", zh: "视频生成" } }, + ]}, ]}, { group: { en: "Run it", zh: "运行" }, items: [ { page: "cli", label: { en: "Command Line (CLI)", zh: "命令行 (CLI)" } }, @@ -159,6 +164,12 @@ var a = el("a", { href: hrefFor(it.page, lang) }, it.label[lang]); if (it.page === cur) a.className = "active"; grp.appendChild(a); + // Optional second level (e.g. the model reference sub-pages). + (it.children || []).forEach(function (ch) { + var c = el("a", { href: hrefFor(ch.page, lang) }, ch.label[lang]); + c.className = ch.page === cur ? "sub active" : "sub"; + grp.appendChild(c); + }); }); sidebar.appendChild(grp); }); diff --git a/website/assets/search-index-zh.js b/website/assets/search-index-zh.js index 7acfe5d0..d207cf24 100644 --- a/website/assets/search-index-zh.js +++ b/website/assets/search-index-zh.js @@ -35,14 +35,22 @@ window.SEARCH_INDEX_ZH = [ { t: "构建原生 GGML / MLX 库", p: "后端", u: "backends.html#native-build", s: "build-windows.ps1 / build-linux.sh / build-macos.sh、CUDA 架构检测，以及自动启用的 Vulkan（用 --no-vulkan 退出；Windows 便携 Vulkan 工具链）。", k: "compile 编译 native cmake cuda arch vulkan glslc no-vulkan TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN" }, { t: "当前二进制发行状态", p: "后端", u: "backends.html#binaries", s: "最新 v3.0.5.0 只有源码下载，没有上传 CLI/server 应用归档；Release 出现资产前请从源码构建。", k: "release 下载 归档 二进制 预构建 source v3.0.5.0" }, - { t: "支持的模型", p: "模型", u: "models.html#table", s: "Gemma 3/4、Qwen 3 / 3.5 / 3.6、GPT OSS、Nemotron-H、Mistral 3、DiffusionGemma、Qwen-Image-Edit。", k: "architectures 架构 家族 gemma qwen gptoss nemotron mistral 图像 编辑" }, - { t: "模型下载（GGUF）", p: "模型", u: "models.html#downloads", s: "各受支持架构的 Hugging Face 链接、精确文件名、mmproj 与 MTP 伴随文件。", k: "huggingface 权重 下载 mmproj mtp draft assistant lora vae" }, - { t: "下载并运行", p: "模型", u: "models.html#download-run", s: "以 Gemma 4 E4B 快速上手开头，附九个模型家族的 hf download 与 CLI / Server 可复制命令。", k: "hf download huggingface_hub cli server 快速 命令 家族" }, - { t: "约 30 秒快速上手：Gemma 4 E4B Q8_0（原生 GGML）", p: "模型", u: "models.html#gemma4-fast-lane", s: "推荐公开 E4B Q8_0 文件、原生构建、CLI/Server 命令、后端选择与可选 mmproj。", k: "gemma4 e4b q8 快速上手 30 秒 快路径 原生 ggml cuda metal vulkan mmproj" }, - { t: "多模态：图像、视频、音频、PDF", p: "模型", u: "models.html#multimodal", s: "Gemma 4 支持图像/视频/音频；其他视觉家族支持图像；CLI/Web UI 支持 PDF 文本提取或页面图像。", k: "vision 视觉 图像 音频 视频 mmproj 投影器 pdf 文档" }, - { t: "图像编辑（Qwen-Image-Edit）", p: "模型", u: "models.html#imageedit", s: "提示词 + 输入图像 -> 编辑后的图像，经 qwen_image MMDiT、Qwen-Image VAE 与 Qwen2.5-VL 文本编码器。", k: "qwen 图像 编辑 mmdit 扩散 vae flowmatch cfg 图片 照片 lora qwen-image-lora lightning step cache easycache fbc dit 步缓存" }, - { t: "思考 / 推理模式", p: "模型", u: "models.html#thinking", s: "Qwen、Gemma 4、GPT OSS、Nemotron-H 带 think 标签的结构化思维链。", k: "reasoning 推理 思维链 think cot" }, - { t: "工具调用 / 函数调用", p: "模型", u: "models.html#toolcalling", s: "模型调用用户自定义工具；跨三种 API 风格的多轮。", k: "functions 函数 工具 agent 智能体" }, + { t: "支持的模型", p: "模型", u: "models.html#table", s: "DeepSeek V4 Flash、Gemma 3/4、Qwen 3 / 3.5 / 3.6、GPT OSS、Nemotron-H、Mistral 3、DiffusionGemma、Muse-Glimmer、Qwen-Image-Edit、Wan 视频。", k: "architectures 架构 家族 deepseek gemma qwen gptoss nemotron mistral 图像 编辑 wan 视频" }, + { t: "哪个更快——每个家族最关键的那一项设置", p: "模型", u: "models.html#fast-lane", s: "每个家族中真正决定端到端耗时的那一个检查点、LoRA 或参数。", k: "fast 快 更快 加速 蒸馏 turbo lightning lora draft 推测解码 性能 哪个" }, + { t: "浏览模型参考", p: "模型", u: "models.html#browse", s: "模型参考分为四个页面：模型下载、文本与 LLM 模型、图像生成、视频生成。", k: "索引 浏览 目录 页面 导航" }, + { t: "模型下载（GGUF）", p: "模型下载", u: "models-downloads.html#downloads", s: "各受支持架构的 Hugging Face 链接、精确文件名，以及 mmproj、VAE、文本编码器与 MTP 草稿等伴随文件。", k: "huggingface 权重 下载 mmproj mtp draft lora vae turbo 蒸馏" }, + { t: "下载并运行", p: "文本与 LLM 模型", u: "models-text.html#download-run", s: "每个文本模型家族的 hf download 与 CLI / Server 可复制命令。", k: "hf download huggingface_hub cli server 快速 命令 家族" }, + { t: "约 30 秒快速上手：Gemma 4 E4B Q8_0（原生 GGML）", p: "文本与 LLM 模型", u: "models-text.html#gemma4-fast-lane", s: "推荐公开 E4B Q8_0 文件、原生构建、CLI/Server 命令、后端选择与可选 mmproj。", k: "gemma4 e4b q8 快速上手 30 秒 快路径 原生 ggml cuda metal vulkan mmproj" }, + { t: "DeepSeek V4 Flash（284B MoE）", p: "文本与 LLM 模型", u: "models-text.html#deepseek4", s: "压缩稀疏注意力的 284B MoE，三种整模型执行器（直连 CUDA、原生 ggml、纯 C# CPU），跨 GPU 自动切分，并支持 DSpark 推测解码。", k: "deepseek deepseek4 v4 flash 284b moe dspark draft-model 切分 gguf 分片 1m 上下文" }, + { t: "多模态：图像、视频、音频、PDF", p: "文本与 LLM 模型", u: "models-text.html#multimodal", s: "Gemma 4 支持图像/视频/音频；其他视觉家族支持图像；CLI/Web UI 支持 PDF 文本提取或页面图像。", k: "vision 视觉 图像 音频 视频 mmproj 投影器 pdf 文档" }, + { t: "思考 / 推理模式", p: "文本与 LLM 模型", u: "models-text.html#thinking", s: "Qwen、Gemma 4、GPT OSS、Nemotron-H 带 think 标签的结构化思维链。", k: "reasoning 推理 思维链 think cot" }, + { t: "工具调用 / 函数调用", p: "文本与 LLM 模型", u: "models-text.html#toolcalling", s: "模型调用用户自定义工具；跨三种 API 风格的多轮。", k: "functions 函数 工具 agent 智能体" }, + { t: "Qwen-Image-Edit —— 下载并运行", p: "图像生成", u: "models-image.html#qwen-image-edit", s: "MMDiT 图像编辑器及其 VAE、Qwen2.5-VL 文本编码器与可选 Lightning LoRA 的 hf download 与 CLI / Server 命令。", k: "qwen 图像 编辑 下载 运行 cli server lora lightning vae mmproj" }, + { t: "图像编辑（Qwen-Image-Edit）", p: "图像生成", u: "models-image.html#imageedit", s: "提示词 + 输入图像 -> 编辑后的图像，经 qwen_image MMDiT、Qwen-Image VAE 与 Qwen2.5-VL 文本编码器。", k: "qwen 图像 编辑 mmdit 扩散 vae flowmatch cfg 图片 照片 lora lightning easycache fbc 步缓存" }, + { t: "Wan 2.1 / 2.2 —— 下载并运行", p: "视频生成", u: "models-video.html#wan", s: "文生视频与图生视频的 hf download 与 CLI / Server 命令，含步数蒸馏的 Turbo 检查点。", k: "wan 下载 运行 cli server video-frames fps t2v i2v turbo 蒸馏 umt5 vae" }, + { t: "视频生成（Wan 2.1 / 2.2）", p: "视频生成", u: "models-video.html#video", s: "提示词或上传的首帧图像生成 H.264 MP4。涵盖 Wan 2.1 T2V、Wan 2.2 TI2V-5B 与 A14B，配合 UMT5-XXL 与因果 3D 视频 VAE。", k: "wan 视频 mp4 文生视频 图生视频 ti2v a14b t2v i2v umt5 vae 扩散" }, + { t: "检查点决定端到端耗时（Wan）", p: "视频生成", u: "models-video.html#video-fast", s: "TI2V-5B 官方配方需要 100 次 DiT 前向；步数蒸馏的 Turbo 检查点只需 4 次，且按文件名自动识别。", k: "wan turbo 蒸馏 lightning lightx2v fastwan 4 步 无引导 cfg 慢 快 小时 分钟 加速" }, + { t: "帧数与分辨率决定其余部分（Wan）", p: "视频生成", u: "models-video.html#video-resolution", s: "token 数为 latent_frames x (h/2) x (w/2)，注意力开销是平方级，因此帧数与画面面积最关键；480p 在训练分布内且便宜得多。", k: "wan 分辨率 帧数 480p 720p token 平方 注意力 开销 更便宜" }, { t: "CLI 示例", p: "CLI", u: "cli.html#examples", s: "文本、图像、视频、音频、PDF、思考、工具、批处理 JSONL、基准测试。", k: "command line 命令行 控制台 示例 pdf 文档" }, { t: "在其他原生后端上运行 Gemma 4 E4B", p: "CLI", u: "cli.html#e4b-fast-lane", s: "Metal、Vulkan 与 ggml_cpu 变体、更省内存的 Q4_K_M，以及多模态所需的 mmproj 说明。", k: "gemma4 e4b 命令行 metal vulkan cpu q4_k_m mmproj q8" }, diff --git a/website/assets/search-index.js b/website/assets/search-index.js index 3661b0db..3e856ef1 100644 --- a/website/assets/search-index.js +++ b/website/assets/search-index.js @@ -35,15 +35,22 @@ window.SEARCH_INDEX = [ { t: "Build the native GGML / MLX libraries", p: "Backends", u: "backends.html#native-build", s: "build-windows.ps1 / build-linux.sh / build-macos.sh, CUDA arch detection, and auto-enabled Vulkan (opt out with --no-vulkan; portable Vulkan toolchain on Windows).", k: "compile native cmake cuda arch vulkan glslc no-vulkan TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN" }, { t: "Current binary release status", p: "Backends", u: "backends.html#binaries", s: "Latest v3.0.5.0 has source downloads but no uploaded CLI/server archives; build from source unless assets appear on Releases.", k: "release download archive binary prebuilt source v3.0.5.0" }, - { t: "Supported models", p: "Models", u: "models.html#table", s: "DeepSeek V4 Flash, Gemma 3/4, Qwen 3 / 3.5 / 3.6, GPT OSS, Nemotron-H, Mistral 3, DiffusionGemma, Qwen-Image-Edit.", k: "architectures families deepseek deepseek4 v4 flash gemma qwen gptoss nemotron mistral image edit" }, - { t: "Model downloads (GGUF)", p: "Models", u: "models.html#downloads", s: "Hugging Face links plus mmproj and MTP-draft companion files for every supported architecture.", k: "huggingface weights download mmproj mtp draft assistant lightning lora vae" }, - { t: "Download & run, per family", p: "Models", u: "models.html#download-run", s: "Copy-paste hf download + TensorSharp.Cli + TensorSharp.Server commands for all nine model families.", k: "hf download huggingface_hub cli server quick start commands per family" }, - { t: "Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)", p: "Models", u: "models.html#gemma4-fast-lane", s: "Recommended public E4B Q8_0 artifact, native build, CLI/server commands, backend choices, and optional mmproj.", k: "gemma4 e4b q8 quick start 30 seconds fast path native ggml cuda metal vulkan mmproj" }, - { t: "DeepSeek V4 Flash (284B MoE)", p: "Models", u: "models.html#deepseek4", s: "Compressed-sparse-attention 284B MoE with three whole-model executors (direct CUDA, native ggml, pure C# CPU), automatic layer split across GPUs, and DSpark speculative decoding.", k: "deepseek deepseek4 v4 flash 284b moe dspark draft-model split gguf shard 1m context lightning indexer csa hca" }, - { t: "Multimodal: image, video, audio, PDF", p: "Models", u: "models.html#multimodal", s: "Image/video/audio for Gemma 4; image for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni; PDF documents via --pdf or Web UI upload.", k: "vision image audio video mmproj projector pdf document" }, - { t: "Image editing (Qwen-Image-Edit)", p: "Models", u: "models.html#imageedit", s: "Prompt + input image -> edited image via the qwen_image MMDiT, a Qwen-Image VAE, and a Qwen2.5-VL text encoder.", k: "qwen image edit mmdit diffusion vae flowmatch cfg picture photo lora qwen-image-lora lightning step cache easycache fbc dit" }, - { t: "Thinking / reasoning mode", p: "Models", u: "models.html#thinking", s: "Structured chain-of-thought with think tags for Qwen, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4.", k: "reasoning chain of thought think cot deepseek" }, - { t: "Tool calling / function calling", p: "Models", u: "models.html#toolcalling", s: "Models invoke user-defined tools; multi-turn across all three API styles. JSON, XML, Harmony, and DeepSeek V4 DSML wire formats.", k: "functions tools agent dsml harmony tool_call xml json" }, + { t: "Supported models", p: "Models", u: "models.html#table", s: "DeepSeek V4 Flash, Gemma 3/4, Qwen 3 / 3.5 / 3.6, GPT OSS, Nemotron-H, Mistral 3, DiffusionGemma, Muse-Glimmer, Qwen-Image-Edit, Wan video.", k: "architectures families deepseek deepseek4 v4 flash gemma qwen gptoss nemotron mistral image edit wan video" }, + { t: "Which one is fastest — the lever that matters per family", p: "Models", u: "models.html#fast-lane", s: "Per family, the one checkpoint, LoRA or flag that decides the wall clock.", k: "fast faster speed speedup distilled turbo lightning lora draft speculative lever performance which" }, + { t: "Browse the model reference", p: "Models", u: "models.html#browse", s: "The model reference split across four pages: downloads, text and LLM models, image generation, video generation.", k: "index browse contents pages navigation" }, + { t: "Model downloads (GGUF)", p: "Model Downloads", u: "models-downloads.html#downloads", s: "Hugging Face links plus mmproj, VAE, text-encoder and MTP-draft companion files for every supported architecture.", k: "huggingface weights download mmproj mtp draft assistant lightning lora vae turbo distilled" }, + { t: "Download & run, per family", p: "Text & LLM Models", u: "models-text.html#download-run", s: "Copy-paste hf download + TensorSharp.Cli + TensorSharp.Server commands for every text model family.", k: "hf download huggingface_hub cli server quick start commands per family" }, + { t: "Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)", p: "Text & LLM Models", u: "models-text.html#gemma4-fast-lane", s: "Recommended public E4B Q8_0 artifact, native build, CLI/server commands, backend choices, and optional mmproj.", k: "gemma4 e4b q8 quick start 30 seconds fast path native ggml cuda metal vulkan mmproj" }, + { t: "DeepSeek V4 Flash (284B MoE)", p: "Text & LLM Models", u: "models-text.html#deepseek4", s: "Compressed-sparse-attention 284B MoE with three whole-model executors (direct CUDA, native ggml, pure C# CPU), automatic layer split across GPUs, and DSpark speculative decoding.", k: "deepseek deepseek4 v4 flash 284b moe dspark draft-model split gguf shard 1m context lightning indexer csa hca" }, + { t: "Multimodal: image, video, audio, PDF", p: "Text & LLM Models", u: "models-text.html#multimodal", s: "Image/video/audio for Gemma 4; image for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni; PDF documents via --pdf or Web UI upload.", k: "vision image audio video mmproj projector pdf document" }, + { t: "Thinking / reasoning mode", p: "Text & LLM Models", u: "models-text.html#thinking", s: "Structured chain-of-thought with think tags for Qwen, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4.", k: "reasoning chain of thought think cot deepseek" }, + { t: "Tool calling / function calling", p: "Text & LLM Models", u: "models-text.html#toolcalling", s: "Models invoke user-defined tools; multi-turn across all three API styles. JSON, XML, Harmony, and DeepSeek V4 DSML wire formats.", k: "functions tools agent dsml harmony tool_call xml json" }, + { t: "Qwen-Image-Edit — download & run", p: "Image Generation", u: "models-image.html#qwen-image-edit", s: "hf download plus CLI and server commands for the MMDiT image editor, its VAE, Qwen2.5-VL text encoder and the optional Lightning LoRA.", k: "qwen image edit download run cli server lora lightning vae mmproj" }, + { t: "Image editing (Qwen-Image-Edit)", p: "Image Generation", u: "models-image.html#imageedit", s: "Prompt + input image -> edited image via the qwen_image MMDiT, a Qwen-Image VAE, and a Qwen2.5-VL text encoder.", k: "qwen image edit mmdit diffusion vae flowmatch cfg picture photo lora qwen-image-lora lightning step cache easycache fbc dit" }, + { t: "Wan 2.1 / 2.2 — download & run", p: "Video Generation", u: "models-video.html#wan", s: "hf download plus CLI and server commands for text-to-video and image-to-video, including the step-distilled Turbo checkpoint.", k: "wan download run cli server video-frames fps t2v i2v turbo distilled umt5 vae" }, + { t: "Video generation (Wan 2.1 / 2.2)", p: "Video Generation", u: "models-video.html#video", s: "Prompt, or an uploaded first frame, to H.264 MP4. Wan 2.1 T2V, Wan 2.2 TI2V-5B and A14B, with UMT5-XXL and a causal 3D video VAE.", k: "wan video mp4 text to video image to video ti2v a14b t2v i2v umt5 vae diffusion" }, + { t: "The checkpoint decides the wall clock (Wan)", p: "Video Generation", u: "models-video.html#video-fast", s: "The base TI2V-5B recipe spends 100 DiT passes; a step-distilled Turbo checkpoint runs the same video in 4, auto-detected from the file name.", k: "wan turbo distilled lightning lightx2v fastwan 4 steps guidance free cfg cache stride slow fast hours minutes speed" }, + { t: "Frames and resolution set the rest (Wan)", p: "Video Generation", u: "models-video.html#video-resolution", s: "Token count is latent_frames x (h/2) x (w/2) and attention is quadratic, so frame count and frame area dominate; 480p is in-distribution and far cheaper.", k: "wan resolution frames 480p 720p tokens quadratic attention cost cheaper size" }, { t: "CLI examples", p: "CLI", u: "cli.html#examples", s: "Text, image, video, audio, PDF, thinking, tools, batch JSONL, benchmarks.", k: "command line console examples pdf document" }, { t: "Configuration file (--config)", p: "CLI", u: "cli.html#config", s: "Read CLI options from a JSON file with variables and auto-downloading models; command line overrides it.", k: "config json variables ${} auto download urls path sha256 reuse cli server" }, diff --git a/website/assets/style.css b/website/assets/style.css index 045020a6..57789f0c 100644 --- a/website/assets/style.css +++ b/website/assets/style.css @@ -135,6 +135,11 @@ kbd { } .sidebar a:hover { background: var(--bg-soft); color: var(--text); text-decoration: none; } .sidebar a.active { background: var(--accent-soft); color: var(--link); font-weight: 600; } +.sidebar a.sub { + margin-left: 10px; padding-left: 12px; font-size: .86rem; + border-left: 2px solid var(--border); border-radius: 0 7px 7px 0; +} +.sidebar a.sub.active { border-left-color: var(--accent); } .content { min-width: 0; padding: 36px 48px 96px; diff --git a/website/backends.html b/website/backends.html index a4899404..f01f2b62 100644 --- a/website/backends.html +++ b/website/backends.html @@ -78,7 +78,7 @@

DeepSeek V4: dedicated whole-model executors

  • --backend ggml_cuda / ggml_vulkan — the native ggml executor: it loads the split GGUF itself, keeps all DSV4 KV state on-device, and runs each prefill/decode micro-batch as a single graph with a shape-signature cache so steady-state decode replays a captured CUDA graph.
  • --backend cpu — a 100% pure-C# executor with no native dependencies, serving quantized weights straight from the memory-mapped GGUF shards.
    • -

    All three layer-split the weights across every visible GPU (the CPU one streams them from the mapped shards), so a model far larger than one card still runs; --tp N or TS_DSV4_NGPU caps how many GPUs are used. Speculative decoding (DSpark) is available on the two GPU engines. → DeepSeek V4 downloads and commands

    +

    All three layer-split the weights across every visible GPU (the CPU one streams them from the mapped shards), so a model far larger than one card still runs; --tp N or TS_DSV4_NGPU caps how many GPUs are used. Speculative decoding (DSpark) is available on the two GPU engines. → DeepSeek V4 downloads and commands

    🔎

    The server reports which backends are actually available on the host in GET /api/models (supportedBackends). If a CUDA or MLX backend is missing, the host did not detect a usable driver/runtime at startup. If ggml_vulkan is missing, the native bridge was not built with Vulkan enabled or no Vulkan 1.3 device/driver was found.

    diff --git a/website/backends_zh-cn.html b/website/backends_zh-cn.html index 62c55d07..5556fb6f 100644 --- a/website/backends_zh-cn.html +++ b/website/backends_zh-cn.html @@ -78,7 +78,7 @@

    DeepSeek V4：专属整模型执行器

  • --backend ggml_cuda / ggml_vulkan —— 原生 ggml 执行器：自行加载分片 GGUF，在设备上持有全部 DSV4 KV 状态，并把每个 prefill/decode 微批作为一张计算图执行，配合按形状签名的缓存，使稳态 decode 直接重放已捕获的 CUDA 图。
  • --backend cpu —— 100% 纯 C# 执行器，零原生依赖，直接从内存映射的 GGUF 分片提供量化权重。
    • -

    三者都会把权重按层切分到所有可见 GPU（CPU 那套则从映射分片流式读取），因此远大于单卡显存的模型依然跑得起来；--tp N 或 TS_DSV4_NGPU 用来限制使用几张卡。投机解码（DSpark）可用于两个 GPU 引擎。→ DeepSeek V4 的下载与命令

    +

    三者都会把权重按层切分到所有可见 GPU（CPU 那套则从映射分片流式读取），因此远大于单卡显存的模型依然跑得起来；--tp N 或 TS_DSV4_NGPU 用来限制使用几张卡。投机解码（DSpark）可用于两个 GPU 引擎。→ DeepSeek V4 的下载与命令

    🔎

    服务器会在 GET /api/models（supportedBackends）中报告主机上实际可用的后端。如果缺少 CUDA 或 MLX 后端，说明主机在启动时未检测到可用的驱动 / 运行时。如果缺少 ggml_vulkan，说明原生桥接库未启用 Vulkan 构建，或未找到支持 Vulkan 1.3 的设备/驱动。

    diff --git a/website/benchmarks.html b/website/benchmarks.html index 70b4affb..64958cba 100644 --- a/website/benchmarks.html +++ b/website/benchmarks.html @@ -66,6 +66,61 @@

    Muse-Glimmer 30B, with and without DFlash

    The pattern: TensorSharp's fused whole-model graph wins prefill below ~2K by 1.16-1.27×, the engines cross somewhere between 2K and 16K, and llama.cpp keeps a 6-8% prefill edge above that; decode is a tie at short context and drifts to 0.86× at 128K. With DFlash speculative decoding TensorSharp is ahead at short and mid prompts and behind from 16K up — but that gap is mostly a runtime cost governor that parks the drafter after a mis-measured probe: on the reps where it stays armed, TensorSharp reaches 94% of llama.cpp at 16K and 96% at 64K. The absolute DFlash figures also depend heavily on the corpus (the 501-token prompt hits 100% draft acceptance on both engines). Both engines run the full 128K context on one card. One caveat on the two long rows: they were taken before a line-ending fix landed in the harness, so on those points TensorSharp prefilled the CRLF form of the same document (1.2% more tokens); throughput is a rate so the effect is small, but the generated continuations are not strictly comparable there. Method, per-rep spread, VRAM and the governor analysis are in the Muse-Glimmer architecture card (docs/models/muse-glimmer.md).

    +

    Wan video: two independent speed levers

    +

    Video is the one workload where the wrong choice costs hours rather than seconds, so it is worth seeing the levers apart. Every row below is the same request — Apple M5 Pro (20-core GPU, 48 GB unified), ggml_metal, Wan2.2-TI2V-5B Q8_0, image-to-video at 1088×832 × 121 frames = 27,404 DiT tokens — and the rows differ only in the engine version and in which checkpoint --model pointed at.

    +
    + + + + + + + +
    ConfigurationDiT passess / passDenoiseVAE decodeEnd to end
    Base checkpoint, before this optimization pass100206.2 s20,615 s863 s≈5 h 58 m
    Base checkpoint, current engine100120.2 s12,020 s563 s≈3 h 30 m
    Step-distilled (Turbo, 4-step) checkpoint4120.2 s481 s563 s17 m 30 s
    +
    +

    The total decomposes into two multipliers that stack: ~1.7× per pass from engine work (F16 attention K/V plus the VAE convolution path) and 25× fewer passes from step distillation. The official Wan2.2-TI2V-5B recipe is 50 steps × 2 classifier-free-guidance passes = 100 DiT passes; a step-distilled checkpoint is trained guidance-free and costs 4. That is not a flag — it is a different --model file. TensorSharp detects it from the DiT file name (turbo, distill, lightning, lightx2v, fastwan, -dmd, or an explicit …-4steps-…), prints step-distilled checkpoint detected -> 4 steps, guidance off on load, and drops guidance automatically; --diffusion-steps / --cfg still override. → Wan downloads

    +
    ⚡

    Once the checkpoint is distilled the VAE decode becomes the bottleneck — 563 s of the 1,050 s run, about 55%. Optimizing the DiT further past this point buys much less than trimming frames or frame area.

    + +

    Resolution and frame count vs. time

    +

    Same M5 Pro, same ggml_metal build, same Turbo checkpoint and the same source image. DiT self-attention costs O(tokens²) and the token count is latent_frames × (h/2) × (w/2), so frame area and frame count dominate everything else:

    +
    + + + + + + + +
    RequestDiT tokensDenoiseVAE decodeTotal
    736×544 × 81f (3.4 s, 480p class)8,21184 s159 s4 m 09 s
    736×544 × 121f (5 s, 480p class)12,121137 s237 s6 m 19 s
    1088×832 × 121f (5 s, 720p class)27,404481 s563 s17 m 30 s
    +
    +

    480p (≈0.4 MP) is a resolution Wan is trained at, so the 736×544 rows are in-distribution output rather than a degraded mode — that is the setting to reach for when a few minutes matters. Quality does fall off below ~0.3 MP, and the pipeline warns when width × height drops under 300,000 px.

    + +

    Where the per-pass 1.7× comes from

    +
    + + + + + + +
    ChangeMeasurementBefore → afterNumerics
    F16 attention K/V in the DiT flash pathone self-attention, seq 27,404 / 24 heads / head dim 128, M5 Pro4,993 → 2,467 ms (2.02×)cosine 0.999964 vs the diffusers reference with either dtype
    Wan VAE convolutions on MPSGraph (Metal)VAE decode, 736×544 × 81f159 → 80 s (1.99×)93.9 dB PSNR, max Δ 0.128 of 255
    +
    +

    A Wan VAE decode profile puts 44.4% of the graph in MUL_MAT and another 30.2% in IM2COL — 74.6% inside the convolution — which is why routing those convs through MPSGraph pays: per shape it reaches 6.2× (512→512 k3 at 320×240 t9), 8.6× (256→256 k3 at 640×480 t9), 10.4× (160→160 k3) and 13.9× (512→512 k1), because MPS reaches ~30 TFLOP/s where ggml's Metal GEMM gets ~4.9. Both knobs have an opt-out for A/B work: TS_WAN_DIT_KV_F16=0 and TS_WAN_VAE_MPS_CONV=0. Q8_0 attention K/V was measured slower than F16 (2,652 ms) and dequantizing the DiT from Q8_0 to F16 moved the matmuls by under 5%, so a higher-precision Wan quant buys essentially no speed.

    + +

    Wan backends and the sd.cpp comparison

    +

    Reference timings on an RTX 2000 Ada (16 GB) with Wan2.1-1.3B F16 at 832×480, 33 frames, 30 UniPC steps — the official 480p recipe:

    +
    + + + + + + + +
    Backends / stepTotal
    ggml_cuda12.0 s445 s
    ggml_vulkan17.2 s625 s
    cuda (direct driver API)19.3 s700 s
    +
    +

    ggml_cuda is the backend to pick for Wan on NVIDIA; cpu / ggml_cpu run correctly but a 480p multi-second video takes tens of minutes there, and mlx is not a supported Wan backend at all. Against stable-diffusion.cpp (master-769, identical GGUFs and settings, 33-frame 480p) sampling is near parity — 281 s vs 258 s — but sd.cpp's VAE decode materializes ≈8 GB of 3D im2col and oversubscribes a 16 GB card into WDDM paging: 51 s vs 1,762 s, which makes TensorSharp 6.0× faster end to end on that run.

    +

    Testing

    The build and test commands below require the .NET 10 SDK for your platform; verify the installation before running them.

    Unit tests (xUnit)

    diff --git a/website/benchmarks_zh-cn.html b/website/benchmarks_zh-cn.html index 82f969c7..8c71fc93 100644 --- a/website/benchmarks_zh-cn.html +++ b/website/benchmarks_zh-cn.html @@ -66,6 +66,61 @@

    Muse-Glimmer 30B，带与不带 DFlash

    规律很清楚：TensorSharp 的融合整模型图在约 2K 以下的 prefill 上领先 1.16-1.27×，两个引擎在 2K 到 16K 之间交叉，再往上 llama.cpp 保持 6-8% 的 prefill 优势；decode 在短上下文时打平，到 128K 降到 0.86×。启用 DFlash 投机解码后，TensorSharp 在短与中等提示上领先，从 16K 起落后——但这个差距主要来自运行期的成本调控器：一次误判的探测会让它暂停起草；在保持启用的那些重复里，TensorSharp 在 16K 达到 llama.cpp 的 94%、64K 达到 96%。DFlash 的绝对数字还高度依赖语料（501 token 那一点在两个引擎上都是 100% 接受率）。两个引擎都能在单卡上跑满 128K。两行长上下文有一个说明：它们是在换行符归一化修复落地之前测的，因此在这两个点上 TensorSharp prefill 的是同一文档的 CRLF 形式（多 1.2% 的 token）；吞吐是速率，影响很小，但那里两侧生成的续写并不严格可比。方法、逐次重复的离散度、显存与调控器分析见 Muse-Glimmer 架构卡片（docs/models/muse-glimmer_zh-cn.md）。

    +

    Wan 视频：两个彼此独立的提速杠杆

    +

    视频是唯一一种选错会浪费数小时而非数秒的工作负载，因此值得把两个杠杆分开来看。下表三行是同一个请求 —— Apple M5 Pro（20 核 GPU、48 GB 统一内存）、ggml_metal、Wan2.2-TI2V-5B Q8_0，图生视频，1088×832 × 121 帧 = 27,404 个 DiT token —— 各行之间只有引擎版本与 --model 指向哪个权重的区别。

    +
    + + + + + + + +
    配置DiT 前向次数秒 / 次去噪VAE 解码端到端
    基础权重，本轮优化之前100206.2 s20,615 s863 s≈5 h 58 m
    基础权重，当前引擎100120.2 s12,020 s563 s≈3 h 30 m
    步数蒸馏（Turbo，4 步）权重4120.2 s481 s563 s17 m 30 s
    +
    +

    总收益可以拆成两个相乘的因子：引擎侧带来的每次前向约 1.7×（F16 注意力 K/V 加上 VAE 卷积路径），以及步数蒸馏带来的前向次数少 25×。Wan2.2-TI2V-5B 的官方配方是 50 步 × 2 次无分类器引导（CFG）分支 = 100 次 DiT 前向；步数蒸馏权重在训练时就不使用引导，只需 4 次。这不是一个开关，而是换一个 --model 文件。TensorSharp 会从 DiT 文件名识别它（turbo、distill、lightning、lightx2v、fastwan、-dmd，或显式的 …-4steps-…），加载时打印 step-distilled checkpoint detected -> 4 steps, guidance off 并自动关闭引导；--diffusion-steps / --cfg 仍可覆盖。→ Wan 下载

    +
    ⚡

    一旦换成蒸馏权重，VAE 解码就成了瓶颈 —— 1,050 s 的总耗时里占 563 s，约 55%。再往下优化 DiT 的收益，远不如减少帧数或帧面积。

    + +

    分辨率与帧数对耗时的影响

    +

    同一台 M5 Pro、同一份 ggml_metal 构建、同一个 Turbo 权重与同一张源图。DiT 自注意力的开销是 O(token²)，而 token 数为 latent_frames × (h/2) × (w/2)，因此帧面积与帧数的影响远超其他因素：

    +
    + + + + + + + +
    请求DiT token 数去噪VAE 解码总计
    736×544 × 81 帧（3.4 s，480p 档）8,21184 s159 s4 m 09 s
    736×544 × 121 帧（5 s，480p 档）12,121137 s237 s6 m 19 s
    1088×832 × 121 帧（5 s，720p 档）27,404481 s563 s17 m 30 s
    +
    +

    480p（约 0.4 MP）是 Wan 训练过的分辨率，因此 736×544 这两行是分布内的正常输出，而不是降级模式 —— 当你只想等几分钟时，就应该选它。低于约 0.3 MP 后画质确实会下滑，宽 × 高小于 300,000 px 时管线会给出告警。

    + +

    每次前向的 1.7× 来自哪里

    +
    + + + + + + +
    改动测量对象前 → 后数值一致性
    DiT flash 路径中的 F16 注意力 K/V单次自注意力，seq 27,404 / 24 头 / head dim 128，M5 Pro4,993 → 2,467 ms（2.02×）对 diffusers 参考实现，两种 dtype 的余弦相似度均为 0.999964
    Wan VAE 卷积走 MPSGraph（Metal）VAE 解码，736×544 × 81 帧159 → 80 s（1.99×）93.9 dB PSNR，最大 Δ 为 255 中的 0.128
    +
    +

    Wan VAE 解码的性能剖析显示：MUL_MAT 占整图 44.4%，IM2COL 再占 30.2% —— 合计 74.6% 都在卷积里，这正是把这些卷积交给 MPSGraph 会有回报的原因：按形状分别达到 6.2×（512→512 k3，320×240 t9）、8.6×（256→256 k3，640×480 t9）、10.4×（160→160 k3）与 13.9×（512→512 k1），因为 MPS 能跑到约 30 TFLOP/s，而 ggml 的 Metal GEMM 只有约 4.9。两处都留了 A/B 用的开关：TS_WAN_DIT_KV_F16=0 与 TS_WAN_VAE_MPS_CONV=0。实测 Q8_0 的注意力 K/V 反而比 F16 慢（2,652 ms），把 DiT 从 Q8_0 反量化到 F16 也只让矩阵乘变化不到 5%，因此更高精度的 Wan 量化基本换不来速度。

    + +

    Wan 的后端选择与 sd.cpp 对比

    +

    RTX 2000 Ada（16 GB）上的参考耗时，Wan2.1-1.3B F16、832×480、33 帧、30 步 UniPC —— 即官方 480p 配方：

    +
    + + + + + + + +
    后端秒 / 步总计
    ggml_cuda12.0 s445 s
    ggml_vulkan17.2 s625 s
    cuda（直连驱动 API）19.3 s700 s
    +
    +

    在 NVIDIA 上跑 Wan 应当选 ggml_cuda；cpu / ggml_cpu 能正确出片，但一段 480p 的数秒视频要几十分钟，而 mlx 根本不是 Wan 支持的后端。与 stable-diffusion.cpp（master-769，相同 GGUF 与设置，33 帧 480p）相比，采样阶段接近持平 —— 281 s vs 258 s —— 但 sd.cpp 的 VAE 解码会实体化约 8 GB 的 3D im2col，把 16 GB 显卡压进 WDDM 换页：51 s vs 1,762 s，因此该次运行中 TensorSharp 端到端快 6.0×。

    +

    测试

    下方构建与测试命令需要按平台安装的 .NET 10 SDK；运行命令前请先验证安装。

    单元测试（xUnit）

    diff --git a/website/cli.html b/website/cli.html index ef60a6ad..0ee9d62a 100644 --- a/website/cli.html +++ b/website/cli.html @@ -128,7 +128,15 @@

    Video generation (Wan 2.1 / 2.2)

    dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \ --model <Wan2.2-TI2V-5B-Q8_0.gguf> --image first_frame.png \ --prompt "the camera pushes in as the waves rise" \ - --video-frames 81 --fps 24 --flow-shift 5.0 --output out.mp4 --backend ggml_cuda + --video-frames 81 --fps 24 --flow-shift 5.0 --output out.mp4 --backend ggml_cuda + +# THE FAST LANE: point --model at a step-distilled checkpoint. Nothing else in the +# command changes; the file name alone switches the run to 4 guidance-free passes. +dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \ + --model <Wan2_2-TI2V-5B-Turbo-Q8_0.gguf> --image first_frame.png \ + --prompt "the camera pushes in as the waves rise" \ + --video-frames 121 --fps 24 --output out.mp4 --backend ggml_metal +
    ⚡

    Step distillation is the single biggest lever, and it is not a flag. A base Wan2.2-TI2V-5B follows the official 50-step × 2-CFG recipe = 100 DiT passes; a distilled Turbo / Lightning / FastWan checkpoint is trained guidance-free and costs 4. TensorSharp detects it from the DiT file name (turbo, distill, lightning, lightx2v, fastwan, -dmd, or an explicit …-4steps-… / …8step… for 1–16) and prints step-distilled checkpoint detected -> 4 steps, guidance off on load. On an M5 Pro at 1088×832 × 121 frames the identical request takes ≈3 h 30 m on the base checkpoint and 17 m 30 s on Turbo. --diffusion-steps / --cfg override the detected values. → Where to download one · the measurements

    DiffusionGemma & inspection

    # DiffusionGemma text-diffusion generation
@@ -208,6 +216,9 @@ 
    Runtime
    
       --list-gpusList the Vulkan devices ggml-vulkan can see (index + adapter name) and exit.
       --helpPrint the full parameter reference (description, default, range, and an example per option) and exit; also shown when the CLI is started with no arguments.
       --kv-cache-dtype <type>KV cache precision: f32, f16, q8_0, or q4_0 (default: auto — the backend/model pick; overrides the KV_CACHE_DTYPE env var). The block-quantized tiers require the native GGML flash-attention path; q4_0 (~1/7 the f32 footprint) targets very long 128K–256K contexts.
+      --n-cpu-moe <N> / -ncmoe <N>Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on the CPU; attention, norms, the router and the shared expert stay on the accelerator. Pass all for every layer. Default: 0 — except DeepSeek V4 on the GPU backends, which auto-offloads the fewest layers that fit the visible VRAM. Env: TS_N_CPU_MOE.
+      --cpu-moe / -cmoeShorthand for --n-cpu-moe all (env TS_CPU_MOE).
+      --cpu-moe-threads <N>Worker threads for the host-side expert matmul (default: one less than the parallelism this process can actually use). Do not exceed the cgroup CPU quota — ggml's pool spins at its barriers, so oversubscription collapses throughput. Env: TS_CPU_MOE_THREADS.
       --interactive / -iStart the interactive REPL (see below).
       --system <text> / --system-file <path>Seed the session's system prompt from text or a file. Only the interactive REPL and DiffusionGemma diffusion mode use it — plain one-shot text mode ignores the system prompt.
       --thinkEnable thinking / reasoning mode (chain-of-thought).
@@ -234,6 +245,7 @@ 
    Sampling
    
       --top-p <f>Nucleus sampling threshold (default 1.0 = disabled).
       --min-p <f>Minimum probability filtering (default 0 = disabled).
       --repeat-penalty <f>Repetition penalty (default 1.0 = none).
+      --penalty-last-n <N>How many of the most recent tokens the repeat / presence / frequency penalties consider (default 64; 0 disables history penalties, -1 uses the whole history).
       --presence-penalty <f> / --frequency-penalty <f>Presence / frequency penalties (default 0 = disabled).
       --seed <N>Random seed (default -1 = non-deterministic).
       --stop <string>Stop sequence (can be repeated).
@@ -251,7 +263,9 @@ 
    DiffusionGemma, benchmarks & logging
    
       --image <path> / --prompt <text> / --output <path>Qwen-Image-Edit: input image, edit instruction, and output PNG (default edited.png). Reuses --diffusion-steps / --diffusion-seed.
       --cfg <F>Qwen-Image-Edit true-CFG guidance scale (omit for auto: 2.5, or 1.0 when a Lightning LoRA is loaded; <= 1 disables the negative pass).
       --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path>Override the resolved Qwen-Image-Edit companion GGUFs (VAE / Qwen2.5-VL text encoder / mmproj).
-      --qwen-image-lora <path>Qwen-Image-Edit Lightning distillation LoRA (.safetensors), merged into the DiT at load time; auto-derives the denoise step count (e.g. 4 or 8) and switches CFG to 1.0 (no negative pass).
+      --qwen-image-lora <path>Qwen-Image-Edit Lightning distillation LoRA (.safetensors). It is applied as a runtime F32 side-path beside each targeted projection — the quantized base weights are left untouched, never merged — and auto-derives the denoise step count from the file name (…-4steps-… / …8step…, 1–16), switching CFG to 1.0 with a fixed timestep shift of 3. That takes the default 30 steps at CFG 2.5 (60 DiT forwards) down to 4–8. Requires the whole-model or fused per-block CUDA forward: on a path without the side-path the model throws rather than emitting noise. Env: TS_QWEN_IMAGE_LORA.
+      --width <px> / --height <px>Fixed Qwen-Image-Edit output size (default 0 = auto: the source size, clamped to what VRAM allows).
+      --offload-cpuStream the Qwen-Image-Edit DiT weights from RAM instead of holding them resident in VRAM: slower per step, but native ~1 MP edits fit on small cards. Default: auto — engages only when the target resolution does not fit beside the resident weights.
       --benchmarkRun a synthetic prefill/decode throughput benchmark.
       --bench-prefill / --bench-decode / --bench-runs <N>Synthetic prefill length, decode length, and run count.
       --bench-kvcache / --bench-kv-turns <N>Multi-turn KV-cache reuse benchmark (with-cache vs forced-reset).
@@ -261,6 +275,25 @@ 
    DiffusionGemma, benchmarks & logging
    
     
   
   
+  
    Video generation (Wan 2.1 / 2.2)
    
+  
    These apply when --model is a wan DiT GGUF. --prompt carries the text, --image supplies the first frame for image-to-video, and --output names the MP4. The denoise step count and guidance come from the model's own recipe unless you override them with --diffusion-steps / --cfg.
    
+  
    
+  
+    
+    
+      
+      
+      
+      
+      
+      
+      
+      
+      
+    
+  
    OptionDescription
    --video-frames <N>Output frame count, snapped to 4k+1 (the VAE's temporal grid); 1 renders a single still. Default: 33, or 49 for Wan2.2-TI2V. Frame count drives both the attention cost and the VAE decode, so it is the second-biggest lever after the checkpoint.
    --fps <N>Playback rate of the saved MP4 (default 16, or 24 for Wan2.2-TI2V — the models' training rates). This changes playback, not the amount of work.
    --flow-shift <F>FlowMatch timestep shift (default 0 = the model's official recipe: 5.0 for Wan 2.2, 12.0 for A14B T2V; Wan 2.1 uses 8.0 for the 1.3B model's video runs, else 3.0 at ≤ 480p and 5.0 above).
    --sampler <name>unipc (default — the official Wan sampler; multistep predictor-corrector, better quality at the same step count) or euler.
    --negative-prompt <text>Negative prompt for classifier-free guidance (default: the official Wan negative prompt). Has no effect on a step-distilled checkpoint, which runs guidance-free.
    --cfg-cache-stride <N>Guidance cache: run the unconditional CFG pass on one step in N and reuse the cached guidance direction in between (the first three steps and the last always recompute it). At 50 steps, 2 runs 77 of the 100 passes (1.30× faster) and 3 runs 70 (1.43×). Default 0 = off. It is an approximation — leave it off when matching a reference sample matters, and note it does nothing on a distilled checkpoint (there is no unconditional pass to cache).
    --diffusion-steps <N> / --cfg <F>Override the recipe or the auto-detected distilled values. On base checkpoints 30 steps instead of 50 is visibly close and 1.7× cheaper.
    --wan-vae <path>Video VAE (wan_2.1_vae.safetensors, or Wan2.2_VAE.safetensors for TI2V-5B). Default: same-directory scan next to the DiT, VAE/ subfolders included. Env: TS_WAN_VAE. Which VAE is required is decided by the DiT itself, not by you.
    --wan-te <path>UMT5-XXL text-encoder GGUF. Default: same-directory scan. Env: TS_WAN_TE. Wan 2.2 A14B also auto-resolves its second high/low-noise expert GGUF by name (TS_WAN_DIT2 overrides; there is no CLI flag for it).
    
+  
    
+  
    Wan runs on ggml_cuda, ggml_vulkan, ggml_metal, ggml_cpu, cuda and cpu — it is the one family that rejects --backend mlx outright — and it takes no --tp. → Wan backend timings and cost model
    
   
    See the full reference, including --test, --test-templates, and chunked-prefill correctness checks, on the API Reference page.
    
 
   
    Interactive REPL commands
    
diff --git a/website/cli_zh-cn.html b/website/cli_zh-cn.html
index c4920c1a..3195a221 100644
--- a/website/cli_zh-cn.html
+++ b/website/cli_zh-cn.html
@@ -128,7 +128,15 @@ 
    视频生成（Wan 2.1 / 2.2）
    
 dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
     --model <Wan2.2-TI2V-5B-Q8_0.gguf> --image first_frame.png \
     --prompt "the camera pushes in as the waves rise" \
-    --video-frames 81 --fps 24 --flow-shift 5.0 --output out.mp4 --backend ggml_cuda
    + --video-frames 81 --fps 24 --flow-shift 5.0 --output out.mp4 --backend ggml_cuda + +# 快车道：把 --model 指向一个步数蒸馏权重。命令的其他部分完全不变； +# 仅凭文件名，这次运行就会切换为 4 次无引导前向。 +dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \ + --model <Wan2_2-TI2V-5B-Turbo-Q8_0.gguf> --image first_frame.png \ + --prompt "the camera pushes in as the waves rise" \ + --video-frames 121 --fps 24 --output out.mp4 --backend ggml_metal +
    ⚡

    步数蒸馏是最大的一个提速杠杆，而且它不是一个开关。基础的 Wan2.2-TI2V-5B 走官方配方 50 步 × 2 次 CFG = 100 次 DiT 前向；蒸馏后的 Turbo / Lightning / FastWan 权重在训练时就不使用引导，只需 4 次。TensorSharp 会从 DiT 文件名识别它（turbo、distill、lightning、lightx2v、fastwan、-dmd，或显式的 …-4steps-… / …8step…，取值 1–16），并在加载时打印 step-distilled checkpoint detected -> 4 steps, guidance off。在 M5 Pro 上、1088×832 × 121 帧的同一个请求，基础权重约需 3 h 30 m，Turbo 只需 17 m 30 s。--diffusion-steps / --cfg 可覆盖识别到的取值。→ 去哪里下载 · 实测数据

    DiffusionGemma 与检查

    # DiffusionGemma 文本扩散生成
@@ -208,6 +216,9 @@ 
    运行时
    
       --list-gpus列出 ggml-vulkan 可见的 Vulkan 设备（索引 + 显卡名称）后退出。
       --help打印完整参数参考（逐项列出说明、默认值、取值范围与示例）后退出；不带任何参数启动 CLI 时也会显示。
       --kv-cache-dtype <type>KV 缓存精度：f32、f16、q8_0 或 q4_0（默认：自动，由后端/模型选择；覆盖 KV_CACHE_DTYPE 环境变量）。块量化档位需要原生 GGML flash-attention 路径；q4_0（约为 f32 占用的 1/7）面向 128K–256K 的超长上下文。
+      --n-cpu-moe <N> / -ncmoe <N>把前 N 层的路由 MoE 专家权重留在系统内存中并在 CPU 上做乘法；注意力、归一化、路由器与共享专家仍留在加速器上。传 all 表示所有层。默认 0 —— 但 GPU 后端上的 DeepSeek V4 例外，它会自动卸载“刚好能放进可见显存”的最少层数。环境变量：TS_N_CPU_MOE。
+      --cpu-moe / -cmoe--n-cpu-moe all 的简写（环境变量 TS_CPU_MOE）。
+      --cpu-moe-threads <N>主机侧专家矩阵乘的工作线程数（默认为本进程实际可用并行度减一）。不要超过 cgroup 的 CPU 配额 —— ggml 的线程池在栅栏处自旋，超订会让吞吐直接崩塌。环境变量：TS_CPU_MOE_THREADS。
       --interactive / -i启动交互式 REPL（见下文）。
       --system <text> / --system-file <path>从文本或文件设定会话的系统提示。仅交互式 REPL 与 DiffusionGemma 扩散模式使用它；普通单次文本模式会忽略系统提示。
       --think启用思考 / 推理模式（思维链）。
@@ -234,6 +245,7 @@ 
    采样
    
       --top-p <f>核采样阈值（默认 1.0 = 禁用）。
       --min-p <f>最小概率过滤（默认 0 = 禁用）。
       --repeat-penalty <f>重复惩罚（默认 1.0 = 无）。
+      --penalty-last-n <N>重复 / 存在 / 频率惩罚考虑最近多少个 token（默认 64；0 关闭历史惩罚，-1 使用全部历史）。
       --presence-penalty <f> / --frequency-penalty <f>存在 / 频率惩罚（默认 0 = 禁用）。
       --seed <N>随机种子（默认 -1 = 非确定性）。
       --stop <string>停止序列（可重复）。
@@ -251,7 +263,9 @@ 
    DiffusionGemma、基准测试与日志
    
       --image <path> / --prompt <text> / --output <path>Qwen-Image-Edit：输入图像、编辑指令与输出 PNG（默认 edited.png）。复用 --diffusion-steps / --diffusion-seed。
       --cfg <F>Qwen-Image-Edit true-CFG 引导尺度（省略则自动取值：2.5，加载 Lightning LoRA 时为 1.0；<= 1 关闭负向分支）。
       --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path>覆盖解析到的 Qwen-Image-Edit 伴随 GGUF（VAE / Qwen2.5-VL 文本编码器 / mmproj）。
-      --qwen-image-lora <path>Qwen-Image-Edit Lightning 蒸馏 LoRA（.safetensors），在加载时合并进 DiT；自动推导去噪步数（如 4 或 8），并将 CFG 切换为 1.0（无负向分支）。
+      --qwen-image-lora <path>Qwen-Image-Edit Lightning 蒸馏 LoRA（.safetensors）。它作为运行期 F32 旁路挂在每个目标投影旁边 —— 量化后的基础权重原封不动，不会被合并 —— 并从文件名自动推导去噪步数（…-4steps-… / …8step…，取值 1–16），同时把 CFG 切换为 1.0 并固定 timestep shift 为 3。这会把默认的 30 步 / CFG 2.5（60 次 DiT 前向）降到 4–8 次。它要求整模或融合分块的 CUDA 前向路径：在没有旁路的路径上模型会直接抛错，而不是输出噪声。环境变量：TS_QWEN_IMAGE_LORA。
+      --width <px> / --height <px>Qwen-Image-Edit 的固定输出尺寸（默认 0 = 自动：源图尺寸，并按显存上限收敛）。
+      --offload-cpu把 Qwen-Image-Edit 的 DiT 权重从内存流式读取，而不是常驻显存：每步更慢，但小显存卡也能做原生约 1 MP 的编辑。默认自动 —— 仅当目标分辨率放不下常驻权重时才启用。
       --benchmark运行合成的 prefill/decode 吞吐基准。
       --bench-prefill / --bench-decode / --bench-runs <N>合成的 prefill 长度、decode 长度与运行次数。
       --bench-kvcache / --bench-kv-turns <N>多轮 KV 缓存复用基准（带缓存 vs 强制重置）。
@@ -261,6 +275,25 @@ 
    DiffusionGemma、基准测试与日志
    
     
   
   
+  
    视频生成（Wan 2.1 / 2.2）
    
+  
    以下参数在 --model 为 wan DiT GGUF 时生效。--prompt 提供文本，--image 提供图生视频的首帧，--output 指定 MP4 文件名。除非用 --diffusion-steps / --cfg 覆盖，去噪步数与引导强度都来自模型自身的配方。
    
+  
    
+  
+    
+    
+      
+      
+      
+      
+      
+      
+      
+      
+      
+    
+  
    选项说明
    --video-frames <N>输出帧数，会对齐到 4k+1（VAE 的时间网格）；1 表示只出一张静态图。默认 33，Wan2.2-TI2V 为 49。帧数同时决定注意力开销与 VAE 解码耗时，是仅次于权重选择的第二大杠杆。
    --fps <N>保存的 MP4 的播放帧率（默认 16，Wan2.2-TI2V 为 24 —— 即模型的训练帧率）。它只改变播放速度，不改变计算量。
    --flow-shift <F>FlowMatch timestep shift（默认 0 = 模型官方配方：Wan 2.2 为 5.0，A14B T2V 为 12.0；Wan 2.1 中 1.3B 模型的视频运行为 8.0，其余 ≤ 480p 为 3.0、更高为 5.0）。
    --sampler <name>unipc（默认 —— Wan 官方采样器；多步预测-校正，在相同步数下质量更好）或 euler。
    --negative-prompt <text>无分类器引导使用的负向提示词（默认为 Wan 官方负向提示词）。对步数蒸馏权重无效，因为它本身就不使用引导。
    --cfg-cache-stride <N>引导缓存：每 N 步才跑一次无条件 CFG 分支，其余步复用缓存的引导方向（前三步与最后一步始终重算）。50 步时，2 只跑 100 次前向中的 77 次（快 1.30×），3 跑 70 次（1.43×）。默认 0 = 关闭。它是一种近似 —— 需要对齐参考样本时请保持关闭；另外它在蒸馏权重上没有意义（根本没有无条件分支可缓存）。
    --diffusion-steps <N> / --cfg <F>覆盖配方或自动识别到的蒸馏取值。基础权重上用 30 步替代 50 步，观感非常接近而开销少 1.7×。
    --wan-vae <path>视频 VAE（wan_2.1_vae.safetensors，TI2V-5B 用 Wan2.2_VAE.safetensors）。默认在 DiT 同目录扫描，包含 VAE/ 子目录。环境变量：TS_WAN_VAE。需要哪个 VAE 由 DiT 自己决定，而不是由你指定。
    --wan-te <path>UMT5-XXL 文本编码器 GGUF。默认在同目录扫描。环境变量：TS_WAN_TE。Wan 2.2 A14B 还会按文件名自动解析第二个 high/low noise 专家 GGUF（用 TS_WAN_DIT2 覆盖；它没有对应的命令行参数）。
    
+  
    
+  
    Wan 可运行于 ggml_cuda、ggml_vulkan、ggml_metal、ggml_cpu、cuda 与 cpu —— 它是唯一会直接拒绝 --backend mlx 的模型族 —— 并且不支持 --tp。→ Wan 后端耗时与成本模型
    
   
    完整参考（含 --test、--test-templates 与分块 prefill 正确性检查）见 API 参考 页。
    
 
   
    交互式 REPL 命令
    
diff --git a/website/features.html b/website/features.html
index 59a858ca..1234305d 100644
--- a/website/features.html
+++ b/website/features.html
@@ -24,7 +24,7 @@ 
    Highlights
    
     
    🖼️
    Multimodal
    Image, video, and audio inputs (Gemma 4); image input for several others.
    
     
    📄
    PDF documents
    Upload PDFs in the Web UI or pass --pdf on the CLI — text PDFs are inlined, scanned pages go to vision models.
    
     
    🎨
    Image editing
    Qwen-Image-Edit turns a prompt + input image into an edited image (MMDiT diffusion).
    
-    
    🎬
    Video generation
    Wan 2.1 / 2.2 turn a prompt — and optionally a first-frame image — into an H.264 MP4.
    
+    
    🎬
    Video generation
    Wan 2.1 / 2.2 turn a prompt — and optionally a first-frame image — into an H.264 MP4. A step-distilled checkpoint cuts a 5-second 720p clip from hours to minutes.
    
     
    💭
    Thinking mode
    Structured chain-of-thought, separated from the visible answer.
    
     
    🛠️
    Tool calling
    Multi-turn function calling across all three API styles.
    
     
    📦
    Native quantized compute
    Q4_K_M, Q8_0, MXFP4, IQ2_XXS and more run in matmul without dequantizing to FP32.
    
@@ -37,21 +37,21 @@ 
    Highlights
    
   
    Models & modalities
    
   
      
     
    • Multi-architecture support — DeepSeek V4 Flash, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit, Wan 2.1 / 2.2 video. → Supported models
      • 
-    
    • DeepSeek V4 Flash (284B MoE) — a compressed-sparse-attention, 1M-context architecture with three dedicated whole-model executors: a direct-CUDA engine (--backend cuda), the native ggml executor (ggml_cuda / ggml_vulkan), and a 100% pure-C# CPU executor (--backend cpu, no native dependencies). Weights layer-split automatically across every visible GPU, and the server hosts it with native per-sequence slots and continuous batching. → DeepSeek V4
      • 
-    
    • Multimodal inference — image, video, and audio inputs for Gemma 4; images for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer. → Multimodal
      • 
+    
    • DeepSeek V4 Flash (284B MoE) — a compressed-sparse-attention, 1M-context architecture with three dedicated whole-model executors: a direct-CUDA engine (--backend cuda), the native ggml executor (ggml_cuda / ggml_vulkan), and a 100% pure-C# CPU executor (--backend cpu, no native dependencies). Weights layer-split automatically across every visible GPU, and the server hosts it with native per-sequence slots and continuous batching. → DeepSeek V4
      • 
+    
    • Multimodal inference — image, video, and audio inputs for Gemma 4; images for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer. → Multimodal
      • 
     
    • PDF document input — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models. Available as a Web UI upload and via the CLI's one-shot --pdf flag; cap the pages read with TS_PDF_MAX_PAGES (default: all). → Web UI
      • 
     
    • Mixture of Experts (MoE) — Gemma 4 MoE (e.g. 26B-A4B), GPT OSS MoE (gpt-oss-20b), Qwen 3.5/3.6 MoE (35B-A3B), and Nemotron-H MoE FFN layers, with a fused batched GPU MoE dispatch.
      • 
     
    • Hybrid SSM-Transformer — Nemotron-H mixes Mamba2 SSM layers, attention layers, and MoE FFN in one model.
      • 
     
    • Hybrid Attention-Recurrent — Qwen 3.5/3.6-family mix full-attention layers with GatedDeltaNet recurrent layers.
      • 
-    
    • Video generation — Wan 2.1 (text → video) and Wan 2.2 TI2V-5B / A14B (text → video and image → video, where the uploaded image becomes the first frame) render an H.264 MP4 from the CLI, /v1/videos/generations, or the Web UI. → Video generation
      • 
+    
    • Video generation — Wan 2.1 (text → video) and Wan 2.2 TI2V-5B / A14B (text → video and image → video, where the uploaded image becomes the first frame) render an H.264 MP4 from the CLI, /v1/videos/generations, or the Web UI. Point --model at a step-distilled checkpoint (Turbo / Lightning / FastWan) and the pipeline auto-detects it, dropping the denoise loop from 100 DiT passes to 4. → Video generation
      • 
     
    • Text-diffusion generation — DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive decode. → DiffusionGemma
      • 
-    
    • Image editing (Qwen-Image-Edit) — a prompt + input image produces an edited image via a 60-block MMDiT diffusion transformer, a Qwen-Image VAE, and a Qwen2.5-VL-7B text encoder (FlowMatch-Euler true-CFG, CUDA-graph-captured DiT forward); an optional Lightning distillation LoRA (--qwen-image-lora) cuts the denoise loop to a few steps at CFG 1.0. → Image editing
      • 
+    
    • Image editing (Qwen-Image-Edit) — a prompt + input image produces an edited image via a 60-block MMDiT diffusion transformer, a Qwen-Image VAE, and a Qwen2.5-VL-7B text encoder (FlowMatch-Euler true-CFG, CUDA-graph-captured DiT forward); an optional Lightning distillation LoRA (--qwen-image-lora) cuts the denoise loop to a few steps at CFG 1.0. → Image editing
      • 
   
    
 
   
    Generation & control
    
   
      
-    
    • Thinking / reasoning mode — structured chain-of-thought with <think> / <|channel> tags (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H). → Thinking mode
      • 
-    
    • Tool calling / function calling — architecture-agnostic output parsing turns raw model output into structured tool_calls, whether the model emits JSON (Qwen 3, Nemotron-H), XML inside a <tool_call> block (Qwen 3.5/3.6), Harmony commentary (GPT OSS), or DSML markup (DeepSeek V4). → Tool calling
      • 
+    
    • Thinking / reasoning mode — structured chain-of-thought with <think> / <|channel> tags (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H). → Thinking mode
      • 
+    
    • Tool calling / function calling — architecture-agnostic output parsing turns raw model output into structured tool_calls, whether the model emits JSON (Qwen 3, Nemotron-H), XML inside a <tool_call> block (Qwen 3.5/3.6), Harmony commentary (GPT OSS), or DSML markup (DeepSeek V4). → Tool calling
      • 
     
    • Configurable sampling — temperature, top-k, top-p, min-p, repetition / presence / frequency penalties, seed, and stop sequences. → Sampling
      • 
     
    • Structured outputs — OpenAI response_format with text, json_object, and validated json_schema. → Structured outputs
      • 
     
    • Chat templates — auto-loaded from GGUF metadata (Jinja2), with hardcoded fallbacks per architecture.
      • 
@@ -68,6 +68,7 @@ 
      Performance & scale
      
     
    • MTP / NextN speculative decoding — multi-token-prediction draft heads accelerate solo decode; lossless because the request's own sampler drives both draft and verify. → Speculative decoding
      • 
     
    • DSpark block speculative decoding — DeepSeek V4's drafter proposes a whole block of tokens per step (a Markov head conditions each block position on the one before it, a confidence head gates how far to draft) and the trunk verifies the block in one batched forward. Loaded as a separate GGUF with --draft-model; measured 1.3–1.4× decode on 4×A40, up to 2.0× on multi-turn chat, with greedy output byte-identical to the baseline. → DSpark
      • 
     
    • Whole-model fused decode graphs — Gemma 4, Qwen 3.5/3.6, and GPT OSS submit an entire decode token as one GGML graph instead of one dispatch per layer, so the GPU never waits on the host between layers. GPT OSS decode: 24 → 154 tok/s on an A40, and flat in context length. → Performance optimizations
      • 
+    
    • Diffusion fast paths (video & image) — a step-distilled Wan checkpoint (Turbo, distill, Lightning, lightx2v, FastWan, -dmd, or an explicit …-4steps-… in the file name) is detected at load and runs 4 guidance-free DiT passes instead of the base recipe's 100 — no flag, just a different --model. Measured on an M5 Pro (ggml_metal, Wan2.2-TI2V-5B Q8_0, 1088×832×121f image-to-video): ≈3 h 30 m on the base checkpoint vs 17 m 30 s on the Turbo one. Independently of that, F16 attention K/V is 2.02× on a 27,404-token self-attention, the Metal MPSGraph VAE convolutions take a 736×544×81f decode from 159 s to 80 s, and --cfg-cache-stride 2 / 3 buys 1.30× / 1.43× on base checkpoints. For images, Qwen-Image-Edit's Lightning LoRA replaces the base 30 steps at CFG 2.5 with 4–8 steps at CFG 1.0, and the CUDA-graph-captured whole-DiT forward took an 8-step denoise from ~153 s to ~63 s. → Video generation
      • 
     
    • Native quantized compute — quantized weights are used directly in matmul without expanding to FP32, saving memory and bandwidth.
      • 
     
    • Optimized pure C# CPU backend — managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, and fused activations.
      • 
     
    • KV cache codecs — pluggable IKvBlockCodec with a built-in TurboQuant (Q2 / Q4 / Q8) compressed codec for paged blocks.
      • 
diff --git a/website/features_zh-cn.html b/website/features_zh-cn.html
index 3bb5581b..94e3548c 100644
--- a/website/features_zh-cn.html
+++ b/website/features_zh-cn.html
@@ -24,7 +24,7 @@ 
      亮点
      
     
      🖼️
      多模态
      图像、视频与音频输入（Gemma 4）；多个其他模型支持图像输入。
      
     
      📄
      PDF 文档
      在 Web UI 上传 PDF，或在 CLI 传 --pdf —— 文本型 PDF 直接内联，扫描页则交给视觉模型。
      
     
      🎨
      图像编辑
      Qwen-Image-Edit 将提示词 + 输入图像转为编辑后的图像（MMDiT 扩散）。
      
-    
      🎬
      视频生成
      Wan 2.1 / 2.2 把提示词——以及可选的首帧图像——变成 H.264 MP4。
      
+    
      🎬
      视频生成
      Wan 2.1 / 2.2 把提示词——以及可选的首帧图像——变成 H.264 MP4。换用步数蒸馏权重，5 秒 720p 视频可从数小时缩短到几十分钟。
      
     
      💭
      思考模式
      结构化的思维链，与可见答案分离。
      
     
      🛠️
      工具调用
      跨三种 API 风格的多轮函数调用。
      
     
      📦
      原生量化计算
      Q4_K_M、Q8_0、MXFP4、IQ2_XXS 等在 matmul 中直接运算，无需反量化到 FP32。
      
@@ -37,21 +37,21 @@ 
      亮点
      
   
      模型与模态
      
   
        
     
      • 多架构支持 —— DeepSeek V4 Flash、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit、Wan 2.1 / 2.2 视频。→ 支持的模型
        • 
-    
      • DeepSeek V4 Flash（284B MoE） —— 一套压缩稀疏注意力、1M 上下文的架构，配有三套专属的整模型执行器：Direct CUDA 引擎（--backend cuda）、原生 ggml 执行器（ggml_cuda / ggml_vulkan），以及 100% 纯 C# 的 CPU 执行器（--backend cpu，零原生依赖）。权重自动按层切分到所有可见 GPU，服务端以原生 per-sequence slot 与连续批处理托管它。→ DeepSeek V4
        • 
-    
      • 多模态推理 —— Gemma 4 支持图像、视频与音频输入；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 支持图像。→ 多模态
        • 
+    
      • DeepSeek V4 Flash（284B MoE） —— 一套压缩稀疏注意力、1M 上下文的架构，配有三套专属的整模型执行器：Direct CUDA 引擎（--backend cuda）、原生 ggml 执行器（ggml_cuda / ggml_vulkan），以及 100% 纯 C# 的 CPU 执行器（--backend cpu，零原生依赖）。权重自动按层切分到所有可见 GPU，服务端以原生 per-sequence slot 与连续批处理托管它。→ DeepSeek V4
        • 
+    
      • 多模态推理 —— Gemma 4 支持图像、视频与音频输入；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 支持图像。→ 多模态
        • 
     
      • PDF 文档输入 —— 原生数字（born-digital）PDF 会完整提取文本层并内联进提示词；扫描版 PDF 则回退为页面图像，交给具备视觉能力的模型。可通过 Web UI 上传使用，也可用 CLI 一次性生成的 --pdf 参数；用 TS_PDF_MAX_PAGES 限制读取的页数（默认：全部）。→ Web UI
        • 
     
      • 专家混合（MoE） —— Gemma 4 MoE（如 26B-A4B）、GPT OSS MoE（gpt-oss-20b）、Qwen 3.5/3.6 MoE（35B-A3B）以及 Nemotron-H MoE FFN 层，配以融合的批量 GPU MoE 调度。
        • 
     
      • 混合 SSM-Transformer —— Nemotron-H 在一个模型中混合 Mamba2 SSM 层、注意力层与 MoE FFN。
        • 
     
      • 混合注意力-循环 —— Qwen 3.5/3.6-family 将全注意力层与 GatedDeltaNet 循环层混合。
        • 
-    
      • 视频生成 —— Wan 2.1（文本 → 视频）与 Wan 2.2 TI2V-5B / A14B（文本 → 视频，以及图像 → 视频，上传的图像作为首帧）可从 CLI、/v1/videos/generations 或 Web UI 输出 H.264 MP4。→ 视频生成
        • 
+    
      • 视频生成 —— Wan 2.1（文本 → 视频）与 Wan 2.2 TI2V-5B / A14B（文本 → 视频，以及图像 → 视频，上传的图像作为首帧）可从 CLI、/v1/videos/generations 或 Web UI 输出 H.264 MP4。把 --model 指向步数蒸馏权重（Turbo / Lightning / FastWan），管线会自动识别，把去噪循环从 100 次 DiT 前向降到 4 次。→ 视频生成
        • 
     
      • 文本扩散生成 —— DiffusionGemma 使用迭代式 EntropyBound 去噪采样器，而非自回归解码。→ DiffusionGemma
        • 
-    
      • 图像编辑（Qwen-Image-Edit） —— 提示词 + 输入图像，经 60 块 MMDiT 扩散 Transformer、Qwen-Image VAE 与 Qwen2.5-VL-7B 文本编码器生成编辑后的图像（FlowMatch-Euler true-CFG、CUDA 图捕获的 DiT 前向）；可选的 Lightning 蒸馏 LoRA（--qwen-image-lora）能以 CFG 1.0 将去噪循环缩减到寥寥几步。→ 图像编辑
        • 
+    
      • 图像编辑（Qwen-Image-Edit） —— 提示词 + 输入图像，经 60 块 MMDiT 扩散 Transformer、Qwen-Image VAE 与 Qwen2.5-VL-7B 文本编码器生成编辑后的图像（FlowMatch-Euler true-CFG、CUDA 图捕获的 DiT 前向）；可选的 Lightning 蒸馏 LoRA（--qwen-image-lora）能以 CFG 1.0 将去噪循环缩减到寥寥几步。→ 图像编辑
        • 
   
      
 
   
      生成与控制
      
   
        
-    
      • 思考 / 推理模式 —— 带 <think> / <|channel> 标签的结构化思维链（Qwen 3、Qwen 3.5/3.6、Gemma 4、GPT OSS、Nemotron-H）。→ 思考模式
        • 
-    
      • 工具调用 / 函数调用 —— 与架构无关的输出解析，无论模型输出的是 JSON（Qwen 3、Nemotron-H）、<tool_call> 块内的 XML（Qwen 3.5/3.6）、Harmony commentary（GPT OSS）还是 DSML 标记（DeepSeek V4），都会转成结构化 tool_calls。→ 工具调用
        • 
+    
      • 思考 / 推理模式 —— 带 <think> / <|channel> 标签的结构化思维链（Qwen 3、Qwen 3.5/3.6、Gemma 4、GPT OSS、Nemotron-H）。→ 思考模式
        • 
+    
      • 工具调用 / 函数调用 —— 与架构无关的输出解析，无论模型输出的是 JSON（Qwen 3、Nemotron-H）、<tool_call> 块内的 XML（Qwen 3.5/3.6）、Harmony commentary（GPT OSS）还是 DSML 标记（DeepSeek V4），都会转成结构化 tool_calls。→ 工具调用
        • 
     
      • 可配置采样 —— 温度、top-k、top-p、min-p、重复 / 存在 / 频率惩罚、随机种子与停止序列。→ 采样
        • 
     
      • 结构化输出 —— OpenAI response_format 支持 text、json_object 与经过校验的 json_schema。→ 结构化输出
        • 
     
      • 聊天模板 —— 从 GGUF 元数据自动加载（Jinja2），并按架构提供硬编码回退。
        • 
@@ -68,6 +68,7 @@ 
        性能与扩展
        
     
      • MTP / NextN 推测解码 —— 多 token 预测草稿头加速单序列解码；因请求自身的采样器同时驱动草稿与验证，故无损。→ 推测解码
        • 
     
      • DSpark 块级投机解码 —— DeepSeek V4 的草稿器每步提议一整块 token（Markov 头让块内每个位置以前一个 token 为条件，置信度头决定起草多远），主干用一次批量前向验证整块。以独立 GGUF 通过 --draft-model 加载；4×A40 实测 decode 提速 1.3–1.4×，多轮对话最高 2.0×，贪心输出与基线逐字节一致。→ DSpark
        • 
     
      • 整模型融合 decode 计算图 —— Gemma 4、Qwen 3.5/3.6 与 GPT OSS 把一个 decode token 作为一次 GGML 计算图提交，而不是每层一次，GPU 因此不会在层间空等主机。GPT OSS decode 在 A40 上从 24 → 154 tok/s，且随上下文长度基本持平。→ 性能优化
        • 
+    
      • 扩散模型快车道（视频与图像） —— 步数蒸馏的 Wan 权重（文件名中带 Turbo、distill、Lightning、lightx2v、FastWan、-dmd，或显式的 …-4steps-…）会在加载时被自动识别，只跑 4 次无引导 DiT 前向，而不是基础配方的 100 次 —— 不需要任何参数，只是换一个 --model。实测（M5 Pro、ggml_metal、Wan2.2-TI2V-5B Q8_0、1088×832×121 帧图生视频）：基础权重约 3 小时 30 分，Turbo 权重 17 分 30 秒。此外，F16 注意力 K/V 在 27,404 token 的自注意力上快 2.02×，Metal MPSGraph VAE 卷积把 736×544×81 帧的解码从 159 秒降到 80 秒，--cfg-cache-stride 2 / 3 在基础权重上再带来 1.30× / 1.43×。图像侧，Qwen-Image-Edit 的 Lightning LoRA 把基础的 30 步 CFG 2.5 换成 4–8 步 CFG 1.0，CUDA 图捕获的整 DiT 前向把 8 步去噪从约 153 秒降到约 63 秒。→ 视频生成
        • 
     
      • 原生量化计算 —— 量化权重直接用于 matmul，无需展开为 FP32，节省内存与带宽。
        • 
     
      • 优化的纯 C# CPU 后端 —— 托管 GEMM 快路径，外加 RMSNorm、RoPE、softmax 与融合激活的融合 SIMD 内核。
        • 
     
      • KV 缓存编解码器 —— 可插拔的 IKvBlockCodec，内置面向分页块的 TurboQuant（Q2 / Q4 / Q8）压缩编解码器。
        • 
diff --git a/website/getting-started.html b/website/getting-started.html
index 311d505f..52a9b8e4 100644
--- a/website/getting-started.html
+++ b/website/getting-started.html
@@ -37,7 +37,7 @@ 
        Install the .NET 10 SDK
        
   
        Other prerequisites
        
   
          
     
        • git, curl, and network access — used to clone TensorSharp and download a model. A full native build also clones ggml into ExternalProjects/ggml/; set TENSORSHARP_GGML_NO_UPDATE=1 to skip later network updates.
          • 
-    
        • A GGUF model file — e.g. from Hugging Face. See Model downloads.
          • 
+    
        • A GGUF model file — e.g. from Hugging Face. See Model downloads.
          • 
   
        
   
        Per-platform toolchains (only for GPU acceleration)
        
   
        
@@ -72,9 +72,15 @@ 
        Full native / GPU path
        
   
        📦
        Current release status: v3.0.5.0 has no uploaded CLI/server archives, so build from source today. The repository contains a Release Binaries workflow, but do not invent an archive URL—use one only when it is visibly listed on the Releases page.
        
 
   
        3 · Download a model
        
-  
        For the quick start, download the recommended benchmark-verified gemma-4-E4B-it-Q8_0.gguf file (7.48 GiB) from the public ggml-org/gemma-4-E4B-it-GGUF repository. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf lives in the same repository, and the Models page lists other options.
        
+  
        For the quick start, download the recommended benchmark-verified gemma-4-E4B-it-Q8_0.gguf file (7.48 GiB) from the public ggml-org/gemma-4-E4B-it-GGUF repository. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf lives in the same repository, and the Models page lists other options.
        
   
        curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
        
   
        🧩
        For multimodal models, download the matching projector (mmproj) and pass its exact path with --mmproj. The server does not auto-detect a projector, and CLI auto-detection recognizes only a few legacy filenames; explicit paths are reliable.
        
+  
        ⚡
        Generating images or video? Pick the fast checkpoint on the way in. Both families ship in a base and a step-distilled form, and the difference is not subtle:
        
+  
          
+    
        • Wan video — a base Wan2.2-TI2V-5B follows the official 50-step × 2-CFG recipe = 100 DiT passes; a Turbo / Lightning / FastWan checkpoint is trained guidance-free and costs 4. TensorSharp detects it from the DiT file name and needs no extra flag. On an M5 Pro a 1088×832 × 121-frame image-to-video takes ≈3 h 30 m on the base checkpoint and 17 m 30 s on Turbo — the same command, a different --model path.
          • 
+    
        • Qwen-Image-Edit — adding the Lightning distillation LoRA with --qwen-image-lora drops the default 30 steps at CFG 2.5 (60 DiT forwards) to a 4- or 8-step run.
          • 
+  
        
+  
        → Model downloads for the exact repos and files · the measurements behind these numbers
        
 
   
        Gemma 4 E4B backend and platform notes
        
   
        The first-run commands below are for Linux + NVIDIA. On Windows + NVIDIA, set $env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA='ON' in PowerShell, then run the same commands without the Bash assignment prefix. On Apple Silicon, omit the CUDA environment assignment and use ggml_metal. For Vulkan on Windows/Linux, set TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON (PowerShell: $env:TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN='ON') and use ggml_vulkan. Without a GPU, use ggml_cpu (native CPU kernels). Text needs no projector. For image, video, or audio, download mmproj-gemma-4-E4B-it-Q8_0.gguf from the same repository and add --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf.
        
@@ -103,6 +109,7 @@ 
        Where to go next
        
     
        ⌨️
        CLI reference
        All flags, the REPL, and batch workflows.
        
     
        🔌
        HTTP API
        Call the server from curl, Python, or SDKs.
        
     
        🧠
        Models
        What's supported and where to download.
        
+    
        📊
        Benchmarks
        Measured speed per model, and the levers that move it.
        
   
        
 
   
        
diff --git a/website/getting-started_zh-cn.html b/website/getting-started_zh-cn.html
index 02b67b82..008fd132 100644
--- a/website/getting-started_zh-cn.html
+++ b/website/getting-started_zh-cn.html
@@ -37,7 +37,7 @@ 
        安装 .NET 10 SDK
        
   
        其他前置要求
        
   
          
     
        • git、curl 与网络访问 —— 用于克隆 TensorSharp 与下载模型。完整原生构建还会把 ggml 克隆到 ExternalProjects/ggml/；之后可设置 TENSORSHARP_GGML_NO_UPDATE=1 跳过网络更新。
          • 
-    
        • 一个 GGUF 模型文件 —— 例如来自 Hugging Face。见模型下载。
          • 
+    
        • 一个 GGUF 模型文件 —— 例如来自 Hugging Face。见模型下载。
          • 
   
        
   
        各平台工具链（仅 GPU 加速需要）
        
   
        
@@ -72,9 +72,15 @@ 
        完整原生 / GPU 路径
        
   
        📦
        当前发行状态：v3.0.5.0 没有上传 CLI/server 应用归档，因此现在请从源码构建。仓库中虽有 Release Binaries 工作流，但不要自行拼接归档 URL；只有在 Releases 页面明确列出后才使用。
        
 
   
        3 · 下载模型
        
-  
        快速上手请从公开的 ggml-org/gemma-4-E4B-it-GGUF 仓库下载推荐的、经基准验证的 gemma-4-E4B-it-Q8_0.gguf（7.48 GiB）。同一仓库还提供更省内存的 gemma-4-E4B-it-Q4_K_M.gguf，更多选项见模型页。
        
+  
        快速上手请从公开的 ggml-org/gemma-4-E4B-it-GGUF 仓库下载推荐的、经基准验证的 gemma-4-E4B-it-Q8_0.gguf（7.48 GiB）。同一仓库还提供更省内存的 gemma-4-E4B-it-Q4_K_M.gguf，更多选项见模型页。
        
   
        curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
        
   
        🧩
        多模态模型还需下载匹配的投影器（mmproj），并用 --mmproj 传入精确路径。服务端不会自动检测投影器，CLI 也只识别少数旧文件名；显式路径最可靠。
        
+  
        ⚡
        要生成图像或视频？在下载阶段就选好快的那一版权重。这两个模型族都同时提供基础版与步数蒸馏版，差距非常大：
        
+  
          
+    
        • Wan 视频 —— 基础的 Wan2.2-TI2V-5B 走官方配方 50 步 × 2 次 CFG = 100 次 DiT 前向；Turbo / Lightning / FastWan 权重在训练时就不使用引导，只需 4 次。TensorSharp 会从 DiT 文件名自动识别，不需要额外参数。在 M5 Pro 上，一段 1088×832 × 121 帧的图生视频，基础权重约需 3 h 30 m，Turbo 只需 17 m 30 s —— 命令完全相同，只是 --model 指向了不同的文件。
          • 
+    
        • Qwen-Image-Edit —— 用 --qwen-image-lora 加载 Lightning 蒸馏 LoRA，可把默认的 30 步 / CFG 2.5（60 次 DiT 前向）降到 4 步或 8 步。
          • 
+  
        
+  
        → 模型下载（具体仓库与文件名）· 这些数字背后的实测
        
 
   
        Gemma 4 E4B 后端与平台说明
        
   
        下一节的首次运行命令面向 Linux + NVIDIA。Windows + NVIDIA 请先在 PowerShell 中设置 $env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA='ON'，再运行不带 Bash 赋值前缀的相同命令。Apple Silicon 请省略 CUDA 环境变量并改用 ggml_metal。Windows/Linux 上使用 Vulkan 时，请设置 TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON（PowerShell：$env:TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN='ON'）并使用 ggml_vulkan。没有 GPU 时使用 ggml_cpu（原生 CPU 内核）。纯文本不需要投影器；图像、视频或音频输入需从同一仓库下载 mmproj-gemma-4-E4B-it-Q8_0.gguf，并添加 --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf。
        
@@ -103,6 +109,7 @@ 
        下一步去哪
        
     
        ⌨️
        CLI 参考
        所有参数、REPL 与批处理工作流。
        
     
        🔌
        HTTP API
        用 curl、Python 或 SDK 调用服务器。
        
     
        🧠
        模型
        支持哪些模型，以及在哪里下载。
        
+    
        📊
        基准测试
        各模型的实测速度，以及影响速度的杠杆。
        
   
        
 
   
        
diff --git a/website/glossary.html b/website/glossary.html
index 0493e701..fa74c2ab 100644
--- a/website/glossary.html
+++ b/website/glossary.html
@@ -34,9 +34,9 @@ 
        Glossary
        
       TTFT"Time to first token" — how long before the answer starts streaming. KV-cache reuse reduces it on follow-up turns.
       BackendThe hardware path that runs the math: CPU, NVIDIA GPU (CUDA), any Vulkan GPU (AMD/Intel/NVIDIA), or Apple Silicon GPU (Metal/MLX). → Backends
       MoE (Mixture of Experts)A model that routes each token to a few specialized sub-networks ("experts") instead of the whole network, giving large capacity at lower compute per token.
-      MultimodalAble to take more than text — images, audio, or video — as input. → Multimodal
-      Thinking / reasoningThe model produces hidden step-by-step reasoning before its final answer; TensorSharp separates the two. → Thinking mode
-      Tool / function callingThe model can ask your application to run a function (e.g. "get_weather") and use the result. → Tool calling
+      MultimodalAble to take more than text — images, audio, or video — as input. → Multimodal
+      Thinking / reasoningThe model produces hidden step-by-step reasoning before its final answer; TensorSharp separates the two. → Thinking mode
+      Tool / function callingThe model can ask your application to run a function (e.g. "get_weather") and use the result. → Tool calling
       Continuous batchingServing many users at once by interleaving their requests on a single hosted model, instead of one at a time. → Deep dive
       Speculative decodingA small "draft" head guesses several tokens; the main model verifies them in one pass — faster output, identical result. → MTP
       Block drafter / DSparkA speculative drafter that proposes a whole block of tokens at once rather than one at a time, with a confidence head deciding how far to draft. DeepSeek V4 ships one as a separate GGUF. → DSpark
@@ -55,7 +55,7 @@ 
        Do I need a GPU?
        
   
        No. TensorSharp runs on plain CPU (--backend cpu or the faster ggml_cpu). A GPU — NVIDIA (CUDA), any Vulkan-capable AMD/Intel/NVIDIA GPU, or Apple Silicon (Metal/MLX) — makes generation substantially faster, especially for larger models. See Backends.
        
 
   
        Which model should I start with?
        
-  
        Start with Gemma 4 E4B Q8_0 (recommended public file: 7.48 GiB) — the verified quick-start model on TensorSharp's native-GGML fast path, with thinking, tools, and optional-projector image/video/audio input. A lower-memory small starter is Qwen3-4B Q4_K_M (2.33 GiB). See the E4B fast lane or model downloads.
        
+  
        Start with Gemma 4 E4B Q8_0 (recommended public file: 7.48 GiB) — the verified quick-start model on TensorSharp's native-GGML fast path, with thinking, tools, and optional-projector image/video/audio input. A lower-memory small starter is Qwen3-4B Q4_K_M (2.33 GiB). See the E4B fast lane or model downloads.
        
 
   
        Is my data private?
        
   
        Yes — inference happens entirely on your machine. Prompts, documents, and images are not sent to any external service. That is the main reason organizations choose local inference. See Business value.
        
diff --git a/website/glossary_zh-cn.html b/website/glossary_zh-cn.html
index 2b603723..78b9c712 100644
--- a/website/glossary_zh-cn.html
+++ b/website/glossary_zh-cn.html
@@ -34,9 +34,9 @@ 
        术语表
        
       TTFT“首 token 时间” —— 答案开始流式输出前需多久。KV 缓存复用会在后续轮次降低它。
       后端（Backend）运行数学运算的硬件路径：CPU、NVIDIA GPU（CUDA）、任何 Vulkan GPU（AMD/Intel/NVIDIA）或 Apple Silicon GPU（Metal/MLX）。→ 后端
       MoE（专家混合）一种把每个 token 路由到少数几个专门子网络（“专家”）而非整张网络的模型，以更低的每 token 计算获得更大容量。
-      多模态（Multimodal）能接收文本以外的输入 —— 图像、音频或视频。→ 多模态
-      思考 / 推理模型在最终答案前产生隐藏的逐步推理；TensorSharp 将两者分离。→ 思考模式
-      工具 / 函数调用模型可请求你的应用运行一个函数（如 “get_weather”）并使用结果。→ 工具调用
+      多模态（Multimodal）能接收文本以外的输入 —— 图像、音频或视频。→ 多模态
+      思考 / 推理模型在最终答案前产生隐藏的逐步推理；TensorSharp 将两者分离。→ 思考模式
+      工具 / 函数调用模型可请求你的应用运行一个函数（如 “get_weather”）并使用结果。→ 工具调用
       连续批处理在单个托管模型上交错运行多个用户的请求，从而同时服务多人，而非一次一个。→ 深入了解
       推测解码一个小“草稿”头猜测若干 token；主模型一次性验证它们 —— 输出更快、结果一致。→ MTP
       块级草稿器 / DSpark一次提议一整块 token 而非逐个的投机草稿器，由置信度头决定起草多远。DeepSeek V4 以独立 GGUF 的形式提供了一个。→ DSpark
@@ -55,7 +55,7 @@ 
        我需要 GPU 吗？
        
   
        不需要。TensorSharp 可在普通 CPU 上运行（--backend cpu 或更快的 ggml_cpu）。GPU —— NVIDIA（CUDA）、任何支持 Vulkan 的 AMD/Intel/NVIDIA GPU，或 Apple Silicon（Metal/MLX）—— 会让生成显著更快，尤其是更大的模型。见后端。
        
 
   
        我该从哪个模型开始？
        
-  
        建议从 Gemma 4 E4B Q8_0（推荐公开文件为 7.48 GiB）开始 —— 它是 TensorSharp 原生 GGML 快路径上经过验证的快速上手模型，支持思维链、工具调用，以及需要可选投影器的图像/视频/音频输入。内存占用更低的小型入门模型是 Qwen3-4B Q4_K_M（2.33 GiB）。见 E4B 快路径或模型下载。
        
+  
        建议从 Gemma 4 E4B Q8_0（推荐公开文件为 7.48 GiB）开始 —— 它是 TensorSharp 原生 GGML 快路径上经过验证的快速上手模型，支持思维链、工具调用，以及需要可选投影器的图像/视频/音频输入。内存占用更低的小型入门模型是 Qwen3-4B Q4_K_M（2.33 GiB）。见 E4B 快路径或模型下载。
        
 
   
        我的数据私密吗？
        
   
        是的 —— 推理完全在你的机器上进行。提示词、文档与图像不会发送到任何外部服务。这正是各组织选择本地推理的主要原因。见商业价值。
        
diff --git a/website/index.html b/website/index.html
index ff72d1ce..b3042bf8 100644
--- a/website/index.html
+++ b/website/index.html
@@ -68,7 +68,7 @@ 
        Explore the wiki
        
     
        🔌
        HTTP API
        Call it from curl, Python, or any Ollama/OpenAI client.
        
     
        🧩
        C# Library
        Embed the engine directly in your .NET application.
        
     
        📚
        API Reference
        Searchable tables of flags, env vars, endpoints, and types.
        
-    
        🧠
        Models
        Supported architectures, downloads, multimodal, and reasoning.
        
+    
        🧠
        Models
        All twelve architectures, downloads, multimodal, reasoning — and which checkpoint to pick when you want it fast.
        
     
        🔗
        Multi-GPU & Multi-Node
        Tensor parallelism across GPUs — CUDA and GGML — and across machines.
        
     
        📖
        Glossary & FAQ
        New to LLMs? Plain-language definitions and common questions.
        
   
        
@@ -110,8 +110,8 @@ 
        Why TensorSharp?
        
     
        💸
        No per-token bill
        Run as much as your hardware allows — predictable cost, no metered API.
        
     
        🔁
        Drop-in compatible
        Speaks the Ollama and OpenAI wire formats, so existing tools and SDKs just work.
        
     
        🖥️
        Runs anywhere
        NVIDIA (CUDA), AMD / Intel / NVIDIA (Vulkan), Apple Silicon (Metal/MLX), or pure CPU — with automatic fallbacks.
        
-    
        🧠
        Modern model support
        DeepSeek V4 Flash, Gemma, Qwen, GPT-OSS, Nemotron-H, Mistral, Muse-Glimmer, plus vision, audio, PDF documents, reasoning & tools.
        
-    
        🎬
        Images and video out, too
        Qwen-Image-Edit rewrites an image from a prompt, and Wan 2.1 / 2.2 generate H.264 video from a prompt or a first-frame image — on the same engine, same GGUF plumbing.
        
+    
        🧠
        Modern model support
        Ten text families — DeepSeek V4 Flash, Gemma 3 / 4, DiffusionGemma, Qwen 3, Qwen 3.5 / 3.6, GPT-OSS, Nemotron-H, Mistral 3, Muse-Glimmer — plus vision, audio, PDF documents, reasoning & tools. Two more generate images and video.
        
+    
        🎬
        Images and video out, too
        Qwen-Image-Edit rewrites an image from a prompt, and Wan 2.1 / 2.2 generate H.264 video from a prompt or a first-frame image — on the same engine, same GGUF plumbing. Pick a step-distilled Wan checkpoint and the same 5-second 720p clip takes 17 min instead of 3½ hours.
        
     
        ⚙️
        Built in .NET
        A native C# engine you can embed in your apps, not just a black-box binary.
        
     
        🔗
        Scales past one GPU
        Tensor parallelism splits a model across several GPUs with --tp N — on the direct CUDA backend and on GGML CUDA / Vulkan — and across machines over a peer-to-peer TCP mesh when one host is not enough.
        
     
        🏁
        Benchmarked vs llama.cpp
        On identical GGUF files and the same GPU it trades wins with the C++ engine — in the current CUDA + Vulkan comparison run (reproducible via benchmarks/engine_comparison): Gemma 4 E4B and the 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner, multi-turn prompts prefill faster on every model (up to 1.49×), and Gemma 4 12B decodes 1.21× faster on Vulkan.
        
diff --git a/website/index_zh-cn.html b/website/index_zh-cn.html
index 6003aff3..58c6dffd 100644
--- a/website/index_zh-cn.html
+++ b/website/index_zh-cn.html
@@ -68,7 +68,7 @@ 
        浏览维基
        
     
        🔌
        HTTP API
        用 curl、Python 或任意 Ollama / OpenAI 客户端调用它。
        
     
        🧩
        C# 库
        将引擎直接嵌入你的 .NET 应用。
        
     
        📚
        API 参考
        可搜索的参数、环境变量、端点与类型表。
        
-    
        🧠
        模型
        支持的架构、下载、多模态与推理能力。
        
+    
        🧠
        模型
        全部十二种架构、下载方式、多模态与推理能力 —— 以及想要更快时该选哪个权重。
        
     
        🔗
        多 GPU 与多节点
        跨 GPU（CUDA 与 GGML）、跨机器的张量并行。
        
     
        📖
        术语表与 FAQ
        初识大模型？这里有通俗的定义和常见问题。
        
   
        
@@ -110,8 +110,8 @@ 
        为什么选择 TensorSharp？
        
     
        💸
        没有按 token 账单
        硬件允许多少就跑多少 —— 成本可预测，没有计量 API。
        
     
        🔁
        即插即用兼容
        支持 Ollama 与 OpenAI 协议，现有工具与 SDK 直接可用。
        
     
        🖥️
        随处可运行
        NVIDIA (CUDA)、AMD / Intel / NVIDIA (Vulkan)、Apple Silicon (Metal/MLX) 或纯 CPU —— 均带自动回退。
        
-    
        🧠
        现代模型支持
        DeepSeek V4 Flash、Gemma、Qwen、GPT-OSS、Nemotron-H、Mistral、Muse-Glimmer，外加视觉、音频、PDF 文档、推理与工具调用。
        
-    
        🎬
        还能产出图像与视频
        Qwen-Image-Edit 按提示词改写图像，Wan 2.1 / 2.2 从提示词或首帧图像生成 H.264 视频——同一个引擎、同一套 GGUF 管线。
        
+    
        🧠
        现代模型支持
        十个文本家族 —— DeepSeek V4 Flash、Gemma 3 / 4、DiffusionGemma、Qwen 3、Qwen 3.5 / 3.6、GPT-OSS、Nemotron-H、Mistral 3、Muse-Glimmer —— 外加视觉、音频、PDF 文档、推理与工具调用。另有两个家族负责生成图像与视频。
        
+    
        🎬
        还能产出图像与视频
        Qwen-Image-Edit 按提示词改写图像，Wan 2.1 / 2.2 从提示词或首帧图像生成 H.264 视频——同一个引擎、同一套 GGUF 管线。改用步数蒸馏（step-distilled）的 Wan 权重，同一段 5 秒 720p 视频只需 17 分钟，而不是 3 个半小时。
        
     
        ⚙️
        用 .NET 构建
        原生 C# 引擎，可嵌入你的应用，而不只是一个黑盒二进制。
        
     
        🔗
        不止一张 GPU
        张量并行用 --tp N 把模型切分到多张 GPU 上（Direct CUDA 与 GGML CUDA / Vulkan 均可）；一台机器不够时，还可通过点对点 TCP 网格跨机器扩展。
        
     
        🏁
        对比 llama.cpp 的基准
        在相同 GGUF 文件与相同 GPU 上与 C++ 引擎互有胜负 —— 在当前的 CUDA + Vulkan 对比运行中（可通过 benchmarks/engine_comparison 复现）：Gemma 4 E4B 与 2-bit 量化的 Qwen 3.6 35B-A3B MoE 在 CUDA 上 prefill 快 1.28×、首 token 早 1.27×，多轮提示的 prefill 在每个模型上都更快（最高 1.49×），Gemma 4 12B 在 Vulkan 上 decode 快 1.21×。
        
diff --git a/website/models-downloads.html b/website/models-downloads.html
new file mode 100644
index 00000000..bd6e5619
--- /dev/null
+++ b/website/models-downloads.html
@@ -0,0 +1,65 @@
+
+
+
+  
+  
+  Model Downloads — TensorSharp Wiki
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        Home › Supported Models › Model Downloads (GGUF)
        
+  
        Model Downloads (GGUF)
        
+  
        One table per artifact: the model itself plus the companion files — projector, VAE, text encoder, second expert, speculative drafter — that each family needs alongside it.
        
+
+  
        Model downloads (GGUF)
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+    
+  
        ArchitectureModelDownloadCompanion files (mmproj / MTP draft)
        DeepSeek V4DeepSeek-V4-Flash-0731 (284B MoE)unsloth/DeepSeek-V4-Flash-0731-GGUFOne subdirectory per quant (UD-Q8_K_XL/, UD-IQ4_XS/, UD-IQ1_S/, …), each a multi-shard set — point --model at the -00001-of- shard. Text only; weights layer-split across every visible GPU
        DeepSeek V4DSpark speculative drafter (optional)bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUFDSpark-drafter-Q2K-Q8-0731.gguf (7.0 GB) for the 0731 release, loaded with --draft-model for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see MODEL_DOWNLOADS.md. Drafters for other architectures are a different design and are not supported
        Gemma 4gemma-4-E4B-itggml-org/gemma-4-E4B-it-GGUFRecommended public artifact for the verified E4B Q8_0 native-GGML tier: gemma-4-E4B-it-Q8_0.gguf. Modalities: mmproj-gemma-4-E4B-it-Q8_0.gguf (same repo). MTP draft: AtomicChat/gemma-4-E4B-it-assistant-GGUF
        Gemma 4gemma-4-12B-it (QAT)unsloth/gemma-4-12B-it-qat-GGUFmmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-12B-it.gguf (both in the same repo)
        Gemma 4gemma-4-31B-itggml-org/gemma-4-31B-it-GGUFmmproj: mmproj-gemma-4-31B-it-Q8_0.gguf (same repo)
        Gemma 4gemma-4-26B-A4B-it (MoE)ggml-org/gemma-4-26B-A4B-it-GGUFmmproj: mmproj-gemma-4-26B-A4B-it-Q8_0.gguf (same repo)
        Gemma 4gemma-4-26B-A4B-it (MoE, QAT)unsloth/gemma-4-26B-A4B-it-qat-GGUFmmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-26B-A4B-it.gguf (same repo). Alt draft: AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — pair drafts only with the matching target size
        Gemma 3gemma-3-4b-it (QAT, official)google/gemma-3-4b-it-qat-q4_0-ggufmmproj: mmproj-model-f16-4B.gguf (same repo). ⚠️ Gated: downloads require a Hugging Face login + accepting Google's Gemma license
        Gemma 3gemma-3-4b-it (non-gated alternative)ggml-org/gemma-3-4b-it-GGUFmmproj: mmproj-model-f16.gguf (same repo)
        Qwen 3Qwen3-4BQwen/Qwen3-4B-GGUF— (text only)
        Qwen 3.5 / 3.6Qwen3.5-9Bunsloth/Qwen3.5-9B-GGUFmmproj: mmproj-F16.gguf (same repo)
        Qwen 3.5 / 3.6Qwen3.5-35B-A3B (MoE)ggml-org/Qwen3.5-35B-A3B-GGUFmmproj: mmproj-Qwen3.5-35B-A3B-Q8_0.gguf (same repo)
        Qwen 3.5 / 3.6Qwen3.6-35B-A3B (MoE, NextN MTP)unsloth/Qwen3.6-35B-A3B-MTP-GGUFmmproj: mmproj-F16.gguf (same repo). These GGUFs retain the embedded NextN block for --mtp-spec; the base-repo GGUFs (unsloth/Qwen3.6-35B-A3B-GGUF) strip it and silently fall back to standard decode
        GPT OSSgpt-oss-20b (MoE)ggml-org/gpt-oss-20b-GGUF— (text only)
        Nemotron-HNemotron-H-8B-Reasoning-128Kbartowski/nvidia_Nemotron-H-8B-…— (text only)
        Nemotron-HNemotron-H-47B-Reasoning-128Kbartowski/nvidia_Nemotron-H-47B-…— (text only)
        Nemotron-HNemotron 3 Nano Omni 30B-A3B (image)unsloth/NVIDIA-Nemotron-3-Nano-Omni-…mmproj: mmproj-BF16.gguf (same repo) — required for image input. Audio is not functional: it needs a Parakeet audio mmproj that this distribution does not ship
        Mistral 3Mistral-Small-3.1-24B-Instructbartowski/mistralai_Mistral-Small-3.1-24B-…Pixtral mmproj: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf (same repo)
        DiffusionGemmadiffusiongemma-26B-A4B-itunsloth/diffusiongemma-26B-A4B-it-GGUF—
        Muse-GlimmerMuse-Glimmer-30Bunsloth/Muse-Glimmer-30B-GGUFmmproj: mmproj-Muse-Glimmer-30B-Q8_0.gguf (same repo) for image input
        Muse-GlimmerDFlash speculative drafter (optional)unsloth/Muse-Glimmer-30B-GGUFdflash-*.gguf — a 5-layer block drafter loaded with --draft-model. Verification is greedy against the target, so the emitted stream is the plain-greedy stream
        Qwen-Image-EditMMDiT DiT (the --model GGUF)unsloth/Qwen-Image-Edit-2511-GGUFe.g. qwen-image-edit-2511-Q4_K_M.gguf
        Qwen-Image-EditQwen-Image VAE (required)QuantStack/Qwen-Image-Edit-GGUFVAE/Qwen_Image-VAE.safetensors — place next to the DiT or pass --qwen-image-vae
        Qwen-Image-EditQwen2.5-VL-7B text encoder (required)unsloth/Qwen2.5-VL-7B-Instruct-GGUFOptional vision mmproj: mmproj-BF16.gguf (same repo) for image-grounded edits
        Qwen-Image-EditLightning LoRA (optional, 4/8-step)lightx2v/Qwen-Image-Edit-2511-LightningQwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors via --qwen-image-lora
        Wan videoStep-distilled Wan DiT — the fast lane (recommended)hum-ma/Wan2.2-TI2V-5B-Turbo-GGUFWan2_2-TI2V-5B-Turbo-Q8_0.gguf (5.40 GB) — note the underscore in Wan2_2; smaller tiers down to Q2_K (1.86 GB). Detected from the file name, so no flag changes: 4 DiT passes instead of 100, guidance off. For A14B image-to-video use jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF and take both the high_noise/ and low_noise/ files. Neither repo ships a VAE or a text encoder — take those from the two rows below
        Wan videoBase Wan DiT (full step recipe)QuantStack/Wan2.2-TI2V-5B-GGUFWan2.2-TI2V-5B-Q8_0.gguf (5.40 GB), with VAE/Wan2.2_VAE.safetensors bundled in the same repo. Also Wan2.2-I2V-A14B and Wan2.2-T2V-A14B (download both the HighNoise and LowNoise experts; both repos bundle the Wan 2.1 VAE), Wan2.1-T2V-14B and Wan2.1-T2V-1.3B
        Wan videoUMT5-XXL text encoder (required)city96/umt5-xxl-encoder-ggufumt5-xxl-encoder-Q8_0.gguf (6.04 GB; Q5_K_M 4.15 GB and Q4_K_M 3.66 GB for tighter memory) — place next to the DiT or pass --wan-te
        Wan videovideo VAE (required)wan_2.1_vae.safetensorsThe DiT decides which one it needs, not you: Wan 2.1 and A14B use wan_2.1_vae.safetensors, TI2V-5B uses Wan2.2_VAE.safetensors. Both QuantStack A14B repos and the TI2V-5B repo already bundle theirs under VAE/, so this separate download is only needed for the Wan 2.1 checkpoints and the distilled repos. Place it next to the DiT (a VAE/ subfolder works) or pass --wan-vae
        
+  
        
+  
        🧩
        Multimodal models need a projector (mmproj) file. Download the matching file and pass its exact path with --mmproj. The server never auto-detects it, and CLI auto-detection recognizes only a small set of legacy filenames.
        
+  
        
+    ← Supported Models
+    Next: Text & LLM Models →
+  
        
+
        
+
+
diff --git a/website/models-downloads_zh-cn.html b/website/models-downloads_zh-cn.html
new file mode 100644
index 00000000..f1aea691
--- /dev/null
+++ b/website/models-downloads_zh-cn.html
@@ -0,0 +1,65 @@
+
+
+
+  
+  
+  模型下载 — TensorSharp 维基
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        首页 › 支持的模型 › 模型下载（GGUF）
        
+  
        模型下载（GGUF）
        
+  
        每个构件一行：模型本体，以及各家族需要与之搭配的配套文件——投影器、VAE、文本编码器、第二专家、推测解码草稿模型。
        
+
+  
        模型下载（GGUF）
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+      
+    
+  
        架构模型下载
        DeepSeek V4DeepSeek-V4-Flash-0731（284B MoE）unsloth/DeepSeek-V4-Flash-0731-GGUF；每个量化档一个子目录（UD-Q8_K_XL/、UD-IQ4_XS/、UD-IQ1_S/ …），均为多分片——--model 指向 -00001-of- 那一片。仅文本；权重按层切分到所有可见 GPU
        DeepSeek V4DSpark 投机草稿器（可选）bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF 的 DSpark-drafter-Q2K-Q8-0731.gguf（7.0 GB，对应 0731 版本），用 --draft-model 加载可获得约 1.3–1.4× 的 decode 提速。另有两家发布的构建（5.6 GB / 10.9 GB）也可直接加载，见 MODEL_DOWNLOADS_zh-cn.md。其他架构的草稿器属于不同设计，暂不支持
        Gemma 4gemma-4-E4B-itggml-org/gemma-4-E4B-it-GGUF；已验证 E4B Q8_0 原生 GGML 规格推荐使用公开文件 gemma-4-E4B-it-Q8_0.gguf；多模态投影器为同仓库的 mmproj-gemma-4-E4B-it-Q8_0.gguf
        Gemma 4gemma-4-12B-it（QAT）unsloth/gemma-4-12B-it-qat-GGUF；同仓库 mmproj-BF16.gguf 与 mtp-gemma-4-12B-it.gguf
        Gemma 4gemma-4-31B-itggml-org/gemma-4-31B-it-GGUF
        Gemma 4gemma-4-26B-A4B-it (MoE)ggml-org/gemma-4-26B-A4B-it-GGUF
        Gemma 4gemma-4-26B-A4B-it（MoE，QAT）unsloth/gemma-4-26B-A4B-it-qat-GGUF；同仓库 mmproj-BF16.gguf 与 MTP draft；另有 AtomicChat assistant
        Gemma 3gemma-3-4b-it非 gated ggml-org 仓库；Q4_K_M 约 2.32 GiB，mmproj 为 mmproj-model-f16.gguf。Google QAT 仓库需要登录并接受许可证
        Gemma 3gemma-3-4b-it（官方 QAT）google/gemma-3-4b-it-qat-q4_0-gguf；mmproj-model-f16-4B.gguf；⚠️ 需要 Hugging Face 登录并接受 Google Gemma 许可证
        Qwen 3Qwen3-4BQwen/Qwen3-4B-GGUF；Qwen3-4B-Q4_K_M.gguf 约 2.33 GiB
        Qwen 3.5 / 3.6Qwen3.5-9Bunsloth/Qwen3.5-9B-GGUF
        Qwen 3.5 / 3.6Qwen3.5-35B-A3B (MoE)ggml-org/Qwen3.5-35B-A3B-GGUF
        Qwen 3.6Qwen3.6-35B-A3B NextN MTPunsloth/Qwen3.6-35B-A3B-MTP-GGUF；UD-Q4_K_M 约 21.11 GiB。基础仓库 GGUF 会剥离 NextN 块
        GPT OSSgpt-oss-20b (MoE)ggml-org/gpt-oss-20b-GGUF
        Nemotron-HNemotron-H-8B-Reasoning-128Kbartowski/nvidia_Nemotron-H-8B-…
        Nemotron-HNemotron-H-47B-Reasoning-128Kbartowski/nvidia_Nemotron-H-47B-…
        Nemotron-HNemotron 3 Nano Omni 30B-A3B（图像）unsloth/NVIDIA-Nemotron-3-Nano-Omni-…；图像输入需要同仓库 mmproj-BF16.gguf；未附真实音频推理所需的 Parakeet mmproj
        Mistral 3Mistral-Small-3.1-24B-Instructbartowski/mistralai_Mistral-Small-3.1-24B-…；Q4_K_M 约 13.35 GiB；同仓库 Pixtral mmproj
        DiffusionGemmadiffusiongemma-26B-A4B-itunsloth/diffusiongemma-26B-A4B-it-GGUF；Q4_K_M 约 15.65 GiB
        Muse-GlimmerMuse-Glimmer-30Bunsloth/Muse-Glimmer-30B-GGUF；图像输入需要同仓库的 mmproj：mmproj-Muse-Glimmer-30B-Q8_0.gguf
        Muse-GlimmerDFlash 投机草稿模型（可选）unsloth/Muse-Glimmer-30B-GGUF 中的 dflash-*.gguf —— 5 层块级草稿模型，用 --draft-model 加载。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致
        Qwen-Image-Edit2511 MMDiT（--model）unsloth/Qwen-Image-Edit-2511-GGUF；qwen-image-edit-2511-Q4_K_M.gguf 约 12.34 GiB
        Qwen-Image-EditQwen-Image VAE（必需）QuantStack/Qwen-Image-Edit-GGUF 中的 VAE/Qwen_Image-VAE.safetensors；通过 --qwen-image-vae 指定
        Qwen-Image-EditQwen2.5-VL-7B 文本编码器（必需）unsloth/Qwen2.5-VL-7B-Instruct-GGUF；可选视觉投影器为同仓库 mmproj-BF16.gguf
        Qwen-Image-EditLightning LoRA（可选，4/8 步）lightx2v/Qwen-Image-Edit-2511-Lightning；如 Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors
        Wan 视频步数蒸馏 Wan DiT —— 快车道（推荐）hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF 中的 Wan2_2-TI2V-5B-Turbo-Q8_0.gguf（5.40 GB）—— 注意 Wan2_2 用的是下划线；最小到 Q2_K（1.86 GB）。从文件名自动识别，无需改任何参数：DiT 前向由 100 次降到 4 次，并关闭引导。A14B 图生视频用 jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF，两个专家（high_noise/ 与 low_noise/）都要下载。这两个仓库都不含 VAE 和文本编码器——请从下面两行取
        Wan 视频基础 Wan DiT（完整步数配方）QuantStack/Wan2.2-TI2V-5B-GGUF 中的 Wan2.2-TI2V-5B-Q8_0.gguf（5.40 GB），同仓库已附带 VAE/Wan2.2_VAE.safetensors。另有 Wan2.2-I2V-A14B 与 Wan2.2-T2V-A14B（需同时下载 HighNoise 与 LowNoise 两个专家；两个仓库均附带 Wan 2.1 VAE）、Wan2.1-T2V-14B 与 Wan2.1-T2V-1.3B
        Wan 视频UMT5-XXL 文本编码器（必需）city96/umt5-xxl-encoder-gguf 中的 umt5-xxl-encoder-Q8_0.gguf（6.04 GB；内存紧张可用 Q5_K_M 4.15 GB 或 Q4_K_M 3.66 GB）—— 放在 DiT 旁边，或用 --wan-te 指定
        Wan 视频视频 VAE（必需）wan_2.1_vae.safetensors；需要哪一个由 DiT 自己决定，不用你选：Wan 2.1 与 A14B 用 wan_2.1_vae.safetensors，TI2V-5B 用 Wan2.2_VAE.safetensors。QuantStack 的两个 A14B 仓库与 TI2V-5B 仓库都已在 VAE/ 下附带，因此只有 Wan 2.1 检查点和蒸馏仓库才需要单独下载。放在 DiT 旁边（VAE/ 子目录亦可）或用 --wan-vae 指定
        
+  
        
+  
        🧩
        多模态模型需要投影器（mmproj）文件。请下载匹配文件并显式传入 --mmproj；服务端不会自动检测，CLI 也只识别少数旧文件名。
        
+  
        
+    ← 支持的模型
+    下一篇：文本与 LLM 模型 →
+  
        
+
        
+
+
diff --git a/website/models-image.html b/website/models-image.html
new file mode 100644
index 00000000..2fcc828c
--- /dev/null
+++ b/website/models-image.html
@@ -0,0 +1,61 @@
+
+
+
+  
+  
+  Image Generation & Editing — TensorSharp Wiki
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        Home › Supported Models › Image Generation & Editing
        
+  
        Image Generation & Editing
        
+  
        Qwen-Image-Edit takes a prompt plus an input image and returns an edited image. This page covers the download, the run recipe, and the two levers that decide how long an edit takes.
        
+  
        💡
        These commands follow the same conventions as the text-model recipes — the Hugging Face CLI, a source build from the repository root, and --backend swapped to match your hardware. See Download & run, per family.
        
+
+  
        Qwen-Image-Edit (image + prompt → edited image)
        
+  
        hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
+hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
+hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
+hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
+hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
+    --prompt "Make the sky a dramatic sunset." --output edited.png \
+    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
+    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
+    --qwen-image-mmproj models/mmproj-BF16.gguf \
+    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
+    --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
+    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
+    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
+    --qwen-image-mmproj models/mmproj-BF16.gguf \
+    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
+    --backend ggml_cuda
        
+
+  
        Image editing (Qwen-Image-Edit)
        
+  
        Qwen-Image-Edit takes a prompt + an input image and returns an edited image — a different output modality from the text LLMs above. The loaded qwen_image GGUF is only the MMDiT (multimodal diffusion transformer); TensorSharp resolves two companion GGUFs alongside it:
        
+  
          
+    
        • Qwen-Image VAE — image ↔ 16-channel latent (the original .safetensors works too).
          • 
+    
        • Qwen2.5-VL-7B text encoder — prompt → 3584-dim conditioning, with an optional mmproj vision tower for image-grounded edits.
          • 
+  
        
+  
        Place the companions next to the DiT GGUF, or point at them with TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ (CLI: --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj). The pipeline VAE-encodes the reference, builds the conditioning, runs a FlowMatch-Euler true-CFG denoise loop (reference-latent concatenation), then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured and flash attention is on by default; the target area auto-clamps to the device VRAM budget unless you pin a width/height.
        
+  
        The speed lever is the Lightning distillation LoRA (--qwen-image-lora / TS_QWEN_IMAGE_LORA, a .safetensors file). TensorSharp parses the trained step count out of its file name (…-4steps-… / …8step…, 1–16 accepted) and switches the sampling defaults to that step count at cfg 1.0 with a fixed timestep shift of 3 — one DiT forward per step, no negative pass. Without it the base recipe is 30 steps at cfg 2.5, so the default 60 DiT forwards drop to 4–8. On the project's CUDA image_edit scenario (Q2_K DiT + 4-step Lightning LoRA, 544×1184, identical inputs and seed) a warm edit completes in 40.44 s against stable-diffusion.cpp's 48.16 s.
        
+  
        The LoRA is applied as a runtime F32 side-path next to each targeted projection (y = W·x + b + (alpha/rank)·up·(down·x)) with the quantized base weights untouched — it is not merged into them, because the Lightning deltas are far below a low-bit quantization step and a merge would be pure requantization noise. That side-path exists only on the whole-model and fused per-block CUDA forwards, so a Lightning LoRA on a fallback path (TS_QWEN_DIT_FUSED_BLOCK=0, TS_QWEN_DIT_NATIVE=0, or a non-CUDA backend) fails loudly rather than emitting noise. Stacking it on a base DiT that is already few-step (a file name containing rapid, turbo, hyper, lightning, lcm, nitro or step) is detected and warned about — such a checkpoint needs no LoRA.
        
+  
        Three fused graphs carry the rest, all on by default: the CUDA-graph-captured whole-DiT forward cut the per-forward cost ~2.9× and an 8-step denoise from ~153 s to ~63 s (TS_QWEN_DIT_WHOLE_CAPTURE=0 to disable); the fused conditioning-encoder trunks took text conditioning from 11.2 s to 2.5 s (TS_QWEN_TE_FUSED=0); and the fused whole-VAE graph took a 928×688 encode from 19.5 s to 0.95 s and the decode from 22.8 s to 1.35 s (TS_QWEN_VAE_FUSED=0). A whole-step DiT cache (TS_QWEN_DIT_CACHE_MODE = easycache / fbc / both) can skip 40–55% of the remaining steps, but it is off by default: on edit workloads it measurably softens fine detail such as faces, so quality comes first and you opt in.
        
+  
        Because the text and vision encoders are freed before the denoise loop starts, a larger text-encoder quantization costs nothing at denoise time while driving the whole edit's fidelity — Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf (4.683 GB) is the recommended choice even on low-VRAM cards; the ~2-bit UD-IQ2_XXS build markedly softens faces.
        
+  
        Run it from the CLI (--image + --prompt), the Web UI image-edit flow (with live denoising previews), or in-process from C# via QwenImageModel.EditImage(). Full details are in the repository's docs/models/qwenimage.md card.
        
+  
        
+    ← Text & LLM Models
+    Next: Video Generation →
+  
        
+
        
+
+
diff --git a/website/models-image_zh-cn.html b/website/models-image_zh-cn.html
new file mode 100644
index 00000000..df1ada20
--- /dev/null
+++ b/website/models-image_zh-cn.html
@@ -0,0 +1,61 @@
+
+
+
+  
+  
+  图像生成与编辑 — TensorSharp 维基
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        首页 › 支持的模型 › 图像生成与编辑
        
+  
        图像生成与编辑
        
+  
        Qwen-Image-Edit 接受提示词加一张输入图像，返回编辑后的图像。本页包含下载、运行示例，以及决定单次编辑耗时的两个关键选项。
        
+  
        💡
        这些命令遵循与文本模型示例相同的约定：Hugging Face CLI、从仓库根目录源码构建，并把 --backend 换成与你硬件匹配的值。参见按家族下载与运行。
        
+
+  
        Qwen-Image-Edit（图像 + 提示词 → 编辑后的图像）
        
+  
        hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
+hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
+hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
+hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
+hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
+    --prompt "把天空改成壮丽的日落。" --output edited.png \
+    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
+    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
+    --qwen-image-mmproj models/mmproj-BF16.gguf \
+    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
+    --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
+    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
+    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
+    --qwen-image-mmproj models/mmproj-BF16.gguf \
+    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
+    --backend ggml_cuda
        
+
+  
        图像编辑（Qwen-Image-Edit）
        
+  
        Qwen-Image-Edit 接收提示词 + 一张输入图像并返回编辑后的图像——这是与上面文本 LLM 不同的输出模态。所加载的 qwen_image GGUF 仅是 MMDiT（多模态扩散 Transformer）；TensorSharp 会在其旁解析两个伴随 GGUF：
        
+  
          
+    
        • Qwen-Image VAE —— 图像 ↔ 16 通道潜变量（原始 .safetensors 也可用）。
          • 
+    
        • Qwen2.5-VL-7B 文本编码器 —— 提示词 → 3584 维条件，外加可选的 mmproj 视觉塔以实现图像接地编辑。
          • 
+  
        
+  
        将伴随文件放在 DiT GGUF 旁，或用 TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ 指定（CLI：--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj）。流水线对参考图做 VAE 编码、构建条件、运行带参考潜变量拼接的 FlowMatch-Euler true-CFG 去噪循环，再 VAE 解码回像素。整个 60 块 DiT 前向被 CUDA 图捕获，flash 注意力默认开启；除非固定宽/高，否则目标面积按设备 VRAM 预算自动钳制。
        
+  
        提速的关键是 Lightning 蒸馏 LoRA（--qwen-image-lora / TS_QWEN_IMAGE_LORA，一个 .safetensors 文件）。TensorSharp 从文件名解析出训练时的步数（…-4steps-… / …8step…，接受 1–16），并把采样默认值切换为该步数、cfg 1.0、时间步 shift 固定为 3——每步一次 DiT 前向，没有负向分支。不加 LoRA 时基础配方是 30 步、cfg 2.5，因此默认的 60 次 DiT 前向降到 4–8 次。在本项目的 CUDA image_edit 场景下（Q2_K DiT + 4 步 Lightning LoRA，544×1184，输入与种子完全一致），一次热态编辑耗时 40.44 s，而 stable-diffusion.cpp 为 48.16 s。
        
+  
        该 LoRA 以运行时 F32 旁路的方式作用在每个目标投影旁（y = W·x + b + (alpha/rank)·up·(down·x)），量化后的基础权重保持不变——它不会被合并进权重，因为 Lightning 的增量远小于低比特量化的一个台阶，合并的结果只是重量化噪声。这条旁路只存在于整模型前向与融合逐块 CUDA 前向上，因此在回退路径上（TS_QWEN_DIT_FUSED_BLOCK=0、TS_QWEN_DIT_NATIVE=0 或非 CUDA 后端）使用 Lightning LoRA 会直接报错，而不是输出噪声。若基础 DiT 本身已经是少步模型（文件名含 rapid、turbo、hyper、lightning、lcm、nitro 或 step），叠加 LoRA 会被检测并告警——这类检查点无需再加 LoRA。
        
+  
        其余提速来自三张默认开启的融合图：CUDA 图捕获的整 DiT 前向把单次前向成本降到约 1/2.9，8 步去噪从约 153 s 降到约 63 s（TS_QWEN_DIT_WHOLE_CAPTURE=0 可关闭）；融合的条件编码器主干把文本条件阶段从 11.2 s 降到 2.5 s（TS_QWEN_TE_FUSED=0）；融合的整 VAE 图把 928×688 的编码从 19.5 s 降到 0.95 s、解码从 22.8 s 降到 1.35 s（TS_QWEN_VAE_FUSED=0）。整步 DiT 缓存（TS_QWEN_DIT_CACHE_MODE = easycache / fbc / both）还能跳过其余步数的 40–55%，但它默认关闭：在编辑类负载上它会可测地弱化人脸等细节，因此以质量优先，需要时再自行开启。
        
+  
        由于文本与视觉编码器在去噪循环开始前就已释放，更大的文本编码器量化档在去噪阶段不增加任何成本，却决定整次编辑的保真度——即便显存有限也推荐 Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf（4.683 GB）；约 2 比特的 UD-IQ2_XXS 会明显弱化人脸。
        
+  
        可从 CLI（--image + --prompt）、Web UI 图像编辑流程（含实时去噪预览），或在进程内从 C# 通过 QwenImageModel.EditImage() 运行。完整细节见仓库的 docs/models/qwenimage_zh-cn.md 卡片。
        
+  
        
+    ← 文本与 LLM 模型
+    下一篇：视频生成 →
+  
        
+
        
+
+
diff --git a/website/models-text.html b/website/models-text.html
new file mode 100644
index 00000000..4cab1114
--- /dev/null
+++ b/website/models-text.html
@@ -0,0 +1,175 @@
+
+
+
+  
+  
+  Text & LLM Models — TensorSharp Wiki
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        Home › Supported Models › Text & LLM Models
        
+  
        Text & LLM Models
        
+  
        Copy-paste download and run recipes for every text architecture, followed by the capabilities they share: image, audio and PDF input, thinking mode, and tool calling.
        
+
+  
        Download & run, per family
        
+  
        Install and verify the .NET 10 SDK for your platform before running a build command. The first block is the verified Gemma 4 E4B quick start. All blocks use the Hugging Face CLI (pip install -U huggingface_hub); the per-family blocks after the first assume a full source build from the repository root. One-shot CLI text prompts are read from a file via --input (put your question in prompt.txt first — --prompt is reserved for Qwen-Image-Edit). Swap --backend for ggml_metal / ggml_vulkan / ggml_cpu to match your hardware (the CLI default is ggml_cpu).
        
+
+  
        Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)
        
+  
        Copying and running the commands takes about 30 seconds; the 7.48 GiB download and the first restore/build take longer and depend on your connection and machine. Repository benchmarks verify TensorSharp's E4B Q8_0 family and execution path; the linked ggml-org repository is the recommended public source. This block is for Linux + NVIDIA:
        
+  
        hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
+TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
+echo "Explain why local inference is useful." > prompt.txt
+
+# Text needs no mmproj.
+dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
+    --input prompt.txt --max-tokens 300 --backend ggml_cuda
+dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
+    --backend ggml_cuda
        
+  
        On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan. For image, video, or audio input, also run hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models and pass --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The server's browser UI is http://localhost:5000/index.html. See Getting Started for Windows PowerShell and full platform syntax.
        
+
+  
        DeepSeek V4 Flash (284B MoE, text, thinking, tools, DSpark)
        
+  
        A 284B mixture-of-experts model with a 128-token raw sliding window plus block-compressed attention (1M advertised context). It does not use the generic per-op forward: TensorSharp runs it through one of three dedicated whole-model executors — --backend cuda (direct CUDA, no ggml), --backend ggml_cuda / ggml_vulkan (native ggml), and --backend cpu (100% pure C#, no native dependencies). All of them layer-split the weights across every visible GPU, so a model far bigger than one card still runs; --tp N (or TS_DSV4_NGPU) caps how many GPUs it uses.
        
+  
        # ~160 GB of weights for the Q8 tier — pick a smaller quant directory if that is too much
+hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
+hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models
+
+# Plain decode — point --model at the FIRST shard
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 200
+
+# DSpark block speculative decoding (~1.3-1.4x decode; needs greedy sampling)
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
+    --input prompt.txt --max-tokens 200 --temperature 0
+
+# Served over HTTP, 4 GPUs, with speculation on
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf
        
+  
        DSpark drafts a whole block of tokens per step and the trunk verifies the block in one batched forward, so greedy output is unchanged. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p, or penalty turns it off); on the server every verify row is drawn with the request's own sampler, so it composes with any sampling settings. See DSpark speculative decoding.
        
+
+  
        Gemma 3 (text + image)
        
+  
        hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
+hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda
        
+
+  
        Qwen 3 (text, thinking, tools)
        
+  
        hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda
        
+
+  
        Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)
        
+  
        hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
+hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
+    --image photo.png --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda
        
+  
        For Qwen 3.6 NextN speculative decoding, download the trunk from the -MTP- repo (the base repo's GGUFs strip the NextN block and silently fall back to standard decode) and start the server with --mtp-spec — the MTP flags exist only on TensorSharp.Server:
        
+  
        hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec
        
+
+  
        GPT OSS (text, thinking always on, tools)
        
+  
        hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda
        
+
+  
        Nemotron-H (text; image on the Omni distribution)
        
+  
        hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
+    --think --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda
        
+  
        For image input use the Nemotron 3 Nano Omni GGUF with its mmproj-BF16.gguf (table above). Audio inference is not functional — the required Parakeet audio mmproj is not shipped with the GGUF distribution.
        
+
+  
        Mistral 3 (text + image)
        
+  
        hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
+hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
+    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
+    --image photo.png --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
+    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda
        
+
+  
        Muse-Glimmer (text + image, thinking, tools, DFlash drafting)
        
+  
        hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
+hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --input prompt.txt --max-tokens 256 --backend ggml_cuda
+# image understanding
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+# DFlash speculative decoding (optional drafter GGUF from the same repo)
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --draft-model models/dflash-kquant.gguf --spec-draft-n-max 15 --input prompt.txt --backend ggml_cuda
        
+
+  
        DiffusionGemma (block text diffusion)
        
+  
        hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
+    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda
        
+
+  
        Multimodal support
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+      
+    
+  
        FamilyInputsNotes
        Gemma 4Image · Video · AudioImages PNG/JPEG/HEIC; Video MP4 (1 fps via OpenCV); Audio WAV 16 kHz mono / MP3 / OGG. E4B projector: mmproj-gemma-4-E4B-it-Q8_0.gguf.
        Gemma 3ImagePNG / JPEG / HEIC. Non-gated 4B projector: mmproj-model-f16.gguf.
        Qwen 3.5 / 3.6ImageDynamic-resolution vision encoder. The 9B / 3.6 repositories use mmproj-F16.gguf.
        Mistral 3ImagePixtral vision encoder. Projector: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf.
        Muse-GlimmerImage50-layer sparse-window ViT with 2D RoPE and a 2×2 pixel shuffle; the image is stretched (no padding, no tiling) to a grid chosen the same way llama.cpp chooses it. Projector: mmproj-Muse-Glimmer-30B-Q8_0.gguf.
        Nemotron-H (Omni)ImageRADIO / v2_vl ViT encoder. Pass the matching --mmproj; image tokens expand at <image> placeholders. Audio is preprocessed only — real audio inference needs a Parakeet audio mmproj that the GGUF distribution does not ship.
        
+  
        
+  
        Send images/audio/video via the CLI (--image, --video, --audio), the Web UI uploads, or the HTTP API (base64 images array for Ollama, image_url data URI for OpenAI). PDF documents are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's --pdf flag (one-shot mode) or the Web UI upload (TS_PDF_MAX_PAGES caps the page count; default: all pages).
        
+
+  
        Thinking / reasoning mode
        
+  
        Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.
        
+  
          
+    
        • Qwen 3 / Qwen 3.5/3.6 / Nemotron-H — <think>…</think> tags.
          • 
+    
        • Gemma 4 — <|channel>thought …<channel|> tags.
          • 
+    
        • GPT OSS — Harmony format: <|channel|>analysis for thinking, <|channel|>final for the answer.
          • 
+    
        • DeepSeek V4 — <think>…</think> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.
          • 
+    
        • Muse-Glimmer — an assistant to=self reasoning channel emitted by the chat template.
          • 
+  
        
+  
        Enable it via --think (CLI), "think": true (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. message.thinking in the Ollama chat response.
        
+
+  
        Tool calling / function calling
        
+  
        Models can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (CLI) or the tools parameter (API). Each architecture uses its own wire format, but the output parser extracts calls into structured tool_calls regardless:
        
+  
          
+    
        • Qwen 3 / Nemotron-H — <tool_call>{"name": …, "arguments": {…}}</tool_call>
          • 
+    
        • Qwen 3.5 / 3.6 — the same <tool_call> block with an XML body: <function=NAME><parameter=key>value</parameter></function> (the JSON form is still accepted).
          • 
+    
        • Gemma 4 — <|tool_call>call:function_name{args}<tool_call|>
          • 
+    
        • GPT OSS (Harmony) — tools declared as a TypeScript namespace; calls emitted on the commentary channel.
          • 
+    
        • Muse-Glimmer — ATEM XML markup declared by the chat template; the parser lifts the calls into the same structured tool_calls shape.
          • 
+    
        • DeepSeek V4 — DSML markup: the system prompt teaches the syntax and carries one JSON schema per function, and the model answers with <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>….
          • 
+  
        
+  
        See Tool calling over HTTP for a complete request/response example and the continuation loop.
        
+
+  
        
+    ← Model Downloads
+    Next: Image Generation →
+  
        
+
        
+
+
diff --git a/website/models-text_zh-cn.html b/website/models-text_zh-cn.html
new file mode 100644
index 00000000..0131da30
--- /dev/null
+++ b/website/models-text_zh-cn.html
@@ -0,0 +1,175 @@
+
+
+
+  
+  
+  文本与 LLM 模型 — TensorSharp 维基
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        首页 › 支持的模型 › 文本与 LLM 模型
        
+  
        文本与 LLM 模型
        
+  
        每种文本架构的下载与运行命令，随后是它们共享的能力：图像、音频与 PDF 输入，思考模式，以及工具调用。
        
+
+  
        下载并运行
        
+  
        运行构建命令前，请先按平台安装并验证 .NET 10 SDK。第一个代码块是已验证的 Gemma 4 E4B 快速上手。所有代码块都使用 Hugging Face CLI（pip install -U huggingface_hub）；其后的各家族代码块假定已完成完整源码构建。从仓库根目录运行；先把问题写入 prompt.txt，并按硬件把 ggml_cuda 换成 ggml_metal、ggml_vulkan 或 ggml_cpu。
        
+
+  
        约 30 秒快速上手：Gemma 4 E4B Q8_0（原生 GGML）
        
+  
        复制并运行这些命令约需 30 秒；7.48 GiB 的模型下载与首次 restore/构建耗时更长，取决于网络与机器。仓库基准已经验证 TensorSharp 的 E4B Q8_0 家族与执行路径；链接的公开 ggml-org 仓库是推荐下载源。下面的代码块面向 Linux + NVIDIA：
        
+  
        hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
+TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
+echo "请说明本地推理的价值。" > prompt.txt
+
+# 纯文本不需要 mmproj。
+dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
+    --input prompt.txt --max-tokens 300 --backend ggml_cuda
+dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
+    --backend ggml_cuda
        
+  
        Apple Silicon 请省略 CUDA 环境变量并使用 ggml_metal；受支持的 Windows/Linux Vulkan GPU 请改为请求 TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON 并使用 ggml_vulkan。图像、视频或音频输入还需运行 hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models，并传入 --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf。服务端的浏览器界面位于 http://localhost:5000/index.html。Windows PowerShell 与完整平台语法见快速开始。
        
+
+  
        DeepSeek V4 Flash（284B MoE，文本、思考、工具、DSpark）
        
+  
        一个 284B 的专家混合模型，配备 128 token 的原始滑动窗口加块压缩注意力（官方宣称 1M 上下文）。它不走通用的逐算子前向：TensorSharp 通过三套专属的整模型执行器之一运行它 —— --backend cuda（Direct CUDA，不依赖 ggml）、--backend ggml_cuda / ggml_vulkan（原生 ggml），以及 --backend cpu（100% 纯 C#，零原生依赖）。三者都会把权重按层切分到所有可见 GPU，因此远大于单卡显存的模型依然跑得起来；--tp N（或 TS_DSV4_NGPU）用来限制使用几张卡。
        
+  
        # Q8 档位约需 160 GB 权重 —— 显存不足时改用更小的量化子目录
+hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
+hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models
+
+# 普通 decode —— --model 指向第一个分片
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 200
+
+# DSpark 块级投机解码（decode 约 1.3-1.4×；需要贪心采样）
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
+    --input prompt.txt --max-tokens 200 --temperature 0
+
+# 通过 HTTP 服务，4 张 GPU，开启投机
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
+    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf
        
+  
        DSpark 每步起草一整块 token，主干用一次批量前向验证整块，因此贪心输出保持不变。在 CLI 上它需要纯 argmax 采样（任何 temperature、top-k/p 或惩罚项都会将其关闭）；在服务端每一行验证都用该请求自己的采样器，因此可与任意采样设置组合。详见 DSpark 投机解码。
        
+
+  
        Gemma 3（文本 + 图像）
        
+  
        hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
+hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda
        
+
+  
        Qwen 3（文本、思维链、工具）
        
+  
        hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda
        
+
+  
        Qwen 3.5 / 3.6（文本 + 图像、思维链、工具；3.6 支持 NextN MTP）
        
+  
        hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
+hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
+    --image photo.png --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda
        
+  
        Qwen 3.6 NextN 必须从保留该块的 -MTP- 仓库下载，并在服务端加入 --mtp-spec；基础仓库 GGUF 会剥离 NextN 块并回退到标准解码：
        
+  
        hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec
        
+
+  
        GPT OSS（文本、始终思考、工具）
        
+  
        hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda
        
+
+  
        Nemotron-H（文本；Omni 分发支持图像）
        
+  
        hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
+    --think --max-tokens 400 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda
        
+  
        图像输入请改用上表 Nemotron 3 Nano Omni GGUF 与 mmproj-BF16.gguf。当前 GGUF 分发未附真实音频推理所需的 Parakeet audio mmproj。
        
+
+  
        Mistral 3（文本 + 图像）
        
+  
        hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
+hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
+    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
+    --image photo.png --max-tokens 300 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
+    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda
        
+
+  
        Muse-Glimmer（文本 + 图像、思维链、工具、DFlash 起草）
        
+  
        hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
+hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --input prompt.txt --max-tokens 256 --backend ggml_cuda
+# 图像理解
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+# DFlash 投机解码（草稿 GGUF 可从同仓库下载）
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
+    --draft-model models/dflash-kquant.gguf --spec-draft-n-max 15 --input prompt.txt --backend ggml_cuda
        
+
+  
        DiffusionGemma（分块文本扩散）
        
+  
        hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
+    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda
        
+
+  
        多模态支持
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+      
+    
+  
        家族输入说明
        Gemma 4图像 · 视频 · 音频图像 PNG/JPEG/HEIC；视频 MP4（经 OpenCV 以 1 fps 采样）；音频 WAV 16 kHz 单声道 / MP3 / OGG。E4B 投影器：mmproj-gemma-4-E4B-it-Q8_0.gguf。
        Gemma 3图像PNG / JPEG / HEIC。非 gated 4B 投影器：mmproj-model-f16.gguf。
        Qwen 3.5 / 3.6图像动态分辨率视觉编码器；9B / 3.6 仓库使用 mmproj-F16.gguf。
        Mistral 3图像Pixtral 视觉编码器。投影器：mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf。
        Muse-Glimmer图像50 层稀疏窗口 ViT，采用 2D RoPE 与 2×2 像素混洗；图像按 llama.cpp 相同的方式选定网格后直接拉伸（不填充、不切块）。投影器：mmproj-Muse-Glimmer-30B-Q8_0.gguf。
        Nemotron-H (Omni)图像RADIO / v2_vl ViT 编码器。传入匹配的 --mmproj；图像 token 在 <image> 占位处展开。当前 GGUF 分发未附真实音频推理需要的 Parakeet mmproj。
        
+  
        
+  
        通过 CLI（--image、--video、--audio、--pdf）、Web UI 上传，或 HTTP API（Ollama 用 base64 images 数组，OpenAI 用 image_url data URI）发送文件。数字版 PDF 会提取文本；扫描版 PDF 会转换为页面图像并需要视觉模型。
        
+
+  
        思考 / 推理模式
        
+  
        具备思考能力的模型（Qwen 3、Qwen 3.5/3.6、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、Muse-Glimmer）会在最终答案前产生结构化的思维链。思考内容与可见回复分离，便于客户端显示或隐藏。
        
+  
          
+    
        • Qwen 3 / Qwen 3.5/3.6 / Nemotron-H —— <think>…</think> 标签。
          • 
+    
        • Gemma 4 —— <|channel>thought …<channel|> 标签。
          • 
+    
        • GPT OSS —— Harmony 格式：<|channel|>analysis 用于思考，<|channel|>final 用于答案。
          • 
+    
        • DeepSeek V4 —— <think>…</think> 标签；不显式开启思考时聊天模板会立即闭合该块，因此推理是按需启用的。
          • 
+    
        • Muse-Glimmer —— 聊天模板输出的 assistant to=self 推理通道。
          • 
+  
        
+  
        通过 --think（CLI）、"think": true（Ollama API / Web UI）或浏览器中的思考开关启用。响应会单独暴露推理 —— 例如 Ollama 聊天响应中的 message.thinking。
        
+
+  
        工具调用 / 函数调用
        
+  
        模型可以调用用户自定义工具，并参与多轮工具调用对话。将工具定义为 JSON，通过 --tools（CLI）或 tools 参数（API）传入。每个架构使用各自的线格式，但输出解析器都会将调用提取为结构化 tool_calls：
        
+  
          
+    
        • Qwen 3 / Nemotron-H —— <tool_call>{"name": …, "arguments": {…}}</tool_call>
          • 
+    
        • Qwen 3.5 / 3.6 —— 同样是 <tool_call> 块，但内容为 XML：<function=NAME><parameter=key>value</parameter></function>（JSON 形式仍被接受）。
          • 
+    
        • Gemma 4 —— <|tool_call>call:function_name{args}<tool_call|>
          • 
+    
        • GPT OSS (Harmony) —— 工具以 TypeScript 命名空间声明；调用在 commentary 通道发出。
          • 
+    
        • DeepSeek V4 —— DSML 标记：系统提示词负责讲解语法并携带每个函数的 JSON schema，模型以 <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>… 作答。
          • 
+    
        • Muse-Glimmer —— 聊天模板声明的 ATEM XML 标记；解析器会把调用提取成同样的结构化 tool_calls。
          • 
+  
        
+  
        完整的请求/响应示例与续接循环见 通过 HTTP 进行工具调用。
        
+
+  
        
+    ← 模型下载
+    下一篇：图像生成 →
+  
        
+
        
+
+
diff --git a/website/models-video.html b/website/models-video.html
new file mode 100644
index 00000000..ad8c2bd8
--- /dev/null
+++ b/website/models-video.html
@@ -0,0 +1,122 @@
+
+
+
+  
+  
+  Video Generation — TensorSharp Wiki
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        Home › Supported Models › Video Generation (Wan 2.1 / 2.2)
        
+  
        Video Generation (Wan 2.1 / 2.2)
        
+  
        Wan turns a prompt — and, on the Wan 2.2 checkpoints, an uploaded first frame — into an H.264 MP4. This page covers the downloads, the run recipes, and the settings that decide the wall clock.
        
+  
        💡
        These commands follow the same conventions as the text-model recipes — the Hugging Face CLI, a source build from the repository root, and --backend swapped to match your hardware. See Download & run, per family.
        
+
+  
        Wan 2.1 / 2.2 (prompt → video, image → video)
        
+  
        Download the Turbo DiT, not the base one. The base Wan2.2-TI2V-5B follows the official 50-step × CFG recipe — 100 DiT passes per video. The step-distilled Turbo build of the same model is trained to run guidance-free in 4 steps, TensorSharp recognizes it from the file name, and the identical request comes back in minutes instead of hours. The Turbo repository ships only the DiT, so the VAE and the text encoder still come from the base repositories:
        
+  
        # The fast lane: 4 DiT passes instead of 100. Note the underscore in "Wan2_2".
+hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models
+hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models
+hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models
+
+# text → video
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --prompt "A red fox trotting through falling snow, cinematic" \
+    --video-frames 81 --fps 24 --output out.mp4 --backend ggml_cuda
+# image → video (the image becomes the first frame; Wan 2.2 models only)
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --image first_frame.png --prompt "the camera pushes in as the waves rise" \
+    --output out.mp4 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --backend ggml_cuda --video-frames 121 --fps 24
        
+  
        At load the console confirms the detection with step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override). Measured on an M5 Pro with ggml_metal, a 1088×832 × 121-frame (5 s, 720p-class) image-to-video request costs 100 DiT passes ≈ 3 h 30 m on the base checkpoint and 4 passes = 17 m 30 s on the Turbo one — the same flags, the same resolution, only the --model path differs. At 480p (736×544 × 121f) the Turbo checkpoint finishes in 6 m 19 s.
        
+  
        To reproduce the base recipe instead — for a reference sample, or on a Wan 2.1 checkpoint that has no distilled build — swap the first download and the --model paths:
        
+  
        hf download QuantStack/Wan2.2-TI2V-5B-GGUF Wan2.2-TI2V-5B-Q8_0.gguf --local-dir models
+
+# 50 steps x 2 CFG passes; --cfg-cache-stride trades a little accuracy for 1.30x / 1.43x
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --prompt "A red fox trotting through falling snow, cinematic" \
+    --video-frames 81 --fps 24 --cfg-cache-stride 2 --output out.mp4 --backend ggml_cuda
        
+  
        For Wan 2.2 A14B image-to-video, download both distilled experts under one --local-dir — the loader finds the low-noise partner from the high-noise file name, so only one --model path is passed. A14B uses the Wan 2.1 VAE:
        
+  
        hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF \
+    high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models
+hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF \
+    low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models
+hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.1_VAE.safetensors \
+    --image first_frame.png --prompt "the camera pushes in as the waves rise" \
+    --output out.mp4 --backend ggml_cuda
        
+  
        ⚡
        Generate at a resolution Wan was trained at. Wan is trained at 480p (832×480) and 720p (1280×704), and TI2V-5B is natively a 720p model. Below ~0.3 MP the DiT is out of distribution and the video gets soft, unstable and color-drifting no matter how many steps you spend — the pipeline warns when you cross that line. Generate at 480p and downscale afterwards rather than generating small.
        
+
+  
        Video generation (Wan 2.1 / 2.2)
        
+  
        Wan turns a prompt — and, on the Wan 2.2 checkpoints, an uploaded first frame — into an H.264 MP4. The --model GGUF is only the DiT; TensorSharp resolves two companions next to it (a VAE/, HighNoise/ or LowNoise/ subfolder is fine):
        
+  
          
+    
        • UMT5-XXL text encoder — prompt → conditioning (--wan-te / TS_WAN_TE). Released from VRAM before the denoise starts.
          • 
+    
        • Causal 3D video VAE — latent ↔ frames (--wan-vae / TS_WAN_VAE). Which one is required is read off the DiT itself, not chosen by you: Wan 2.1 and A14B use wan_2.1_vae; TI2V-5B uses Wan2.2_VAE.
          • 
+    
        • The A14B second expert — the two 14B experts must both be present. Put them in one directory, or in HighNoise/ + LowNoise/ subdirectories, and the loader pairs them from the file names (TS_WAN_DIT2 overrides; there is no CLI flag).
          • 
+  
        
+  
        
+  
+    
+    
+      
+      
+      
+    
+  
        FamilyLatentModesNotes
        Wan 2.1 T2V (1.3B / 14B)16 ch, 8×8×4text → videosingle DiT
        Wan 2.2 TI2V-5B48 ch, 16×16×4text → video, image → videodense 5B, 24 fps, 720p-capable
        Wan 2.2 A14B (T2V / I2V)16 ch (36 ch I2V input)text → video, image → videotwo 14B experts switched at a timestep boundary
        
+  
        
+  
        Each denoise step is one resident-weight ggml graph (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V image-to-video); the video VAE encode and decode are each a single graph. The stages hand VRAM over in sequence — text encoder, then DiT, then VAE — so peak usage is roughly max(TE, DiT + attention, VAE): TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 minutes, and both A14B experts run sequentially on the same card. Numerics are verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decode 59.9 dB PSNR), and Wan 2.1 runs 6.0× faster end-to-end than stable-diffusion.cpp on the identical workload.
        
+
+  
        The checkpoint decides the wall clock
        
+  
        Wan's DiT token count is latent_frames × (h/2) × (w/2) and self-attention costs O(tokens²), so a 5-second 720p video is a genuinely large job: 1088×832 × 121 frames is 27 404 tokens, and the official TI2V-5B recipe spends 50 steps × 2 classifier-free-guidance passes = 100 DiT passes on it. A step-distilled checkpoint is trained to run guidance-free in 4 steps, so the same video costs 4 passes — 1/25th of the denoising work. TensorSharp detects one from the DiT file name (turbo, distill, lightning, lightx2v, fastwan, -dmd, or an explicit …-4steps-… for 1–16 steps), prints step-distilled checkpoint detected -> 4 steps, guidance off at load, and applies the recipe itself; --diffusion-steps and --cfg still override.
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+    
+  
        M5 Pro, ggml_metal, TI2V-5B Q8_0, 1088×832 × 121f (27 404 tokens, image → video)Base, beforeBase, nowTurbo, now
        DiT passes100 (50 steps × CFG)1004 (guidance-free)
        per pass206.2 s120.2 s120.2 s
        denoise total20 615 s12 020 s481 s
        VAE decode, 121 frames863 s563 s563 s
        end to end≈ 5 h 58 m≈ 3 h 30 m17 m 30 s
        
+  
        
+  
        The two levers are independent: ~1.7× per pass from keeping the DiT's flash-attention keys and values in F16 (2.02× on a single 27 k-token self-attention, at cosine 0.999964 against the diffusers reference — TS_WAN_DIT_KV_F16=0 restores F32) and from running the Metal VAE convolutions through MPSGraph instead of ggml's im2col+GEMM lowering (VAE decode 159 s → 80 s at 736×544 × 81f, numerics unchanged at 93.9 dB PSNR — TS_WAN_VAE_MPS_CONV=0 reverts), and 25× fewer passes from the distilled checkpoint. Once distilled, the VAE decode is the bottleneck at ~55% of the run, not the DiT.
        
+
+  
        Frames and resolution set the rest
        
+  
        
+  
+    
+    
+      
+      
+      
+    
+  
        Output (M5 Pro, same Turbo checkpoint and image)TokensDenoiseVAE decodeTotal
        736×544 × 81f (3.4 s, 480p class)8 21184 s159 s4 m 09 s
        736×544 × 121f (5 s, 480p class)12 121137 s237 s6 m 19 s
        1088×832 × 121f (5 s, 720p class)27 404481 s563 s17 m 30 s
        
+  
        
+  
        480p (≈0.4 MP) is a resolution Wan is trained at, so the first two rows are in-distribution rather than a degraded mode — that is the setting to reach for when a few minutes matters. Quality falls off below ~0.3 MP, where the DiT is out of distribution and the video turns soft, unstable and color-drifting however many steps you spend; the pipeline prints a warning when you cross that line. Generate at a supported resolution and downscale afterwards (--width 480 --height 704 rather than 320×480).
        
+  
        To make a large request cheaper, in order of effect: (1) use a step-distilled checkpoint — 100 passes become 4, dwarfing everything else; (2) fewer frames — 121 → 61 roughly quarters the attention work and halves the VAE decode; (3) a smaller frame area, but not below ~0.3 MP; (4) fewer steps, on base checkpoints only — 30 instead of 50 is visibly close and 1.7× cheaper; (5) --cfg-cache-stride 2 or 3, which runs the unconditional pass on one step in N and reuses the cached guidance direction between — 77 of the 100 passes (1.30×) or 70 (1.43×) at 50 steps. It is an approximation, so leave it off when matching a reference sample matters, and it does nothing on a distilled checkpoint, which is already guidance-free.
        
+  
        On NVIDIA, ggml_cuda is the fastest Wan backend: on an RTX 2000 Ada 16 GB running the official 480p Wan2.1-1.3B recipe it takes 12.0 s/step against ggml_vulkan's 17.2 and the direct cuda backend's 19.3. Wan does not run on mlx at all, and the cpu / ggml_cpu backends are for functional use only.
        
+
+  
        Drive it from the CLI (--prompt, optional --image, --video-frames, --fps, --flow-shift, --sampler, --negative-prompt), the HTTP API (/v1/videos/generations), or the Web UI chat with an image upload. On the server, --video-frames and --fps set defaults rather than caps — a request carrying its own frames or fps overrides them independently, and with both omitted the model's own recipe applies (49 frames at 24 fps for Wan2.2-TI2V, 33 at 16 fps otherwise). Frame counts snap to the VAE's temporal grid (4k+1). Full details are in the repository's docs/models/wan.md card.
        
+  
        
+    ← Image Generation
+    Next: Command Line →
+  
        
+
        
+
+
diff --git a/website/models-video_zh-cn.html b/website/models-video_zh-cn.html
new file mode 100644
index 00000000..33d5da75
--- /dev/null
+++ b/website/models-video_zh-cn.html
@@ -0,0 +1,122 @@
+
+
+
+  
+  
+  视频生成 — TensorSharp 维基
+  
+  
+  
+  
+  
+  
+
+
+
+
        
+  
        首页 › 支持的模型 › 视频生成（Wan 2.1 / 2.2）
        
+  
        视频生成（Wan 2.1 / 2.2）
        
+  
        Wan 把提示词——在 Wan 2.2 检查点上还可以加一张首帧图像——变成 H.264 MP4。本页包含下载、运行示例，以及决定端到端耗时的设置。
        
+  
        💡
        这些命令遵循与文本模型示例相同的约定：Hugging Face CLI、从仓库根目录源码构建，并把 --backend 换成与你硬件匹配的值。参见按家族下载与运行。
        
+
+  
        Wan 2.1 / 2.2（提示词 → 视频、图像 → 视频）
        
+  
        请下载 Turbo 版 DiT，而不是基础版。基础的 Wan2.2-TI2V-5B 走官方的 50 步 × CFG 配方，也就是每段视频 100 次 DiT 前向；同一模型的步数蒸馏 Turbo 版本经过训练可在无引导的 4 步内完成，TensorSharp 会从文件名识别出来，于是同一条请求从数小时变成数分钟。Turbo 仓库只提供 DiT，VAE 与文本编码器仍从基础仓库获取：
        
+  
        # 快车道：DiT 前向 4 次而不是 100 次。注意 “Wan2_2” 用的是下划线。
+hf download hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF Wan2_2-TI2V-5B-Turbo-Q8_0.gguf --local-dir models
+hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models
+hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models
+
+# 文本 → 视频
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --prompt "A red fox trotting through falling snow, cinematic" \
+    --video-frames 81 --fps 24 --output out.mp4 --backend ggml_cuda
+# 图像 → 视频（上传的图像作为首帧；仅 Wan 2.2 模型支持）
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --image first_frame.png --prompt "the camera pushes in as the waves rise" \
+    --output out.mp4 --backend ggml_cuda
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2_2-TI2V-5B-Turbo-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --backend ggml_cuda --video-frames 121 --fps 24
        
+  
        加载时控制台会打印 step-distilled checkpoint detected -> 4 steps, guidance off (--diffusion-steps / --cfg override) 确认识别成功。在 M5 Pro 上用 ggml_metal 实测：一条 1088×832、121 帧（5 秒，720p 级）的图生视频请求，基础检查点需要 100 次 DiT 前向、约 3 小时 30 分，Turbo 检查点只需 4 次、17 分 30 秒——参数相同、分辨率相同，只有 --model 路径不同。若改到 480p（736×544、121 帧），Turbo 检查点 6 分 19 秒即可完成。
        
+  
        若要复现基础配方——例如生成参考样本，或使用没有蒸馏版本的 Wan 2.1 检查点——只需替换第一条下载命令与 --model 路径：
        
+  
        hf download QuantStack/Wan2.2-TI2V-5B-GGUF Wan2.2-TI2V-5B-Q8_0.gguf --local-dir models
+
+# 50 步 x 2 次 CFG 前向；--cfg-cache-stride 以少量精度换取 1.30x / 1.43x
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
+    --prompt "A red fox trotting through falling snow, cinematic" \
+    --video-frames 81 --fps 24 --cfg-cache-stride 2 --output out.mp4 --backend ggml_cuda
        
+  
        Wan 2.2 A14B 图生视频请把两个蒸馏专家下载到同一个 --local-dir——加载器会依据高噪专家的文件名找到对应的低噪专家，因此只需传一个 --model 路径。A14B 使用 Wan 2.1 的 VAE：
        
+  
        hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF \
+    high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models
+hf download jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF \
+    low_noise/wan2.2_i2v_A14b_low_noise_lightx2v_4step-Q4_K_M.gguf --local-dir models
+hf download QuantStack/Wan2.2-I2V-A14B-GGUF VAE/Wan2.1_VAE.safetensors --local-dir models
+
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/high_noise/wan2.2_i2v_A14b_high_noise_lightx2v_4step-Q4_K_M.gguf \
+    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.1_VAE.safetensors \
+    --image first_frame.png --prompt "the camera pushes in as the waves rise" \
+    --output out.mp4 --backend ggml_cuda
        
+  
        ⚡
        请在 Wan 训练过的分辨率上生成。Wan 的训练分辨率是 480p（832×480）与 720p（1280×704），TI2V-5B 原生就是 720p 模型。低于约 0.3 MP 时 DiT 已经超出分布，无论花多少步，画面都会发糊、不稳定并出现偏色——低于该阈值时流水线会给出警告。请先在受支持的分辨率上生成，再缩小，而不是直接生成小尺寸。
        
+
+  
        视频生成（Wan 2.1 / 2.2）
        
+  
        Wan 把提示词——在 Wan 2.2 检查点上还可以加一张上传的首帧图像——变成 H.264 MP4。--model 指定的 GGUF 只是 DiT，TensorSharp 会在其旁边解析两个伴随文件（放在 VAE/、HighNoise/ 或 LowNoise/ 子目录中同样可以）：
        
+  
          
+    
        • UMT5-XXL 文本编码器 —— 提示词 → 条件向量（--wan-te / TS_WAN_TE）。去噪开始前即从显存释放。
          • 
+    
        • 因果 3D 视频 VAE —— 潜变量 ↔ 帧（--wan-vae / TS_WAN_VAE）。需要哪一个由 DiT 自身读出，不用你指定：Wan 2.1 与 A14B 使用 wan_2.1_vae，TI2V-5B 使用 Wan2.2_VAE。
          • 
+    
        • A14B 的第二个专家 —— 两个 14B 专家必须同时存在。把它们放在同一目录，或放进 HighNoise/ 与 LowNoise/ 子目录，加载器会依据文件名配对（TS_WAN_DIT2 可覆盖；没有对应的 CLI 参数）。
          • 
+  
        
+  
        
+  
+    
+    
+      
+      
+      
+    
+  
        系列潜空间模式说明
        Wan 2.1 T2V（1.3B / 14B）16 通道，8×8×4文本 → 视频单个 DiT
        Wan 2.2 TI2V-5B48 通道，16×16×4文本 → 视频、图像 → 视频稠密 5B，24 fps，可出 720p
        Wan 2.2 A14B（T2V / I2V）16 通道（I2V 输入 36 通道）文本 → 视频、图像 → 视频两个 14B 专家在时间步边界切换
        
+  
        
+  
        每个去噪步是一张常驻权重的 ggml 图（CUDA 图捕获、flash attention；TI2V 图生视频还带 per-token 时间步调制），视频 VAE 的编码与解码各是一张图。各阶段按顺序交接显存——先文本编码器，再 DiT，最后 VAE——因此峰值约为 max(TE, DiT + attention, VAE)：TI2V-5B 在 16 GB GPU 上 8 分钟内生成 81 帧 480p 图生视频，A14B 的两个专家也能在同一张卡上顺序执行。数值上与 diffusers 对齐（DiT 余弦相似度 > 0.995，VAE 编码器 > 0.999，解码 59.9 dB PSNR），同等负载下 Wan 2.1 端到端比 stable-diffusion.cpp 快 6.0×。
        
+
+  
        检查点决定端到端耗时
        
+  
        Wan 的 DiT token 数为 latent_frames × (h/2) × (w/2)，自注意力开销为 O(tokens²)，因此 5 秒 720p 视频确实是个大活：1088×832、121 帧就是 27 404 个 token，而 TI2V-5B 的官方配方要在其上跑 50 步 × 2 次无分类器引导前向 = 100 次 DiT 前向。步数蒸馏检查点经过训练可在无引导的 4 步内完成，因此同一段视频只需 4 次前向——去噪工作量只有 1/25。TensorSharp 会从 DiT 文件名识别（turbo、distill、lightning、lightx2v、fastwan、-dmd，或显式的 …-4steps-…，步数取 1–16），在加载时打印 step-distilled checkpoint detected -> 4 steps, guidance off 并自动套用该配方；--diffusion-steps 与 --cfg 仍可覆盖。
        
+  
        
+  
+    
+    
+      
+      
+      
+      
+      
+    
+  
        M5 Pro、ggml_metal、TI2V-5B Q8_0、1088×832×121 帧（27 404 token，图 → 视频）基础版，优化前基础版，现在Turbo，现在
        DiT 前向次数100（50 步 × CFG）1004（无引导）
        单次前向206.2 s120.2 s120.2 s
        去噪总耗时20 615 s12 020 s481 s
        VAE 解码 121 帧863 s563 s563 s
        端到端约 5 小时 58 分约 3 小时 30 分17 分 30 秒
        
+  
        
+  
        两条加速路径互相独立：单次前向约 1.7× 来自把 DiT flash 注意力的 K/V 保持为 F16（单次 27k token 自注意力实测 2.02×，与 diffusers 参考实现的余弦相似度 0.999964——TS_WAN_DIT_KV_F16=0 可恢复 F32），以及在 Metal 上把 Wan VAE 卷积改走 MPSGraph 而非 ggml 的 im2col+GEMM 降级路径（736×544×81 帧下 VAE 解码 159 s → 80 s，数值不变，93.9 dB PSNR——TS_WAN_VAE_MPS_CONV=0 可回退）；另一条是蒸馏检查点带来的前向次数减少 25×。一旦用上蒸馏检查点，瓶颈就变成 VAE 解码（约占整轮的 55%），而不再是 DiT。
        
+
+  
        帧数与分辨率决定其余部分
        
+  
        
+  
+    
+    
+      
+      
+      
+    
+  
        输出（M5 Pro，同一 Turbo 检查点与图像）Token去噪VAE 解码总计
        736×544 × 81 帧（3.4 秒，480p 级）8 21184 s159 s4 分 09 秒
        736×544 × 121 帧（5 秒，480p 级）12 121137 s237 s6 分 19 秒
        1088×832 × 121 帧（5 秒，720p 级）27 404481 s563 s17 分 30 秒
        
+  
        
+  
        480p（约 0.4 MP）本身就是 Wan 训练过的分辨率，因此前两行属于分布内的正常档位，而不是降级模式——想在几分钟内出片就该选它。质量真正开始下滑是在约 0.3 MP 以下：那里 DiT 已超出分布，无论花多少步，画面都会发糊、不稳定并偏色，流水线也会给出警告。请在受支持的分辨率上生成后再缩小（用 --width 480 --height 704 而不是 320×480）。
        
+  
        要让一次大请求更便宜，按效果排序：(1) 换成步数蒸馏检查点——100 次前向变 4 次，压倒其他一切；(2) 减少帧数——121 → 61 大致把注意力工作量降到四分之一，VAE 解码减半；(3) 减小画面面积，但不要低于约 0.3 MP；(4) 减少步数，仅限基础检查点——30 步与 50 步观感接近，成本低 1.7×；(5) --cfg-cache-stride 2 或 3，即每 N 步才跑一次无条件前向、其余步复用缓存的引导方向——50 步下 100 次前向只跑 77 次（1.30×）或 70 次（1.43×）。它是近似方法，需要对齐参考样本时请关掉；在本就无引导的蒸馏检查点上也没有作用。
        
+  
        在 NVIDIA 上 ggml_cuda 是最快的 Wan 后端：RTX 2000 Ada 16 GB 跑官方 480p Wan2.1-1.3B 配方时为 12.0 s/步，ggml_vulkan 为 17.2，直连 cuda 后端为 19.3。Wan 完全不支持 mlx；cpu / ggml_cpu 后端只用于功能性验证。
        
+
+  
        可通过 CLI（--prompt、可选 --image、--video-frames、--fps、--flow-shift、--sampler、--negative-prompt）、HTTP API（/v1/videos/generations）或 Web UI 聊天上传图像来驱动。在服务端，--video-frames 与 --fps 设定的是默认值而非上限——请求自带 frames 或 fps 时各自覆盖；两者都不给时套用模型自身的配方（Wan2.2-TI2V 为 49 帧 24 fps，其余为 33 帧 16 fps）。帧数会对齐到 VAE 的时间网格（4k+1）。完整说明见仓库中的 docs/models/wan_zh-cn.md 卡片。
        
+  
        
+    ← 图像生成
+    下一篇：命令行 →
+  
        
+
        
+
+
diff --git a/website/models.html b/website/models.html
index 1d3841f0..525b2f1e 100644
--- a/website/models.html
+++ b/website/models.html
@@ -17,7 +17,16 @@
   
        Home › Supported Models
        
   
        Supported Models
        
   
        TensorSharp loads models in GGUF format and auto-detects the architecture from the file's general.architecture metadata. Pick a quantization that fits your hardware (Q4_K_M for low memory, Q8_0 for higher quality).
        
-  
        📘
        Gemma 4 E4B is the example model in From Tensors to Tokens. Read the book for a guided build from tensor fundamentals through multimodal inference; use this page for current downloads and capability details. Explore the book →
        
+  
        📘
        Gemma 4 E4B is the example model in From Tensors to Tokens. Read the book for a guided build from tensor fundamentals through multimodal inference; use this reference for current downloads and capability details. Explore the book →
        
+
+  
        Browse the model reference
        
+  
        The full reference is split across four pages. Start with downloads if you know which model you want, or with a category page if you are still choosing.
        
+  
        
+    
        ⬇️
        Model Downloads
        Every GGUF, with the projector, VAE, encoder and drafter files each family needs.
        
+    
        🧠
        Text & LLM Models
        Ten architectures, their run recipes, plus multimodal input, thinking and tool calling.
        
+    
        🖼️
        Image Generation
        Qwen-Image-Edit, the Lightning LoRA, and the DiT step cache.
        
+    
        🎬
        Video Generation
        Wan 2.1 / 2.2, and the distilled checkpoint that turns hours into minutes.
        
+  
        
 
   
        Supported architectures
        
   
        
@@ -27,277 +36,43 @@ 
        Supported architectures
        
       DeepSeek V4 Flashdeepseek4DeepSeek-V4-Flash (284B MoE, 256 experts, compressed sparse attention, 1M context)Text onlyYesYes (DSML)Yes (DSpark block drafter, separate GGUF)
       Gemma 4gemma4gemma-4-E4B, 12B, 31B, 26B-A4B (MoE)Image, Video, AudioYesYesYes (separate draft)
       Gemma 3gemma3gemma-3-4bImageNoNo—
-      Qwen 3qwen3Qwen3-4BText onlyYesYes—
+      Qwen 3qwen3, qwen2, qwen2vl, qwen2_vlQwen3-4B. Qwen2 / Qwen2.5-VL GGUFs load through the same class as text-only chatText onlyYesYes—
       Qwen 3.5 / 3.6qwen35, qwen35moe, qwen3nextQwen3.5-9B, Qwen3.5/3.6-35B-A3B (MoE)ImageYesYesYes on 3.6 (embedded NextN — only in GGUFs that retain the NextN block, e.g. the -MTP- repos)
       GPT OSSgptoss, gpt-ossgpt-oss-20b (MoE)Text onlyYes (always)Yes—
       Nemotron-Hnemotron_h, nemotron_h_moeNemotron-H-8B, 47B, Nemotron 3 Nano OmniImage (Omni)YesYes—
       Mistral 3mistral3Mistral-Small-3.1-24B-InstructImageNoNo—
-      Muse-Glimmermuse-glimmerMuse-Glimmer-30B (interleaved SWA + NoPE, attention output gate)ImageYesYes (ATEM)Yes (DFlash block drafter, separate GGUF)
+      Muse-Glimmermuse-glimmer, muse_glimmerMuse-Glimmer-30B (interleaved SWA + NoPE, attention output gate)ImageYesYes (ATEM)Yes (DFlash block drafter, separate GGUF)
       DiffusionGemmadiffusion-gemma, diffusion_gemmadiffusion-gemma text-diffusion GGUFsText onlyNoNo—
       Qwen-Image-Editqwen_image, qwen-imageqwen-image-edit MMDiT (+ VAE & Qwen2.5-VL companions)Image edit (image+text → image)NoNo—
-      Wan videowanWan 2.1 T2V 1.3B/14B, Wan 2.2 TI2V-5B, Wan 2.2 A14B T2V/I2V (two 14B experts) (+ UMT5-XXL & video-VAE companions)Video out (text → video, image → video)NoNo—
+      Wan videowan, wan2.1, wan2.2Wan 2.1 T2V 1.3B/14B, Wan 2.2 TI2V-5B, Wan 2.2 A14B T2V/I2V (two 14B experts) (+ UMT5-XXL & video-VAE companions)Video out (text → video, image → video)NoNo—
     
   
   
        
   
        Detailed per-model architecture cards (forward graph, components, parameters, and how TensorSharp optimizes prefill/decode) live under docs/models/ in the repository.
        
 
-  
        Model downloads (GGUF)
        
-  
        
-  
-    
-    
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-    
-  
        ArchitectureModelDownloadCompanion files (mmproj / MTP draft)
        DeepSeek V4DeepSeek-V4-Flash-0731 (284B MoE)unsloth/DeepSeek-V4-Flash-0731-GGUFOne subdirectory per quant (UD-Q8_K_XL/, UD-IQ4_XS/, UD-IQ1_S/, …), each a multi-shard set — point --model at the -00001-of- shard. Text only; weights layer-split across every visible GPU
        DeepSeek V4DSpark speculative drafter (optional)bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUFDSpark-drafter-Q2K-Q8-0731.gguf (7.0 GB) for the 0731 release, loaded with --draft-model for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see MODEL_DOWNLOADS.md. Drafters for other architectures are a different design and are not supported
        Gemma 4gemma-4-E4B-itggml-org/gemma-4-E4B-it-GGUFRecommended public artifact for the verified E4B Q8_0 native-GGML tier: gemma-4-E4B-it-Q8_0.gguf. Modalities: mmproj-gemma-4-E4B-it-Q8_0.gguf (same repo). MTP draft: AtomicChat/gemma-4-E4B-it-assistant-GGUF
        Gemma 4gemma-4-12B-it (QAT)unsloth/gemma-4-12B-it-qat-GGUFmmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-12B-it.gguf (both in the same repo)
        Gemma 4gemma-4-31B-itggml-org/gemma-4-31B-it-GGUFmmproj: mmproj-gemma-4-31B-it-Q8_0.gguf (same repo)
        Gemma 4gemma-4-26B-A4B-it (MoE)ggml-org/gemma-4-26B-A4B-it-GGUFmmproj: mmproj-gemma-4-26B-A4B-it-Q8_0.gguf (same repo)
        Gemma 4gemma-4-26B-A4B-it (MoE, QAT)unsloth/gemma-4-26B-A4B-it-qat-GGUFmmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-26B-A4B-it.gguf (same repo). Alt draft: AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — pair drafts only with the matching target size
        Gemma 3gemma-3-4b-it (QAT, official)google/gemma-3-4b-it-qat-q4_0-ggufmmproj: mmproj-model-f16-4B.gguf (same repo). ⚠️ Gated: downloads require a Hugging Face login + accepting Google's Gemma license
        Gemma 3gemma-3-4b-it (non-gated alternative)ggml-org/gemma-3-4b-it-GGUFmmproj: mmproj-model-f16.gguf (same repo)
        Qwen 3Qwen3-4BQwen/Qwen3-4B-GGUF— (text only)
        Qwen 3.5 / 3.6Qwen3.5-9Bunsloth/Qwen3.5-9B-GGUFmmproj: mmproj-F16.gguf (same repo)
        Qwen 3.5 / 3.6Qwen3.5-35B-A3B (MoE)ggml-org/Qwen3.5-35B-A3B-GGUFmmproj: mmproj-Qwen3.5-35B-A3B-Q8_0.gguf (same repo)
        Qwen 3.5 / 3.6Qwen3.6-35B-A3B (MoE, NextN MTP)unsloth/Qwen3.6-35B-A3B-MTP-GGUFmmproj: mmproj-F16.gguf (same repo). These GGUFs retain the embedded NextN block for --mtp-spec; the base-repo GGUFs (unsloth/Qwen3.6-35B-A3B-GGUF) strip it and silently fall back to standard decode
        GPT OSSgpt-oss-20b (MoE)ggml-org/gpt-oss-20b-GGUF— (text only)
        Nemotron-HNemotron-H-8B-Reasoning-128Kbartowski/nvidia_Nemotron-H-8B-…— (text only)
        Nemotron-HNemotron-H-47B-Reasoning-128Kbartowski/nvidia_Nemotron-H-47B-…— (text only)
        Nemotron-HNemotron 3 Nano Omni 30B-A3B (image)unsloth/NVIDIA-Nemotron-3-Nano-Omni-…mmproj: mmproj-BF16.gguf (same repo) — required for image input. Audio is not functional: it needs a Parakeet audio mmproj that this distribution does not ship
        Mistral 3Mistral-Small-3.1-24B-Instructbartowski/mistralai_Mistral-Small-3.1-24B-…Pixtral mmproj: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf (same repo)
        DiffusionGemmadiffusiongemma-26B-A4B-itunsloth/diffusiongemma-26B-A4B-it-GGUF—
        Muse-GlimmerMuse-Glimmer-30Bunsloth/Muse-Glimmer-30B-GGUFmmproj: mmproj-Muse-Glimmer-30B-Q8_0.gguf (same repo) for image input
        Muse-GlimmerDFlash speculative drafter (optional)unsloth/Muse-Glimmer-30B-GGUFdflash-*.gguf — a 5-layer block drafter loaded with --draft-model. Verification is greedy against the target, so the emitted stream is the plain-greedy stream
        Qwen-Image-EditMMDiT DiT (the --model GGUF)unsloth/Qwen-Image-Edit-2511-GGUFe.g. qwen-image-edit-2511-Q4_K_M.gguf
        Qwen-Image-EditQwen-Image VAE (required)QuantStack/Qwen-Image-Edit-GGUFVAE/Qwen_Image-VAE.safetensors — place next to the DiT or pass --qwen-image-vae
        Qwen-Image-EditQwen2.5-VL-7B text encoder (required)unsloth/Qwen2.5-VL-7B-Instruct-GGUFOptional vision mmproj: mmproj-BF16.gguf (same repo) for image-grounded edits
        Qwen-Image-EditLightning LoRA (optional, 4/8-step)lightx2v/Qwen-Image-Edit-2511-LightningQwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors via --qwen-image-lora
        Wan videoWan DiT (the --model GGUF)QuantStack/Wan2.2-TI2V-5B-GGUFAlso Wan2.2-I2V-A14B (download both the HighNoise and LowNoise experts) and Wan2.1-T2V-14B
        Wan videoUMT5-XXL text encoder (required)city96/umt5-xxl-encoder-ggufumt5-xxl-encoder-Q8_0.gguf — place next to the DiT or pass --wan-te
        Wan videovideo VAE (required)wan_2.1_vae.safetensorsWan 2.1 and A14B use wan_2.1_vae.safetensors; TI2V-5B uses Wan2.2_VAE.safetensors (bundled in the TI2V-5B repo). Place next to the DiT (a VAE/ subfolder works) or pass --wan-vae
        
-  
        
-  
        🧩
        Multimodal models need a projector (mmproj) file. Download the matching file and pass its exact path with --mmproj. The server never auto-detects it, and CLI auto-detection recognizes only a small set of legacy filenames.
        
-
-  
        Download & run, per family
        
-  
        Install and verify the .NET 10 SDK for your platform before running a build command. The first block is the verified Gemma 4 E4B quick start. All blocks use the Hugging Face CLI (pip install -U huggingface_hub); the per-family blocks after the first assume a full source build from the repository root. One-shot CLI text prompts are read from a file via --input (put your question in prompt.txt first — --prompt is reserved for Qwen-Image-Edit). Swap --backend for ggml_metal / ggml_vulkan / ggml_cpu to match your hardware (the CLI default is ggml_cpu).
        
-
-  
        Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)
        
-  
        Copying and running the commands takes about 30 seconds; the 7.48 GiB download and the first restore/build take longer and depend on your connection and machine. Repository benchmarks verify TensorSharp's E4B Q8_0 family and execution path; the linked ggml-org repository is the recommended public source. This block is for Linux + NVIDIA:
        
-  
        hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
-TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
-echo "Explain why local inference is useful." > prompt.txt
-
-# Text needs no mmproj.
-dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
-    --input prompt.txt --max-tokens 300 --backend ggml_cuda
-dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
-    --backend ggml_cuda
        
-  
        On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan. For image, video, or audio input, also run hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models and pass --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The server's browser UI is http://localhost:5000/index.html. See Getting Started for Windows PowerShell and full platform syntax.
        
-
-  
        DeepSeek V4 Flash (284B MoE, text, thinking, tools, DSpark)
        
-  
        A 284B mixture-of-experts model with a 128-token raw sliding window plus block-compressed attention (1M advertised context). It does not use the generic per-op forward: TensorSharp runs it through one of three dedicated whole-model executors — --backend cuda (direct CUDA, no ggml), --backend ggml_cuda / ggml_vulkan (native ggml), and --backend cpu (100% pure C#, no native dependencies). All of them layer-split the weights across every visible GPU, so a model far bigger than one card still runs; --tp N (or TS_DSV4_NGPU) caps how many GPUs it uses.
        
-  
        # ~160 GB of weights for the Q8 tier — pick a smaller quant directory if that is too much
-hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
-hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models
-
-# Plain decode — point --model at the FIRST shard
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --input prompt.txt --max-tokens 200
-
-# DSpark block speculative decoding (~1.3-1.4x decode; needs greedy sampling)
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
-    --input prompt.txt --max-tokens 200 --temperature 0
-
-# Served over HTTP, 4 GPUs, with speculation on
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf
        
-  
        DSpark drafts a whole block of tokens per step and the trunk verifies the block in one batched forward, so greedy output is unchanged. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p, or penalty turns it off); on the server every verify row is drawn with the request's own sampler, so it composes with any sampling settings. See DSpark speculative decoding.
        
-
-  
        Gemma 3 (text + image)
        
-  
        hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
-hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda
        
-
-  
        Qwen 3 (text, thinking, tools)
        
-  
        hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda
        
-
-  
        Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)
        
-  
        hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
-hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
-    --image photo.png --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda
        
-  
        For Qwen 3.6 NextN speculative decoding, download the trunk from the -MTP- repo (the base repo's GGUFs strip the NextN block and silently fall back to standard decode) and start the server with --mtp-spec — the MTP flags exist only on TensorSharp.Server:
        
-  
        hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec
        
-
-  
        GPT OSS (text, thinking always on, tools)
        
-  
        hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda
        
-
-  
        Nemotron-H (text; image on the Omni distribution)
        
-  
        hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
-    --think --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda
        
-  
        For image input use the Nemotron 3 Nano Omni GGUF with its mmproj-BF16.gguf (table above). Audio inference is not functional — the required Parakeet audio mmproj is not shipped with the GGUF distribution.
        
-
-  
        Mistral 3 (text + image)
        
-  
        hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
-hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
-    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
-    --image photo.png --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
-    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda
        
-
-  
        Muse-Glimmer (text + image, thinking, tools, DFlash drafting)
        
-  
        hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
-hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \r
-    --input prompt.txt --max-tokens 256 --backend ggml_cuda
-# image understanding
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \r
-    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \r
-    --max-tokens 300 --backend ggml_cuda
-# DFlash speculative decoding (optional drafter GGUF from the same repo)
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \r
-    --draft-model models/dflash-kquant.gguf --spec-draft-n-max 15 --input prompt.txt --backend ggml_cuda
        
-
-  
        DiffusionGemma (block text diffusion)
        
-  
        hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
-    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda
        
-
-  
        Qwen-Image-Edit (image + prompt → edited image)
        
-  
        hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
-hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
-hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
-hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
-hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
-    --prompt "Make the sky a dramatic sunset." --output edited.png \
-    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
-    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
-    --qwen-image-mmproj models/mmproj-BF16.gguf \
-    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
-    --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
-    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
-    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
-    --qwen-image-mmproj models/mmproj-BF16.gguf \
-    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
-    --backend ggml_cuda
        
-
-  
        Wan 2.1 / 2.2 (prompt → video, image → video)
        
-  
        hf download QuantStack/Wan2.2-TI2V-5B-GGUF Wan2.2-TI2V-5B-Q8_0.gguf --local-dir models
-hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models
-hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models
-
-# text → video
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \r
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \r
-    --prompt "A red fox trotting through falling snow, cinematic" \r
-    --video-frames 81 --fps 24 --output out.mp4 --backend ggml_cuda
-# image → video (the image becomes the first frame; Wan 2.2 models only)
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \r
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \r
-    --image first_frame.png --prompt "the camera pushes in as the waves rise" \r
-    --output out.mp4 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \r
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors --backend ggml_cuda
        
-
-  
        Multimodal support
        
+  
        Which one is fastest — the lever that matters per family
        
+  
        Every family has one setting that dominates its wall-clock time. The first row is the one people miss most often: on Wan, the checkpoint you download decides whether a 5-second 720p video takes three and a half hours or seventeen minutes, and it needs no flag at all.
        
   
        
   
-    
+    
-      
-      
-      
-      
-      
-      
+      
+      
+      
+      
+      
+      
+      
+      
+      
        FamilyInputsNotes
        FamilyFast laneMeasured effect
        
     
        Gemma 4Image · Video · AudioImages PNG/JPEG/HEIC; Video MP4 (1 fps via OpenCV); Audio WAV 16 kHz mono / MP3 / OGG. E4B projector: mmproj-gemma-4-E4B-it-Q8_0.gguf.
        Gemma 3ImagePNG / JPEG / HEIC. Non-gated 4B projector: mmproj-model-f16.gguf.
        Qwen 3.5 / 3.6ImageDynamic-resolution vision encoder. The 9B / 3.6 repositories use mmproj-F16.gguf.
        Mistral 3ImagePixtral vision encoder. Projector: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf.
        Muse-GlimmerImage50-layer sparse-window ViT with 2D RoPE and a 2×2 pixel shuffle; the image is stretched (no padding, no tiling) to a grid chosen the same way llama.cpp chooses it. Projector: mmproj-Muse-Glimmer-30B-Q8_0.gguf.
        Nemotron-H (Omni)ImageRADIO / v2_vl ViT encoder. Pass the matching --mmproj; image tokens expand at <image> placeholders. Audio is preprocessed only — real audio inference needs a Parakeet audio mmproj that the GGUF distribution does not ship.
        Wan videoLoad a step-distilled checkpoint — a DiT file name containing Turbo, distill, Lightning, lightx2v, FastWan, -dmd or …-4steps-… is auto-detected. No flag100 DiT passes → 4, guidance off. The same 1088×832×121f image-to-video request on M5 Pro / ggml_metal: ≈ 3 h 30 m → 17 m 30 s
        Wan video--cfg-cache-stride 2 / 3 — base (non-distilled) checkpoints only1.30× / 1.43× at 50 steps (77 / 70 of the 100 passes). Approximate; pointless on a distilled checkpoint, which is already guidance-free
        Qwen-Image-EditLightning distillation LoRA via --qwen-image-loraBase recipe 30 steps at cfg 2.5 = 60 DiT forwards → 4–8 forwards at cfg 1.0. Warm 544×1184 4-step edit: 40.44 s vs stable-diffusion.cpp's 48.16 s
        DeepSeek V4DSpark block drafter via --draft-model (cuda / ggml_cuda only)4×A40, 200 greedy tokens: decode 26.0 → 34.0 tok/s (cuda) and 26.4 → 37.1 tok/s (ggml_cuda) at 69% acceptance; 1.5–2.0× across a 5-turn chat. Output is unchanged
        Muse-GlimmerDFlash block drafter via --draft-model (pass no sampler flags — it needs pure greedy)RTX PRO 6000, 128 greedy tokens: 35.0 → 50.9 tok/s at a 60-token prompt and 24.9 → 43.5 at 2 050. On Apple Silicon it does not pay today (20.7 → 13.9 tok/s) — run plain decode there
        Qwen 3.6 / Gemma 4--mtp-spec on TensorSharp.Server — Qwen 3.6 uses the NextN block embedded in an -MTP- GGUF, Gemma 4 a separate --mtp-draft-model draftEngages on solo (non-concurrent) sequences and only on GGML CUDA / GGML Metal and the direct cuda backend; off on CPU / MLX. Tune with --mtp-draft (8) and --mtp-pmin (0.75)
        Every text familyPick the right --backend: ggml_cuda on NVIDIA, ggml_metal on Apple Silicon, ggml_cpu (not cpu) on CPURTX 3080 Laptop, gemma-4-26B-A4B QAT: ggml_cuda decode 78.7 vs the direct cuda backend's 35.3 tok/s, prefill 1832 vs 128. Apple Silicon, Muse-Glimmer-30B: ggml_metal prefill 413.6 vs mlx's 29.0
        Any MoE that does not fit--n-cpu-moe N / --cpu-moe — keeps the first N layers' routed experts in system RAMTrades decode for VRAM, but wins outright when it stops a spill: gpt-oss-20b 16.2 → 2.9 GB on a 16 GB card, and --n-cpu-moe 12 turns 0.3 tok/s of WDDM paging into 25.4
        Multiple GPUs--tp N (env TENSORSHARP_TP_DEGREE) on the direct cuda backend and on GGML CUDA / VulkanMuse-Glimmer-30B UD-IQ2_XXS on 2× RTX PRO 4000: prefill 1171 → 1569 (1.34×), decode 40.2 → 63.2 tok/s (1.57×). It also runs models that fit on no single card at all
        
             
   
        
   
        
-  
        Send images/audio/video via the CLI (--image, --video, --audio), the Web UI uploads, or the HTTP API (base64 images array for Ollama, image_url data URI for OpenAI). PDF documents are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's --pdf flag (one-shot mode) or the Web UI upload (TS_PDF_MAX_PAGES caps the page count; default: all pages).
        
-
-  
        Image editing (Qwen-Image-Edit)
        
-  
        Qwen-Image-Edit takes a prompt + an input image and returns an edited image — a different output modality from the text LLMs above. The loaded qwen_image GGUF is only the MMDiT (multimodal diffusion transformer); TensorSharp resolves two companion GGUFs alongside it:
        
-  
          
-    
        • Qwen-Image VAE — image ↔ 16-channel latent (the original .safetensors works too).
          • 
-    
        • Qwen2.5-VL-7B text encoder — prompt → 3584-dim conditioning, with an optional mmproj vision tower for image-grounded edits.
          • 
-  
        
-  
        Place the companions next to the DiT GGUF, or point at them with TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ (CLI: --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj). The pipeline VAE-encodes the reference, builds the conditioning, runs a FlowMatch-Euler true-CFG denoise loop (reference-latent concatenation), then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured and flash attention is on by default; the target area auto-clamps to the device VRAM budget unless you pin a width/height.
        
-  
        An optional Lightning distillation LoRA (--qwen-image-lora / TS_QWEN_IMAGE_LORA, .safetensors) is merged into the DiT at load time — it auto-derives the reduced denoise step count (e.g. 4 or 8) and switches CFG to 1.0, skipping the negative pass. The denoise loop is further accelerated by a whole-step DiT cache (EasyCache-style; TS_QWEN_DIT_CACHE_MODE = easycache / fbc) that skips DiT steps when the output is predictable from the previous step.
        
-  
        Run it from the CLI (--image + --prompt), the Web UI image-edit flow (with live denoising previews), or in-process from C# via QwenImageModel.EditImage(). Full details are in the repository's docs/models/qwenimage.md card.
        
-
-  
        Video generation (Wan 2.1 / 2.2)
        
-  
        Wan turns a prompt — and, on the Wan 2.2 checkpoints, an uploaded first frame — into an H.264 MP4. The --model GGUF is only the DiT; TensorSharp resolves two companions next to it (a VAE/, HighNoise/ or LowNoise/ subfolder is fine):
        
-  
          
-    
        • UMT5-XXL text encoder — prompt → conditioning (--wan-te / TS_WAN_TE). Released from VRAM before the denoise starts.
          • 
-    
        • Causal 3D video VAE — latent ↔ frames (--wan-vae / TS_WAN_VAE). Wan 2.1 and A14B use wan_2.1_vae; TI2V-5B uses Wan2.2_VAE.
          • 
-  
        
-  
        
-  
-    
-    
-      
-      
-      
-    
-  
        FamilyLatentModesNotes
        Wan 2.1 T2V (1.3B / 14B)16 ch, 8×8×4text → videosingle DiT
        Wan 2.2 TI2V-5B48 ch, 16×16×4text → video, image → videodense 5B, 24 fps, 720p-capable
        Wan 2.2 A14B (T2V / I2V)16 ch (36 ch I2V input)text → video, image → videotwo 14B experts switched at a timestep boundary
        
-  
        
-  
        Each denoise step is one resident-weight ggml graph (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V image-to-video); the video VAE encode and decode are each a single graph. The stages hand VRAM over in sequence — text encoder, then DiT, then VAE — so peak usage is roughly max(TE, DiT + attention, VAE): TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 minutes, and both A14B experts run sequentially on the same card. Numerics are verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decode 59.9 dB PSNR), and Wan 2.1 runs 6.0× faster end-to-end than stable-diffusion.cpp on the identical workload.
        
-  
        Drive it from the CLI (--prompt, optional --image, --video-frames, --fps, --flow-shift, --sampler, --negative-prompt), the HTTP API (/v1/videos/generations), or the Web UI chat with an image upload. Full details are in the repository's docs/models/wan.md card.
        
-
-  
        Thinking / reasoning mode
        
-  
        Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.
        
-  
          
-    
        • Qwen 3 / Qwen 3.5/3.6 / Nemotron-H — <think>…</think> tags.
          • 
-    
        • Gemma 4 — <|channel>thought …<channel|> tags.
          • 
-    
        • GPT OSS — Harmony format: <|channel|>analysis for thinking, <|channel|>final for the answer.
          • 
-    
        • DeepSeek V4 — <think>…</think> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.
          • 
-    
        • Muse-Glimmer — an assistant to=self reasoning channel emitted by the chat template.
          • 
-  
        
-  
        Enable it via --think (CLI), "think": true (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. message.thinking in the Ollama chat response.
        
-
-  
        Tool calling / function calling
        
-  
        Models can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (CLI) or the tools parameter (API). Each architecture uses its own wire format, but the output parser extracts calls into structured tool_calls regardless:
        
-  
          
-    
        • Qwen 3 / Nemotron-H — <tool_call>{"name": …, "arguments": {…}}</tool_call>
          • 
-    
        • Qwen 3.5 / 3.6 — the same <tool_call> block with an XML body: <function=NAME><parameter=key>value</parameter></function> (the JSON form is still accepted).
          • 
-    
        • Gemma 4 — <|tool_call>call:function_name{args}<tool_call|>
          • 
-    
        • GPT OSS (Harmony) — tools declared as a TypeScript namespace; calls emitted on the commentary channel.
          • 
-    
        • Muse-Glimmer — ATEM XML markup declared by the chat template; the parser lifts the calls into the same structured tool_calls shape.
          • 
-    
        • DeepSeek V4 — DSML markup: the system prompt teaches the syntax and carries one JSON schema per function, and the model answers with <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>….
          • 
-  
        
-  
        See Tool calling over HTTP for a complete request/response example and the continuation loop.
        
+  
        Wan, Qwen-Image-Edit and DiffusionGemma are constructed without a tensor-parallel degree, so --tp does not apply to them. Wan is also the one family that refuses a backend outright: mlx is not supported.
        
 
   
        
     ← Compute Backends
-    Next: Command Line →
+    Next: Model Downloads →
   
        
 
 
diff --git a/website/models_zh-cn.html b/website/models_zh-cn.html
index ee51d09e..6209e3e5 100644
--- a/website/models_zh-cn.html
+++ b/website/models_zh-cn.html
@@ -17,7 +17,16 @@
   
        首页 › 支持的模型
        
   
        支持的模型
        
   
        TensorSharp 加载 GGUF 格式的模型，并从文件的 general.architecture 元数据自动识别架构。选择适合你硬件的量化（低内存用 Q4_K_M，更高质量用 Q8_0）。
        
-  
        📘
        Gemma 4 E4B 是《From Tensors to Tokens》采用的示例模型。配套图书提供从张量基础到多模态推理的连贯构建过程；本页则提供最新下载与能力细节。了解本书 →
        
+  
        📘
        Gemma 4 E4B 是《From Tensors to Tokens》采用的示例模型。配套图书提供从张量基础到多模态推理的连贯构建过程；这组参考页面则提供最新下载与能力细节。了解本书 →
        
+
+  
        浏览模型参考
        
+  
        完整参考分为四个页面。如果你已经确定要用哪个模型，从"模型下载"开始；如果还在挑选，从对应的分类页面开始。
        
+  
        
+    
        ⬇️
        模型下载
        所有 GGUF，以及各家族所需的投影器、VAE、编码器与草稿模型。
        
+    
        🧠
        文本与 LLM 模型
        十种架构及其运行示例，加上多模态输入、思考模式与工具调用。
        
+    
        🖼️
        图像生成
        Qwen-Image-Edit、Lightning LoRA，以及 DiT 步缓存。
        
+    
        🎬
        视频生成
        Wan 2.1 / 2.2，以及把数小时变成几分钟的蒸馏检查点。
        
+  
        
 
   
        支持的架构
        
   
        
@@ -27,277 +36,43 @@ 
        支持的架构
        
       DeepSeek V4 Flashdeepseek4DeepSeek-V4-Flash（284B MoE，256 专家，压缩稀疏注意力，1M 上下文）仅文本是是（DSML）是（DSpark 块级草稿器，独立 GGUF）
       Gemma 4gemma4gemma-4-E4B、12B、31B、26B-A4B (MoE)图像、视频、音频是是是（独立草稿）
       Gemma 3gemma3gemma-3-4b图像否否—
-      Qwen 3qwen3Qwen3-4B仅文本是是—
+      Qwen 3qwen3、qwen2、qwen2vl、qwen2_vlQwen3-4B。Qwen2 / Qwen2.5-VL 的 GGUF 也由同一实现加载，按纯文本对话使用仅文本是是—
       Qwen 3.5 / 3.6qwen35、qwen35moe、qwen3nextQwen3.5-9B、Qwen3.5/3.6-35B-A3B (MoE)图像是是3.6 支持（内嵌 NextN——仅限保留 NextN 块的 GGUF，例如 -MTP- 仓库）
       GPT OSSgptoss、gpt-ossgpt-oss-20b (MoE)仅文本是（始终）是—
       Nemotron-Hnemotron_h、nemotron_h_moeNemotron-H-8B、47B、Nemotron 3 Nano Omni图像（Omni）是是—
       Mistral 3mistral3Mistral-Small-3.1-24B-Instruct图像否否—
-      Muse-Glimmermuse-glimmerMuse-Glimmer-30B（交错滑动窗口 + NoPE 全注意力层，注意力输出门控）图像是是（ATEM）是（DFlash 块草稿模型，独立 GGUF）
+      Muse-Glimmermuse-glimmer、muse_glimmerMuse-Glimmer-30B（交错滑动窗口 + NoPE 全注意力层，注意力输出门控）图像是是（ATEM）是（DFlash 块草稿模型，独立 GGUF）
       DiffusionGemmadiffusion-gemma、diffusion_gemmadiffusion-gemma 文本扩散 GGUF仅文本否否—
       Qwen-Image-Editqwen_image、qwen-imageqwen-image-edit MMDiT（+ VAE 与 Qwen2.5-VL 伴随文件）图像编辑（图像+文本 → 图像）否否—
-      Wan 视频wanWan 2.1 T2V 1.3B/14B、Wan 2.2 TI2V-5B、Wan 2.2 A14B T2V/I2V（两个 14B 专家）（+ UMT5-XXL 与视频 VAE 伴随文件）视频输出（文本 → 视频、图像 → 视频）否否—
+      Wan 视频wan、wan2.1、wan2.2Wan 2.1 T2V 1.3B/14B、Wan 2.2 TI2V-5B、Wan 2.2 A14B T2V/I2V（两个 14B 专家）（+ UMT5-XXL 与视频 VAE 伴随文件）视频输出（文本 → 视频、图像 → 视频）否否—
     
   
   
        
   
        各模型的详细架构卡（前向图、组件、参数，以及 TensorSharp 如何优化 prefill/decode）位于仓库的 docs/models/ 下。
        
 
-  
        模型下载（GGUF）
        
-  
        
-  
-    
-    
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-      
-    
-  
        架构模型下载
        DeepSeek V4DeepSeek-V4-Flash-0731（284B MoE）unsloth/DeepSeek-V4-Flash-0731-GGUF；每个量化档一个子目录（UD-Q8_K_XL/、UD-IQ4_XS/、UD-IQ1_S/ …），均为多分片——--model 指向 -00001-of- 那一片。仅文本；权重按层切分到所有可见 GPU
        DeepSeek V4DSpark 投机草稿器（可选）bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF 的 DSpark-drafter-Q2K-Q8-0731.gguf（7.0 GB，对应 0731 版本），用 --draft-model 加载可获得约 1.3–1.4× 的 decode 提速。另有两家发布的构建（5.6 GB / 10.9 GB）也可直接加载，见 MODEL_DOWNLOADS_zh-cn.md。其他架构的草稿器属于不同设计，暂不支持
        Gemma 4gemma-4-E4B-itggml-org/gemma-4-E4B-it-GGUF；已验证 E4B Q8_0 原生 GGML 规格推荐使用公开文件 gemma-4-E4B-it-Q8_0.gguf；多模态投影器为同仓库的 mmproj-gemma-4-E4B-it-Q8_0.gguf
        Gemma 4gemma-4-12B-it（QAT）unsloth/gemma-4-12B-it-qat-GGUF；同仓库 mmproj-BF16.gguf 与 mtp-gemma-4-12B-it.gguf
        Gemma 4gemma-4-31B-itggml-org/gemma-4-31B-it-GGUF
        Gemma 4gemma-4-26B-A4B-it (MoE)ggml-org/gemma-4-26B-A4B-it-GGUF
        Gemma 4gemma-4-26B-A4B-it（MoE，QAT）unsloth/gemma-4-26B-A4B-it-qat-GGUF；同仓库 mmproj-BF16.gguf 与 MTP draft；另有 AtomicChat assistant
        Gemma 3gemma-3-4b-it非 gated ggml-org 仓库；Q4_K_M 约 2.32 GiB，mmproj 为 mmproj-model-f16.gguf。Google QAT 仓库需要登录并接受许可证
        Gemma 3gemma-3-4b-it（官方 QAT）google/gemma-3-4b-it-qat-q4_0-gguf；mmproj-model-f16-4B.gguf；⚠️ 需要 Hugging Face 登录并接受 Google Gemma 许可证
        Qwen 3Qwen3-4BQwen/Qwen3-4B-GGUF；Qwen3-4B-Q4_K_M.gguf 约 2.33 GiB
        Qwen 3.5 / 3.6Qwen3.5-9Bunsloth/Qwen3.5-9B-GGUF
        Qwen 3.5 / 3.6Qwen3.5-35B-A3B (MoE)ggml-org/Qwen3.5-35B-A3B-GGUF
        Qwen 3.6Qwen3.6-35B-A3B NextN MTPunsloth/Qwen3.6-35B-A3B-MTP-GGUF；UD-Q4_K_M 约 21.11 GiB。基础仓库 GGUF 会剥离 NextN 块
        GPT OSSgpt-oss-20b (MoE)ggml-org/gpt-oss-20b-GGUF
        Nemotron-HNemotron-H-8B-Reasoning-128Kbartowski/nvidia_Nemotron-H-8B-…
        Nemotron-HNemotron-H-47B-Reasoning-128Kbartowski/nvidia_Nemotron-H-47B-…
        Nemotron-HNemotron 3 Nano Omni 30B-A3B（图像）unsloth/NVIDIA-Nemotron-3-Nano-Omni-…；图像输入需要同仓库 mmproj-BF16.gguf；未附真实音频推理所需的 Parakeet mmproj
        Mistral 3Mistral-Small-3.1-24B-Instructbartowski/mistralai_Mistral-Small-3.1-24B-…；Q4_K_M 约 13.35 GiB；同仓库 Pixtral mmproj
        DiffusionGemmadiffusiongemma-26B-A4B-itunsloth/diffusiongemma-26B-A4B-it-GGUF；Q4_K_M 约 15.65 GiB
        Muse-GlimmerMuse-Glimmer-30Bunsloth/Muse-Glimmer-30B-GGUF图像输入需要同仓库的 mmproj：mmproj-Muse-Glimmer-30B-Q8_0.gguf
        Muse-GlimmerDFlash 投机草稿模型（可选）unsloth/Muse-Glimmer-30B-GGUFdflash-*.gguf —— 5 层块级草稿模型，用 --draft-model 加载。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致
        Qwen-Image-Edit2511 MMDiT（--model）unsloth/Qwen-Image-Edit-2511-GGUF；qwen-image-edit-2511-Q4_K_M.gguf 约 12.34 GiB
        Qwen-Image-EditQwen-Image VAE（必需）QuantStack/Qwen-Image-Edit-GGUF 中的 VAE/Qwen_Image-VAE.safetensors；通过 --qwen-image-vae 指定
        Qwen-Image-EditQwen2.5-VL-7B 文本编码器（必需）unsloth/Qwen2.5-VL-7B-Instruct-GGUF；可选视觉投影器为同仓库 mmproj-BF16.gguf
        Qwen-Image-EditLightning LoRA（可选，4/8 步）lightx2v/Qwen-Image-Edit-2511-Lightning；如 Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors
        Wan 视频Wan DiT（即 --model 的 GGUF）QuantStack/Wan2.2-TI2V-5B-GGUF另有 Wan2.2-I2V-A14B（需同时下载 HighNoise 与 LowNoise 两个专家）与 Wan2.1-T2V-14B
        Wan 视频UMT5-XXL 文本编码器（必需）city96/umt5-xxl-encoder-ggufumt5-xxl-encoder-Q8_0.gguf —— 放在 DiT 旁边，或用 --wan-te 指定
        Wan 视频视频 VAE（必需）wan_2.1_vae.safetensorsWan 2.1 与 A14B 使用 wan_2.1_vae.safetensors；TI2V-5B 使用 Wan2.2_VAE.safetensors（随 TI2V-5B 仓库提供）。放在 DiT 旁边（VAE/ 子目录亦可）或用 --wan-vae 指定
        
-  
        
-  
        🧩
        多模态模型需要投影器（mmproj）文件。请下载匹配文件并显式传入 --mmproj；服务端不会自动检测，CLI 也只识别少数旧文件名。
        
-
-  
        下载并运行
        
-  
        运行构建命令前，请先按平台安装并验证 .NET 10 SDK。第一个代码块是已验证的 Gemma 4 E4B 快速上手。所有代码块都使用 Hugging Face CLI（pip install -U huggingface_hub）；其后的各家族代码块假定已完成完整源码构建。从仓库根目录运行；先把问题写入 prompt.txt，并按硬件把 ggml_cuda 换成 ggml_metal、ggml_vulkan 或 ggml_cpu。
        
-
-  
        约 30 秒快速上手：Gemma 4 E4B Q8_0（原生 GGML）
        
-  
        复制并运行这些命令约需 30 秒；7.48 GiB 的模型下载与首次 restore/构建耗时更长，取决于网络与机器。仓库基准已经验证 TensorSharp 的 E4B Q8_0 家族与执行路径；链接的公开 ggml-org 仓库是推荐下载源。下面的代码块面向 Linux + NVIDIA：
        
-  
        hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
-TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
-echo "请说明本地推理的价值。" > prompt.txt
-
-# 纯文本不需要 mmproj。
-dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
-    --input prompt.txt --max-tokens 300 --backend ggml_cuda
-dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
-    --backend ggml_cuda
        
-  
        Apple Silicon 请省略 CUDA 环境变量并使用 ggml_metal；受支持的 Windows/Linux Vulkan GPU 请改为请求 TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON 并使用 ggml_vulkan。图像、视频或音频输入还需运行 hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models，并传入 --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf。服务端的浏览器界面位于 http://localhost:5000/index.html。Windows PowerShell 与完整平台语法见快速开始。
        
-
-  
        DeepSeek V4 Flash（284B MoE，文本、思考、工具、DSpark）
        
-  
        一个 284B 的专家混合模型，配备 128 token 的原始滑动窗口加块压缩注意力（官方宣称 1M 上下文）。它不走通用的逐算子前向：TensorSharp 通过三套专属的整模型执行器之一运行它 —— --backend cuda（Direct CUDA，不依赖 ggml）、--backend ggml_cuda / ggml_vulkan（原生 ggml），以及 --backend cpu（100% 纯 C#，零原生依赖）。三者都会把权重按层切分到所有可见 GPU，因此远大于单卡显存的模型依然跑得起来；--tp N（或 TS_DSV4_NGPU）用来限制使用几张卡。
        
-  
        # Q8 档位约需 160 GB 权重 —— 显存不足时改用更小的量化子目录
-hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
-hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models
-
-# 普通 decode —— --model 指向第一个分片
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --input prompt.txt --max-tokens 200
-
-# DSpark 块级投机解码（decode 约 1.3-1.4×；需要贪心采样）
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
-    --input prompt.txt --max-tokens 200 --temperature 0
-
-# 通过 HTTP 服务，4 张 GPU，开启投机
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
-    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf
        
-  
        DSpark 每步起草一整块 token，主干用一次批量前向验证整块，因此贪心输出保持不变。在 CLI 上它需要纯 argmax 采样（任何 temperature、top-k/p 或惩罚项都会将其关闭）；在服务端每一行验证都用该请求自己的采样器，因此可与任意采样设置组合。详见 DSpark 投机解码。
        
-
-  
        Gemma 3（文本 + 图像）
        
-  
        hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
-hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda
        
-
-  
        Qwen 3（文本、思维链、工具）
        
-  
        hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda
        
-
-  
        Qwen 3.5 / 3.6（文本 + 图像、思维链、工具；3.6 支持 NextN MTP）
        
-  
        hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
-hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
-    --image photo.png --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda
        
-  
        Qwen 3.6 NextN 必须从保留该块的 -MTP- 仓库下载，并在服务端加入 --mtp-spec；基础仓库 GGUF 会剥离 NextN 块并回退到标准解码：
        
-  
        hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec
        
-
-  
        GPT OSS（文本、始终思考、工具）
        
-  
        hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda
        
-
-  
        Nemotron-H（文本；Omni 分发支持图像）
        
-  
        hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
-    --think --max-tokens 400 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda
        
-  
        图像输入请改用上表 Nemotron 3 Nano Omni GGUF 与 mmproj-BF16.gguf。当前 GGUF 分发未附真实音频推理所需的 Parakeet audio mmproj。
        
-
-  
        Mistral 3（文本 + 图像）
        
-  
        hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
-hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
-    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
-    --image photo.png --max-tokens 300 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
-    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda
        
-
-  
        Muse-Glimmer（文本 + 图像、思维链、工具、DFlash 起草）
        
-  
        hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
-hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
-    --input prompt.txt --max-tokens 256 --backend ggml_cuda
-# 图像理解
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
-    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \
-    --max-tokens 300 --backend ggml_cuda
-# DFlash 投机解码（草稿 GGUF 可从同仓库下载）
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
-    --draft-model models/dflash-kquant.gguf --spec-draft-n-max 15 --input prompt.txt --backend ggml_cuda
        
-
-  
        DiffusionGemma（分块文本扩散）
        
-  
        hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
-    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda
        
-
-  
        Qwen-Image-Edit（图像 + 提示词 → 编辑后的图像）
        
-  
        hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
-hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
-hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
-hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
-hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
-
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
-    --prompt "把天空改成壮丽的日落。" --output edited.png \
-    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
-    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
-    --qwen-image-mmproj models/mmproj-BF16.gguf \
-    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
-    --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
-    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
-    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
-    --qwen-image-mmproj models/mmproj-BF16.gguf \
-    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
-    --backend ggml_cuda
        
-
-  
        Wan 2.1 / 2.2（提示词 → 视频、图像 → 视频）
        
-  
        hf download QuantStack/Wan2.2-TI2V-5B-GGUF Wan2.2-TI2V-5B-Q8_0.gguf --local-dir models
-hf download QuantStack/Wan2.2-TI2V-5B-GGUF VAE/Wan2.2_VAE.safetensors --local-dir models
-hf download city96/umt5-xxl-encoder-gguf umt5-xxl-encoder-Q8_0.gguf --local-dir models
-
-# 文本 → 视频
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
-    --prompt "A red fox trotting through falling snow, cinematic" \
-    --video-frames 81 --fps 24 --output out.mp4 --backend ggml_cuda
-# 图像 → 视频（上传的图像作为首帧；仅 Wan 2.2 模型支持）
-dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors \
-    --image first_frame.png --prompt "the camera pushes in as the waves rise" \
-    --output out.mp4 --backend ggml_cuda
-dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Wan2.2-TI2V-5B-Q8_0.gguf \
-    --wan-te models/umt5-xxl-encoder-Q8_0.gguf --wan-vae models/VAE/Wan2.2_VAE.safetensors --backend ggml_cuda
        
-
-  
        多模态支持
        
+  
        哪个更快——每个家族最关键的那一项设置
        
+  
        每个模型家族都有一项主导端到端耗时的设置。第一行是最容易被忽略的一项：在 Wan 上，你下载的是哪个检查点，直接决定一段 5 秒 720p 视频是跑三个半小时还是十七分钟——而且不需要加任何参数。
        
   
        
   
-    
+    
-      
-      
-      
-      
-      
-      
+      
+      
+      
+      
+      
+      
+      
+      
+      
        家族输入说明
        家族快车道实测效果
        
     
        Gemma 4图像 · 视频 · 音频图像 PNG/JPEG/HEIC；视频 MP4（经 OpenCV 以 1 fps 采样）；音频 WAV 16 kHz 单声道 / MP3 / OGG。E4B 投影器：mmproj-gemma-4-E4B-it-Q8_0.gguf。
        Gemma 3图像PNG / JPEG / HEIC。非 gated 4B 投影器：mmproj-model-f16.gguf。
        Qwen 3.5 / 3.6图像动态分辨率视觉编码器；9B / 3.6 仓库使用 mmproj-F16.gguf。
        Mistral 3图像Pixtral 视觉编码器。投影器：mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf。
        Muse-Glimmer图像50 层稀疏窗口 ViT，采用 2D RoPE 与 2×2 像素混洗；图像按 llama.cpp 相同的方式选定网格后直接拉伸（不填充、不切块）。投影器：mmproj-Muse-Glimmer-30B-Q8_0.gguf。
        Nemotron-H (Omni)图像RADIO / v2_vl ViT 编码器。传入匹配的 --mmproj；图像 token 在 <image> 占位处展开。当前 GGUF 分发未附真实音频推理需要的 Parakeet mmproj。
        Wan 视频加载步数蒸馏检查点——DiT 文件名中含 Turbo、distill、Lightning、lightx2v、FastWan、-dmd 或 …-4steps-… 即被自动识别，无需任何参数100 次 DiT 前向 → 4 次，关闭引导。同一条 1088×832×121 帧图生视频请求，M5 Pro / ggml_metal：约 3 小时 30 分 → 17 分 30 秒
        Wan 视频--cfg-cache-stride 2 / 3——仅适用于未蒸馏的基础检查点50 步下 1.30× / 1.43×（100 次前向只跑 77 / 70 次）。属于近似方法；在本就无引导的蒸馏检查点上没有意义
        Qwen-Image-Edit用 --qwen-image-lora 加载 Lightning 蒸馏 LoRA基础配方 30 步、cfg 2.5，即 60 次 DiT 前向 → 4–8 次、cfg 1.0。544×1184、4 步的热态编辑：40.44 s，对比 stable-diffusion.cpp 的 48.16 s
        DeepSeek V4用 --draft-model 加载 DSpark 块级草稿器（仅 cuda / ggml_cuda）4×A40、200 个贪心 token：decode 26.0 → 34.0 tok/s（cuda）、26.4 → 37.1 tok/s（ggml_cuda），接受率 69%；5 轮对话中可达 1.5–2.0×。输出保持不变
        Muse-Glimmer用 --draft-model 加载 DFlash 块级草稿器（不要传任何采样参数，它需要纯贪心）RTX PRO 6000、128 个贪心 token：60 token 提示下 35.0 → 50.9 tok/s，2 050 token 下 24.9 → 43.5。在 Apple Silicon 上目前并不划算（20.7 → 13.9 tok/s），那里请用普通 decode
        Qwen 3.6 / Gemma 4在 TensorSharp.Server 上加 --mtp-spec——Qwen 3.6 用 -MTP- GGUF 内嵌的 NextN 块，Gemma 4 用独立的 --mtp-draft-model 草稿仅在单独（无并发）序列上启用，且仅限 GGML CUDA / GGML Metal 与直连 cuda 后端；CPU / MLX 上不启用。可用 --mtp-draft（默认 8）与 --mtp-pmin（默认 0.75）调节
        所有文本家族选对 --backend：NVIDIA 用 ggml_cuda，Apple Silicon 用 ggml_metal，CPU 用 ggml_cpu（而非 cpu）RTX 3080 Laptop、gemma-4-26B-A4B QAT：ggml_cuda decode 78.7 tok/s，直连 cuda 后端 35.3；prefill 1832 对 128。Apple Silicon、Muse-Glimmer-30B：ggml_metal prefill 413.6，mlx 仅 29.0
        装不下的 MoE--n-cpu-moe N / --cpu-moe——把前 N 层的路由专家权重留在系统内存以 decode 换显存；但只要能避免显存溢出就是净赚：gpt-oss-20b 在 16 GB 卡上 16.2 → 2.9 GB，--n-cpu-moe 12 把 WDDM 换页导致的 0.3 tok/s 变成 25.4
        多 GPU在直连 cuda 与 GGML CUDA / Vulkan 上使用 --tp N（环境变量 TENSORSHARP_TP_DEGREE）Muse-Glimmer-30B UD-IQ2_XXS、2× RTX PRO 4000：prefill 1171 → 1569（1.34×），decode 40.2 → 63.2 tok/s（1.57×）。它还能跑单卡完全装不下的模型
        
             
   
        
   
        
-  
        通过 CLI（--image、--video、--audio、--pdf）、Web UI 上传，或 HTTP API（Ollama 用 base64 images 数组，OpenAI 用 image_url data URI）发送文件。数字版 PDF 会提取文本；扫描版 PDF 会转换为页面图像并需要视觉模型。
        
-
-  
        图像编辑（Qwen-Image-Edit）
        
-  
        Qwen-Image-Edit 接收提示词 + 一张输入图像并返回编辑后的图像——这是与上面文本 LLM 不同的输出模态。所加载的 qwen_image GGUF 仅是 MMDiT（多模态扩散 Transformer）；TensorSharp 会在其旁解析两个伴随 GGUF：
        
-  
          
-    
        • Qwen-Image VAE —— 图像 ↔ 16 通道潜变量（原始 .safetensors 也可用）。
          • 
-    
        • Qwen2.5-VL-7B 文本编码器 —— 提示词 → 3584 维条件，外加可选的 mmproj 视觉塔以实现图像接地编辑。
          • 
-  
        
-  
        将伴随文件放在 DiT GGUF 旁，或用 TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ 指定（CLI：--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj）。流水线对参考图做 VAE 编码、构建条件、运行带参考潜变量拼接的 FlowMatch-Euler true-CFG 去噪循环，再 VAE 解码回像素。整个 60 块 DiT 前向被 CUDA 图捕获，flash 注意力默认开启；除非固定宽/高，否则目标面积按设备 VRAM 预算自动钳制。
        
-  
        可选的 Lightning 蒸馏 LoRA（--qwen-image-lora / TS_QWEN_IMAGE_LORA，.safetensors）会在加载时合并进 DiT——自动推导减少后的去噪步数（如 4 或 8），并将 CFG 切换为 1.0，跳过负向分支。去噪循环还可由整步 DiT 缓存（EasyCache 风格；TS_QWEN_DIT_CACHE_MODE = easycache / fbc）进一步加速：当某个 DiT 步的输出可由上一步预测时直接跳过。
        
-  
        可从 CLI（--image + --prompt）、Web UI 图像编辑流程（含实时去噪预览），或在进程内从 C# 通过 QwenImageModel.EditImage() 运行。完整细节见仓库的 docs/models/qwenimage_zh-cn.md 卡片。
        
-
-  
        视频生成（Wan 2.1 / 2.2）
        
-  
        Wan 把提示词——在 Wan 2.2 检查点上还可以加一张上传的首帧图像——变成 H.264 MP4。--model 指定的 GGUF 只是 DiT，TensorSharp 会在其旁边解析两个伴随文件（放在 VAE/、HighNoise/ 或 LowNoise/ 子目录中同样可以）：
        
-  
          
-    
        • UMT5-XXL 文本编码器 —— 提示词 → 条件向量（--wan-te / TS_WAN_TE）。去噪开始前即从显存释放。
          • 
-    
        • 因果 3D 视频 VAE —— 潜变量 ↔ 帧（--wan-vae / TS_WAN_VAE）。Wan 2.1 与 A14B 使用 wan_2.1_vae，TI2V-5B 使用 Wan2.2_VAE。
          • 
-  
        
-  
        
-  
-    
-    
-      
-      
-      
-    
-  
        系列潜空间模式说明
        Wan 2.1 T2V（1.3B / 14B）16 通道，8×8×4文本 → 视频单个 DiT
        Wan 2.2 TI2V-5B48 通道，16×16×4文本 → 视频、图像 → 视频稠密 5B，24 fps，可出 720p
        Wan 2.2 A14B（T2V / I2V）16 通道（I2V 输入 36 通道）文本 → 视频、图像 → 视频两个 14B 专家在时间步边界切换
        
-  
        
-  
        每个去噪步是一张常驻权重的 ggml 图（CUDA 图捕获、flash attention；TI2V 图生视频还带 per-token 时间步调制），视频 VAE 的编码与解码各是一张图。各阶段按顺序交接显存——先文本编码器，再 DiT，最后 VAE——因此峰值约为 max(TE, DiT + attention, VAE)：TI2V-5B 在 16 GB GPU 上 8 分钟内生成 81 帧 480p 图生视频，A14B 的两个专家也能在同一张卡上顺序执行。数值上与 diffusers 对齐（DiT 余弦相似度 > 0.995，VAE 编码器 > 0.999，解码 59.9 dB PSNR），同等负载下 Wan 2.1 端到端比 stable-diffusion.cpp 快 6.0×。
        
-  
        可通过 CLI（--prompt、可选 --image、--video-frames、--fps、--flow-shift、--sampler、--negative-prompt）、HTTP API（/v1/videos/generations）或 Web UI 聊天上传图像来驱动。完整说明见仓库中的 docs/models/wan_zh-cn.md 卡片。
        
-
-  
        思考 / 推理模式
        
-  
        具备思考能力的模型（Qwen 3、Qwen 3.5/3.6、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、Muse-Glimmer）会在最终答案前产生结构化的思维链。思考内容与可见回复分离，便于客户端显示或隐藏。
        
-  
          
-    
        • Qwen 3 / Qwen 3.5/3.6 / Nemotron-H —— <think>…</think> 标签。
          • 
-    
        • Gemma 4 —— <|channel>thought …<channel|> 标签。
          • 
-    
        • GPT OSS —— Harmony 格式：<|channel|>analysis 用于思考，<|channel|>final 用于答案。
          • 
-    
        • DeepSeek V4 —— <think>…</think> 标签；不显式开启思考时聊天模板会立即闭合该块，因此推理是按需启用的。
          • 
-    
        • Muse-Glimmer —— 聊天模板输出的 assistant to=self 推理通道。
          • 
-  
        
-  
        通过 --think（CLI）、"think": true（Ollama API / Web UI）或浏览器中的思考开关启用。响应会单独暴露推理 —— 例如 Ollama 聊天响应中的 message.thinking。
        
-
-  
        工具调用 / 函数调用
        
-  
        模型可以调用用户自定义工具，并参与多轮工具调用对话。将工具定义为 JSON，通过 --tools（CLI）或 tools 参数（API）传入。每个架构使用各自的线格式，但输出解析器都会将调用提取为结构化 tool_calls：
        
-  
          
-    
        • Qwen 3 / Nemotron-H —— <tool_call>{"name": …, "arguments": {…}}</tool_call>
          • 
-    
        • Qwen 3.5 / 3.6 —— 同样是 <tool_call> 块，但内容为 XML：<function=NAME><parameter=key>value</parameter></function>（JSON 形式仍被接受）。
          • 
-    
        • Gemma 4 —— <|tool_call>call:function_name{args}<tool_call|>
          • 
-    
        • GPT OSS (Harmony) —— 工具以 TypeScript 命名空间声明；调用在 commentary 通道发出。
          • 
-    
        • DeepSeek V4 —— DSML 标记：系统提示词负责讲解语法并携带每个函数的 JSON schema，模型以 <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>… 作答。
          • 
-    
        • Muse-Glimmer —— 聊天模板声明的 ATEM XML 标记；解析器会把调用提取成同样的结构化 tool_calls。
          • 
-  
        
-  
        完整的请求/响应示例与续接循环见 通过 HTTP 进行工具调用。
        
+  
        Wan、Qwen-Image-Edit 与 DiffusionGemma 构造时不接受张量并行度，因此 --tp 对它们无效。Wan 也是唯一会直接拒绝某个后端的家族：不支持 mlx。
        
 
   
        
     ← 计算后端
-    下一篇：命令行 →
+    下一篇：模型下载 →
   
        
 
 
diff --git a/website/overview.html b/website/overview.html
index 903a6412..d4801c6d 100644
--- a/website/overview.html
+++ b/website/overview.html
@@ -74,7 +74,7 @@ 
        Project structure
        
     
       TensorSharp.Core/Tensor library, ops, memory, device abstraction, CPU SIMD/quantized kernels.
       TensorSharp.Runtime/GGUF, tokenizers, templates, sampling; Paged/ KV primitives and Scheduling/ the inference engine + MTP core.
-      TensorSharp.Models/Models/<Family>/One folder per architecture (DeepSeek4, Gemma3/4, Qwen3/35, GptOss, Nemotron, Mistral3, DiffusionGemma, QwenImage), each with a legacy and a batched forward — except DeepSeek4, which uses whole-model executors instead.
+      TensorSharp.Models/Models/<Family>/One folder per architecture (DeepSeek4, Gemma3/4, Qwen3/35, GptOss, Nemotron, Mistral3, MuseGlimmer, DiffusionGemma, QwenImage, WanVideo). The autoregressive text families carry a legacy and a batched forward; DeepSeek4 uses whole-model executors instead, and the media families (DiffusionGemma, QwenImage, WanVideo) run diffusion pipelines rather than a decode loop.
       TensorSharp.GGML.Native/The native C++ bridge to ggml (matmul, fused transformer kernels, paged attention, MoE, Mamba2, GatedDeltaNet, diffusion).
       TensorSharp.Server/ASP.NET Core server: program bootstrap, model service, inference-engine host, chat pipeline, telemetry.
       docs/Per-model architecture cards, paged-attention deep dive, env-var matrix, benchmark matrix.
@@ -87,10 +87,11 @@ 
        Current status & capabilities
        
   
-      
+      
-      
+      
+      
diff --git a/website/overview_zh-cn.html b/website/overview_zh-cn.html
index 0d8feabc..d518692b 100644
--- a/website/overview_zh-cn.html
+++ b/website/overview_zh-cn.html
@@ -74,7 +74,7 @@ 
        项目结构
        
-      
+      
@@ -87,10 +87,11 @@ 
        当前状态与能力
        
     
        AreaStatus
        
     
        Model familiesDeepSeek V4 Flash (deepseek4), Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family (qwen35, qwen35moe, qwen3next), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), Mistral 3, Muse-Glimmer (muse-glimmer), Qwen-Image-Edit (qwen_image image editing), and Wan 2.1 / 2.2 (wan video generation).
        Model familiesTwelve, dispatched by ModelBase.Create() from the GGUF's general.architecture: DeepSeek V4 Flash (deepseek4), Gemma 3 (gemma3), Gemma 4 (gemma4), DiffusionGemma (diffusion-gemma, diffusion_gemma), Qwen 3 (qwen3, and Qwen2 / Qwen2.5-VL checkpoints as text-only chat via qwen2, qwen2vl, qwen2_vl), Qwen 3.5/3.6-family (qwen35, qwen35moe, qwen3next), GPT OSS (gptoss, gpt-oss), Nemotron-H incl. Nemotron 3 Nano Omni (nemotron_h, nemotron_h_moe), Mistral 3 (mistral3), Muse-Glimmer (muse-glimmer, muse_glimmer), Qwen-Image-Edit image editing (qwen_image, qwen-image), and Wan 2.1 / 2.2 video generation (wan, wan2.1, wan2.2). → Supported architectures
        
       
        Inference hostsCLI, interactive REPL, ASP.NET Core web UI, Ollama-style API, OpenAI Chat Completions-style API.
        
       
        BackendsPure C# CPU, direct CUDA/cuBLAS (cuda), MLX Metal (mlx), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU.
        MultimodalGemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer image input. PDF documents via Web UI upload and CLI --pdf (text extracted and inlined; scanned pages rendered as images for vision models). Media out: Qwen-Image-Edit (image) and Wan 2.1 / 2.2 (H.264 MP4 video, text → video and image → video). → Video generation
        Speed leversThe choice that matters most is usually the checkpoint, not the flag. A step-distilled Wan DiT (Turbo / Lightning / lightx2v / FastWan / …-4steps-… in the file name) is detected at load and runs 4 guidance-free DiT passes instead of the base recipe's 100: the same 1088×832×121-frame image-to-video request measured ≈3 h 30 m on the base Wan2.2-TI2V-5B and 17 m 30 s on the Turbo one (M5 Pro, ggml_metal). Qwen-Image-Edit's Lightning LoRA (--qwen-image-lora) likewise replaces 30 steps at CFG 2.5 with 4–8 at CFG 1.0. For text models the levers are the backend (ggml_cuda on NVIDIA, ggml_metal on Apple Silicon, ggml_cpu rather than the pure-managed cpu), whole-model fused decode graphs (GPT OSS 24 → 154 tok/s on an A40), speculative decoding, --tp N, and --n-cpu-moe for a model that would otherwise not fit. → Models · Benchmarks
        MultimodalGemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer image input. PDF documents via Web UI upload and CLI --pdf (text extracted and inlined; scanned pages rendered as images for vision models). Media out: Qwen-Image-Edit (image) and Wan 2.1 / 2.2 (H.264 MP4 video, text → video and image → video). → Video generation
        
       
        Continuous batchingvLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (on by default; opt-out via --no-continuous-batching). DeepSeek V4 serves through its own native per-sequence slots on the same engine.
        
       
        Speculative decodingMTP / NextN draft heads for Qwen 3.6 (embedded) and Gemma 4 (separate draft GGUF); DSpark block drafting for DeepSeek V4 and DFlash block drafting for Muse-Glimmer (separate drafter GGUF via --draft-model on the cuda / ggml_cuda backends). Verification is greedy against the target, so the emitted stream is the plain-greedy stream. Off by default — opt in with the server's --mtp-spec flag (env TS_MTP_SPEC), or by passing --draft-model on the CLI. → DSpark
        
       
        Tensor parallelismMegatron-LM column/row-parallel TP on the direct cuda backend and on GGML CUDA / Vulkan (--tp N / TENSORSHARP_TP_DEGREE, on both the CLI and the server), plus multi-node distributed TP over a peer-to-peer TCP mesh (--tp-node-id / --tp-peers), for every autoregressive architecture. Fused per-rank graphs and MoE expert parallelism make --tp 2 decode faster than one GPU on Gemma 4. Optional Redis-backed shared KV cache and Responses API store. → Multi-GPU & Multi-Node
        
             
       
        TensorSharp.Core/张量库、运算、内存、设备抽象、CPU SIMD / 量化内核。
        
       
        TensorSharp.Runtime/GGUF、分词器、模板、采样；Paged/ KV 原语与 Scheduling/ 推理引擎 + MTP 核心。
        TensorSharp.Models/Models/<Family>/每个架构一个文件夹（DeepSeek4、Gemma3/4、Qwen3/35、GptOss、Nemotron、Mistral3、DiffusionGemma、QwenImage），各含一个 legacy 与一个批量前向——DeepSeek4 例外，它使用整模型执行器。
        TensorSharp.Models/Models/<Family>/每个架构一个文件夹（DeepSeek4、Gemma3/4、Qwen3/35、GptOss、Nemotron、Mistral3、MuseGlimmer、DiffusionGemma、QwenImage、WanVideo）。自回归文本家族各含一个 legacy 与一个批量前向；DeepSeek4 例外，它使用整模型执行器；媒体家族（DiffusionGemma、QwenImage、WanVideo）走的是扩散管线而非 decode 循环。
        
       
        TensorSharp.GGML.Native/到 ggml 的原生 C++ 桥接（matmul、融合 transformer 内核、分页注意力、MoE、Mamba2、GatedDeltaNet、扩散）。
        
       
        TensorSharp.Server/ASP.NET Core 服务器：程序引导、模型服务、推理引擎宿主、聊天流水线、遥测。
        
       
        docs/各模型架构卡、分页注意力深入解析、环境变量矩阵、基准矩阵。
        
   
        
-      
+      
-      
+      
+      
diff --git a/website/server.html b/website/server.html
index 00a37637..4327bfe4 100644
--- a/website/server.html
+++ b/website/server.html
@@ -68,9 +68,40 @@ 
        Web UI features
      • Message editing and deletion with regeneration from any point in the conversation.
      • DiffusionGemma denoising previews when a diffusion-gemma GGUF is hosted (the whole assistant message is replaced on each step, then finalized).
      • Qwen-Image-Edit flow when a qwen_image DiT is hosted: attach an image, type the edit instruction, and temporary edited images (live denoising previews, up to 8 frames) refresh in place until the final PNG appears with a download link.
        • 
+    
      • Wan video flow when a wan DiT is hosted: type the prompt (optionally attach an image to use as the first frame) and the browser streams per-step denoise progress over SSE until the MP4 appears with a download link. The prompt and that image are the only two things the browser sends — see Video generation below.
      • Free scrolling — read earlier replies while new tokens stream; auto-scroll resumes at the bottom.
        • 
 
+  
        Video generation (Wan): what the UI sets, and what only the API sets
        
+  
        Host a wan DiT and the Web UI turns into a video generator. It is worth knowing exactly how little of the request the browser controls, because a Wan job can run for minutes or for hours and the browser gives you no way to change that mid-flight.
        
+  
        ⏱️
        The browser sends only two fields: prompt and (when you attach an image) imagePath for the first frame. Frame count, fps, resolution, denoise steps, CFG, seed, flow shift, sampler, negative prompt and the guidance cache are not exposed in the UI — every one of them comes from the startup flags or the model's own recipe. If you need to set them per request, call POST /api/video-generate or /v1/videos/generations directly.
        
+  
        What the process decides at startup, and how:
        
+  
        
+  
        
     
        领域状态
        
     
        模型家族DeepSeek V4 Flash（deepseek4）、Gemma 3/4、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family（qwen35、qwen35moe、qwen3next）、GPT OSS、Nemotron-H（含 Nemotron 3 Nano Omni）、Mistral 3、Muse-Glimmer（muse-glimmer），以及 Qwen-Image-Edit（qwen_image 图像编辑）与 Wan 2.1 / 2.2（wan 视频生成）。
        模型家族共 12 个，由 ModelBase.Create() 依据 GGUF 的 general.architecture 分发：DeepSeek V4 Flash（deepseek4）、Gemma 3（gemma3）、Gemma 4（gemma4）、DiffusionGemma（diffusion-gemma、diffusion_gemma）、Qwen 3（qwen3；Qwen2 / Qwen2.5-VL 权重亦可经 qwen2、qwen2vl、qwen2_vl 以纯文本对话方式加载）、Qwen 3.5/3.6-family（qwen35、qwen35moe、qwen3next）、GPT OSS（gptoss、gpt-oss）、Nemotron-H 含 Nemotron 3 Nano Omni（nemotron_h、nemotron_h_moe）、Mistral 3（mistral3）、Muse-Glimmer（muse-glimmer、muse_glimmer）、Qwen-Image-Edit 图像编辑（qwen_image、qwen-image），以及 Wan 2.1 / 2.2 视频生成（wan、wan2.1、wan2.2）。→ 支持的架构
        
       
        推理宿主CLI、交互式 REPL、ASP.NET Core Web UI、Ollama 式 API、OpenAI Chat Completions 式 API。
        
       
        后端纯 C# CPU、直接 CUDA/cuBLAS（cuda）、MLX Metal（mlx）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C#——都会把权重按层切分到所有可见 GPU。
        多模态Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 图像输入。PDF 文档可经 Web UI 上传或 CLI --pdf 传入（提取文本并内联；扫描页渲染为图像交给视觉模型）。媒体输出：Qwen-Image-Edit（图像）与 Wan 2.1 / 2.2（H.264 MP4 视频，文本 → 视频与图像 → 视频）。→ 视频生成
        提速手段真正决定快慢的往往是选哪个权重，而不是加哪个参数。步数蒸馏的 Wan DiT（文件名含 Turbo / Lightning / lightx2v / FastWan / …-4steps-…）在加载时被自动识别，只跑 4 次无引导 DiT 前向，而非基础配方的 100 次：同一个 1088×832×121 帧的图生视频请求，基础 Wan2.2-TI2V-5B 实测约 3 小时 30 分，Turbo 权重为 17 分 30 秒（M5 Pro、ggml_metal）。Qwen-Image-Edit 的 Lightning LoRA（--qwen-image-lora）同理，把 30 步 CFG 2.5 换成 4–8 步 CFG 1.0。文本模型这边的手段是：选对后端（NVIDIA 用 ggml_cuda，Apple Silicon 用 ggml_metal，CPU 推理用 ggml_cpu 而非纯托管的 cpu）、整模型融合 decode 计算图（GPT OSS 在 A40 上 24 → 154 tok/s）、推测解码、--tp N，以及让放不下的模型跑起来的 --n-cpu-moe。→ 模型 · 基准测试
        多模态Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 图像输入。PDF 文档可经 Web UI 上传或 CLI --pdf 传入（提取文本并内联；扫描页渲染为图像交给视觉模型）。媒体输出：Qwen-Image-Edit（图像）与 Wan 2.1 / 2.2（H.264 MP4 视频，文本 → 视频与图像 → 视频）。→ 视频生成
        
       
        连续批处理vLLM 式分页 KV 缓存、块哈希前缀共享、迭代级调度器（默认开启；可用 --no-continuous-batching 关闭）。DeepSeek V4 在同一引擎上通过其原生 per-sequence slot 提供服务。
        
       
        推测解码Qwen 3.6（内嵌）与 Gemma 4（独立草稿 GGUF）的 MTP / NextN 草稿头；DeepSeek V4 的 DSpark 与 Muse-Glimmer 的 DFlash 块级起草（在 cuda / ggml_cuda 后端上通过 --draft-model 加载独立草稿 GGUF）。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致。默认关闭——服务端用 --mtp-spec（环境变量 TS_MTP_SPEC）启用，CLI 直接传 --draft-model 即可。→ DSpark
        
       
        张量并行Direct cuda 后端与 GGML CUDA / Vulkan 后端上的 Megatron-LM 列/行并行 TP（--tp N / TENSORSHARP_TP_DEGREE，CLI 与服务端均支持），以及基于点对点 TCP 网格的多节点分布式 TP（--tp-node-id / --tp-peers），覆盖全部自回归架构。融合的按 rank 计算图与 MoE 专家并行让 Gemma 4 上 --tp 2 的 decode 快于单卡。可选 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ 多 GPU 与多节点
        
     
     
     
     
   
        
+    
+    
+      
+      
+      
+      
+    
+  
        What decides itSettingNotes
        --model (fixed for the process)Denoise passes, guidanceThe single biggest cost factor. A base Wan2.2-TI2V-5B runs the official 50 steps × 2 CFG passes = 100 DiT passes; a step-distilled Turbo / Lightning / FastWan checkpoint is detected from the file name and runs 4 guidance-free passes. On an M5 Pro at 1088×832 × 121 frames that is ≈3 h 30 m versus 17 m 30 s for the identical request. → measurements · downloads
        --video-frames NDefault frame countA server-wide default, not a cap: a request that carries frames overrides it. Snapped to 4k+1. With the flag omitted the model recipe applies — 49 frames for Wan2.2-TI2V, 33 otherwise.
        --fps NDefault playback rateAlso a default, overridden independently by a request's fps. Omitted, the recipe gives 24 fps for Wan2.2-TI2V and 16 otherwise. FPS changes playback, not the amount of work.
        The request (API only)Everything elsewidth, height, frames, steps, cfg, cfg2, seed, fps, flowShift, sampler, negativePrompt, cfgCacheStride.
        
+  
+  
        The Web-UI-shaped endpoint takes the fields under their JSON names; the same body works on /api/video-generate (one MP4 back) and /api/video-generate/stream (SSE denoise progress). Generations are serialized process-wide, so one job runs at a time:
        
+  
        curl -s http://localhost:5000/api/video-generate \
+  -H "Content-Type: application/json" \
+  -d '{
+        "prompt": "the camera pushes in as the waves rise",
+        "width": 736, "height": 544,
+        "frames": 81, "fps": 24,
+        "steps": 30, "seed": 42,
+        "flowShift": 5.0, "sampler": "unipc",
+        "negativePrompt": "static, blurry",
+        "cfgCacheStride": 2
+      }'
        
+  
        An "image" field carries a base64 first frame (a data:…;base64, prefix is accepted); "imagePath" is the Web UI's form and must reference a file already returned by /api/upload. The OpenAI-shaped POST /v1/videos/generations takes the same parameters but spells the size as "size": "832x480" and the negative prompt as "negative_prompt". Full request/response shapes are on the HTTP API page.
        
+  
        💡
        Before starting a long job, size it. Cost is dominated by DiT tokens (latent_frames × (h/2) × (w/2)) and self-attention is O(tokens²), so frames and frame area matter far more than steps. In order of effect: use a step-distilled checkpoint, cut frames, cut frame area (but not below ~0.3 MP), cut steps on base checkpoints, then "cfgCacheStride": 2 or 3 for 1.30× / 1.43×. 480p (≈0.4 MP) is a resolution Wan is trained at, so it is a real output mode rather than a degraded one.
        
+
   
        Configuration file (--config)
        
   
        Instead of a long command line, pass a JSON file with --config. The CLI reads the same format. Command-line options always win—file values are applied first, then anything you also pass on the command line overrides them, so one file can be reused across hosts while you override just what differs. Repeat --config to layer files (later files win). Comments and trailing commas are allowed.
        
   
        # Read all options from a file; override just the backend for this host
@@ -122,7 +153,11 @@ 
        Server options
        
       --paged-kv-block-size / --paged-kv-ram-mb / --paged-kv-ssd-dir / --paged-kv-ssd-mbLegacy standalone paged-KV tuning. Use the TS_SCHED_* engine settings below for current server requests.
       --paged-kv-quant-bits <b>Legacy standalone TurboQuant setting; this server flag accepts 0, 4, or 8 (the runtime env var and CLI additionally accept 2).
       --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path>Override the resolved Qwen-Image-Edit companion GGUFs (VAE / Qwen2.5-VL text encoder / mmproj).
-      --qwen-image-lora <path>Qwen-Image-Edit Lightning distillation LoRA (.safetensors) merged into the DiT; auto-derives the denoise step count and switches CFG to 1.0.
+      --qwen-image-lora <path>Qwen-Image-Edit Lightning distillation LoRA (.safetensors), applied as a runtime side-path beside the untouched quantized base weights (not merged). Auto-derives the denoise step count from the file name and switches CFG to 1.0, taking the default 60 DiT forwards down to 4–8. Env: TS_QWEN_IMAGE_LORA.
+      --video-frames <N>Wan: default output frame count when a request omits frames, snapped to 4k+1. Model default: 33, or 49 for Wan2.2-TI2V. A request value overrides it — this is a default, not a cap.
+      --fps <N>Wan: default MP4 playback rate when a request omits fps. Model default: 16, or 24 for Wan2.2-TI2V. FPS changes playback rate, not generation work.
+      --wan-vae <path>Wan video VAE (wan_2.1_vae.safetensors, or Wan2.2_VAE.safetensors for TI2V-5B). Default: same-directory scan next to the DiT, VAE/ subfolders included. Env: TS_WAN_VAE.
+      --wan-te <path>UMT5-XXL text-encoder GGUF for Wan. Default: same-directory scan (env TS_WAN_TE). Wan 2.2 A14B also auto-resolves its second high/low-noise expert GGUF by name (TS_WAN_DIT2).
     
   
   
diff --git a/website/server_zh-cn.html b/website/server_zh-cn.html
index 93edefaa..68b6b2a4 100644
--- a/website/server_zh-cn.html
+++ b/website/server_zh-cn.html
@@ -68,9 +68,40 @@ 
        Web UI 功能
        
     
      • 消息编辑与删除，并可从对话中任意位置重新生成。
        • 
     
      • 当托管 diffusion-gemma GGUF 时，提供 DiffusionGemma 去噪预览（每步替换整条助手消息，最后定稿）。
        • 
     
      • 托管 qwen_image DiT 时提供 Qwen-Image-Edit：上传图像并输入编辑指令后，最多 8 帧实时去噪预览会原位刷新，最终 PNG 提供下载链接。
        • 
+    
      • 托管 wan DiT 时提供 Wan 视频生成：输入提示词（可选附一张图像作为首帧），浏览器通过 SSE 实时显示逐步去噪进度，直到 MP4 出现并提供下载链接。浏览器只发送这两样东西 —— 详见下文视频生成。
        • 
     
      • 自由滚动 —— 在新 token 流式输出时阅读较早的回复；自动滚动会在回到底部时恢复。
        • 
   
      
 
+  
      视频生成（Wan）：网页界面能设什么，什么只能靠 API
      
+  
      托管一个 wan DiT，网页界面就变成了一个视频生成器。有必要清楚地知道浏览器实际控制了请求中多小的一部分：一次 Wan 任务可能跑几分钟，也可能跑几小时，而浏览器在任务开始后无法再改变这一点。
      
+  
      ⏱️
      浏览器只发送两个字段：prompt，以及（当你附加图像时）作为首帧的 imagePath。帧数、fps、分辨率、去噪步数、CFG、seed、flow shift、采样器、负向提示词与引导缓存都没有在界面上暴露 —— 它们全部来自启动参数或模型自身的配方。若要按请求设置这些值，请直接调用 POST /api/video-generate 或 /v1/videos/generations。
      
+  
      进程在启动时就已决定的部分：
      
+  
      
+  
+    
+    
+      
+      
+      
+      
+    
+  
      由什么决定设置项说明
      --model（进程内固定）去噪前向次数、引导最大的单项成本因素。基础的 Wan2.2-TI2V-5B 走官方配方 50 步 × 2 次 CFG = 100 次 DiT 前向；步数蒸馏的 Turbo / Lightning / FastWan 权重会从文件名被识别出来，只跑 4 次无引导前向。在 M5 Pro 上、1088×832 × 121 帧的同一个请求，这就是约 3 h 30 m 与 17 m 30 s 的差别。→ 实测数据 · 下载
      --video-frames N默认帧数这是服务端范围的默认值，不是上限：请求里带 frames 就会覆盖它。数值会对齐到 4k+1。不传该参数时按模型配方来 —— Wan2.2-TI2V 为 49 帧，其余为 33 帧。
      --fps N默认播放帧率同样是默认值，可被请求中的 fps 独立覆盖。不传时配方给出 Wan2.2-TI2V 24 fps、其余 16 fps。fps 只改变播放速度，不改变计算量。
      请求本身（仅 API）其余全部width、height、frames、steps、cfg、cfg2、seed、fps、flowShift、sampler、negativePrompt、cfgCacheStride。
      
+  
      
+  
      Web UI 形态的接口按上述 JSON 字段名接收参数；同一个请求体在 /api/video-generate（返回一个 MP4）与 /api/video-generate/stream（SSE 去噪进度）上都可用。视频生成在进程内串行执行，同一时刻只跑一个任务：
      
+  
      curl -s http://localhost:5000/api/video-generate \
+  -H "Content-Type: application/json" \
+  -d '{
+        "prompt": "the camera pushes in as the waves rise",
+        "width": 736, "height": 544,
+        "frames": 81, "fps": 24,
+        "steps": 30, "seed": 42,
+        "flowShift": 5.0, "sampler": "unipc",
+        "negativePrompt": "static, blurry",
+        "cfgCacheStride": 2
+      }'
      
+  
      "image" 字段可直接携带 base64 首帧（允许带 data:…;base64, 前缀）；"imagePath" 是 Web UI 使用的形式，必须引用 /api/upload 已经返回过的文件。OpenAI 形态的 POST /v1/videos/generations 接收同样的参数，但尺寸写作 "size": "832x480"、负向提示词写作 "negative_prompt"。完整的请求/响应结构见 HTTP API 页。
      
+  
      💡
      开跑长任务之前，先估算规模。开销主要由 DiT token 数决定（latent_frames × (h/2) × (w/2)），而自注意力是 O(token²)，因此帧数与帧面积的影响远大于步数。按效果排序：换用步数蒸馏权重、减少帧数、减小帧面积（但不要低于约 0.3 MP）、在基础权重上减少步数，最后是 "cfgCacheStride": 2 或 3（1.30× / 1.43×）。480p（约 0.4 MP）是 Wan 训练过的分辨率，属于真正的输出档位，而非降级模式。
      
+
   
      配置文件（--config）
      
   
      可以用 --config 传入一个 JSON 文件，取代冗长的命令行；CLI 读取相同格式。命令行参数始终优先——先应用文件中的值，命令行上再次给出的参数会覆盖它们，因此可以在多台主机上复用同一个文件，只覆盖需要变化的部分。--config 可重复以叠加多个文件（后者优先）。允许注释与尾随逗号。
      
   
      # 从文件读取全部参数；本次仅覆盖后端
@@ -122,7 +153,11 @@ 
      服务器选项
      
       --paged-kv-block-size / --paged-kv-ram-mb / --paged-kv-ssd-dir / --paged-kv-ssd-mb旧的独立 paged-KV 调参。当前服务请求请使用下方 TS_SCHED_* 引擎设置。
       --paged-kv-quant-bits <b>旧的独立 TurboQuant 设置；服务端参数接受 0、4 或 8（运行时环境变量与 CLI 还接受 2）。
       --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path>覆盖解析到的 Qwen-Image-Edit 伴随 GGUF（VAE / Qwen2.5-VL 文本编码器 / mmproj）。
-      --qwen-image-lora <path>Qwen-Image-Edit Lightning 蒸馏 LoRA（.safetensors），合并进 DiT；自动推导去噪步数，并将 CFG 切换为 1.0。
+      --qwen-image-lora <path>Qwen-Image-Edit Lightning 蒸馏 LoRA（.safetensors），作为运行期旁路挂在原封不动的量化基础权重旁边（不合并）。它从文件名自动推导去噪步数并把 CFG 切换为 1.0，使默认的 60 次 DiT 前向降到 4–8 次。环境变量：TS_QWEN_IMAGE_LORA。
+      --video-frames <N>Wan：请求未带 frames 时的默认输出帧数，会对齐到 4k+1。模型默认 33，Wan2.2-TI2V 为 49。请求值会覆盖它 —— 这是默认值，不是上限。
+      --fps <N>Wan：请求未带 fps 时的默认 MP4 播放帧率。模型默认 16，Wan2.2-TI2V 为 24。fps 只改变播放速度，不改变生成的计算量。
+      --wan-vae <path>Wan 视频 VAE（wan_2.1_vae.safetensors，TI2V-5B 用 Wan2.2_VAE.safetensors）。默认在 DiT 同目录扫描，包含 VAE/ 子目录。环境变量：TS_WAN_VAE。
+      --wan-te <path>Wan 使用的 UMT5-XXL 文本编码器 GGUF。默认在同目录扫描（环境变量 TS_WAN_TE）。Wan 2.2 A14B 还会按文件名自动解析第二个 high/low noise 专家 GGUF（TS_WAN_DIT2）。