perf(qwen3): skip redundant decode d2h/host copies in greedy path - #1
Open
lyfne123 wants to merge 5 commits into
Open
perf(qwen3): skip redundant decode d2h/host copies in greedy path#1lyfne123 wants to merge 5 commits into
lyfne123 wants to merge 5 commits into
Conversation
Add an end-to-end DeepSeek V4 HTTP completion test that starts the serving command with the documented TP=8 configuration, sends the prompt 'Huawei is' with greedy decoding, and requires the exact six-token completion ' a leading global provider of ICT'. Bound server startup and generation, preserve HTTP error response bodies, and print only a bounded server-log tail on failure. Make process-group shutdown best-effort so stuck or failed waits cannot mask the original accuracy failure. Add focused CPU coverage for HTTP error diagnostics, bounded log reads, and the forced-kill timeout path. Run the DeepSeek guard through task-submit with automatic eight-device allocation outside the configured whitelist, a 20-minute allocation wait, a 30-minute runtime limit, the CI model at /data/l00955553/model/dsv4-flash-w8a8, and the runtime settings required by the documented server configuration. Move both Qwen guards into their own automatic task-submit reservations and add a run-scoped marker so cancelled CI jobs can reap orphaned submissions. Make serving workers close executor-owned runtime resources on every exit path, including initialization and inference failures, so distributed workers do not retain NPU memory after tests finish. Cover normal, exceptional, and repeated cleanup behavior with focused unit tests, and extend the NPU job timeout and dependencies for the HTTP guard.
ci: add DeepSeek V4 generation accuracy guard
…ecode-fwd-restore Adapt Qwen3-14B serving to pypto-lib#747 (decode_layer -> decode_fwd rename)
The fused Qwen3-14B decode kernel does device-side embedding and greedy
sampling, so several per-step host copies are pure waste. Eliminate three,
all host-only (no kernel change, no recompile):
1. Greedy logits d2h. In the device-greedy path the token id is produced
inside the kernel (decode_sampled_ids_buffer), so host code never reads
the [BATCH, VOCAB] f32 logits. Point the kernel's logits `out` slot at a
reusable worker-resident DeviceTensor instead of the shared host buffer;
the runtime only d2h-copies host shared-memory args, so the ~9.7 MB/step
copy (16 x 152064 x 4B) disappears. Non-greedy still uses the host buffer.
2. next_hidden d2h. Decode never returns next_hidden (run_decode always
passes None to _integrated_sample_result), so its Out buffer is scratch.
Route it to a device-resident buffer unconditionally, dropping a
~160 KB/step copy.
3. Dead decode hidden copy. _prepare_decode_inputs built a [B, hidden]
bf16 buffer from batch.hidden_states (a zeros placeholder for
device-embedding executors) that no caller reads and the kernel does not
consume. Stop building/returning it.
DecodeResult.hidden_states / .logits become Optional; serving_worker reads
host logits lazily (only on the non-greedy fallback), so None is safe.
Verified on NPU (a2a3, greedy temperature=0): correct output
("The capital of France is" -> "Paris ... Washington"), 23 decode steps
through the new path, no errors. Run-to-run token divergence was confirmed
to be inherent NPU non-determinism (the same unmodified binary diverges
identically across devices at a near-tie argmax), not caused by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lyfne123
force-pushed
the
perf/qwen3-decode-skip-redundant-d2h
branch
from
July 13, 2026 03:05
9b66241 to
66976a7
Compare
…gression) The decode d2h/host-copy removal is numerically neutral: the kernel's greedy argmax reads identical logit values whether the logits buffer is host- or device-resident, next_hidden is write-only, and the removed decode hidden copy was never consumed. Local reruns pass chunked-prefill 3/3 on this branch and 2/2 on main; the prior CI failure was an inherent chunked-vs-full-prefill near-tie flip on the first token. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lyfne123
force-pushed
the
perf/qwen3-decode-skip-redundant-d2h
branch
from
July 13, 2026 06:12
c85cabe to
94e8975
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on hw-native-sys#68 (needs its
decode_fwdrestore + pypto-lib bump). The fused Qwen3-14B decode kernel does device-side embedding and greedy sampling, so several per-step host copies are pure waste. This removes three, all host-only — no kernel change, no recompile. All Qwen entry points default totemperature=0(greedy device sampling), so this is the hot path in normal runs.16 × 152064 × 4B ≈ 9.7 MBnext_hiddend2h16 × 5120 × 2B ≈ 160 KBhiddencopy[16, 5120]alloc + copy + floatHow
The runtime only device→host copies host shared-memory args; worker-resident
DeviceTensorargs stay on device. So:decode_sampled_ids_buffer(device), and host code never reads the full[BATCH, VOCAB]logits. Point the kernel's logitsoutslot at a reusable worker-residentDeviceTensor(_decode_device_scratch) instead of the shared host buffer. The kernel's internal argmax stays device-local; the d2h disappears. Non-greedy still uses the host buffer and returns real logits.run_decodealways passesNoneto_integrated_sample_result), so itsOutbuffer is scratch. Route it to a device-resident buffer unconditionally._prepare_decode_inputsbuilt a[B, hidden]bf16 buffer frombatch.hidden_states(a zeros placeholder for device-embedding executors) that the kernel does not consume and no caller reads. Stop building/returning it.DecodeResult.hidden_states/.logitsbecomeOptional;serving_workerreads host logits lazily (only on the non-greedy fallback), soNoneis safe.engine._sample_batch_rowsalready short-circuits on device-sampled ids before touching logits.Verification (NPU, a2a3, greedy
temperature=0)"The capital of France is"→"Paris … Washington", 24 tokens over 23 decode steps through the new path, no errors.next_hiddenis write-only.Scope