[feat][model]v0.26.0 camasync dsv4 - #300
Conversation
<!-- markdownlint-disable --> PLEASE FILL IN THE PR DESCRIPTION AND MAKE SURE THE CHECKLIST ITEMS HAVE BEEN CONSIDERED. ## Purpose <!-- What changed and why? Link the issue/RFC/design notes when available. --> ## Issue - Related issue(s): # - Closing keyword, only if fully resolved: Closes # ## Scope - In scope: - Out of scope: ## Implementation Notes <!-- Call out vLLM extension points, plugin-owned classes, compatibility shims, or any behavior that intentionally differs from the original AFD commit. --> ## Test Plan <!-- Commands or manual checks planned. Include CPU-only and GPU-gated coverage separately when relevant. --> ## Test Result <!-- Paste command results, skip reasons, links to GPU validation, or a short explanation if not run. --> ## Docs Impact - Files updated: - If none, reason: --- <details> <summary>Essential PR Checklist</summary> - [ ] Purpose is clear and linked to public context when possible. - [ ] Scope is bounded. - [ ] Compatibility with vLLM v0.26.0 is considered. - [ ] No changes are made to the vLLM source checkout. - [ ] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [ ] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [ ] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [ ] Validation evidence is included, including skipped GPU tests when applicable. - [ ] Documentation impact is stated. </details> --------- Signed-off-by: bjf-frz <frz123db@gmail.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
<!-- markdownlint-disable --> ## Purpose Remove the device-wide NPU synchronization performed after every connector-driven FFN layer output. The per-layer barrier serializes CAM combine sends and prevents async MoE ubatches from overlapping work across layers. ## Issue - Related issue(s): vllm-project#283 - This PR only addresses the per-layer FFN synchronization part of the issue. ## Scope - In scope: - Remove `torch.npu.synchronize()` after `send_ffn_work_item_output()`. - Out of scope: - The connector-step synchronization in `AFDNPUFFNWorker`, which remains required. - Multiprocessing/spawn compatibility changes. - KV connector or PD-disaggregation changes. ## Implementation Notes The change is limited to the plugin-owned `AFDNPUFFNModelRunner`. The worker loop still synchronizes after a complete connector-driven step, preserving the step-level completion boundary while allowing different FFN layers/ubatches to pipeline. ## Test Plan - Run CPU-safe lint and unit-test collection locally. - Deploy the DSV4 Flash AFD Prefill Attention+FFN recipe on an Ascend A3 node without a KV connector. - Send one smoke request followed by 64 concurrent completion requests. - Check Attention and FFN logs for CAM buffer, timeout, traceback, and output errors. ## Test Result - `python3 -m ruff check afd_plugin/v1/worker/npu/ffn_model_runner.py`: passed. - NPU worker/connector unit modules: skipped during collection on the local macOS environment because the NPU runtime is unavailable. - Ascend A3 DSV4 Flash Prefill: - Attention topology: DP2TP4. - FFN topology: EP8. - KV connector disabled. - 64/64 concurrent requests returned HTTP 200 with valid completion payloads. - Latency: p50 17.34 s, max 19.55 s. - No CAM buffer, timeout, or output-structure errors were observed. ## Docs Impact - Files updated: none. - No documentation change is needed for this internal synchronization removal. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.26.0 is considered. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
vllm-project#293) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: ShwStone <haowenshi@outlook.com>
…llm-project#294) <!-- markdownlint-disable --> PLEASE FILL IN THE PR DESCRIPTION AND MAKE SURE THE CHECKLIST ITEMS HAVE BEEN CONSIDERED. ## Purpose This PR includes the following changes: 1. Isolate the DeepSeek V4 Async CAM forward path - Move the model-owned two-stage Async CAM execution from `deepseek_v4.py` into `deepseek_v4_async_cam_forward.py`. - Keep the native vLLM-Ascend forward unchanged for requests without Async CAM ubatch metadata. - Preserve the existing Attention → CAM dispatch → remote FFN → CAM combine schedule. 2. Improve the DSV4 forward integration - Lazily load the Async CAM forward implementation only when the request uses AFD Async MoE ubatching. - Preserve the upstream forward signature and avoid copying the large native DSV4 forward implementation. - Add a regression test covering native-forward and Async-CAM delegation. 3. Remove redundant AFD registration - Remove the repeated `register_afd()` call from model configuration handling. - AFD registration remains handled by the installed plugin entrypoint. ## Issue - Related issue(s): # - Closing keyword, only if fully resolved: Closes # ## Scope - In scope: - Out of scope: ## Implementation Notes <!-- Call out vLLM extension points, plugin-owned classes, compatibility shims, or any behavior that intentionally differs from the original AFD commit. --> ## Test Plan Cross-node AFD deployment: - Attention: DP3 TP8 - FFN: DP8 EP8 - `HCCL_BUFFSIZE=4096` - `max-num-batched-tokens=8192` ## Test Result Request validation: - Input length: 1024 - Output length: 128 - Concurrency: 2 - Requests: 4 - Successful: 4/4 - Failed: 0 - Output throughput: 9.80 tok/s - Total token throughput: 88.18 tok/s - Mean TTFT: 967.81 ms - Mean TPOT: 198.08 ms ## Docs Impact - Files updated: - If none, reason: --- <details> <summary>Essential PR Checklist</summary> - [ ] Purpose is clear and linked to public context when possible. - [ ] Scope is bounded. - [ ] Compatibility with vLLM v0.26.0 is considered. - [ ] No changes are made to the vLLM source checkout. - [ ] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [ ] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [ ] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [ ] Validation evidence is included, including skipped GPU tests when applicable. - [ ] Documentation impact is stated. </details> --------- Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: bjf-frz <frz123db@gmail.com>
|
Please fix precommit |
jiangkuaixue123
left a comment
jiangkuaixue123
left a comment
There was a problem hiding this comment.
This PR should stay scoped to DSV4 over CAMAsync. Please remove the CAMP2P-only model/control-plane path and its tests; those changes are a separate scenario and materially alter the existing CAMP2P topology and common metadata contract. There is also a backend-isolation regression in the patched EngineCore path that would break GPU-only installations.
| prefix=f"{prefix}.mlp", | ||
| ) | ||
| else: | ||
| self.mlp = RemoteFFNProxy(layer_idx=layer_idx) |
There was a problem hiding this comment.
[blocking] Please remove this non-gate DSV4 path from this PR. The CAMAsync validation above requires compute_gate_on_attention=true, so this branch—and the hash-routing input_ids handling later in compute_ffn_output—implements the separate DSV4-over-CAMP2P scenario. Keeping it here is what pulls the CAMP2P-specific control-plane changes into a CAMAsync PR.
| return _AFDFFNKVCacheConfig() | ||
| # ### PATCH END: AFD FFN late-loaded KV cache bypass | ||
|
|
||
| import vllm_ascend.patch.platform.patch_kv_cache_utils # noqa: F401 |
There was a problem hiding this comment.
[blocking] register_afd() installs this EngineCore patch for every backend. All non-FFN engines now execute this import before KV-cache initialization, so a CUDA-only deployment—where vllm_ascend is intentionally absent—fails with ModuleNotFoundError just because the AFD plugin is enabled. Please scope this import to an NPU-only initialization path (and mark it as an AFD-specific patch delta) or remove it from the copied generic upstream path.
There was a problem hiding this comment.
[blocking]
register_afd()installs this EngineCore patch for every backend. All non-FFN engines now execute this import before KV-cache initialization, so a CUDA-only deployment—wherevllm_ascendis intentionally absent—fails withModuleNotFoundErrorjust because the AFD plugin is enabled. Please scope this import to an NPU-only initialization path (and mark it as an AFD-specific patch delta) or remove it from the copied generic upstream path.
Removed the explicit patch_kv_cache_utils import. VLLM_PLUGINS is an exact allowlist, so ascend alone does not enable Ascend general plugins. The deployment now enables them explicitly:
VLLM_PLUGINS=ascend,ascend_kv_connector,ascend_model_loader,ascend_service_profiling,ascend_model,afd
This applies Ascend global patches in spawned EngineCore processes through the intended _ensure_global_patch() path.
|
If any of the review comments are unclear, or if you believe the current implementation is necessary for the CAMAsync design, please feel free to leave the code unchanged for now and reply directly in the corresponding inline review thread. We can discuss the rationale and agree on the intended scope there before you make any changes. |
Signed-off-by: bjf-frz <frz123db@gmail.com>
|
Please fix DCO |
Signed-off-by: bjf-frz <frz123db@gmail.com>
jiangkuaixue123
left a comment
jiangkuaixue123
left a comment
There was a problem hiding this comment.
One additional scope comment on the shared process-group helper.
jiangkuaixue123
left a comment
jiangkuaixue123
left a comment
There was a problem hiding this comment.
One additional scope cleanup.
Signed-off-by: bjf-frz <frz123db@gmail.com>
4b770c9 to
b893b11
Compare
Fixed. |
Signed-off-by: bjf-frz <frz123db@gmail.com>
|
LGTM！ |
2 participants
PLEASE FILL IN THE PR DESCRIPTION AND MAKE SURE THE CHECKLIST ITEMS HAVE BEEN CONSIDERED.
Purpose
Issue
Scope
Implementation Notes
Test Plan
Test Result
Docs Impact
Essential PR Checklist