Feature, motivation, and pitch
Ascend NPU ModelRunnerV2 support landed in #257, but the current path explicitly rejects enable_dbo=true and use_ubatching=true. This leaves DBO workloads on ModelRunnerV1 and prevents the Ascend ModelRunnerV2 path from reaching the DBO milestone described in Phase 3 of #240.
Add native DBO support to AFDNPUAttentionModelRunnerV2 for the synchronous CAMP2pAFDConnector path. The implementation should preserve native vLLM/vLLM-Ascend ownership of ModelRunnerV2 request state, input preparation, per-uBatch Attention metadata, output merging, NPU streams/events, and ACL graph lifecycle.
Related work:
Proposed change
- Enable
enable_dbo / native ubatching for the Ascend ModelRunnerV2 Attention path.
- Prefer an
AscendUBatchRunner owned by vLLM-Ascend, exposed through a narrow factory or extension hook.
- Integrate AFD transaction, stage, token-shape, DP, and connector-control metadata per uBatch without copying the ModelRunnerV2 execution loop.
- Initially support exactly two uBatches with synchronous
CAMP2pAFDConnector.
- Deliver eager DBO first; qualify
FULL_DECODE_ONLY DBO only after eager correctness is established.
- Keep ModelRunnerV1 DBO and ModelRunnerV2 no-DBO behavior unchanged.
- Fail closed for unsupported connectors, graph modes, topologies, or uBatch counts.
vLLM compatibility and extension points
Preferred extension point:
- A vLLM-Ascend-owned
AscendUBatchRunner and an overridable uBatch-runner construction hook in the native NPU ModelRunnerV2 path.
- Narrow AFD composition/hooks for connector lifecycle, control publication, AFD forward-context metadata, profiling, and cleanup.
Compat shim needed:
- A version-pinned shim may be used if the pinned vLLM-Ascend runtime does not yet expose the required factory or lifecycle hooks.
Monkey patch needed:
- Avoid if possible. If unavoidable, keep it exact-version, minimal, and documented with an upstream/removal plan.
- Do not copy native
execute_model, input preparation, Attention metadata construction, output merging, or ACL graph-manager lifecycle into afd-plugin.
The implementation targets the repository's pinned vLLM/vLLM-Ascend runtime pair and must not modify either upstream source tree from this repository.
Alternatives considered
- Keep DBO on ModelRunnerV1 only: lowest implementation risk, but leaves Ascend MRV2 feature parity incomplete.
- Port the existing NPU V1 DBO loop into ModelRunnerV2: rejected because it duplicates upstream execution ownership and creates a large upgrade surface.
- Copy the GPU
UBatchRunner into afd-plugin: rejected because CUDA-specific stream/graph assumptions do not define the Ascend runtime contract.
- Implement eager and ACL graph DBO together: increases the debugging surface; eager-first provides a smaller correctness gate.
Validation plan
Focused tests:
- Configuration accepts the supported two-uBatch MRV2 DBO setup and rejects unsupported combinations before connector rendezvous or weight loading.
- Per-uBatch AFD metadata preserves real and padded token lengths, stage identity, transaction identity, and DP rank agreement.
- Connector control is published before each uBatch data transfer and outside ACL graph bodies.
- Failure paths clean up sibling execution, streams/events, forward contexts, graph state, and connector transactions.
Ascend hardware E2E:
- DeepSeek-V2-Lite, synchronous
CAMP2pAFDConnector, 1A1F, DP1/TP1, eager DBO.
- Decode, prefill, mixed batches, uneven splits, padding-only tails, repeated requests, profiling, and clean shutdown.
- Deterministic output comparison against approved native/no-DBO or ModelRunnerV1 references.
- DP2/TP1 and TP2/DP1 coverage before broad topology claims.
- Full non-accuracy NPU E2E gate, then GSM8K-7 accuracy.
- After eager acceptance,
FULL_DECODE_ONLY capture/replay, graph-key separation by uBatch count/shape, graph misses, and shape changes.
Acceptance criteria:
- Exactly two uBatches execute correctly through native Ascend ModelRunnerV2 DBO ownership.
- No copy of the native MRV2 execution loop or Ascend Attention/graph internals is introduced.
- AFD connector payloads and control metadata are correct for every uBatch.
- Eager correctness, cleanup, and accuracy gates pass on Ascend hardware.
- Graph DBO is not claimed until its separate capture/replay gate passes.
- Existing Ascend MRV2 no-DBO and ModelRunnerV1 DBO tests remain green.
Before submitting
Feature, motivation, and pitch
Ascend NPU ModelRunnerV2 support landed in #257, but the current path explicitly rejects
enable_dbo=trueanduse_ubatching=true. This leaves DBO workloads on ModelRunnerV1 and prevents the Ascend ModelRunnerV2 path from reaching the DBO milestone described in Phase 3 of #240.Add native DBO support to
AFDNPUAttentionModelRunnerV2for the synchronousCAMP2pAFDConnectorpath. The implementation should preserve native vLLM/vLLM-Ascend ownership of ModelRunnerV2 request state, input preparation, per-uBatch Attention metadata, output merging, NPU streams/events, and ACL graph lifecycle.Related work:
Proposed change
enable_dbo/ native ubatching for the Ascend ModelRunnerV2 Attention path.AscendUBatchRunnerowned by vLLM-Ascend, exposed through a narrow factory or extension hook.CAMP2pAFDConnector.FULL_DECODE_ONLYDBO only after eager correctness is established.vLLM compatibility and extension points
Preferred extension point:
AscendUBatchRunnerand an overridable uBatch-runner construction hook in the native NPU ModelRunnerV2 path.Compat shim needed:
Monkey patch needed:
execute_model, input preparation, Attention metadata construction, output merging, or ACL graph-manager lifecycle into afd-plugin.The implementation targets the repository's pinned vLLM/vLLM-Ascend runtime pair and must not modify either upstream source tree from this repository.
Alternatives considered
UBatchRunnerinto afd-plugin: rejected because CUDA-specific stream/graph assumptions do not define the Ascend runtime contract.Validation plan
Focused tests:
Ascend hardware E2E:
CAMP2pAFDConnector, 1A1F, DP1/TP1, eager DBO.FULL_DECODE_ONLYcapture/replay, graph-key separation by uBatch count/shape, graph misses, and shape changes.Acceptance criteria:
Before submitting