Skip to content

[Feature]: Add DBO support for Ascend NPU ModelRunnerV2 #265

Description

@jiangkuaixue123

Feature, motivation, and pitch

Ascend NPU ModelRunnerV2 support landed in #257, but the current path explicitly rejects enable_dbo=true and use_ubatching=true. This leaves DBO workloads on ModelRunnerV1 and prevents the Ascend ModelRunnerV2 path from reaching the DBO milestone described in Phase 3 of #240.

Add native DBO support to AFDNPUAttentionModelRunnerV2 for the synchronous CAMP2pAFDConnector path. The implementation should preserve native vLLM/vLLM-Ascend ownership of ModelRunnerV2 request state, input preparation, per-uBatch Attention metadata, output merging, NPU streams/events, and ACL graph lifecycle.

Related work:

Proposed change

  • Enable enable_dbo / native ubatching for the Ascend ModelRunnerV2 Attention path.
  • Prefer an AscendUBatchRunner owned by vLLM-Ascend, exposed through a narrow factory or extension hook.
  • Integrate AFD transaction, stage, token-shape, DP, and connector-control metadata per uBatch without copying the ModelRunnerV2 execution loop.
  • Initially support exactly two uBatches with synchronous CAMP2pAFDConnector.
  • Deliver eager DBO first; qualify FULL_DECODE_ONLY DBO only after eager correctness is established.
  • Keep ModelRunnerV1 DBO and ModelRunnerV2 no-DBO behavior unchanged.
  • Fail closed for unsupported connectors, graph modes, topologies, or uBatch counts.

vLLM compatibility and extension points

Preferred extension point:

  • A vLLM-Ascend-owned AscendUBatchRunner and an overridable uBatch-runner construction hook in the native NPU ModelRunnerV2 path.
  • Narrow AFD composition/hooks for connector lifecycle, control publication, AFD forward-context metadata, profiling, and cleanup.

Compat shim needed:

  • A version-pinned shim may be used if the pinned vLLM-Ascend runtime does not yet expose the required factory or lifecycle hooks.

Monkey patch needed:

  • Avoid if possible. If unavoidable, keep it exact-version, minimal, and documented with an upstream/removal plan.
  • Do not copy native execute_model, input preparation, Attention metadata construction, output merging, or ACL graph-manager lifecycle into afd-plugin.

The implementation targets the repository's pinned vLLM/vLLM-Ascend runtime pair and must not modify either upstream source tree from this repository.

Alternatives considered

  • Keep DBO on ModelRunnerV1 only: lowest implementation risk, but leaves Ascend MRV2 feature parity incomplete.
  • Port the existing NPU V1 DBO loop into ModelRunnerV2: rejected because it duplicates upstream execution ownership and creates a large upgrade surface.
  • Copy the GPU UBatchRunner into afd-plugin: rejected because CUDA-specific stream/graph assumptions do not define the Ascend runtime contract.
  • Implement eager and ACL graph DBO together: increases the debugging surface; eager-first provides a smaller correctness gate.

Validation plan

Focused tests:

  • Configuration accepts the supported two-uBatch MRV2 DBO setup and rejects unsupported combinations before connector rendezvous or weight loading.
  • Per-uBatch AFD metadata preserves real and padded token lengths, stage identity, transaction identity, and DP rank agreement.
  • Connector control is published before each uBatch data transfer and outside ACL graph bodies.
  • Failure paths clean up sibling execution, streams/events, forward contexts, graph state, and connector transactions.

Ascend hardware E2E:

  • DeepSeek-V2-Lite, synchronous CAMP2pAFDConnector, 1A1F, DP1/TP1, eager DBO.
  • Decode, prefill, mixed batches, uneven splits, padding-only tails, repeated requests, profiling, and clean shutdown.
  • Deterministic output comparison against approved native/no-DBO or ModelRunnerV1 references.
  • DP2/TP1 and TP2/DP1 coverage before broad topology claims.
  • Full non-accuracy NPU E2E gate, then GSM8K-7 accuracy.
  • After eager acceptance, FULL_DECODE_ONLY capture/replay, graph-key separation by uBatch count/shape, graph misses, and shape changes.

Acceptance criteria:

  • Exactly two uBatches execute correctly through native Ascend ModelRunnerV2 DBO ownership.
  • No copy of the native MRV2 execution loop or Ascend Attention/graph internals is introduced.
  • AFD connector payloads and control metadata are correct for every uBatch.
  • Eager correctness, cleanup, and accuracy gates pass on Ascend hardware.
  • Graph DBO is not claimed until its separate capture/replay gate passes.
  • Existing Ascend MRV2 no-DBO and ModelRunnerV1 DBO tests remain green.

Before submitting

  • I searched existing issues and RFCs.
  • I identified whether this belongs in plugin-owned code, compat helpers, or compat patches.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions