[RFC]: Async AFD for Prefill on Ascend NPU

[puxiaokun3721]

Summary

This RFC proposes an asynchronous execution model for Attention-FFN Disaggregation (AFD) on Ascend NPU, scoped to prefill. It defines a semantic work-item contract (one Attention DP replica per layer), moves shared-expert computation to the Attention side, defines a routed-expert-only operator boundary, and sets two FFN execution modes (eager and aclgraph) as goals. It also proposes maintaining the asynchronous operators in a separate open-source CAM operator repository, shipped to the plugin as a wheel plus a C++ package.

The community is asked to confirm the execution model, the operator/resource boundary, the operator packaging, the shared-expert ownership, and the staged delivery.

Motivation

Attention-FFN Disaggregation (AFD) separates the stateful Attention path from the stateless FFN/MoE path so that the two roles can use different resource ratios, parallel strategies, and execution policies. Synchronous and asynchronous AFD address different inefficiencies.

Synchronous AFD primarily improves FFN efficiency by changing the Attention-to-FFN resource ratio. More Attention workers can feed fewer FFN workers, raising the FFN batch size and improving weight reuse. This is especially useful during decode, where a small FFN batch is commonly memory-bandwidth bound.

That resource separation does not, by itself, remove synchronization stalls. In a colocated or synchronous AFD execution, MoE dispatch and combine create barriers between Attention data-parallel replicas. Online requests have different prompt lengths and arrive at different times, so Attention DP replicas do not finish a layer at the same time. A fast replica waits for the slowest replica even when the FFN side already has enough ready work to execute. The result is straggler-induced idle time caused by load imbalance rather than by the speed of a single communication operation.

This RFC proposes an asynchronous AFD execution model for prefill on Ascend NPU. The readiness unit is one Attention DP replica. A replica becomes ready only after all participating Attention ranks have published the dispatch shards required to form its complete payload. The participant set may use tensor, context, sequence, or combined parallelism. Once the replica is ready, the FFN side may consume and execute it without waiting for other Attention DP replicas. Routing metadata travels in band with the data. The FFN runtime consumes ready work items from the connector instead of waiting for all Attention DP replicas to advance through each MoE layer in a globally synchronized execution wave.

Prefill is the initial scope because one ready Attention DP replica normally provides enough tokens to run FFN efficiently. Decode has a different trade-off: consuming one small DP contribution immediately may reduce waiting but leave FFN memory-bandwidth bound. Dynamic re-batching across multiple ready DP replicas is a promising direction, but its latency/throughput policy is still under exploration and is not standardized by this RFC.

Goals

• Enable FFN workers to consume each complete Attention DP replica payload independently, removing global barriers across Attention DP replicas while preserving per-request and per-layer correctness.
• Define a maintainable semantic contract between the model runtime, CAMAsyncAFDConnector, and the asynchronous dispatch/combine operators.
• Support both FFN execution modes — eager and aclgraph capture-and-replay — so the FFN runtime does not read the layer identity or token counts back to the host to drive control flow.
• Make the CAM asynchronous operator sources, Python bindings, build integration, and tests available to afd-plugin through the CAM operator repository.
• Define reproducible correctness, reliability, compatibility, and performance acceptance criteria for the vLLM 0.26.0 plugin baseline.

Non-goals

• The initial scope is asynchronous AFD for prefill. This RFC does not define a decode scheduling or dynamic re-batching policy; decode remains a possible follow-up direction (see "Decode follow-up").
• This RFC does not deprecate or replace synchronous AFD connectors such as CAMP2pAFDConnector. Synchronous and asynchronous AFD address different bottlenecks and may coexist.
• This RFC does not introduce new TP/EP topology rules, expert placement policies, or Attention/FFN resource allocation strategies.
• This RFC does not redesign MoE routing or eliminate the underlying workload imbalance. It reduces idle time caused by global synchronization barriers when such imbalance occurs.
• This RFC defines the semantic behavior of asynchronous AFD, but does not standardize the current operator flag layout, peer-accessible memory offsets, or fixed implementation constants as a public ABI.

Proposed change

Architecture

The data path for one work item (one Attention DP replica, one MoE layer) has three logical stages:

Attention DP replica                       FFN ranks

Attention + routing
        |
        | async dispatch (data + routing metadata)
        v
  dispatch_send  ----------------------->  dispatch_recv
                                                 |
                                                 | layer-select routed MoE compute
                                                 v
  combine_recv  <-------------------------  combine_send
        |
        v
weighted routed result + Attention-side local shared expert result

Many such work-item paths may overlap: a ready Attention DP replica can be consumed by the FFN side while other Attention DP replicas are still computing their current layer, which is what removes the cross-DP barrier. The figure above shows one work item; the concurrency across work items is the source of the asynchronous benefit.

The implementation uses CAMAsyncAFDConnector and the following operator roles. The names below are the protocol-layer roles; the concrete operator bindings are provided by the CAM operator package and do not need to be resolved in this RFC.

• Four communication operators for the routed-expert data plane:
  • dispatch_send: publish Attention activations and routing metadata;
  • dispatch_recv: select a ready Attention DP replica and materialize expert-grouped FFN inputs;
  • combine_send: publish processed routed expert outputs;
  • combine_recv: reconstruct token order, reduce the routed-expert results of each token's top-k selection, and return the per-token routed output.
• Two layer-select routed-expert compute operators, used on the FFN side to execute the routed MoE computation while keeping the layer identity on the device:
  • npu_grouped_matmul_swiglu_quant_v2_multi_layer (routed expert W1);
  • npu_grouped_matmul_multi_layer (routed expert W2).

"Layer-select" here means the operator receives the address table of every layer's weights plus a device-side layer_index, and uses that index to fetch the current layer's weight/scale before running the single-layer GEMM. This keeps the layer identity on the device and lets a graph capture the loop over work items instead of the host re-issuing each layer.

Shared-expert computation is moved to the Attention side and is not carried on the routed data plane (see "Shared-expert ownership" below). The operators use peer-accessible device memory and in-band metadata. The connector has no separate DP control plane. The FFN runtime consumes ready work items from the connector; the layer identity and per-expert token counts for a work item are consumed on the FFN device, not on the host.

Asynchronous work-item contract

The RFC defines semantic readiness and completion rather than the current flag layout:

1. Each participating Attention rank must finish writing its dispatch data and routing metadata before publishing its ready signal.
2. An Attention DP replica is ready for FFN consumption only after all of its dispatch participants have published their ready signals.
3. FFN ranks may consume any ready Attention DP replica without waiting for all replicas. The selection policy must not starve a ready replica indefinitely.
4. A work item identifies at least the source Attention DP replica and model layer. The layer identity and per-expert token counts are consumed on the FFN device as part of the work item; the FFN runtime does not read them back to the host to drive control flow for that work item.
5. Attention may consume a combine result only after every participating FFN rank has completed the matching work item.

Exact ready/completion flags, memory-ordering primitives, buffer layouts, and how each work item is bound to a specific layer remain operator implementation details.

Shared-expert ownership

In the target protocol, shared-expert computation is owned by the Attention side. The Attention role holds the shared-expert weights and computes the shared branch locally from the layer's attention output. The routed data plane does not transfer shared-expert activations, scales, or results; the FFN side processes routed experts only. On the Attention side, combine_recv reduces the routed-expert outputs of each token's top-k selection, and the framework then adds the locally computed shared-expert result to that routed result.

Consequently the asynchronous CAM protocol is routed-expert-only:

• CAM dispatch/combine handles routed experts only;
• routed expert IDs use a zero-based range without a synthetic shared slot;
• the Attention runtime computes the shared expert branch locally and merges it after receiving the routed result;
• the FFN runtime does not receive, compute, or transfer shared-expert data.

This is a departure from the current CAMAsyncAFDConnector, which transfers shared-expert inputs and outputs over the data plane and computes the shared branch on the FFN side. Migrating to the routed-only protocol changes expert indexing, completion conditions, output schemas, window sizing, and metadata lengths, and is tracked as a child design/implementation issue under this RFC.

Operator source and packaging

The asynchronous operators require the four communication operators and the two layer-select routed-expert operators. They are maintained in a separate open-source CAM operator repository (https://gitcode.com/openeuler/umdk/tree/master/src/cam) rather than in the afd-plugin source tree, and are shipped to afd-plugin as a wheel (Python bindings) plus a C++ package (aclnn/kernel). afd-plugin calls them directly through the CAM interface, not via torch_npu operator bindings.

The rationale is that placing all custom operators in the plugin's own custom-operator directory would bloat that tree and substantially increase build time, and it would move the maintenance ownership of these operators to the community repository. Keeping them in the separate CAM repository keeps maintenance with the authors. The small a2e/e2a operators used by the synchronous CAMP2pAFDConnector remain in the plugin because of their size; they are outside the scope of this RFC.

Bringing the asynchronous operators into the open-source build requires:

1. import the operator source with provenance and license review;
2. provide the whl (bindings) and C++ package for the four communication and two layer-select operators;
3. reconcile the operator namespace between the CAM repository and the afd-plugin call sites;
4. integrate the supported Ascend build so the plugin can consume the shipped packages;
5. keep the bundled binary path only where needed for controlled migration or parity comparison;
6. publish supported CANN, torch_npu, Python, SoC, dtype, model, and topology matrices rather than relying on implicit constants.

Open sourcing the operators is an enabling deliverable of the architecture, not the sole purpose of this RFC.

FFN execution modes

The asynchronous protocol defines two FFN-side execution modes, both goals of this work. Both differ from the current implementation, in which the FFN runtime reads the layer identity and per-expert token counts back to the host for each work item and the host then issues the per-layer operators.

• Eager mode: the FFN runtime consumes ready work items from the connector. The receive operators return buffers sized by the maximum reserved token count, so their shapes are static; the layer identity and per-expert token counts are consumed on the FFN device as part of the work item, and the host does not read them back to drive control flow.
• aclgraph mode: the FFN-side dispatch/combine loop is captured once on the device and replayed on subsequent invocations, rather than re-executed as an eagerly launched sequence per layer. This removes the per-layer host orchestration and the host re-issue of the per-item operators.

Both modes rely on fixed-shape buffers sized by the maximum reserved token count, and both keep the layer identity and per-expert token counts on the device.

The graph capture is provided through the CAM operator deliverables (the operators expose the capture support needed by the graph runtime), while the operators themselves remain callable through the CAM interface — the graph is captured by the framework layer over a graph-capturable CAM path, and does not require resolving those operators through torch_npu bindings.

Requirements for the aclgraph mode in particular:

• the MoE-side input/output buffers are sized by the maximum reserved token count so their shapes are static and graph-capturable;
• the FFN-side loop runs a compile-time-fixed number of iterations, so the graph is captured once and reused, not recompiled per work item;
• shape changes that require a new graph are handled by bucketing inputs into a finite set of reserved shapes rather than recompiling on every change;
• the receive operators block on the device (spinning on the ready flag) and do not consume the host;
• buffer reuse across iterations is serialized so a later iteration cannot overwrite an in-flight transfer or its results.

Staged delivery

Each phase should be reviewable independently. Child issues and PRs can be linked under the corresponding phase as implementation proceeds.

Phase 0 — Finalize semantic, source, and execution-mode boundaries

• agree on Attention DP replica readiness and the work-item identity, including source Attention DP replica and model layer;
• confirm the routed-expert-only CAM protocol and Attention-side shared expert ownership;
• define the responsibilities of the model runtime, CAMAsyncAFDConnector, the four communication operators, and the two layer-select routed-expert operators;
• confirm connector ownership of HCCL communicator initialization, peer-accessible communication-buffer setup, lifetime, and teardown;
• settle operator provenance, license, package (wheel + C++), and namespace requirements, including the separate CAM operator repository;
• agree on the two FFN execution modes (eager and aclgraph) and on the operator interfaces that must support both.

This phase is complete when the community agrees on the architecture, source, and execution-mode boundaries needed to begin reviewable implementation work.

Phase 1 — Release the routed-expert-only CAM operator source code

• publish the four communication operators and the two layer-select routed-expert operators (host-side code, tiling, Ascend C kernels);
• provide the bindings and the C++/aclnn package through the CAM operator repository;
• integrate with connector-managed HCCL communication resources;
• document and validate supported operator configurations and communication-buffer capacity requirements;
• add reproducible package builds and operator-level correctness tests.

This phase is complete when all six operators build from the CAM operator repository and pass their documented operator-level tests without transferring shared expert data.

Phase 2 — Integrate the operators and move shared experts to Attention

The sourced operators are routed-expert-only, so switching to them and moving the shared-expert branch to the Attention side land together as one change:

• switch CAMAsyncAFDConnector to the sourced routed-only operators;
• move the shared-expert branch to the Attention side: Attention holds the shared-expert weights, computes the shared branch locally from the layer attention output, and merges it with the routed result after combine;
• remove shared-expert transfer and compute from the FFN runtime;
• validate asynchronous prefill AFD end to end on the vLLM 0.26.0 plugin baseline;
• retain the binary path only where needed for controlled migration or parity comparison.

This phase is complete when the sourced operators are integrated and the routed-only path passes end-to-end correctness with shared experts computed on Attention, without changing the non-AFD or synchronous AFD paths.

Phase 3 — Qualify the feature, complete source migration, and FFN modes

• validate timeout, shutdown, failure cleanup, uneven DP progress, and repeated execution;
• document the supported vLLM, vLLM-Ascend, CANN, torch_npu, Python, SoC, dtype, model, and topology combinations;
• perform reproducible plugin-native performance characterization;
• implement and qualify both FFN execution modes: eager, and the aclgraph capture-and-replay path with fixed-shape buffers, compile-time-fixed iteration count, shape bucketing, and device-side blocking receive;
• remove or deprecate bundled binary artifacts only after source-built parity and migration requirements are satisfied.

This phase is complete when the supported configurations have reproducible correctness, reliability, compatibility, and performance results, the binary migration policy is documented, and both FFN execution modes are qualified.

Decode follow-up

Asynchronous AFD for decode remains a possible follow-up direction. Its scheduling, dynamic re-batching, and latency/throughput policies are outside the scope of this RFC and require a separate design discussion.

Decisions requested from the community

This RFC asks for feedback on the following decisions. The first four are the primary items; the fifth is secondary.

1. Should barrier removal at the Attention-DP-group granularity be the stable execution model for Ascend NPU asynchronous prefill AFD?
2. Is the division of responsibility between the model runtime, CAMAsyncAFDConnector, the connector-driven FFN execution loop, the four communication operators, and the two layer-select routed-expert operators maintainable?
3. Should the asynchronous operators be maintained in a separate open-source CAM operator repository and shipped to afd-plugin as a wheel plus a C++ package, rather than living in the afd-plugin source tree? If so, what package and namespace migration policy should be used?
4. Is a routed-expert-only CAM operator boundary, with shared-expert computation owned by the Attention side, acceptable as the long-term protocol?
5. Is the staged delivery (Phase 0–3) an acceptable way to land this?

Plugin boundary

Upstream-owned

• Request scheduling, request state, and the standard worker lifecycle.
• Native Attention and MoE computation outside the AFD transfer boundary.
• Base NPU model runner behavior unrelated to AFD.
• KV-cache, sampling, and output lifecycle.
• Standard process group, forward context, and DP behavior for non-AFD paths.

Plugin-owned

• AFD configuration, topology validation, and connector selection.
• CAMAsyncAFDConnector and its semantic work-item contract.
• AFD-specific Attention/FFN worker and model runner subclasses or integration hooks, including the connector-driven FFN execution path.
• Model-side routed dispatch/combine integration, including Attention-side shared-expert weights, local computation, and result merging.
• HCCL communicator initialization, peer-accessible communication-buffer setup, lifetime, and teardown for the CAM connector backend.
• Consuming the asynchronous operator packages (wheel + C++) from the CAM operator repository, and the afd-plugin call sites and backend-specific integration tests.
• Supported-configuration matrices, reproducible recipes, and qualification artifacts.

Compatibility helpers

• Narrow adapters for pinned vLLM and vLLM-Ascend process group, runner, forward context, and platform APIs.

Compatibility patches

• The current async-DP engine patch that disables native DP wave coordination for AFD Attention execution.
• The current forward context patch that avoids constructing synchronized native DP metadata for this asynchronous path.

These patches let an AFD async-DP Attention rank run as a connector-driven engine process in vLLM 0.26.0; they are framework-layer adaptations, not part of the operator or work-item contract proposed here. The target protocol still requires them (or their upstreamed equivalents): migrating to routed-only operators and the two FFN execution modes does not remove the need for them. Their future removal is tracked separately against upstream vLLM extension points.

Every patch must remain version-gated and carry an upstream or removal plan. This RFC does not propose unrelated changes to vLLM core.

Explicit class paths

• afd_plugin.connectors.npu.async_cam.CAMAsyncAFDConnector
• afd_plugin.v1.worker.npu.attention_worker.AFDNPUAttentionWorker
• afd_plugin.v1.worker.npu.ffn_worker.AFDNPUFFNWorker
• corresponding NPU model runners and model-side asynchronous CAM forward helpers

Risks and alternatives

Risks

• Deadlock and stale state: the current device-side waits are unbounded; rank failure or an invalid state transition can block all progress.
• Ordering and window reuse: independent DP progress can overwrite a window unless ownership and completion are explicit.
• Fairness: repeatedly selecting the first ready DP replica may starve a slow or unlucky group.
• Framework/operator drift: changing operator metadata without a versioned semantic contract can silently break the connector.
• Compatibility cost: async execution currently relies on plugin patches to vLLM engine and forward context behavior.
• Maintenance and licensing: source import must resolve provenance, third-party code, and differences between existing license headers.

Alternatives considered

1. Keep colocated or synchronous AFD execution. This is simpler and remains a valid fallback, but it does not remove cross-DP straggler barriers.
2. Tune only the Attention:FFN rank ratio. This improves FFN efficiency but addresses a different bottleneck from asynchronous barrier removal.
3. Use more fixed microbatches behind a synchronous barrier. This may hide some communication but still couples progress to a global wave and cannot consume arbitrary ready DP replicas.
4. Keep distributing binary operators. This minimizes near-term build work but prevents community debugging, adaptation, review, and contribution.
5. Wait for a complete decode design. Prefill already has a distinct and useful execution model; coupling it to unresolved decode scheduling would delay review without improving the prefill contract.

Feedback period

Suggested feedback period: two weeks after the RFC is posted. The issue should remain open as the umbrella tracker, while accepted implementation work moves to linked child issues and PRs. Performance results may be added when the vLLM/afd-plugin reproduction is complete.

CC list

• @jiangkuaixue123
• @hsliuustc0106
• @francisleely

[jiangkuaixue123]

Overall, I have no objections to the proposal. The delivery and packaging approach for the CAM operator source code can be discussed further.

[ShwStone]

Shared-expert ownership
In the target protocol, shared-expert computation is owned by the Attention side. The Attention role holds the shared-expert weights and computes the shared branch locally from the layer's attention output. The routed data plane does not transfer shared-expert activations, scales, or results; the FFN side processes routed experts only. On the Attention side, combine_recv reduces the routed-expert outputs of each token's top-k selection, and the framework then adds the locally computed shared-expert result to that routed result.

I would like to understand the expected scheduling of shared-expert computation when ubatch is enabled. Could this interfere with the ideal DBO? I am concerned that this implementation may introduce performance regression compared to the current one.

[puxiaokun3721]

@ShwStone Thanks for raising this concern. Our current planned schedule is:

Attention(ubatch 0)
-> dispatch routed work(ubatch 0)
-> compute shared expert locally(ubatch 0)
-> Attention(ubatch 1)
-> combine_recv(ubatch 0)
-> routed result + shared result(ubatch 0)

Moving the shared expert to the Attention side removes shared-expert activations, scales, and results from the CAM data plane, reduces communication and window usage, and reduces FFN-side workload. It also provides an opportunity to overlap the local shared-expert computation with the remote routed-expert computation.

We agree that the impact on DBO performance needs to be validated. We will compare this schedule with the current shared-on-FFN implementation through experiments and report the TTFT, throughput, and execution timeline before qualifying the change.

[Socratesa]

Is the current CAMAsyncAFDConnector not fully implemented yet? I think this should be the ASAP communication library, from a Huawei paper on AFD's application in prefill. Or is the current RFC the official implementation of ASAP, while the current repo is still an experimental version? How does the performance compare?

ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill.

[puxiaokun3721]

@Socratesa Thanks for the comment. Let me clarify the current status.

This asynchronous prefill AFD design was incubated jointly with the ASAP paper team in an internal Huawei project. We are now preparing to land it in the AFD-PLUGIN community. So this RFC is the community-facing proposal for that design, driven from this repository. The core functionality of CAMAsyncAFDConnector is implemented and usable, but it is not yet the release form the RFC proposes. We are doing the necessary code refactoring for open-source release, along with performance work on the asynchronous communication operators.

We will benchmark against the vLLM-Ascend non-AFD baseline (same model, quantization, and workload). Once the refactoring lands, we will publish the performance data and the corresponding recipe, including TTFT / throughput and the results under asymmetric attention:FFN resource ratios.

[Socratesa]

@Socratesa Thanks for the comment. Let me clarify the current status.

This asynchronous prefill AFD design was incubated jointly with the ASAP paper team in an internal Huawei project. We are now preparing to land it in the AFD-PLUGIN community. So this RFC is the community-facing proposal for that design, driven from this repository. The core functionality of CAMAsyncAFDConnector is implemented and usable, but it is not yet the release form the RFC proposes. We are doing the necessary code refactoring for open-source release, along with performance work on the asynchronous communication operators.

We will benchmark against the vLLM-Ascend non-AFD baseline (same model, quantization, and workload). Once the refactoring lands, we will publish the performance data and the corresponding recipe, including TTFT / throughput and the results under asymmetric attention:FFN resource ratios.

Thank you very much for your reply.

1. Can I understand it as meaning that the existing CAMAsyncAFDConnector has already reached the level of the paper, but needs to be refactored for various reasons? I care more about whether the existing implementation can meet the level of the paper

2. I saw that there is an AsyncAFD version based on Nvidia GPUs in the community. What is its relationship with the current RFC? Will it follow the same design principles, with only the backend being different?

[puxiaokun3721]

1. Can I understand it as meaning that the existing CAMAsyncAFDConnector has already reached the level of the paper, but needs to be refactored for various reasons? I care more about whether the existing implementation can meet the level of the paper
2. I saw that there is an AsyncAFD version based on Nvidia GPUs in the community. What is its relationship with the current RFC? Will it follow the same design principles, with only the backend being different?

The paper's experiments were run on an in-house inference framework similar to vLLM. The two frameworks differ in some features, which may lead to different end-to-end gains for the asynchronous design. We are implementing this asynchronous design on vllm-afd-plugin and will measure its performance on this framework directly — the results we publish will be the reproducible data from this repository.

The paper and this RFC both focus on the implementation and performance on Ascend NPU. The GPU AsyncAFD version in the community is a port of this design to GPUs, currently following the same design principles, with the differences mainly in the communication backend and operator implementations.