Skip to content

feat(npu): A5 (Ascend950) support for CAMP2p via HCCL p2p - #303

Open
ksiyuan wants to merge 1 commit into
vllm-project:mainfrom
ksiyuan:a5-adapt
Open

feat(npu): A5 (Ascend950) support for CAMP2p via HCCL p2p#303
ksiyuan wants to merge 1 commit into
vllm-project:mainfrom
ksiyuan:a5-adapt

Conversation

@ksiyuan

@ksiyuan ksiyuan commented Sep 1, 2026

Copy link
Copy Markdown

Purpose

Enable the synchronous CAMP2p AFD connector on Atlas A5 (Ascend950). On A5 the a2e/e2a custom-op transport cannot move data between the AFD Attention and FFN ranks (see "Why not a2e/e2a" below), so on A5 the four a2e/e2a call sites are replaced with plain HCCL point-to-point (dist.send/recv) over the existing per-ubatch afd HCCL groups. The FFN side runs its MoE internally through the standard vLLM-Ascend EP path, where the native ops are proven. Validated by a
2-rank eager 1A+1F DeepSeek-V2-Lite completion smoke on Ascend 950PR.

Why not a2e/e2a (and not native MC2 dispatch/combine) on A5

a2e/e2a are a peer-window transport: data movement and completion handshake are done by AI-core loads/stores to the remote rank's HCCL window, per the ABI documented in the op headers (HcclA2CombineOpParam.windowsIn[rankId], where non-self entries are declared "cross-card mapping addresses"). On A5 the HCCL resource violates its own documented ABI: those peer entries are populated but are not in the local device's addressable map, and any AI-core MTE access is rejected at the hardware level (507035, vector-core MTE out-of-range; driver-confirmed, reproduced across machines in both directions). The same resource exposes this generation's real transport (AiRMA/RDMA QP + ibVerbs
descriptors), and no production code in the ecosystem relies on raw kernel peer-window stores — cross-device movement goes through the HCCL engine / native CANN ops instead.

The torch_npu native ops (npu_moe_distribute_dispatch_v2/combine_v2) are available on A5 but only support standard EP where every group member is an expert rank; the AFD mixed group (Attention ranks own no experts and receive zero tokens) re-triggers 507035 in the dispatch kernel.

Hence Route B2: plain HCCL p2p for Attention<->FFN hidden-state movement (only hidden_states crosses; gate stays on FFN), with FFN-internal MoE in the standard EP configuration.

Issue

Scope

  • In scope: CAMP2p connector A5 path (eager); A5 rank mapping + per-peer token counts from the DP metadata control plane; unit tests.
  • Out of scope: ACL graph capture on A5 (HCCL p2p is not graph-capturable; eager only — noted in camp2p_a5.py); multi-rank / DBO validation; 910C behavior is unchanged.

Implementation Notes

  • camp2p.py: A5 branches in send_attn_output/recv_ffn_output/ recv_attn_output/send_ffn_output; init_afd_connector skips the custom-op loader on A5 (a2e/e2a register only ascend910_93).
  • camp2p_a5.py (new): is_a5(); contiguous rank mapping (attention i -> FFN i // (attn_size//ffn_size), matching _num_tokens_for_ffn_rank); per-peer DP token counts; p2p_send/p2p_recv primitives.
  • FFN per-peer seq_lens are carried in AFDTransferMetadata.seq_lens and used to split results back (no equal-ratio split).

Test Plan

  • Unit: tests/unit/connectors/test_camp2p_connector.py (mapping round-trip, DP->TP count expansion, missing-metadata fallback); ruff clean.
  • Hardware: 2-rank eager 1A+1F DeepSeek-V2-Lite completion smoke on A5 (950PR).
  • Pending (follow-up): multi-rank 2A1F/2A2F, DBO, 910C regression.

Test Result

  • PASS: 2-rank eager completion round-trip on Ascend 950PR; unit tests added; ruff/pylance clean.

Docs Impact

  • No docs updated in this PR; A5-specific limitations (eager-only, 507035 rationale) are documented in the camp2p_a5.py module docstring.

@ksiyuan
ksiyuan force-pushed the a5-adapt branch 2 times, most recently from a506be1 to 7d60274 Compare September 2, 2026 08:14
a2e/e2a custom ops and native MC2 dispatch/combine both fail on the AFD mixed group on A5 (507035 MTE out-of-range; remote HCCL windows unreachable). Replace the four a2e/e2a call sites on A5 with plain dist.send/recv over the per-ubatch 'afd' HCCL groups. Validated by a 2-rank eager 1A+1F DeepSeek-V2-Lite completion smoke on Ascend 950PR.

- camp2p.py: A5 branches in send_attn_output/recv_ffn_output/recv_attn_output/send_ffn_output; init_afd_connector skips the custom-op loader on A5 (a2e/e2a register only ascend910_93).
- camp2p.py A5 helpers: is_a5(), contiguous rank mapping (attention i -> ffn i//(attn//ffn), matching _num_tokens_for_ffn_rank), shared per-peer DP token counts (attention_token_counts), p2p primitives; unit tests added.
- Gate stays on FFN (only hidden_states crosses the wire). 910C custom-op path unchanged.
- Known gaps: ACL graph capture, DBO, multi-rank validation, 910C regression.

Signed-off-by: ksiyuan <ksiyuan@umich.edu>
@ksiyuan ksiyuan changed the title feat(npu): A5 (Ascend950) support for CAMP2p via HCCL p2p (Route B2) feat(npu): A5 (Ascend950) support for CAMP2p via HCCL p2p Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant