[Performance] Provisioning the PTO-ISA async-SDMA workspace degrades AICore fault recovery ~1000x (0.3s → ~306s) on a2a3

[ChaoWao]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Summary

Provisioning the PTO-ISA async-SDMA scratch workspace on an a2a3 device makes AICore fault recovery ~1000× slower (~0.3 s → ~306 s), and also changes the surfaced fault error code (507046 → 507015).

SdmaWorkspaceManager::Init() creates 48 STARS streams via aclrtCreateStreamWithConfig(..., ACL_STREAM_DEVICE_USE_ONLY). Those streams are device-visible and sit in the device fault/sync domain. Once any AICore task faults and poisons the card, tearing the 48 streams down — whether by aclrtResetDeviceForce/rtDeviceReset at finalize or by aclrtDestroyStream — blocks ~306 s. Merely provisioning the workspace (no SDMA op ever submitted) is enough to trigger this on the next AICore fault, because the cost is in creating/holding the streams, not using them.

This is latent on main (the SDMA workspace is provisioned only on the HCCL comm path today), and was surfaced while making single-card provisioning eager in PR #1406 (branch feat-sdma-prefetch-workspace-addr), which pointed the tests/st/aicore_op_timeout / tests/st/runtime_fatal_codes fault-injection STs at an SDMA-provisioned device.

Git Commit ID

simpler upstream/main 4dbcc9c8; pto-isa pin 83d01313 (root cause is in SdmaWorkspaceManager::CreateStarsStreams). Reproduced on PR #1406 head.

CANN Version

cann-9.0.0

Host Platform

Linux (aarch64)

Reproduction

1. On a2a3 hardware, get the PTO-ISA async-SDMA workspace provisioned on a device (the eager per-device path in PR Add: opt-in async-DMA SDMA workspace via Worker enable_sdma #1406, or the HCCL comm path on main).
2. Run a kernel that deliberately hangs an AICore op and let the timeout chain reap it, e.g.:

   task-submit --device auto --device-num 1 --run \
     "python -m pytest tests/st/aicore_op_timeout/test_aicore_op_timeout.py --platform a2a3 --device \$TASK_DEVICE"
   

3. Watch the device log (ASCEND_PROCESS_LOG_PATH): the fault (507015) surfaces in a few seconds, but DeviceReset/rtDeviceReset (or the stream teardown, if destroyed first) then blocks ~306 s before returning 507007.

Controlled comparison (same aicore_op_timeout test, a2a3 hardware):

SDMA workspace on device	fault→reset done	run() error code	pytest total
No (as on main single-card)	~0.3 s	507046 (STREAM_SYNC_TIMEOUT)	~9 s
Yes (48 STARS streams)	~306 s	507015 (AICORE_EXCEPTION)	~318 s

Additional findings while narrowing the root cause:

• Destroying the 48 streams before the reset does not help — the ~306 s just moves from rtDeviceReset into aclrtDestroyStream (poisoned-device teardown blocks either way).
• The CANN-shipped SdmaWorkspaceManager (cann-9.0.0/include/pto/npu/...) creates the streams via rtsStreamCreate(RT_STREAM_CP_PROCESS_USE = 0x800) instead of ACL_STREAM_DEVICE_USE_ONLY. Swapping only that flag avoids the fault-domain interference but instead breaks simpler's AICore register MMIO setup (init_aicore_register_addresses → halMemCtl rc=13), failing even a normal prefetch run. So neither known stream flag lets the 48 STARS streams coexist with simpler's device infrastructure during/after a fault.

Expected Performance

AICore fault recovery on a2a3 should be ~0.3 s (as it is when no SDMA workspace is provisioned), and the fault should surface as it does without SDMA (507046 at the host stream sync).

Actual Performance

With the SDMA workspace provisioned, fault recovery takes ~306 s (~1000× slower) and the code changes to 507015. Any a2a3 workload that has the SDMA workspace on its device pays this on the next AICore fault.

---

Related: #1315 (gating the SDMA overlay behind a macro — different root cause, same subsystem).

[ChaoWao]

I pushed the simpler-side containment to #1406.

The implementation now declares async-DMA requirements per CoreCallable and provisions the 48-stream SDMA workspace only immediately before the first marked TMR run. Ordinary initialization, registration, communication domains, and non-SDMA faults no longer create those CP-process streams. Poisoned runners are rejected before provisioning, provider state is invalidated after reset, and unsupported HBG/simulation/a5 declarations fail at registration.

Hardware acceptance on a2a3:

• Non-SDMA AICore fault plus a post-fault marked-callable attempt completed in 12.6 s end-to-end; Worker.close() stayed below the 20 s regression limit, and the marked attempt failed before provisioning.
• The real prefetch path still created/queryed all 48 STARS streams and produced bit-exact output.
• A retained-DSO two-Worker test re-provisioned correctly after reset.
• The two-device SDMA completion demo remained bit-exact on both ranks.

The detailed teardown trace and rejected alternatives are recorded in docs/investigations/2026-07-a2a3-sdma-fault-teardown.md in the PR.

This mitigates the regression for workloads that have not actually used SDMA; it does not close the underlying fault-domain defect. Once SDMA has genuinely been provisioned, a later AICore fault still waits about 306 s in the CANN v9.0 remote logic-CQ teardown. Full closure needs both a bounded/state-aware runtime FreeLogicCq path while the device is running down and driver cancellation/wakeup of pending remote TRS events. I am leaving this issue open for that work.