[Bug] Allreduce L3 cases intermittently hit an AICore UB out-of-bounds in full-suite onboard runs

[ChaoWao]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

Two L3 collective cases — TestAllreduceOnephaseP2 (2 ranks) and TestAllreduceRingP4 (4 ranks) — intermittently fail with an AICore VEC addressing fault, not a timeout:

VEC instruction error: the ub address out of bounds
  core id 15, blk:1, errcode 0x4000000000000000
→ aicpu exception → 507018 → sched_error_code=100 SCHEDULER_TIMEOUT

The 507018 / SCHEDULER_TIMEOUT tail is the consequence, not the mechanism: per .claude/rules/running-onboard.md, 507018 is a generic host-side code, and here the device log names a UB out-of-bounds as the thing that actually fired. A vector unit reading outside its UB allocation is a real addressing bug, so this should not be filed away as scheduling noise even though it presents as flaky.

Only reproduced in a full pytest examples tests/st run with many devices in flight; never in isolation.

This is likely to get more frequent, not less. The failures were seen while validating a wide parallel scheduler, and #1309 has since removed the user-facing block_dim knob — every run now takes the whole device (24 clusters on a2a3) instead of whatever a case pinned. Whatever contention widens this window is now the default configuration for every case in the suite.

Steps to Reproduce

# Full suite, 8 devices — the failing cases are multi-device L3 running
# concurrently with unrelated cases under the resource scheduler.
task-submit --timeout 3600 --max-time 3600 --device auto --device-num 8 \
    --run "python -m pytest examples tests/st --platform a2a3 --device \$TASK_DEVICE"

Observed once in a full-suite run; a second full-suite run immediately after passed 54/54, and pytest tests/st/worker/collectives/allreduce --platform a2a3 alone passed 10/10 on the same build. Expect to repeat the full suite several times to hit it.

To capture the device-side evidence, redirect the device log out of the shared default first:

LOGDIR="$PWD/outputs/allreduce_ub/ascend"; mkdir -p "$LOGDIR"
export ASCEND_PROCESS_LOG_PATH="$LOGDIR"

Expected Behavior

TestAllreduceOnephaseP2 and TestAllreduceRingP4 pass regardless of what else is running concurrently on other devices. Kernel UB addressing must not depend on scheduler width or on unrelated cases sharing the box.

Actual Behavior

VEC instruction error: the ub address out of bounds
  core id 15, blk:1, errcode 0x4000000000000000
→ aicpu exception → 507018 → sched_error_code=100 SCHEDULER_TIMEOUT

TestAllreduceOnephaseP2 (2 devices) and TestAllreduceRingP4 (4 devices) both failed in the same run.

Git Commit ID

d60b8acdd6f11ee0b3ec1e5bb1b32a1c3e0e9c26

CANN Version

(not recorded at the time of the run)

Driver Version

npu-smi 26.0.rc1

Host Platform

Linux (aarch64)

Additional Context

First observed 2026-07-24 during a full onboard a2a3 suite run validating the prepare_launch_shape / available_*_count change on #1458. The commit above is current main; nothing merged since addresses this path.

Ruled out as caused by that PR: src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp is the only bind-time reader of worker_count, so moving the core-geometry publication ahead of the bind is a no-op for tensormap_and_ringbuffer — which is the runtime both of these collectives use (@scene_test(level=3, runtime="tensormap_and_ringbuffer")).

What would narrow it down: the failing core id 15, blk:1 plus the UB offset in the device log should identify which kernel and which tile index overran. Both cases share _ALLREDUCE_MODES incores under tests/st/worker/collectives/allreduce/kernels/aiv/, so a rank-count-dependent or peer-count-dependent UB offset computation is the first place to look — the two failing configurations are 2-rank onephase and 4-rank ring, while other collective cases in the same run passed.

Related (both closed, for error-class context only): #1036 (AICore VEC UB-not-aligned → 507018, single-rank, reproducible), #897 (allreduce SCHEDULER_TIMEOUT, root cause was idle threads, not a VEC fault).

Tracked locally in KNOWN_ISSUES.md since 2026-07-24; filing here so it is visible.

[ChaoWao]

Investigated this today. No root cause yet, but two things are now settled and worth
recording so the next person doesn't re-derive them.

The suggested lead is ruled out: these kernels have no data-dependent UB address

The issue proposes "a rank-count-dependent or peer-count-dependent UB offset
computation is the first place to look". There isn't one. All five modes under
tests/st/worker/collectives/allreduce/kernels/aiv/ bind every tile to a
compile-time constant:

kernel	TASSIGN offsets	tile extent	highest UB byte touched
allreduce_onephase_kernel.cpp	0x0, 0x10000, 0x20000	Tile<Vec, float, 1, 256> = 1 KiB	0x20400
allreduce_ring_kernel.cpp	0x0, 0x10000	same	0x10400
allreduce_twophase_kernel.cpp	0x0, 0x10000, 0x20000	same	0x20400
allreduce_bidirectional_ring_kernel.cpp	0x0, 0x10000	same	0x10400
allreduce_ibing_kernel.cpp	0x0, 0x10000	same	0x10400

nranks only ever reaches the runtime tile dims (chunk_elems = ALLREDUCE_COUNT / nranks), which shrink the tile — they never move a base or
grow one past the template extent, and every kernel guards nranks against
kMaxSupportedRanks and the divisibility precondition before computing
anything. The whole footprint sits under 132 KiB of the 192 KiB AIV UB, for
every rank count, on both failing modes.

So the faulting instruction did not come from these kernels' own addressing.
That leaves: (a) the AICore was made to execute something other than the kernel
we think it was — i.e. a function_bin_addr / dispatch-payload problem, the
class #1036 ran into, or (b) the device-log line belongs to a different kernel
or a post-mortem register dump of a core killed mid-spin, and the real mechanism
is the wait that never completed.

Did not reproduce

15 full-suite iterations on a2a3 (6 + 9; a 16th was cut short when its
task-submit lease was torn down), all passing, each under a single
task-submit lease over 8 devices:

task-submit --timeout 10800 --max-time 10800 --device auto --device-num 8 \
    --run "loop.sh N \$TASK_DEVICE"   # loop body:
python -m pytest examples tests/st --platform a2a3 --device "$DEVICES"

~6 min per iteration, ASCEND_PROCESS_LOG_PATH pointed at a per-iteration dir
so a failing run's device log would have been isolated. Nothing fired. This is
consistent with the report ("expect to repeat the full suite several times"),
it just means the window is narrower than 15 runs on this box.

What would settle it next time it fires

The host-side 507018 is generic; the device log is the only thing that
separates the two candidates above. Worth capturing specifically:

• PrintErrorInfoForDavinciTask: fault kernel_name, hash, and
  GetBinAndKernelNameExceptionArgs: binSize. If binSize matches
  build/lib/a2a3/onboard/tensormap_and_ringbuffer/aicore_kernel.o, the report
  is naming the polling-dispatch executor and the faulting code is whatever it
  jumped to — which is candidate (a), not a kernel-arithmetic bug. [Bug] AICore VEC UB-not-aligned (507018) on step3p5 single-rank, while qwen3/32b passes on the same chip #1036 has the
  worked example of making that distinction.
• PrintCoreInfo: pc current, and whether the offset lands inside the
  executor's own binary size or far outside it.
• Counts of FATAL: Task Allocator Deadlock and Timeout (N cycles): producer/consumers. Per .claude/rules/running-onboard.md, if neither fired
  and only HandleTaskTimeout did, the SCHEDULER_TIMEOUT tail is a long or
  stalled op rather than a proven deadlock — which would point at (b).

Unrelated latent bug found on the way

All three comm backends zero a domain window after the point where peers can
reach it (a2a3/a5: after the handle-exchange file barrier; sim: after the
ready_count barrier), so a wipe could in principle erase a peer's barrier
signal and hang the owner's TWAIT forever. It is not the cause here:
Orchestrator.allocate_domain joins every rank's chip child before any kernel
is submitted, so no kernel can write into a peer's window before that peer's
wipe. Filed as hardening in #1500; it does not close this issue.

[ChaoWao]

Status after the investigation above, and a recommendation on how to spend time here.

#1501 records the triage procedure that investigation had to derive from scratch, as a new section in docs/troubleshooting/device-error-codes.md — the guide covered capacity codes and stalls but had nothing for an AICore addressing fault, which is the third way a run reaches the same generic 507018. It carries F1 (kernel arithmetic vs the core jumping somewhere else, via binSize), F2 (ruling the kernel in or out statically, which is what cleared these collectives), and F3 (real fault vs a post-mortem dump of a core reaped mid-spin).

It also states plainly why this issue lost its evidence, since that is the mechanical reason it is still open: outputs/<case>_<ts>/ is only created when a DFX flag is on, so a plain onboard run leaves its device log in the shared ~/ascend/log/debug/ among every other user's on the box. The one real occurrence produced exactly the artifact that would have decided F1, and it was unattributable by the time anyone looked. Whoever runs a reproduction attempt has to export ASCEND_PROCESS_LOG_PATH first, or a hit is wasted.

A correction to this issue's own text, so it does not misdirect the next reader: the "Additional Context" section I wrote suggests "a rank-count-dependent or peer-count-dependent UB offset computation is the first place to look". The investigation above disproved that — all five modes bind every tile to a compile-time constant base, nranks only divides the chunk count, and the whole footprint stays under 132 KiB of the 192 KiB UB at every rank count. That lead is closed.

Recommendation: do not chase this with blind full-suite loops. 15 iterations at ~6 min each under an exclusive 8-device lease produced nothing, on a shared box where that time has real cost. With the kernels cleared statically, the remaining candidates are a dispatch-payload problem or a reaped-mid-spin artifact, and neither becomes more likely with repetition alone.

If someone does want to force it, targeted contention beats repetition: run the allreduce L3 cases in a loop while other device-heavy cases hammer the same devices, which attacks the "contention" hypothesis directly rather than re-rolling the whole suite. But note that hypothesis rests on a single observation, and the isolated-device-log step is a precondition either way.

Otherwise this is best left to fire naturally in CI or someone's full-suite run, now that #1501 makes such an occurrence readable.

[ChaoWao]

A lead nobody has checked: this is plausibly already fixed by #1477. Posting the evidence rather than leaving this open indefinitely as "cannot reproduce".

The timeline is the whole argument

	when
the one and only observed failure	2026-07-24
#1477 merged (Fix: size CoreTracker by the device's clusters, not by a uint64_t)	2026-07-26T01:21Z
15 clean full-suite iterations	2026-07-26T14:39Z — 13 hours after the fix

Every non-reproducing run postdates the fix. The only failure predates it by two days.

And these cases were exposed to that bug

CallConfig::block_dim = 0 was the "auto" sentinel meaning take the whole device, and resolve_block_dim() returned max_block_dim_ for it — 24 clusters on a2a3. The allreduce cases never pinned block_dim; their config carries only device_count. So they ran at 24 clusters, on every run, including 2026-07-24.

#1477's CoreTracker held its state in a uint64_t, which at PLATFORM_CORES_PER_BLOCKDIM = 3 caps at 21 clusters. Cluster indices past that wrote out of bounds, onto adjacent tracker state, and assign_own_clusters() was missing the capacity guard its serial sibling had. The bisection when I fixed it was 23 clusters pass / 24 clusters fail — and auto width on a2a3 is exactly 24.

So on 2026-07-24 these cases were running the one configuration that tripped an out-of-bounds write, every time.

Why the symptom fits

This lands squarely on the investigation's candidate (a) — "the AICore was made to execute something other than the kernel we think it was" — and is consistent with what is already established:

• F2 cleared the kernels' own arithmetic. Constant TASSIGN bases, nranks only dividing the chunk count, ≤132 KiB of 192 KiB UB at every rank count. A bad UB address therefore did not come from the kernel computing one, which is exactly what corrupted dispatch state would produce.
• Both failures end in SCHEDULER_TIMEOUT. Fix: size CoreTracker by the device's clusters, not by a uint64_t #1477's own signature was 210x AICORE_EXIT to core 0 → SCHEDULER_TIMEOUT.
• Rarity fits. An out-of-bounds write corrupts whatever is adjacent; most of the time that stalls or does nothing visible, and only some corruption patterns turn into a faulting address.

What is not proven

I cannot trace the specific chain from that out-of-bounds write to VEC instruction error ... core id 15, blk:1, and I have not reproduced the failure on pre-#1477 code. This is a strong temporal correlation plus a plausible mechanism, not a demonstration.

Two ways forward

Close it as probably-fixed, reopen on recurrence. The evidence above plus 15 clean full-suite iterations is a reasonable bar, and #1501 now makes a recurrence readable if it happens.

Or settle it with a targeted experiment, which is cheap compared to what has already been spent: build ae2611d8^ (the commit before the CoreTracker fix), run only tests/st/worker/collectives/allreduce at auto width on a2a3 under task-submit, with ASCEND_PROCESS_LOG_PATH pointed at a per-run directory. If it reproduces there and not on current main, the question is closed properly. That is a hypothesis-driven run against a named suspect, not the blind full-suite repetition I argued against above.

I would rather do the second before closing, if a device is available — the first is an inference and this issue has already cost one investigation that ended in "no root cause".

[ChaoWao]

Closing as very likely fixed by #1477, on the timeline evidence above, without running the targeted pre-#1477 reproduction.

To be explicit about what that means: the fix is inferred, not demonstrated. The argument is

1. the only observed failure predates Fix: size CoreTracker by the device's clusters, not by a uint64_t #1477 by two days,
2. all 15 clean full-suite iterations postdate it by 13 hours,
3. these cases provably ran in the exact configuration that tripped it — block_dim = 0 meant "whole device" = 24 clusters on a2a3, against a CoreTracker that overran its uint64_t past 21, and the bisection when I fixed it was 23-pass / 24-fail,
4. F2 had already cleared the kernels' own addressing, which is what corrupted dispatch state rather than kernel arithmetic looks like.

What is missing is the one experiment that would settle it: build ae2611d8^, run tests/st/worker/collectives/allreduce at auto width, and see it fail there but not on main. That was offered and declined, which is a reasonable call — the cost of that run against the probability the inference is wrong.

If this signature recurs, reopen rather than filing fresh. The reasoning above becomes evidence against the CoreTracker explanation, and the investigation should go straight to candidate (a) — a dispatch-payload / function_bin_addr problem — since F2 has already ruled the kernels out statically and that need not be redone.

The triage procedure derived here is in docs/troubleshooting/device-error-codes.md (#1501), including the reason this issue lost its evidence: the harness only creates outputs/<case>_<ts>/ when a DFX flag is on, so a plain onboard run leaves its device log in the shared ~/ascend/log/debug/. Anyone chasing the next one should export ASCEND_PROCESS_LOG_PATH before the run, or a hit is wasted the same way this one was.