From 3e38c65b3e637ead42dcf07c41ce7b9e5c526be2 Mon Sep 17 00:00:00 2001
From: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
Date: Sun, 26 Jul 2026 08:05:24 -0700
Subject: [PATCH] Add: AICore fault triage to the device error code guide
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The guide covered capacity codes and stalls but had nothing for an AICore
addressing fault, which is the third way a run reaches the same generic
507018. #1489 spent an investigation re-deriving the procedure from
scratch and still could not close, so record it.

Three steps, ordered so the cheap disqualifications come first:

- F1 separates "the kernel computed a bad address" from "the core was
  made to execute something that is not the kernel". `binSize` in
  GetBinAndKernelNameExceptionArgs matching the runtime's own
  aicore_kernel.o means the report is naming the polling-dispatch
  executor, so the fault is a dispatch-payload problem and no amount of
  reading the kernel's arithmetic will find it. #1036 is the worked
  example.
- F2 rules the kernel in or out statically. Constant TASSIGN bases plus
  template tile extents do not make an out-of-range address impossible —
  the constants can overrun on their own — but they make it decidable on
  paper, because runtime values that only shrink a tile move no base and
  grow no extent. Summing the highest byte reached and comparing against
  the UB size settles it either way without touching the device. This is
  what cleared the allreduce collectives in #1489.
- F3 separates a real fault from a post-mortem register dump of a core
  reaped mid-spin, by counting which detector actually fired.

#1489 has since been closed as very likely fixed by #1477 — a CoreTracker
whose uint64_t state overran past 21 clusters while those cases ran at the
device's full 24. F2's verdict held: the kernels were never at fault, and
the answer was in F1's second case all along. The section says so, because
a worked example that names its own outcome is worth more than one that
trails off.

Also state where the device log has to be redirected and why the harness
will not do it: outputs/<case>_<ts>/ exists only when a DFX flag is on, so
a plain onboard run leaves its log in the shared ~/ascend/log/debug/ with
every other user's, and the evidence is unattributable once the run ends.
That is how #1489's only real occurrence lost the one artifact that would
have decided F1.

Renumber the trailing section and fix the in-page link that pointed at
its old number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 docs/troubleshooting/device-error-codes.md | 86 +++++++++++++++++++++-
 1 file changed, 84 insertions(+), 2 deletions(-)

diff --git a/docs/troubleshooting/device-error-codes.md b/docs/troubleshooting/device-error-codes.md
index 81f48ad7a5..89e2fcea7b 100644
--- a/docs/troubleshooting/device-error-codes.md
+++ b/docs/troubleshooting/device-error-codes.md
@@ -85,7 +85,7 @@ The counters *are* the decision: priority is `running > ready > waiting > orch n
 done`. `running=1` is always S1; `running=0, ready>0` is S3.
 
 **Only S1 and S3 are reproducible in practice.** S4, S5 and unknown are defensive
-labels — see [4. Codes with no end-to-end test](#4-codes-with-no-end-to-end-test).
+labels — see [5. Codes with no end-to-end test](#5-codes-with-no-end-to-end-test).
 
 ### Host-side CANN codes
 
@@ -225,7 +225,89 @@ released its fanout reference. Check for a hung producer first (that is S1 above
 then verify the consumer really declares the dependency and exits. If the kernel is
 merely slow, raising the timeout will prove it.
 
-## 4. Codes with no end-to-end test
+## 4. Chasing down an AICore fault (UB out of bounds, VEC/CUBE instruction error)
+
+An AICore addressing fault reaches the host as the same generic `507018` a stall
+does, usually with a `SCHEDULER_TIMEOUT` tail, because the faulting core stops
+answering and a watchdog reaps the op afterwards. The device log is the only
+place the fault itself appears:
+
+```text
+VEC instruction error: the ub address out of bounds
+  core id 15, blk:1, errcode 0x4000000000000000
+→ aicpu exception → 507018 → sched_error_code=100 SCHEDULER_TIMEOUT
+```
+
+**Redirect the device log before the run that you expect to fail.** The harness
+does not do this for you: `outputs/<case>_<ts>/` is only created when a DFX flag
+is on, so a plain onboard run leaves its device log in the shared
+`~/ascend/log/debug/device-<id>/` among every other user's. A fault you cannot
+attribute to a file is a fault you cannot diagnose, and the evidence is gone
+once the run ends:
+
+```bash
+LOGDIR="$PWD/outputs/<label>/ascend"; mkdir -p "$LOGDIR"   # driver fopens; it will not mkdir
+export ASCEND_PROCESS_LOG_PATH="$LOGDIR"
+```
+
+See the "Device logs" section of
+[`.claude/rules/running-onboard.md`](../../.claude/rules/running-onboard.md).
+
+### F1: is it the kernel's own addressing, or did the core jump somewhere else?
+
+These two look identical from the host and have nothing in common as bugs, so
+settle it before reading any kernel arithmetic:
+
+- **`PrintErrorInfoForDavinciTask`** — `fault kernel_name`, `hash`, and
+  `GetBinAndKernelNameExceptionArgs: binSize`. If `binSize` matches the
+  runtime's own `aicore_kernel.o` rather than your kernel, the report is naming
+  the **polling-dispatch executor**, and the faulting code is whatever it jumped
+  to — a dispatch-payload / `function_bin_addr` problem, not kernel arithmetic.
+- **`PrintCoreInfo`** — `pc current`. Whether that offset lands inside the
+  executor's binary size or far outside it tells you the same thing
+  independently.
+
+[#1036](https://github.com/hw-native-sys/simpler/issues/1036) has the worked
+example of making this distinction.
+
+### F2: rule the kernel's own addressing in or out statically
+
+Before instrumenting, check whether the kernel *can* compute an out-of-range UB
+address at all. Compile-time `TASSIGN` offsets and template-derived tile extents
+do not make that impossible — the constants themselves can overrun — but they
+make it **decidable on paper**, because runtime values that only shrink a tile
+(a per-rank chunk count, a shorter row) move no base and grow no extent.
+
+So compute it. Sum the highest byte each `TASSIGN` base plus its tile extent
+reaches and compare against the AIV UB size:
+
+- **Fits for every input the guards admit** → the address did not come from
+  here, and you are in F1's second case.
+- **Does not fit** → you have found the bug without touching the device.
+
+The allreduce collectives were cleared this way in
+[#1489](https://github.com/hw-native-sys/simpler/issues/1489): all five modes
+bind every tile to a constant base, `nranks` only divides the chunk count, and
+the whole footprint stays under 132 KiB of the 192 KiB UB at every rank count.
+
+That case is worth following to its end, because F2's verdict held. The fault
+was not in the kernels; it was almost certainly
+[#1477](https://github.com/hw-native-sys/simpler/pull/1477), a `CoreTracker`
+whose `uint64_t` state overran past 21 clusters while those cases ran at the
+device's full 24. Corrupted dispatch state, F1's second case — reached by
+believing F2 rather than re-reading the kernel arithmetic a second time.
+
+### F3: separate a real fault from a post-mortem register dump
+
+A core killed while spinning can produce a fault-looking line that describes the
+state it was reaped in rather than an instruction it executed. Count the
+detectors, per [`running-onboard.md`](../../.claude/rules/running-onboard.md):
+if `FATAL: Task Allocator Deadlock` and `Timeout (N cycles):
+producer/consumers` are both **zero** and only `HandleTaskTimeout` fired, then
+nothing on the device proved a fault or a deadlock — treat the wait that never
+completed as the primary object of the investigation, not the addressing line.
+
+## 5. Codes with no end-to-end test
 
 Three codes and three stall sub-classes have no ST — not from neglect, but because
 they cannot be provoked. Recording why, so the next person does not re-derive it