diff --git a/docs/troubleshooting/device-error-codes.md b/docs/troubleshooting/device-error-codes.md index 81f48ad7a..89e2fcea7 100644 --- a/docs/troubleshooting/device-error-codes.md +++ b/docs/troubleshooting/device-error-codes.md @@ -85,7 +85,7 @@ The counters *are* the decision: priority is `running > ready > waiting > orch n done`. `running=1` is always S1; `running=0, ready>0` is S3. **Only S1 and S3 are reproducible in practice.** S4, S5 and unknown are defensive -labels — see [4. Codes with no end-to-end test](#4-codes-with-no-end-to-end-test). +labels — see [5. Codes with no end-to-end test](#5-codes-with-no-end-to-end-test). ### Host-side CANN codes @@ -225,7 +225,89 @@ released its fanout reference. Check for a hung producer first (that is S1 above then verify the consumer really declares the dependency and exits. If the kernel is merely slow, raising the timeout will prove it. -## 4. Codes with no end-to-end test +## 4. Chasing down an AICore fault (UB out of bounds, VEC/CUBE instruction error) + +An AICore addressing fault reaches the host as the same generic `507018` a stall +does, usually with a `SCHEDULER_TIMEOUT` tail, because the faulting core stops +answering and a watchdog reaps the op afterwards. The device log is the only +place the fault itself appears: + +```text +VEC instruction error: the ub address out of bounds + core id 15, blk:1, errcode 0x4000000000000000 +→ aicpu exception → 507018 → sched_error_code=100 SCHEDULER_TIMEOUT +``` + +**Redirect the device log before the run that you expect to fail.** The harness +does not do this for you: `outputs/_/` is only created when a DFX flag +is on, so a plain onboard run leaves its device log in the shared +`~/ascend/log/debug/device-/` among every other user's. A fault you cannot +attribute to a file is a fault you cannot diagnose, and the evidence is gone +once the run ends: + +```bash +LOGDIR="$PWD/outputs/