Update: re-harvest qwen3_14b_decode and scale it to all 40 layers

.claude/skills/l0-swimlane/SKILL.md

@@ -81,7 +81,6 @@ Verified examples (slots read from source):
 | Test | Loop bound | Flag |
 | ---- | ---------- | ---- |
 | `paged_attention_unroll` | `aic_qk_matmul.cpp` `args[4] = n_blocks` (scalar) | `--set-arg 4=4` |
-| `qwen3_14b_decode` (fa_fused) | `fa_fused_aic/aiv.cpp` outer loop `for(i=block_idx; i<v1[0]; i+=24)`; `v1[0]` = slot 0 `fa_total` (a 1-elem **INT32 tensor** read as the work-item count) | `--set-arg 0=96` → `ceil(96/24)=4` blocks. Slot 0 is 0 in replay → empty trace without this |
 | `batch_paged_attention` | `context_lens` **tensor** (slot 1; 2nd `params_sf` `add_input`); the SF kernel (func 1) derives per-batch blocks from its content | `--set-arg 1=512` |
 
 ### `--spmd-block-num N` — SPMD grid width

.claude/skills/multi-repo-qwen-setup/SKILL.md

@@ -25,17 +25,21 @@ styles**. They measure different things; don't confuse them:
 | repo | simpler (this) | pypto-serving | pypto-lib |
 | entry | qwen3_14b_decode | npu_generate.py | decode_fwd.py |
 | cross-repo? | no | yes | yes |
-| measures | 2-layer decode correctness/timing | end-to-end TPOT | device decode TPOT + sched DFX |
+| measures | full 40-layer decode correctness/timing | end-to-end TPOT | device decode TPOT + sched DFX |
 | see | this section | sections 2-5 | section 6 |
 
 - **Path 0 — in-repo example (simpler itself, NO cross-repo).** Start here
   if you just want qwen3 decode running on simpler. simpler ships a
   self-contained SceneTestCase at
-  `examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/` — a fused chunk
-  of **two** Qwen3-14B decode layers (harvested pypto codegen: 8 AIC + 27
-  AIV + orchestration, golden in `simpler_setup/goldens/qwen3_14b_decode.py`).
-  No pypto / pypto-lib / JIT descent — it builds and runs like any simpler
-  example. Onboard rules still apply (per-die lock + `onboard-arch-precheck`):
+  `examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/` — **all 40**
+  Qwen3-14B decode layers as one fused dispatch (harvested pypto codegen: 18
+  AIC + 16 AIV + orchestration, plus the vendored CANN FusedInferAttentionScore
+  extern under `kernels/paged_attention_cce/`; golden in
+  `simpler_setup/goldens/qwen3_14b_decode.py`). No pypto / pypto-lib / JIT
+  descent — it builds and runs like any simpler example. Budget ~5 min wall and
+  a 38 GiB fixture (weights + paged KV, bf16), and note it needs CANN devkit
+  headers to build the attention extern. Onboard rules still apply (per-die
+  lock + `onboard-arch-precheck`):
 
   ```bash
   .claude/skills/onboard-arch-precheck/check.sh a2a3 || exit 1

.github/workflows/ci.yml

@@ -595,11 +595,31 @@ jobs:
           # them on their own dedicated device(s) in the SDMA step below, so an
           # SDMA fault's slow teardown can never mix with ordinary fault recovery.
           SDMA_IGNORE="--ignore=examples/a2a3/tensormap_and_ringbuffer/prefetch_async_demo --ignore=examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo"
+          # qwen3_14b_decode runs all 40 Qwen3-14B decode layers against a 38 GiB
+          # fixture and takes ~5 min on its own — over half this sweep's 600 s
+          # session budget. It gets its own step below with its own budget, so the
+          # short timeout here keeps its job: catching a genuine hang.
+          HEAVY_IGNORE="--ignore=examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode"
           if [ "$(uname -m)" = "x86_64" ]; then
-            python -m pytest examples tests/st $SDMA_IGNORE --platform a2a3 --device ${DEVICE_RANGE} -v --require-pto-isa --pto-session-timeout 600
+            python -m pytest examples tests/st $SDMA_IGNORE $HEAVY_IGNORE --platform a2a3 --device ${DEVICE_RANGE} -v --require-pto-isa --pto-session-timeout 600
           else
             task-submit --timeout 1800 --max-time 1800 --device auto --device-num "$DEVICE_NUM" \
-              --run "python -m pytest examples tests/st $SDMA_IGNORE --platform a2a3 --device \$TASK_DEVICE -v --require-pto-isa --pto-session-timeout 600"
+              --run "python -m pytest examples tests/st $SDMA_IGNORE $HEAVY_IGNORE --platform a2a3 --device \$TASK_DEVICE -v --require-pto-isa --pto-session-timeout 600"
+          fi
+
+      # qwen3_14b_decode is the full 40-layer Qwen3-14B decode — a 38 GiB fixture
+      # (x40-stacked weights + paged KV) and ~5 min wall. It runs here on its own
+      # device with its own session budget rather than eating the general sweep's.
+      - name: Qwen3-14B 40-layer decode (a2a3)
+        run: |
+          source /usr/local/Ascend/cann/set_env.sh
+          source .venv/bin/activate
+          QWEN_TEST="examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode"
+          if [ "$(uname -m)" = "x86_64" ]; then
+            python -m pytest $QWEN_TEST --platform a2a3 --device ${DEVICE_RANGE} -v --require-pto-isa --pto-session-timeout 1800
+          else
+            task-submit --timeout 3600 --max-time 3600 --device auto --device-num 1 \
+              --run "python -m pytest $QWEN_TEST --platform a2a3 --device \$TASK_DEVICE -v --require-pto-isa --pto-session-timeout 1800"
           fi
 
       # SDMA pytest — every SDMA test runs here on its own dedicated device(s),

.pre-commit-config.yaml

@@ -23,23 +23,25 @@ repos:
           entry: python tests/lint/check_english_only.py
           language: python
           language_version: python3
-          exclude: ^(3rdparty/|docs/zh-cn/|README\.zh-CN\.md)
+          exclude: ^(3rdparty/|docs/zh-cn/|README\.zh-CN\.md)|vendor/
 
     - repo: https://github.com/pre-commit/pre-commit-hooks
       rev: v4.6.0
       hooks:
         - id: check-added-large-files
         - id: check-yaml
         - id: end-of-file-fixer
+          exclude: ^3rdparty/|vendor/
         - id: trailing-whitespace
+          exclude: ^3rdparty/|vendor/
 
     # C++ linting
     - repo: https://github.com/pre-commit/mirrors-clang-format
       rev: v21.1.0
       hooks:
         - id: clang-format
           types_or: [c++, c]
-          exclude: ^3rdparty/
+          exclude: ^3rdparty/|vendor/
 
     - repo: local
       hooks:
@@ -70,7 +72,7 @@ repos:
       rev:  2.0.0
       hooks:
         - id: cpplint
-          exclude: ^3rdparty/
+          exclude: ^3rdparty/|vendor/
           args:
             - --root=include
             - --linelength=120

docs/dfx/dfx-buffer-capacity-audit.md

@@ -113,7 +113,12 @@ overwrite count.
 | paged_attention_unroll_manual_scope | 3/1 drop0 | 0/49 | 3/21 | 7/2 | 3/1 ovf0 |
 | benchmark_bgemm | 3/1 drop0 | 0/33 | 3/17 | 7/1 | 3/0 ovf0 |
 | paged_attention_ringbuffer | 3/1 drop0 | 0/25 | 3/12 | 7/1 | 3/1 ovf0 |
-| qwen3_14b_decode | 3/1 drop0 | 0/23 | 3/11 | 7/1 | 3/1 ovf0 |
+
+`qwen3_14b_decode` was measured here too, but only as a 2-layer chunk; it now
+runs all 40 layers, so its peaks need re-measuring before they mean anything and
+the stale row was dropped. Expect dep_gen and l2_swimlane to overflow on the
+full graph. It was not the worst case in any column, so the margins below are
+unaffected.
 
 Converted to worst-case margins (`free = lowest/cap`, `ready = 1 − peak/cap`,
 verdict reads the weaker dimension):

docs/dfx/l0-swimlane-profiling.md

@@ -257,7 +257,7 @@ categories — one representative each, with its verified `--func-id`. The
 runnable commands follow the table, wrapped in `task-submit` (the step-5
 `msprof` collect takes a device on the shared box). Most use
 `--platform a2a3sim` (the dump runs off-NPU); `alternating_matmul_add`,
-`paged_attention_unroll`, and `qwen3_14b_decode`, whose `CASES` declare
+and `paged_attention_unroll`, whose `CASES` declare
 **no `a2a3sim`**, are grouped separately and use `--platform a2a3` after an
 arch-precheck (the case must declare the `--platform` you pass — §3.1).
 
@@ -271,7 +271,6 @@ arch-precheck (the case must declare the `--platform` you pass — §3.1).
 | SPMD mix, 2 AIV share a source | `spmd_multiblock_mix` | `--func-id 0,1,2` | func 1 & 2 are distinct ids but **both `kernel_spmd_mix.cpp`** → the 2 AIV collapse to one (both lanes run it). Routes by `get_sub_block_id` (slot 49) → in replay both lanes read `sub_block_id=0`; AIV0/AIV1 differ only by write offset, so the pipeline stays representative. (The same-source collapse also covers the duplicate-func_id `[0,1,1]` shape an SPMD mix produces when `aiv0 = aiv1`.) |
 | Paged-attn, loop = scalar | `paged_attention_unroll` | `--func-id 0 --set-arg 4=4` | QK stage; `n_blocks` scalar (slot 4) → shrink to 4 ([§7.2](#72---set-arg-floor-for-a-loop-count-without-distortion)) |
 | Paged-attn, loop = control tensor | `batch_paged_attention` | `--func-id 1 --set-arg 1=512 --case CaseSmall1` | SF reads `context_lens` (**slot 1**) content (`aiv_softmax_prepare.cpp`); `--set-arg 1=512` fills it uniformly → shrinks the derived per-batch block count |
-| Real SPMD workload | `qwen3_14b_decode` | `--func-id 16,17 --set-arg 0=96` | the `fa_fused` mix `{16,17,17}` (AIC + 2 same-source AIV → collapses). a2a3-only. `--set-arg 0=96` sets slot 0 `fa_total` (the outer work-item count) → `ceil(96/24)=4` fa blocks → real QK/PV `MMAD` + online-softmax (`VMAX`/`VEXP`/`VSUB`/`VCADD`) lanes. Slot 0 defaults to 0 in replay → empty trace, so this `--set-arg` is required. camodel simulates ~19k instrs cycle-by-cycle — expect minutes, not a hang (§7.1) |
 
 Runnable commands (one per category):
 
@@ -302,12 +301,15 @@ L0a="python -m simpler_setup.tools.l0_swimlane --platform a2a3 -g"
 task-submit --device auto --max-time 1800 --run "$L0a --func-id 0 --test $T/alternating_matmul_add/test_alternating_matmul_add.py"
 # Paged-attn, loop = scalar (shrink n_blocks to 4)
 task-submit --device auto --max-time 1800 --run "$L0a --func-id 0 --set-arg 4=4 --test $T/paged_attention_unroll/test_paged_attention_unroll.py"
-# Real SPMD workload — qwen3 fa_fused mix {16,17,17}. --set-arg 0=96 sets slot 0
-# (fa_total = outer work-item count) → ceil(96/24)=4 fa blocks → real MMAD + online
-# softmax. Slot 0 is 0 in replay otherwise → empty trace. camodel takes minutes (§7.1).
-task-submit --device auto --max-time 1800 --run "$L0a --func-id 16,17 --set-arg 0=96 --test $E/qwen3_14b_decode/test_qwen3_14b_decode.py"
 ```
 
+`qwen3_14b_decode` used to serve as the "real SPMD workload" row here, driving
+its generated `fa_fused` mix with `--set-arg 0=96` on the `fa_total` work-item
+count. Its attention is now a CANN `FusedInferAttentionScore` extern whose work
+distribution comes from tiling metadata that `paged_attention_tiling_cce` builds
+at runtime, not from a replayable scalar, so there is no `--set-arg` that shrinks
+it — it is no longer an l0 target.
+
 **Not l0 targets (excluded).** Runtime-mechanics tests (`orch_so_cache`,
 `prepared_callable`, `dynamic_register`, `l3_group`, `l3_dependency`,
 `l3_l2_orch_comm`, `aicore_op_timeout`, `scope_stats`); comm / notify

docs/testing.md

@@ -633,6 +633,17 @@ Key fields:
 - `CASES[].platforms`: which platforms each case supports (sim names end in "sim")
 - `runtime`: which runtime to use
 - `CALLABLE.orchestration.source` / `CALLABLE.incores[].source`: paths relative to the test file
+- `CALLABLE.incores[].extra_include_dirs`: extra `-I` paths for that one kernel,
+  on top of the runtime/platform includes every kernel already gets. Entries may
+  be relative to the test file, or use `$VAR` — `$ASCEND_HOME_PATH/aarch64-linux/asc/include`
+  keeps a CANN-linked kernel off machine-specific paths. They are resolved when
+  the kernel is **compiled**, not when its module is imported, so a case that
+  declares an SDK dependency is still collectable on sim/macOS runners that do
+  not have that SDK. Paths that do not exist are dropped, letting one candidate
+  list span SDK layouts; an unset variable raises, and so does a declared set
+  where nothing survived (that means the SDK is missing). Used by
+  [`qwen3_14b_decode`](../examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/)
+  for its CANN attention extern.
 
 ### Output Validation