Add: READMEs for the a5 bgemm and deferred-completion demos

examples/a5/tensormap_and_ringbuffer/README.md

@@ -16,15 +16,15 @@ For the `Worker` API underneath the framework, see
 | Example | What it teaches |
 | ------- | --------------- |
 | [`vector_example/`](vector_example/) | The smallest complete kernel: `f = (a+b+1)*(a+b+2) + (a+b)`. Runs on sim. |
-| [`bgemm/`](bgemm/) | Batched tiled matmul `C = A @ B` — a fixed 4×4×4 grid of 64×64 tiles over 2 batches, with AIC doing the matmul and AIV the accumulation. The clearest two-core-type example here. Runs on sim. |
+| [`bgemm/`](bgemm/) | Batched tiled matmul `C = A @ B` over a fixed 4×4×4 grid of 64×64 tiles. **One source in `kernels/mix/`, compiled twice** — cube half `TPUSH`es the intermediate into `VEC_FIFO`, vector half `TPOP`s it, so the matmul product never round-trips through GM. Runs on sim. |
 
 ## Paged attention
 
 | Example | What it teaches |
 | ------- | --------------- |
 | [`paged_attention/`](paged_attention/) | Online softmax with mixed AIC/AIV execution, bfloat16, at production scale. Onboard only. |
-| [`paged_attention_manual_scope/`](paged_attention_manual_scope/) | The same computation with explicit scope control — see [`docs/manual-scope.md`](../../../docs/manual-scope.md). Also runs on sim. |
-| [`paged_attention_unroll_manual_scope/`](paged_attention_unroll_manual_scope/) | Manual scope plus loop unrolling. Onboard only. |
+| [`paged_attention_manual_scope/`](paged_attention_manual_scope/) | The same computation with explicit scope control — all four kernels byte-identical to the baseline's, only the orchestration differs (52 lines of 288). See [`docs/manual-scope.md`](../../../docs/manual-scope.md). Also runs on sim. |
+| [`paged_attention_unroll_manual_scope/`](paged_attention_unroll_manual_scope/) | A second implementation, not a patch on the baseline: KV blocks batched into groups of `N_UNROLL`, four tasks per group instead of per block, with the kernels rewritten to match. Onboard only. |
 
 ## Asynchronous completion and cross-card transfer
 
@@ -33,8 +33,8 @@ rather than on task end. All need two dies.
 
 | Example | Mechanism |
 | ------- | --------- |
-| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. |
-| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** instead — the a5-only counterpart, and the reason this pair is worth reading side by side. Carries an extra `skipif` guard beyond the platform marker. |
+| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. Needs `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON` at build **and** run time; skipped otherwise. |
+| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** — `kernel_consumer.cpp` is byte-identical to the SDMA demo's, so the transport is the only variable. The two overlays are **mutually exclusive in one build**, so comparing them means rebuilding. |
 | [`async_notify_demo/`](async_notify_demo/) | Notification counters alongside deferred completion. |
 | [`deferred_notify_demo/`](deferred_notify_demo/) | The same shape on the simulator. |
 

examples/a5/tensormap_and_ringbuffer/bgemm/README.md

@@ -0,0 +1,66 @@
+# bgemm — one source, two cores, no round trip through GM
+
+Batched tiled matmul `C = A @ B` over a fixed 4×4×4 grid of 64×64 tiles, 2
+batches. The arithmetic is unremarkable. What makes this the example to read on
+a5 is the directory it keeps its kernel in:
+
+```text
+kernels/mix/kernel_bgemm.cpp        # not kernels/aic/ + kernels/aiv/
+```
+
+## One file, compiled twice
+
+The same source is registered as **two** incores with different `core_type`s,
+and the preprocessor picks the half each build compiles:
+
+| `func_id` | Name | `core_type` | Guard | Does |
+| --------- | ---- | ----------- | ----- | ---- |
+| 0 | `GEMM` | `aic` | `__DAV_CUBE__` | `TLOAD`, `TMATMUL`, `TPUSH` |
+| 1 | `ADD` | `aiv` | `__DAV_VEC__` | `TPOP`, `TADD`, `TSTORE` |
+
+Both entries name the same `source`. Both see the same three-tensor `args[]` —
+`args[0] = A`, `args[1] = B`, `args[2] = C` — which is what "cooperative mix"
+means here: the halves are not two kernels passing data, they are two views of
+one kernel.
+
+**The intermediate `P = A[m,k] @ B[k,n]` never touches GM.** The cube half
+`TPUSH`es it into `VEC_FIFO` and the vector half `TPOP`s it straight out. Only
+the accumulator `C` is read and written through global memory, once per
+`GRID_K` iteration.
+
+That is the whole point of the mix form: a matmul feeding an elementwise
+accumulate is the canonical case where a GM round trip for the intermediate is
+pure cost.
+
+`"mix"` is a recognised sentinel elsewhere too — `deps_viewer` and
+`l0_swimlane` collapse a multi-core-type task to it, and
+`sched_overhead_analysis` labels it `MIX`.
+
+## `C` is INOUT, not OUT
+
+The vector half `TLOAD`s `C` from GM, adds, and stores it back, `GRID_K` times.
+`C` is a zero-initialised accumulator, so those host-provided zeros must be
+staged host-to-device — making it **read-before-write**. Declaring it `OUT`
+would skip the H2D stage and accumulate onto whatever the buffer happened to
+hold.
+
+## Run
+
+```bash
+pytest examples/a5/tensormap_and_ringbuffer/bgemm --platform a5sim
+pytest examples/a5/tensormap_and_ringbuffer/bgemm --platform a5 --device 0
+```
+
+One case, `default`, on both `a5sim` and `a5` — one of the few a5 examples you
+can run without a die. Shapes are compile-time constants (`TILE_* = 64`,
+`GRID_* = 4`, `BATCH = 2`) and must match the kernel; the case takes no params.
+
+Wrap hardware runs in `task-submit` on a shared box.
+
+## Compare with
+
+[`examples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm/`](../../../a2a3/tensormap_and_ringbuffer/benchmark_bgemm/)
+is a2a3's matmul example and is **not** a port of this one. It splits AIC and
+AIV into separate sources, routes the intermediate through GM, and exists to
+sweep shapes rather than to demonstrate the mix form. Same operation, opposite
+emphasis.

examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md

@@ -0,0 +1,52 @@
+# sdma_async_completion_demo — deferred completion over SDMA
+
+Two ranks, one transfer, one dependency:
+
+```text
+producer:  TGET_ASYNC the peer rank's input from the HCCL window into local
+           `out`, then register the PTO AsyncEvent via defer_pto_async_event
+consumer:  depends on the producer's output, writes result = out + 1
+```
+
+Checking `out` and `result` tests two separate things: `out` proves SDMA
+completion polling saw the transfer land, `result` proves the deferred-release
+dependency held the consumer until it had. A consumer that ran early would
+still produce a plausible `result` from a partially written `out`, which is why
+both are checked.
+
+The remote address is plain symmetric-window arithmetic — take the local
+pointer's offset from `windowsIn[rankId]` and add it to `windowsIn[peer_rank]`.
+Every rank's window is laid out identically, so an offset is rank-independent.
+
+## Gated behind an overlay that is off by default
+
+Unlike its a2a3 namesake, the a5 demo needs the PTO async-SDMA workspace
+compiled into the host runtime:
+
+| Gate | Effect |
+| ---- | ------ |
+| `@pytest.mark.platforms(["a5"])` | deselected on any other `--platform` |
+| `@pytest.mark.device_count(2)` | needs two dies |
+| `@pytest.mark.skipif(...)` | skipped unless `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` is `1` / `ON` / `TRUE` / `YES` |
+
+The CMake option defaults `OFF`, so a stock build skips this test even on a5
+hardware — **a green CI run says nothing about SDMA completion on a5.**
+
+```bash
+SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON pip install --no-build-isolation -e .
+SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON \
+  pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo \
+  --platform a5 --device 0-1
+```
+
+The variable is read twice: `simpler_setup/runtime_builder.py` forwards it to
+CMake so the overlay is compiled in, and the test reads it from the environment
+to decide whether to skip.
+
+Wrap the hardware run in `task-submit` on a shared box.
+
+## Compare with
+
+- [`../urma_deferred_completion_demo/`](../urma_deferred_completion_demo/) — the same protocol over URMA. `kernel_consumer.cpp` is byte-identical; only the transfer kernel, its completion header, and the build flag differ. **The two overlays are mutually exclusive in one build**, so comparing them means rebuilding — that README has the detail.
+- [`examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/`](../../../a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/) — the a2a3 port of this demo, which needs no overlay flag.
+- [`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the overlays are gated off and the checklist for re-enabling them.

examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/README.md

@@ -0,0 +1,89 @@
+# urma_deferred_completion_demo — the same protocol, over URMA
+
+The a5-only twin of
+[`../sdma_async_completion_demo/`](../sdma_async_completion_demo/). Both run the
+identical two-rank protocol; they differ in which transport moves the bytes and
+which completion path reports it done.
+
+```text
+producer:  TGET_ASYNC the peer's input from the window into local `out`,
+           register the AsyncEvent through the deferred-completion path
+consumer:  depends on that producer output, writes result = out + 1
+```
+
+Checking both `out` and `result` is what makes it a test of two things at once:
+`out` proves completion polling saw the transfer land, and `result` proves the
+deferred-release dependency held the consumer back until it had.
+
+## What actually differs from the SDMA demo
+
+`kernels/aiv/kernel_consumer.cpp` is **byte-identical** between the two
+directories. Only the transfer kernel and the orchestration change:
+
+| What | URMA | SDMA |
+| ---- | ---- | ---- |
+| Transfer kernel | `kernel_urma_tget_async.cpp` | `kernel_sdma_tget_async.cpp` |
+| Completion header | `backend/urma/urma_completion_kernel.h` | `backend/sdma/sdma_completion_kernel.h` |
+| Build flag | `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` | `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` |
+
+Read them side by side and the transport is the only variable — which is
+exactly what you want when deciding which one a workload should use.
+
+## Both overlays are off by default, and they are mutually exclusive
+
+This is the part to know before trying to run either.
+
+```cmake
+option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE "..." OFF)
+option(SIMPLER_ENABLE_PTO_URMA_WORKSPACE "..." OFF)
+if(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE AND SIMPLER_ENABLE_PTO_URMA_WORKSPACE)
+    message(FATAL_ERROR "... mutually exclusive because CommContext exposes a
+                         single workSpace/workSpaceSize pair")
+endif()
+```
+
+— `src/a5/platform/onboard/host/CMakeLists.txt`. **One host runtime build can
+carry one overlay, not both**, because `CommContext` has a single
+`workSpace` / `workSpaceSize` pair to hand the kernel. So the two demos cannot
+pass in the same build; comparing them means rebuilding.
+
+The kernel checks for this at run time too: `workSpace == 0` means the overlay
+was not built in.
+
+## Consequently it is doubly gated, and never runs in CI
+
+| Gate | Effect |
+| ---- | ------ |
+| `@pytest.mark.platforms(["a5"])` | deselected on any other `--platform` |
+| `@pytest.mark.device_count(2)` | needs two dies |
+| `@pytest.mark.skipif(not _urma_workspace_enabled())` | skipped unless `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` is one of `1` / `ON` / `TRUE` / `YES` in the environment |
+| `run()` raises | if `platform != "a5"` or the device count is not 2, and re-checks the env var before doing any work |
+
+Since the CMake option defaults `OFF`, a stock build skips this test even on a5
+hardware. **A green CI run says nothing about URMA.** Treat it as a manual
+bring-up check, not as coverage.
+
+## Run
+
+```bash
+# 1. rebuild the host runtime with the overlay on
+SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON pip install --no-build-isolation -e .
+
+# 2. run it, with the same variable visible to pytest (the skipif reads the env)
+SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON \
+  pytest examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo \
+  --platform a5 --device 0-1
+```
+
+The variable is needed twice for different reasons: `runtime_builder.py`
+forwards it to CMake so the overlay is compiled in, and the test reads it from
+the environment to decide whether to skip. Setting only one of the two gives
+you either a skipped test or a `workSpace == 0` failure.
+
+Wrap the hardware run in `task-submit` on a shared box.
+
+## See also
+
+[`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the
+overlays are gated off, what is gated, and the checklist for re-enabling them
+(issue [#1315](https://github.com/hw-native-sys/simpler/issues/1315)).