diff --git a/examples/a5/tensormap_and_ringbuffer/README.md b/examples/a5/tensormap_and_ringbuffer/README.md index 714f7b1dbe..e13857a00f 100644 --- a/examples/a5/tensormap_and_ringbuffer/README.md +++ b/examples/a5/tensormap_and_ringbuffer/README.md @@ -16,15 +16,15 @@ For the `Worker` API underneath the framework, see | Example | What it teaches | | ------- | --------------- | | [`vector_example/`](vector_example/) | The smallest complete kernel: `f = (a+b+1)*(a+b+2) + (a+b)`. Runs on sim. | -| [`bgemm/`](bgemm/) | Batched tiled matmul `C = A @ B` — a fixed 4×4×4 grid of 64×64 tiles over 2 batches, with AIC doing the matmul and AIV the accumulation. The clearest two-core-type example here. Runs on sim. | +| [`bgemm/`](bgemm/) | Batched tiled matmul `C = A @ B` over a fixed 4×4×4 grid of 64×64 tiles. **One source in `kernels/mix/`, compiled twice** — cube half `TPUSH`es the intermediate into `VEC_FIFO`, vector half `TPOP`s it, so the matmul product never round-trips through GM. Runs on sim. | ## Paged attention | Example | What it teaches | | ------- | --------------- | | [`paged_attention/`](paged_attention/) | Online softmax with mixed AIC/AIV execution, bfloat16, at production scale. Onboard only. | -| [`paged_attention_manual_scope/`](paged_attention_manual_scope/) | The same computation with explicit scope control — see [`docs/manual-scope.md`](../../../docs/manual-scope.md). Also runs on sim. | -| [`paged_attention_unroll_manual_scope/`](paged_attention_unroll_manual_scope/) | Manual scope plus loop unrolling. Onboard only. | +| [`paged_attention_manual_scope/`](paged_attention_manual_scope/) | The same computation with explicit scope control — all four kernels byte-identical to the baseline's, only the orchestration differs (52 lines of 288). See [`docs/manual-scope.md`](../../../docs/manual-scope.md). Also runs on sim. | +| [`paged_attention_unroll_manual_scope/`](paged_attention_unroll_manual_scope/) | A second implementation, not a patch on the baseline: KV blocks batched into groups of `N_UNROLL`, four tasks per group instead of per block, with the kernels rewritten to match. Onboard only. | ## Asynchronous completion and cross-card transfer @@ -33,8 +33,8 @@ rather than on task end. All need two dies. | Example | Mechanism | | ------- | --------- | -| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. | -| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** instead — the a5-only counterpart, and the reason this pair is worth reading side by side. Carries an extra `skipif` guard beyond the platform marker. | +| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. Needs `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON` at build **and** run time; skipped otherwise. | +| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** — `kernel_consumer.cpp` is byte-identical to the SDMA demo's, so the transport is the only variable. The two overlays are **mutually exclusive in one build**, so comparing them means rebuilding. | | [`async_notify_demo/`](async_notify_demo/) | Notification counters alongside deferred completion. | | [`deferred_notify_demo/`](deferred_notify_demo/) | The same shape on the simulator. | diff --git a/examples/a5/tensormap_and_ringbuffer/bgemm/README.md b/examples/a5/tensormap_and_ringbuffer/bgemm/README.md new file mode 100644 index 0000000000..8e2c0d0f92 --- /dev/null +++ b/examples/a5/tensormap_and_ringbuffer/bgemm/README.md @@ -0,0 +1,66 @@ +# bgemm — one source, two cores, no round trip through GM + +Batched tiled matmul `C = A @ B` over a fixed 4×4×4 grid of 64×64 tiles, 2 +batches. The arithmetic is unremarkable. What makes this the example to read on +a5 is the directory it keeps its kernel in: + +```text +kernels/mix/kernel_bgemm.cpp # not kernels/aic/ + kernels/aiv/ +``` + +## One file, compiled twice + +The same source is registered as **two** incores with different `core_type`s, +and the preprocessor picks the half each build compiles: + +| `func_id` | Name | `core_type` | Guard | Does | +| --------- | ---- | ----------- | ----- | ---- | +| 0 | `GEMM` | `aic` | `__DAV_CUBE__` | `TLOAD`, `TMATMUL`, `TPUSH` | +| 1 | `ADD` | `aiv` | `__DAV_VEC__` | `TPOP`, `TADD`, `TSTORE` | + +Both entries name the same `source`. Both see the same three-tensor `args[]` — +`args[0] = A`, `args[1] = B`, `args[2] = C` — which is what "cooperative mix" +means here: the halves are not two kernels passing data, they are two views of +one kernel. + +**The intermediate `P = A[m,k] @ B[k,n]` never touches GM.** The cube half +`TPUSH`es it into `VEC_FIFO` and the vector half `TPOP`s it straight out. Only +the accumulator `C` is read and written through global memory, once per +`GRID_K` iteration. + +That is the whole point of the mix form: a matmul feeding an elementwise +accumulate is the canonical case where a GM round trip for the intermediate is +pure cost. + +`"mix"` is a recognised sentinel elsewhere too — `deps_viewer` and +`l0_swimlane` collapse a multi-core-type task to it, and +`sched_overhead_analysis` labels it `MIX`. + +## `C` is INOUT, not OUT + +The vector half `TLOAD`s `C` from GM, adds, and stores it back, `GRID_K` times. +`C` is a zero-initialised accumulator, so those host-provided zeros must be +staged host-to-device — making it **read-before-write**. Declaring it `OUT` +would skip the H2D stage and accumulate onto whatever the buffer happened to +hold. + +## Run + +```bash +pytest examples/a5/tensormap_and_ringbuffer/bgemm --platform a5sim +pytest examples/a5/tensormap_and_ringbuffer/bgemm --platform a5 --device 0 +``` + +One case, `default`, on both `a5sim` and `a5` — one of the few a5 examples you +can run without a die. Shapes are compile-time constants (`TILE_* = 64`, +`GRID_* = 4`, `BATCH = 2`) and must match the kernel; the case takes no params. + +Wrap hardware runs in `task-submit` on a shared box. + +## Compare with + +[`examples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm/`](../../../a2a3/tensormap_and_ringbuffer/benchmark_bgemm/) +is a2a3's matmul example and is **not** a port of this one. It splits AIC and +AIV into separate sources, routes the intermediate through GM, and exists to +sweep shapes rather than to demonstrate the mix form. Same operation, opposite +emphasis. diff --git a/examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md b/examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md new file mode 100644 index 0000000000..65038bbb52 --- /dev/null +++ b/examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/README.md @@ -0,0 +1,52 @@ +# sdma_async_completion_demo — deferred completion over SDMA + +Two ranks, one transfer, one dependency: + +```text +producer: TGET_ASYNC the peer rank's input from the HCCL window into local + `out`, then register the PTO AsyncEvent via defer_pto_async_event +consumer: depends on the producer's output, writes result = out + 1 +``` + +Checking `out` and `result` tests two separate things: `out` proves SDMA +completion polling saw the transfer land, `result` proves the deferred-release +dependency held the consumer until it had. A consumer that ran early would +still produce a plausible `result` from a partially written `out`, which is why +both are checked. + +The remote address is plain symmetric-window arithmetic — take the local +pointer's offset from `windowsIn[rankId]` and add it to `windowsIn[peer_rank]`. +Every rank's window is laid out identically, so an offset is rank-independent. + +## Gated behind an overlay that is off by default + +Unlike its a2a3 namesake, the a5 demo needs the PTO async-SDMA workspace +compiled into the host runtime: + +| Gate | Effect | +| ---- | ------ | +| `@pytest.mark.platforms(["a5"])` | deselected on any other `--platform` | +| `@pytest.mark.device_count(2)` | needs two dies | +| `@pytest.mark.skipif(...)` | skipped unless `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` is `1` / `ON` / `TRUE` / `YES` | + +The CMake option defaults `OFF`, so a stock build skips this test even on a5 +hardware — **a green CI run says nothing about SDMA completion on a5.** + +```bash +SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON pip install --no-build-isolation -e . +SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON \ + pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo \ + --platform a5 --device 0-1 +``` + +The variable is read twice: `simpler_setup/runtime_builder.py` forwards it to +CMake so the overlay is compiled in, and the test reads it from the environment +to decide whether to skip. + +Wrap the hardware run in `task-submit` on a shared box. + +## Compare with + +- [`../urma_deferred_completion_demo/`](../urma_deferred_completion_demo/) — the same protocol over URMA. `kernel_consumer.cpp` is byte-identical; only the transfer kernel, its completion header, and the build flag differ. **The two overlays are mutually exclusive in one build**, so comparing them means rebuilding — that README has the detail. +- [`examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/`](../../../a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/) — the a2a3 port of this demo, which needs no overlay flag. +- [`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the overlays are gated off and the checklist for re-enabling them. diff --git a/examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/README.md b/examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/README.md new file mode 100644 index 0000000000..f3022bb9af --- /dev/null +++ b/examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/README.md @@ -0,0 +1,89 @@ +# urma_deferred_completion_demo — the same protocol, over URMA + +The a5-only twin of +[`../sdma_async_completion_demo/`](../sdma_async_completion_demo/). Both run the +identical two-rank protocol; they differ in which transport moves the bytes and +which completion path reports it done. + +```text +producer: TGET_ASYNC the peer's input from the window into local `out`, + register the AsyncEvent through the deferred-completion path +consumer: depends on that producer output, writes result = out + 1 +``` + +Checking both `out` and `result` is what makes it a test of two things at once: +`out` proves completion polling saw the transfer land, and `result` proves the +deferred-release dependency held the consumer back until it had. + +## What actually differs from the SDMA demo + +`kernels/aiv/kernel_consumer.cpp` is **byte-identical** between the two +directories. Only the transfer kernel and the orchestration change: + +| What | URMA | SDMA | +| ---- | ---- | ---- | +| Transfer kernel | `kernel_urma_tget_async.cpp` | `kernel_sdma_tget_async.cpp` | +| Completion header | `backend/urma/urma_completion_kernel.h` | `backend/sdma/sdma_completion_kernel.h` | +| Build flag | `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` | `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` | + +Read them side by side and the transport is the only variable — which is +exactly what you want when deciding which one a workload should use. + +## Both overlays are off by default, and they are mutually exclusive + +This is the part to know before trying to run either. + +```cmake +option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE "..." OFF) +option(SIMPLER_ENABLE_PTO_URMA_WORKSPACE "..." OFF) +if(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE AND SIMPLER_ENABLE_PTO_URMA_WORKSPACE) + message(FATAL_ERROR "... mutually exclusive because CommContext exposes a + single workSpace/workSpaceSize pair") +endif() +``` + +— `src/a5/platform/onboard/host/CMakeLists.txt`. **One host runtime build can +carry one overlay, not both**, because `CommContext` has a single +`workSpace` / `workSpaceSize` pair to hand the kernel. So the two demos cannot +pass in the same build; comparing them means rebuilding. + +The kernel checks for this at run time too: `workSpace == 0` means the overlay +was not built in. + +## Consequently it is doubly gated, and never runs in CI + +| Gate | Effect | +| ---- | ------ | +| `@pytest.mark.platforms(["a5"])` | deselected on any other `--platform` | +| `@pytest.mark.device_count(2)` | needs two dies | +| `@pytest.mark.skipif(not _urma_workspace_enabled())` | skipped unless `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` is one of `1` / `ON` / `TRUE` / `YES` in the environment | +| `run()` raises | if `platform != "a5"` or the device count is not 2, and re-checks the env var before doing any work | + +Since the CMake option defaults `OFF`, a stock build skips this test even on a5 +hardware. **A green CI run says nothing about URMA.** Treat it as a manual +bring-up check, not as coverage. + +## Run + +```bash +# 1. rebuild the host runtime with the overlay on +SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON pip install --no-build-isolation -e . + +# 2. run it, with the same variable visible to pytest (the skipif reads the env) +SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON \ + pytest examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo \ + --platform a5 --device 0-1 +``` + +The variable is needed twice for different reasons: `runtime_builder.py` +forwards it to CMake so the overlay is compiled in, and the test reads it from +the environment to decide whether to skip. Setting only one of the two gives +you either a skipped test or a `workSpace == 0` failure. + +Wrap the hardware run in `task-submit` on a shared box. + +## See also + +[`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the +overlays are gated off, what is gated, and the checklist for re-enabling them +(issue [#1315](https://github.com/hw-native-sys/simpler/issues/1315)).