diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..8172d37 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,70 @@ +name: Publish documentation + +on: + push: + branches: [main] + paths: + - docs/** + - mkdocs.yml + - requirements-docs.txt + - scripts/** + - benchmark/** + - README.md + - .github/workflows/pages.yml + pull_request: + paths: + - docs/** + - mkdocs.yml + - requirements-docs.txt + - scripts/** + - benchmark/** + - README.md + - .github/workflows/pages.yml + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: requirements-docs.txt + - name: Install documentation dependencies + run: pip install -r requirements-docs.txt + - name: Build strict documentation site + run: mkdocs build --strict + - name: Verify golden-manual coverage and links + run: python3 scripts/verify_golden_manual.py + - name: Configure Pages + if: github.event_name != 'pull_request' + uses: actions/configure-pages@v5 + - name: Upload Pages artifact + if: github.event_name != 'pull_request' + uses: actions/upload-pages-artifact@v3 + with: + path: site + + deploy: + if: github.event_name != 'pull_request' + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + permissions: + pages: write + id-token: write + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index f0d0540..20d3b0c 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,9 @@ # Build outputs output/ **/output/ +/site/ +__pycache__/ +*.py[cod] # Temporary files *.log diff --git a/README.md b/README.md index 34e2a8e..5f2dc17 100644 --- a/README.md +++ b/README.md @@ -1,315 +1,242 @@ # SuperNPUBench -SuperNPUBench is a high-performance operator library and benchmark platform for -NPU tile-programming ISA. It ships **two architecture backends** under `benchmark/` -(two-level-arch = LinxISA, one-level-arch = PTO ISA) plus an instruction-level -**microbenchmark** suite, all driven by the same Linx toolchain. +SuperNPUBench is the LinxISA/PTO operator and microbenchmark workload suite. +The checked-in Linx target surface follows the current v0.57 contract: -## Repository Structure +- target triple: `linx64-linx-none-elf` +- compiler: the superproject LLVM build in `compiler/llvm` +- C library/sysroot: the superproject musl build in `out/libc/musl` +- tile assembly: named v0.57 TMA, CUBE, and TEPL block headers +- tile operands: canonical `B.IOT` descriptors with ordinary `r` constraints -``` +Retired target triples, numeric PTO selectors, vendor-only register +constraints, and compatibility block spellings are not supported. + +## Layout + +```text SuperNPUBench/ ├── benchmark/ -│ ├── two-level-arch/ # Linx two-level block ISA (was benchmark-linxisa) -│ │ ├── kernels/ # header-only operator implementations -│ │ ├── test/ # test suites + build system -│ │ │ ├── common/ # shared Makefile.common, _start.s, benchmark.h -│ │ │ └── kernel/ # per-operator test cases -│ │ └── compile_all.sh -│ ├── one-level-arch/ # PTO one-level tile ISA (was benchmark-ptoisa) -│ │ ├── kernels/ -│ │ ├── test/ -│ │ └── compile_all.sh -│ └── (no docs/ — comparison docs were removed) -├── microbenchmark/ # instruction-level micro-bench (cube/vector/memory/scalar) -├── architecture/ # ISA reference docs -│ ├── linxisa-reference/ # LinxISA programming guide + ISA reference -│ └── ptoisa-reference/ # PTO ISA programming guide -└── compile_all.sh # top-level: two-level | one-level | all +│ ├── two-level-arch/ # LinxISA block templates and PTO kernels +│ └── one-level-arch/ # PTO-oriented companion kernels +├── microbenchmark/ # scalar, memory, vector, and cube cases +├── architecture/ # workload-oriented programming notes +└── compile_all.sh ``` -> Build outputs (`output/`, `**/output/`) and `.DS_Store` are gitignored. +Build outputs under `output/` are ignored. -## Architecture Backends +## Toolchain setup -### two-level-arch (LinxISA) -- Block-structured ISA with heterogeneous cores: BCC (main), Cube (matrix), Vector, MTC/TMA (data transfer). -- Programming model: block instructions (VPAR/VSEQ, CUBE, TMA, TEPL). -- Reference: [`architecture/linxisa-reference/`](architecture/linxisa-reference/). +SuperNPUBench is a submodule of the LinxISA superproject. Build LLVM and musl +from the superproject root, then export their locations: -### one-level-arch (PTO ISA) -- Tile-centric ISA with explicit memory hierarchy: Vec (UB), Mat (L1), Left (L0A), Right (L0B), Acc (L0C). -- Programming model: tile operations, Auto/Manual modes. -- Reference: [`architecture/ptoisa-reference/`](architecture/ptoisa-reference/). - -Both backends share the same operator set and test layout; their kernel -implementations differ in ISA style. +```bash +cmake --build compiler/llvm/build-linxisa-clang \ + --target clang lld llvm-objdump llvm-objcopy -j8 -## Operator Overview +export COMPILER_DIR="$PWD/compiler/llvm/build-linxisa-clang/bin" +export LINX_SYSROOT="$PWD/out/libc/musl/install/phase-b" +``` -Each backend implements 10 operator categories: +The Makefiles add: -| Operator | Description | -|----------|-------------| -| **matmul** | FP4/BF16/FP32/FP16/FP8 matrix multiply; quantization, mixed precision, A/B reuse | -| **fa** | Flash Attention; 2D unroll, HIF4 quantization, unaligned boundary | -| **transpose** | 3D~6D tensor transpose; multiple dtypes | -| **reduction** | Row/column max & sum; single-tree, unaligned, cumsum, reduceprod | -| **gelu** | GELU activation; exact (erf) and tanh approximation | -| **broadcast** | 2D~5D broadcast; vectorized variants | -| **gather** | Data gathering; large-scale, power-of-2 dims | -| **concat** | Concatenation; gather/scatter modes | -| **control** | `hashtable_lookup_simd` (pure tile-op, single-tier gfsim) | -| **sort** | `topk` | +```text +-target linx64-linx-none-elf +--sysroot=$LINX_SYSROOT +-fenable-matrix +-O2 +-std=c++20 +``` -## Setup Environment +`COMPILER_DIR` must contain `clang`, `clang++`, `llvm-objdump`, and +`llvm-objcopy`. The workload does not download or select an external compiler. -SuperNPUBench compiles with the **Linx toolchain** (`linx_blockisa_llvm_musl`, -clang-15, target `linx64v5-unknown-linux-musl`). Build it once from the -[`linx-toolchain-build`](https://github.com/LinxISA/linx-toolchain-build) repo, -which clones the matching ISA sources and produces the `linx_blockisa_llvm_musl` -install tree that `COMPILER_DIR` points at. +## Build examples -### 1. Clone the build repo +Compile one two-level operator: ```bash -git clone https://github.com/LinxISA/linx-toolchain-build.git -cd linx-toolchain-build +cd benchmark/two-level-arch/test/kernel/matmul +make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32 \ + M=256 N=256 K=256 tM=64 tN=64 tK=64 ``` -### 2. Install host build tools +Compile the matching one-level operator: ```bash -sudo apt-get install -y git make cmake ninja-build gcc g++ python3 autoconf m4 +cd benchmark/one-level-arch/test/kernel/matmul +make TESTCASE=matmul TYPE=HIF4_HIF4 \ + M=256 N=2048 K=2048 tM=128 tN=128 tK=128 ``` -### 3. Initialize component sources - -`make init-src` clones the five component repos under `src/` on their pinned -branches (run it again any time to fetch updates): - -| Directory | Repository | Branch | -| --- | --- | --- | -| `src/llvm-project` | `LinxISA/llvm-project` | `dev-llvm15_56` | -| `src/musl` | `LinxISA/linx-musl` | `linx` | -| `src/jemalloc` | `LinxISA/jemalloc` | `linx` | -| `src/linux-linxisa` | `LinxISA/linux` | `main` | -| `src/Linx-TileOP-API` | `LinxISA/Linx-TileOP-API` | `linx` | +Run batch builds: ```bash -make init-src +./compile_all.sh two-level +./compile_all.sh one-level +./compile_all.sh all ``` -### 4. Build the toolchain - -Only `linx64v5-linux-musl` is supported by the top-level Makefile: - -```bash -make WITH_TARGET=linx64v5-linux-musl -``` +Artifacts are written below `benchmark//output/`. -This builds, in order: LLVM/clang/lld → kernel headers → musl → compiler-rt → -libc++/libc++abi/libunwind → jemalloc → Linx-TileOP-API headers. Progress is -tracked by stamp files under `stamps/`, so re-running `make` resumes from the -last completed step; `make clean` rebuilds from scratch. The install tree is -written to `output/linx_blockisa_llvm_musl/`: +## Representative v0.57 hardware-path check -``` -output/linx_blockisa_llvm_musl/ -├── bin/ # clang, clang++, ld.lld, llvm-ar/nm/ranlib, -│ # linx64v5-linux-musl-clang(++) symlinks -├── lib/ # clang runtime, libc++, ... -└── sysroot/ # musl + kernel headers + runtime libs -``` +The hashtable kernel exercises named TEPL blocks plus TLOAD/TSTORE: -### 5. Point SuperNPUBench at the toolchain +```bash +"$COMPILER_DIR/clang++" -c \ + -target linx64-linx-none-elf \ + --sysroot="$LINX_SYSROOT" \ + -idirafter "$LINX_SYSROOT/usr/include" \ + -fenable-matrix -O2 -std=c++20 \ + -Ibenchmark/two-level-arch/include \ + -Ibenchmark/two-level-arch/test/common \ + -Ibenchmark/two-level-arch/test/common/src \ + -Ibenchmark/two-level-arch/kernels \ + -Ibenchmark/two-level-arch/models \ + -D__linx -DENABLE_TENSOR_INSTR \ + -DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=256 \ + benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp \ + -o /tmp/supernpubench-v057.o + +"$COMPILER_DIR/llvm-objdump" -d /tmp/supernpubench-v057.o +``` + +The disassembly must contain named blocks such as `BSTART.TLOAD`, +`BSTART.TSTORE`, `BSTART.TSEL`, and `BSTART.TCMP`; numeric TEPL selectors are +not an acceptable result. + +## Microbenchmarks + +The microbenchmark tree covers: + +| Family | Representative operations | +| --- | --- | +| scalar | integer, floating-point, conversion, and memory latency/throughput | +| memory | TLOAD, TSTORE, TMOV, gather, and scatter | +| vector | named TEPL elementwise, scalar-tile, reduction, and expansion ops | +| cube | TMATMUL, bias/accumulate variants, and ACCCVT | ```bash -export COMPILER_DIR=$(pwd)/output/linx_blockisa_llvm_musl/bin -$COMPILER_DIR/clang --version -# clang version 15.0.4 (linx64v5-musl-local ...) -# Target: linx64v5-unknown-linux-musl +cd microbenchmark +make TESTCASE=tmatmul_fp16_64x64x64 +bash compile_all.sh all ``` -Then proceed to [Quick Start](#quick-start). +See [microbenchmark/README.md](microbenchmark/README.md) for case-generation +details. -### (Optional) Package + +## One-level benchmark catalog -```bash -make package # -> output/linx_blockisa_llvm_musl.tar.gz -``` +The one-level manifests contain **53 build variants**. Every name below +has a source-complete page with its build command and PTO intrinsic surface in +the website's **Benchmarks** section. -### Platform notes (macOS) +
One-level / broadcast (4 names, 6 variants) -The build also works on Apple Silicon, but the system `make` (GNU 3.81) and -`tar` (libarchive) differ from the Linux defaults, so two extra steps are -needed: +`broadcast`, `broadcast_vec_019`, `broadcast_vec_039`, `broadcast_vec_07` -- **GNU make >= 4** is required by the kernel-headers step - (`GNU Make >= 4.0 is required`). `brew install make` (provides `gmake`) and - run the build with `gmake`, or put `gmake` on `PATH` ahead of `/usr/bin/make` - (the kernel Makefile invokes `make` literally). -- **GNU tar** is required by `make package` (`tar --format=gnu`). `brew install - gnu-tar` and prepend `$(brew --prefix)/opt/gnu-tar/libexec/gnubin` to `PATH`. -- **sancov host-compile fix**: Apple clang rejects an initializer-list in - `llvm/tools/sancov/sancov.cpp` (`chosen constructor is explicit in - copy-initialization`). If the LLVM build fails there, change the two - `SpecialCaseList::createOrDie({{...}}, ...)` call sites to pass an explicit - `std::vector{...}`, then resume with - `ninja -C build/build-llvm-musl`. +
-## Quick Start +
One-level / concat (2 names, 4 variants) -### 1. Environment +`concat_gather`, `concat_scatter` -Build the Linx toolchain once (see [Setup Environment](#setup-environment)), then -point `COMPILER_DIR` at it: +
-```bash -export COMPILER_DIR=/path/to/linx_blockisa_llvm_musl/bin -``` +
One-level / control (1 name, 6 variants) -### 2. Compile an operator +`hashtable_lookup_simd` -```bash -# two-level-arch (LinxISA) -cd benchmark/two-level-arch/test/kernel/matmul -make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32 M=256 N=256 K=256 tM=64 tN=64 tK=64 +
-# one-level-arch (PTO ISA) -cd benchmark/one-level-arch/test/kernel/matmul -make TESTCASE=matmul TYPE=HIF4_HIF4 M=256 N=2048 K=2048 tM=128 tN=128 tK=128 -``` +
One-level / element_wise/gelu (1 name, 1 variant) -### 3. Batch / full compilation +`gelu` -```bash -# per-operator batch -cd benchmark/two-level-arch/test/kernel/matmul && bash compile.all +
-# all operators of one backend -./compile_all.sh two-level # two-level-arch only -./compile_all.sh one-level # one-level-arch only -./compile_all.sh all # both (default) -``` +
One-level / fa (2 names, 9 variants) -Artifacts land in `benchmark//output/kernel//elf/`. +`fa_2d_unroll`, `fa_HIF4_HIF4` -## Microbenchmark +
-`microbenchmark/` is an instruction-level bench organized by ISA family, -generated by `gen_cases.py`. All 231 cases compile & link on the Linx toolchain. +
One-level / gather (1 name, 1 variant) -| family | covers | cases | -| --- | --- | ---: | -| cube (CUBE) | TMATMUL / TMATMUL_BIAS / TMATMUL_MX / ACCCVT | 9 | -| vector (TEPL) | elementwise / tile-scalar / reduce / expand (toolchain-exposed subset) | 73 | -| memory (TLSU) | TLOAD / TSTORE / TMOV / MGATHER / MSCATTER (+mask, layout) | 25 | -| scalar (GPR) | int ALU / load-store / float / conversion × throughput+latency | 124 | -| **total** | | **231** | +`gather` -```bash -cd microbenchmark && make TESTCASE=tmatmul_fp16_64x64x64 # one case -cd microbenchmark && bash compile_all.sh all # all families -``` +
-See [`microbenchmark/README.md`](microbenchmark/README.md) for details. +
One-level / matmul (1 name, 16 variants) -## Running on the Models +`matmul` -Compiled ELF binaries run on the **SuperScalarModel** simulator suite (no longer -on LinxCoreModel). Build `gfrun`/`gfsim` from the -[SuperScalarModel](../SuperScalarModel) repo, then point them at the ELF: +
-- `gfrun` — functional model (correctness) -- `gfsim` — cycle-accurate model (timing) +
One-level / reduction/reducemax_col (1 name, 1 variant) -```bash -# from the SuperScalarModel repo root (where bin/ lives) -bin/gfrun -f /path/to/SuperNPUBench/benchmark/two-level-arch/output/kernel//elf/.elf -bin/gfsim -f /path/to/SuperNPUBench/benchmark/two-level-arch/output/kernel//elf/.elf -``` +`reducemax_col` -### Tile-op kernels: single-tier gfsim mode +
-Kernels written purely with tile ops using TEPL template instructions (e.g. -`control/hashtable_lookup_simd`) run on the VectorLite engine, which gfsim only -steps in **single-tier mode**: +
One-level / reduction/reducemax_row (1 name, 1 variant) -```bash -bin/gfsim -f -s core.singleTierMode=true -``` +`reducemax_row` -Without this flag the engine is inert and the run deadlocks. `gfrun` does not -need the flag. +
-## Build System +
One-level / reduction/reducesum_col (1 name, 2 variants) -### Makefile parameters +`reducesum_col` -| Parameter | Description | Example | -|-----------|-------------|---------| -| `TESTCASE` | Test case name | `matmul`, `fa_2d_unroll` | -| `TYPE` | Operator type (matmul) | `HIF4_HIF4`, `A16W4`, `MASK` | -| `MODE` | Operator mode | `MASK_FP32`, `BF16x2_NOGATHER` | -| `M/N/K` | Matrix dimensions | `M=256 N=2048 K=2048` | -| `tM/tN/tK` | Tile sizes | `tM=128 tN=128 tK=128` | -| `COMPILER_DIR` | Compiler path | `/path/to/linx/bin` | -| `PLAT` | Platform | `linx` (default), `cpu` | +
-### Build targets +
One-level / reduction/reducesum_row (1 name, 1 variant) -```bash -make TESTCASE= all # compile -make TESTCASE= diss # disassembly -make TESTCASE= sim # run in simulator -make TESTCASE= debug # debug mode -make clean # clean current operator -make clean_all # clean all -``` +`reducesum_row` -## Documentation +
-- **LinxISA**: [programming guide (zh)](architecture/linxisa-reference/programming_guide.md) | [programming guide (en)](architecture/linxisa-reference/programming_guide_en.md) | [ISA reference](architecture/linxisa-reference/isa_reference.md) -- **PTO ISA**: [programming guide (zh)](architecture/ptoisa-reference/programming_guide.md) | [programming guide (en)](architecture/ptoisa-reference/programming_guide_en.md) +
One-level / sort (1 name, 1 variant) -## Known Issues +`topk` -- **`fa_2d_unroll` compiler crash** (Issue #6): `X=1,Y=1` / `X=2,Y=1` trigger - `Assertion failed: (Reg != 0 && "LinxV5 CallingConv Fail!")`. Avoid `Ydim=1`. -- **`control/hashtable_lookup_simd`**: pure tile-op kernel; run gfsim with - `-s core.singleTierMode=true` (see above). Its `.data` files are generated by - the Makefile (`gen_data.py`) on first build. -- **microbenchmark `tmatmul_acc`**: skipped — toolchain `matmul.ac` backend crash. -- **microbenchmark vector subset**: only toolchain-exposed TEPL opcodes are - generated; `TGEMV*`/`TMOV` not yet exposed (TCOPY fallback). See - [`microbenchmark/README.md`](microbenchmark/README.md). +
-## Toolchain +
One-level / transpose (1 name, 4 variants) -- Compiler: `linx_blockisa_llvm_musl` (clang-15, linx64v5-musl) -- Flags: `-mlxbc -fenable-matrix -O2 -mllvm -enable-all-vector-as-tilereg=true -std=c++20` -- Target: Linx64 V5 +`transpose` -## Development Guide +
-### Adding an operator + -1. Add header-only kernel under both `benchmark//kernels//`. -2. Create test dir under both `benchmark//test/kernel//` with - `Makefile`, `compile.all`, `src/`. -3. Mirror the two backends. +## Models -### Conventions +ELF files can be passed to the superproject model or emulator appropriate for +the test lane. Do not hard-code developer-local simulator paths in scripts. -- Header-only kernels; PTO tile-programming paradigm. -- Build artifacts not tracked (`.gitignore`). -- Keep both backends in parallel. +## Development rules -## Related Links +1. Keep the two architecture trees structurally aligned where they expose the + same operator. +2. Use named v0.57 block headers; do not introduce raw numeric PTO selectors. +3. Use standard inline-assembly constraints. Tile values use `r`, and B.IOT + destinations use the Linx `%q` operand modifier to print only the queue bank. +4. Keep host fallbacks behind `!defined(__linx)`; Linx builds must compile the + architectural tile path. +5. Run `git diff --check`, the representative hardware-path compile above, and + the relevant `compile.all` scripts before submitting changes. + +## Documentation -- [LinxISA](https://linxisa.github.io/linx-isa/) · [PTO ISA](https://pto-isa.github.io/docs/isa/tile/) +- [Superscalar NPU programming guide](https://pto-isa.github.io/SuperNPUBench/) +- Install site dependencies with `python3 -m pip install -r requirements-docs.txt` +- Build the site locally with `python3 -m mkdocs serve` ## License -See LICENSE. +See `LICENSE`. diff --git a/architecture/linxisa-reference/isa_reference.md b/architecture/linxisa-reference/isa_reference.md index 64d459a..f1f5383 100644 --- a/architecture/linxisa-reference/isa_reference.md +++ b/architecture/linxisa-reference/isa_reference.md @@ -287,11 +287,9 @@ CUBE 块使用 L0 buffer 进行矩阵运算： TMA 块支持高效的数据搬运： -``` -bstart.tma - // 配置 DMA 参数 - // 执行数据传输 -bstop +```asm +BSTART.TLOAD INT32 + // B.ARG / B.DIM / B.IOR / B.IOT descriptors ``` ### 4.2 搬运模式 @@ -376,5 +374,5 @@ bstop --- -**文档版本**: 1.0 +**文档版本**: 1.0 **最后更新**: 2026-01-08 diff --git a/architecture/linxisa-reference/programming_guide.md b/architecture/linxisa-reference/programming_guide.md index 6193c08..07e13b0 100644 --- a/architecture/linxisa-reference/programming_guide.md +++ b/architecture/linxisa-reference/programming_guide.md @@ -141,12 +141,9 @@ CUBE 块支持的数据类型组合： TMA（Tile Memory Access）块用于高效的数据搬运： -```cpp -// 从全局内存加载到 Tile Register -bstart.tma - // DMA 传输配置 - // 执行数据搬运 -bstop +```asm +BSTART.TLOAD INT32 + // B.ARG / B.DIM / B.IOR / B.IOT 描述符 ``` ### 5.2 搬运模式 @@ -228,5 +225,5 @@ result[j] = sum(C[:, j]) --- -**文档版本**: 1.0 +**文档版本**: 1.0 **最后更新**: 2026-01-08 diff --git a/architecture/linxisa-reference/programming_guide_en.md b/architecture/linxisa-reference/programming_guide_en.md index 2fa7860..ffc5e7f 100644 --- a/architecture/linxisa-reference/programming_guide_en.md +++ b/architecture/linxisa-reference/programming_guide_en.md @@ -142,12 +142,9 @@ Data type combinations supported by CUBE block: The TMA (Tile Memory Access) block is used for efficient data transfer: -```cpp -// Load from global memory to Tile Register -bstart.tma - // DMA transfer configuration - // Execute data transfer -bstop +```asm +BSTART.TLOAD INT32 + // B.ARG / B.DIM / B.IOR / B.IOT descriptors ``` ### 5.2 Transfer Modes @@ -229,5 +226,5 @@ result[j] = sum(C[:, j]) --- -**Document Version**: 1.0 +**Document Version**: 1.0 **Last Updated**: 2026-01-08 diff --git a/benchmark/one-level-arch/compile_all.sh b/benchmark/one-level-arch/compile_all.sh index a3cca63..488720f 100755 --- a/benchmark/one-level-arch/compile_all.sh +++ b/benchmark/one-level-arch/compile_all.sh @@ -1,7 +1,7 @@ -#!/bin/bash -# PTO ISA compilation script for all kernel operators +#!/usr/bin/env bash +set -uo pipefail -# Don't use set -e as some operators may fail to compile +# PTO ISA compilation script for all kernel operators SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" : "${COMPILER_DIR:?Set COMPILER_DIR to the in-repo Linx compiler bin directory}" @@ -38,6 +38,7 @@ compile_operator() { echo "✓ $operator_name compilation completed" else echo "✗ $operator_name compilation failed" + return 1 fi else echo "Warning: No compile.all found in $operator_path" @@ -46,19 +47,20 @@ compile_operator() { } # Compile all operators -compile_operator "$REPO_ROOT/test/kernel/matmul" "matmul" -compile_operator "$REPO_ROOT/test/kernel/broadcast" "broadcast" -compile_operator "$REPO_ROOT/test/kernel/concat" "concat" -compile_operator "$REPO_ROOT/test/kernel/gather" "gather" -compile_operator "$REPO_ROOT/test/kernel/transpose" "transpose" -compile_operator "$REPO_ROOT/test/kernel/element_wise/gelu" "gelu" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_col" "reducemax_col" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_row" "reducemax_row" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_col" "reducesum_col" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_row" "reducesum_row" -compile_operator "$REPO_ROOT/test/kernel/control" "control" -compile_operator "$REPO_ROOT/test/kernel/fa" "fa" -compile_operator "$REPO_ROOT/test/kernel/sort" "sort" +failures=() +compile_operator "$REPO_ROOT/test/kernel/matmul" "matmul" || failures+=("matmul") +compile_operator "$REPO_ROOT/test/kernel/broadcast" "broadcast" || failures+=("broadcast") +compile_operator "$REPO_ROOT/test/kernel/concat" "concat" || failures+=("concat") +compile_operator "$REPO_ROOT/test/kernel/gather" "gather" || failures+=("gather") +compile_operator "$REPO_ROOT/test/kernel/transpose" "transpose" || failures+=("transpose") +compile_operator "$REPO_ROOT/test/kernel/element_wise/gelu" "gelu" || failures+=("gelu") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_col" "reducemax_col" || failures+=("reducemax_col") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_row" "reducemax_row" || failures+=("reducemax_row") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_col" "reducesum_col" || failures+=("reducesum_col") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_row" "reducesum_row" || failures+=("reducesum_row") +compile_operator "$REPO_ROOT/test/kernel/control" "control" || failures+=("control") +compile_operator "$REPO_ROOT/test/kernel/fa" "fa" || failures+=("fa") +compile_operator "$REPO_ROOT/test/kernel/sort" "sort" || failures+=("sort") echo "" echo "==========================================" @@ -68,3 +70,8 @@ echo "" echo "Generated ELF files:" find "$REPO_ROOT/output" -name "*.elf" -type f | wc -l echo "ELF files are located in: $REPO_ROOT/output/" + +if ((${#failures[@]})); then + echo "Failed operators: ${failures[*]}" >&2 + exit 1 +fi diff --git a/benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp b/benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp index 77ec34c..17ae721 100644 --- a/benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp +++ b/benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp @@ -30,68 +30,14 @@ // │ TADD │ API 有，二层实现 │ jcore/TAdd.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ MGATHER │ 已支持(一层) │ template_asm.h 用 asm volatile │ -// │ │ │ (BSTART.TMA) 实现； │ +// │ │ │ (BSTART.MGATHER) 实现； │ // │ │ │ 但缺少 Coalesce::Elem 模板参数支持 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TSTORE │ API 有(名不同)， │ Pto ISA 名 TSTORE； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYOUT； │ -// │ │ │ jcore/TCopyOut.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TSTORE； │ +// │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ // └──────────┴──────────────────┴──────────────────────────────────────────┘ // -// PTO ISA 文档签名 (Declared in include/pto/pto_instr.hpp): -// -// TCI: -// template -// PTO_INST RecordEvent TCI(TileData &dst, T start, WaitEvents &... events); -// -// TEXPANDS: -// template -// PTO_INST RecordEvent TEXPANDS(TileData &dst, typename TileData::DType scalar, -// WaitEvents &... events); -// -// TREMS: -// template -// PTO_INST RecordEvent TREMS(TileDataDst &dst, TileDataSrc &src, -// typename TileDataSrc::DType scalar, -// TileDataTmp &tmp, WaitEvents &... events); -// -// TDIVS: -// template -// PTO_INST RecordEvent TDIVS(TileDataDst &dst, TileDataSrc &src0, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TMULS: -// template -// PTO_INST RecordEvent TMULS(TileDataDst &dst, TileDataSrc &src0, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TADD: -// template -// PTO_INST RecordEvent TADD(TileDataDst &dst, TileDataSrc0 &src0, -// TileDataSrc1 &src1, WaitEvents &... events); -// -// MGATHER: -// template -// PTO_INST RecordEvent MGATHER(TileDst &dst, GlobalData &src, -// TileInd &indexes, WaitEvents &... events); -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); -// // ============================================================================ #include // Tile, GlobalTensor 等类型 (当前编译器已有) @@ -148,7 +94,7 @@ void gen_offset_pto( // ---- Step 2: TEXPANDS 初始化 out = 0 ---- // [当前编译器] PTO ISA 名 TEXPANDS，当前编译器名 TEXPANDSCALAR；jcore 为 __vec__ - TEXPANDS(out, (off_t)0); + TEXPANDSCALAR(out, (off_t)0); size_t stride = 1; @@ -233,7 +179,7 @@ void broadcast( MGATHER(outTile, inGm, offsetTile); // TSTORE: 将 outTile 写回 global memory - // [当前编译器] PTO ISA 名 TSTORE，当前编译器名 TCOPYOUT；jcore 为 __vec__ + // [当前编译器] PTO ISA 名 TSTORE，当前编译器名 TSTORE；jcore 为 __vec__ TSTORE(gO, outTile); } if constexpr (rmd_M) { diff --git a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp index 908e75f..64240fd 100644 --- a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp +++ b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp @@ -2,7 +2,7 @@ // Broadcast (B,1,K) -> (B,N,K) — PTO 一层编程模型 // // 原始 broadcast_vec_019.hpp 策略: -// TCOPYIN (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TCOPYOUT (kTileBatch,N*K) +// TLOAD (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TSTORE (kTileBatch,N*K) // __vec__ 块: dst[batch*RowStride + copy*K + x] = src[batch*RowStride + x] // // PTO 一层策略: @@ -18,39 +18,18 @@ // │ 指令 │ │ │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYIN； │ -// │ │ │ jcore/TCopyIn.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TLOAD； │ +// │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TINSERT │ 完全缺失 │ pto_tileop.hpp 中无此 API； │ // │ │ │ 当前编译器无 TINSERT 实现 │ // │ │ │ (仅有反向操作 TEXTRACT) │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYOUT； │ -// │ │ │ jcore/TCopyOut.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TSTORE； │ +// │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ // └──────────┴──────────────────┴──────────────────────────────────────────┘ // -// PTO ISA 文档签名 (Declared in include/pto/pto_instr.hpp): -// -// TLOAD: -// template -// PTO_INST RecordEvent TLOAD(TileData &dst, GlobalData &src, -// WaitEvents &... events); -// -// TINSERT: -// template -// PTO_INST RecordEvent TINSERT(DstTileData &dst, SrcTileData &src, -// uint16_t indexRow, uint16_t indexCol, -// WaitEvents &... events); -// 语义: dst[indexRow+i, indexCol+j] = src[i,j] -// for 0 <= i < src.ValidRow, 0 <= j < src.ValidCol -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); // ============================================================================ #include @@ -116,7 +95,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); // TLOAD: GM -> UB, 加载 (kTileBatch, kInner) 输入 tile - // [当前编译器] 名为 TCOPYIN, jcore 为 __vec__ + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ TLOAD(inTile, gsrc); // TINSERT × kBCast: 将输入 tile 插入输出 tile 的 N 个列偏移 @@ -128,7 +107,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, } // TSTORE: UB -> GM, 写回 (kTileBatch, kBCast*kInner) 输出 tile - // [当前编译器] 名为 TCOPYOUT, jcore 为 __vec__ + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ TSTORE(gdst, outTile); } diff --git a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp index edc99a8..6f23e89 100644 --- a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp +++ b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp @@ -2,7 +2,7 @@ // Broadcast (B,1,K) -> (B,N,K) — PTO 一层编程模型 // // 原始 broadcast_vec_039.hpp 策略: -// TCOPYIN (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TCOPYOUT (kTileBatch,N*K) +// TLOAD (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TSTORE (kTileBatch,N*K) // __vec__ 块: dst[y*RowStride + x] = src[y*RowStride + (x & (K-1))] // (K 为 2 的幂, 用位与代替取余) // @@ -19,39 +19,18 @@ // │ 指令 │ │ │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYIN； │ -// │ │ │ jcore/TCopyIn.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TLOAD； │ +// │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TINSERT │ 完全缺失 │ pto_tileop.hpp 中无此 API； │ // │ │ │ 当前编译器无 TINSERT 实现 │ // │ │ │ (仅有反向操作 TEXTRACT) │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYOUT； │ -// │ │ │ jcore/TCopyOut.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TSTORE； │ +// │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ // └──────────┴──────────────────┴──────────────────────────────────────────┘ // -// PTO ISA 文档签名 (Declared in include/pto/pto_instr.hpp): -// -// TLOAD: -// template -// PTO_INST RecordEvent TLOAD(TileData &dst, GlobalData &src, -// WaitEvents &... events); -// -// TINSERT: -// template -// PTO_INST RecordEvent TINSERT(DstTileData &dst, SrcTileData &src, -// uint16_t indexRow, uint16_t indexCol, -// WaitEvents &... events); -// 语义: dst[indexRow+i, indexCol+j] = src[i,j] -// for 0 <= i < src.ValidRow, 0 <= j < src.ValidCol -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); // ============================================================================ #include @@ -117,7 +96,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); // TLOAD: GM -> UB, 加载 (kTileBatch, kInner) 输入 tile - // [当前编译器] 名为 TCOPYIN, jcore 为 __vec__ + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ TLOAD(inTile, gsrc); // TINSERT × kBCast: 将输入 tile 插入输出 tile 的 N 个列偏移 @@ -129,7 +108,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, } // TSTORE: UB -> GM, 写回 (kTileBatch, kBCast*kInner) 输出 tile - // [当前编译器] 名为 TCOPYOUT, jcore 为 __vec__ + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ TSTORE(gdst, outTile); } diff --git a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp index ee43b44..bb203e1 100644 --- a/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp +++ b/benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp @@ -2,7 +2,7 @@ // Broadcast (N,1) -> (N,C) — PTO 一层编程模型 // // 原始 broadcast_vec_07.hpp 策略: -// TCOPYIN (kTileRows,1) -> __vec__ 行广播 -> TCOPYOUT (kTileRows,C) +// TLOAD (kTileRows,1) -> __vec__ 行广播 -> TSTORE (kTileRows,C) // __vec__ 块: dst[x + y*RowStride] = src[y*RowStride] (col 0 -> all cols) // // PTO 一层策略: @@ -16,38 +16,18 @@ // │ 指令 │ │ │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYIN； │ -// │ │ │ jcore/TCopyIn.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TLOAD； │ +// │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │TROWEXPAND│ API 有(名不同)， │ PTO ISA 名 TROWEXPAND； │ // │ │ 二层实现 │ 当前编译器名 TEXPANDROW； │ // │ │ │ jcore/TExpandRow.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ -// │ │ 二层实现 │ 当前编译器名 TCOPYOUT； │ -// │ │ │ jcore/TCopyOut.hpp 用 __vec__ 实现 │ +// │ │ 二层实现 │ 当前编译器名 TSTORE； │ +// │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ // └──────────┴──────────────────┴──────────────────────────────────────────┘ // -// PTO ISA 文档签名 (Declared in include/pto/pto_instr.hpp): -// -// TLOAD: -// template -// PTO_INST RecordEvent TLOAD(TileData &dst, GlobalData &src, -// WaitEvents &... events); -// -// TROWEXPAND: -// template -// PTO_INST RecordEvent TROWEXPAND(TileDataDst &dst, TileDataSrc &src, -// WaitEvents &... events); -// 约束: dst[i,j] = src[i,0]; dst 与 src 必须为 Vec tile, ND layout; -// A2A3: srcValidRow == dstValidRow; srcValidCol >= 1 -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); // ============================================================================ #include @@ -103,7 +83,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, gm_out gdst(out_ptr + i * kTileRows * kC); // TLOAD: GM -> UB, 加载 (kTileRows, 1) 输入 tile - // [当前编译器] 名为 TCOPYIN, jcore 为 __vec__ + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ TLOAD(inTile, gsrc); // TROWEXPAND: 将每行 col 0 广播到全部 kC 列 -> (kTileRows, kC) @@ -111,7 +91,7 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, TROWEXPAND(outTile, inTile); // TSTORE: UB -> GM, 写回 (kTileRows, kC) 输出 tile - // [当前编译器] 名为 TCOPYOUT, jcore 为 __vec__ + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ TSTORE(gdst, outTile); } diff --git a/benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp b/benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp index aee1988..69a5e48 100644 --- a/benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp +++ b/benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp @@ -56,7 +56,7 @@ void concat_gather( OffsetTile contribution; TCI(linear_index, output_base); - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); // 计算输入 stride（标量核心，运行时计算） uint32_t input_stride[DATA_DIM]; @@ -132,7 +132,7 @@ void concat_gather( TailOffsetTile contribution; TCI(linear_index, output_base); - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); uint32_t input_stride[DATA_DIM]; input_stride[DATA_DIM - 1] = 1; diff --git a/benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp b/benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp index 0501538..266cc11 100644 --- a/benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp +++ b/benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp @@ -60,7 +60,7 @@ void concat_scatter( OffsetTile contribution; TCI(linear_index, static_cast(input_base)); - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); // 计算输出 stride（标量核心，运行时计算） uint16_t output_stride[DATA_DIM]; @@ -116,7 +116,7 @@ void concat_scatter( TailOffsetTile contribution; TCI(linear_index, static_cast(input_base)); - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); uint16_t output_stride[DATA_DIM]; output_stride[DATA_DIM - 1] = 1; diff --git a/benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp b/benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp index 14b09ea..87ed7b3 100644 --- a/benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp +++ b/benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp @@ -2,19 +2,20 @@ #define HASHTABLE_LOOKUP_SIMD_HPP #include +#include "template_asm.h" #include // ============================================================================ // Pure-SIMD (tile-level) hashtable lookup. // // Everything below is expressed with whole-tile operations from the tile-op API -// (TMULS/TADDS/TXOR/TSLL/TSRL/TOR/TAND/TREM for the hash, MGATHER/TCMP/TSELECT for +// (TMULS/TADDS/TXOR/TSHLS/TSHRS/TOR/TAND/TREM for the hash, MGATHER/TCMP/TSELECT for // the probe). There are NO __vec__ SIMT kernels, no <<<>>> launches, and no per-lane // control flow (blkv_get_index / blkv_rdadd / data-dependent branches). The probe loop // is a host loop bounded by kMaxProbe with NO data-dependent early break: a tile-level // early exit would need to reduce the "still-searching" mask to a host scalar, and the // tile-op API has no tile->scalar-register reduction (only TROWSUM->tile and -// TCOPYOUT->memory). Since TSELECT keeps already-found lanes immune to later probes, +// TSTORE->memory). Since TSELECT keeps already-found lanes immune to later probes, // running all kMaxProbe iterations is correct; the loop is pure tile ops + a host counter. // ============================================================================ @@ -68,8 +69,9 @@ inline void murmurRound(TileU32& h, TileU32& block, TileU32& k, TileU32& tA, Til // (h>>1) < 2^31 so it is a non-negative int32; 2*(m-1)+1 < 2^31 so the second // operand stays non-negative -> signed TREM yields the correct unsigned result. template -void uModU32(typename HashFindTypes::TileI32& dst, - typename HashFindTypes::TileU32& h) +__attribute__((always_inline)) inline void +uModU32(typename HashFindTypes::TileI32& dst, + typename HashFindTypes::TileU32& h) { using Types = HashFindTypes; using TileU32 = typename Types::TileU32; @@ -79,12 +81,12 @@ void uModU32(typename HashFindTypes::TileI32& dst, TileI32 halfI, bitI, capI, tmp; TSHRS(halfU, h, 1u); // h >> 1 (0 .. 2^31-1) - TEXPANDS(oneU, 1u); + TEXPANDSCALAR(oneU, 1u); TAND(bitU, h, oneU); // h & 1 - TCVT(halfI, halfU); // -> non-negative int32 - TCVT(bitI, bitU); - TEXPANDS(capI, static_cast(kMod)); + TCAST(halfI, halfU); // -> non-negative int32 + TCAST(bitI, bitU); + TEXPANDSCALAR(capI, static_cast(kMod)); TREM(tmp, halfI, capI); // (h>>1) % m TMULS(tmp, tmp, 2); // 2 * r1 @@ -94,8 +96,10 @@ void uModU32(typename HashFindTypes::TileI32& dst, // Compute per-key probe byte-offset = (MurmurHash3(key) % kCap) * sizeof(TableEntry) template -void computeProbeOffsets(typename HashFindTypes::TileI32& probeOff, - typename HashFindTypes::TileI64& queryKeys) +__attribute__((always_inline)) inline void +computeProbeOffsets( + typename HashFindTypes::TileI32& probeOff, + typename HashFindTypes::TileI64& queryKeys) { using Types = HashFindTypes; using TileU32 = typename Types::TileU32; @@ -106,11 +110,11 @@ void computeProbeOffsets(typename HashFindTypes::TileI32& // block0 = low 32 bits, block1 = high 32 bits of each 8-byte key TileU32 block0, block1; - TCVT(block0, queryKeys); // static_cast truncates to low 32 + TCAST(block0, queryKeys); // static_cast truncates to low 32 TSHRS(keyHi, queryKeys, 32u); // logical >> 32 (unsigned), high 32 -> low - TCVT(block1, keyHi); + TCAST(block1, keyHi); - TEXPANDS(h, DEFAULT_HASH_SEED); + TEXPANDSCALAR(h, DEFAULT_HASH_SEED); TCVT(blk, block0); murmurRound(h, blk, k, tA, tB); TCVT(blk, block1); @@ -118,7 +122,7 @@ void computeProbeOffsets(typename HashFindTypes::TileI32& // finalize: h ^= len(=8); h ^= h>>16; h *= ..; h ^= h>>13; h *= ..; h ^= h>>16 TileU32 lenTile; - TEXPANDS(lenTile, 8u); + TEXPANDSCALAR(lenTile, 8u); TXOR(h, h, lenTile); TSHRS(tmp, h, 16u); TXOR(h, h, tmp); TMULS(h, h, 0x85ebca6bu); @@ -163,15 +167,15 @@ void runHashFind(int32_t __out__ *out, computeProbeOffsets(probeOffTile, queryKeyTile); - TEXPANDS(outTile, kNotFound); + TEXPANDSCALAR(outTile, kNotFound); // TCMP EQ supports int32 only, so compare the 64-bit keys as two int32 halves. TileI32 queryLo, queryHi; { TileI64 qHi; - TCVT(queryLo, queryKeyTile); // low 32 bits + TCAST(queryLo, queryKeyTile); // low 32 bits TSHRS(qHi, queryKeyTile, 32u); - TCVT(queryHi, qHi); // high 32 bits + TCAST(queryHi, qHi); // high 32 bits } TileI64 tableKeyTile; @@ -180,22 +184,22 @@ void runHashFind(int32_t __out__ *out, TileI32 tableValTile; TileI32 matchLo, matchHi, matchMask; TileI32 capBytesTile; - TEXPANDS(capBytesTile, static_cast(kCap) * static_cast(sizeof(TableEntry))); + TEXPANDSCALAR(capBytesTile, static_cast(kCap) * static_cast(sizeof(TableEntry))); for (int probe = 0; probe < kMaxProbe; probe++) { MGATHER(tableKeyTile, tableKeyGlobal, probeOffTile); MGATHER(tableValTile, tableValGlobal, probeOffTile); // 64-bit key equality = (low32 == low32) && (high32 == high32) - TCVT(tableLo, tableKeyTile); + TCAST(tableLo, tableKeyTile); TSHRS(tableKeyHi, tableKeyTile, 32u); - TCVT(tableHi, tableKeyHi); - TCMP(matchLo, queryLo, tableLo); - TCMP(matchHi, queryHi, tableHi); + TCAST(tableHi, tableKeyHi); + TCMP(matchLo, queryLo, tableLo, CmpMode::EQ); + TCMP(matchHi, queryHi, tableHi, CmpMode::EQ); TAND(matchMask, matchLo, matchHi); - TSEL(outTile, matchMask, tableValTile); // match ? value : keep + TSELECT(outTile, matchMask, tableValTile, outTile); // match ? value : keep // NOTE: no early break. A data-dependent early exit would require reducing the - // "still-searching" mask to a host scalar (TROWSUMEXPAND -> TCOPYOUT -> scalar + // "still-searching" mask to a host scalar (TROWSUMEXPAND -> TSTORE -> scalar // load), i.e. a tile->memory->scalar round-trip every iteration. The tile-op API // has no tile->scalar-register reduction, so that round-trip is the only option // and it is exactly the fragile path we avoid. TSELECT already makes already-found diff --git a/benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp b/benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp index 6e67503..5ce3d08 100644 --- a/benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp +++ b/benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp @@ -2,7 +2,7 @@ // GELU 算子 — PTO 一层编程模型 // // 原始 gelu.hpp 策略: -// TCOPYIN (half) -> __vec__ gelu_simd (多项式拟合) -> TCOPYOUT (half) +// TLOAD (half) -> __vec__ gelu_simd (多项式拟合) -> TSTORE (half) // __vec__ 块逐元素: fp16→fp32, clamp, Horner 多项式, exp, 除法, fp32→fp16 // // PTO 一层策略: @@ -15,8 +15,8 @@ // │ Pto ISA │ 当前编译器状态 │ 说明 │ // │ 指令 │ │ │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ -// │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD；当前编译器名 TCOPYIN； │ -// │ │ 二层实现 │ jcore/TCopyIn.hpp 用 __vec__ 实现 │ +// │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD；当前编译器名 TLOAD； │ +// │ │ 二层实现 │ jcore/TLoad.hpp 用 __vec__ 实现 │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ // │ TCVT │ API 有(签名不同)，│ PTO ISA: TCVT(dst,src,tmp,mode,satMode) │ // │ │ 二层实现 │ 当前编译器: TCVT(dst,src) 无 tmp/mode； │ @@ -40,71 +40,10 @@ // │ │ │ (template_asm.h 有 TRECIP_TEPL 内联汇编, │ // │ │ │ 但不在 pto_tileop.hpp API 中) │ // ├──────────┼──────────────────┼──────────────────────────────────────────┤ -// │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE；当前编译器名 TCOPYOUT；│ -// │ │ 二层实现 │ jcore/TCopyOut.hpp 用 __vec__ 实现 │ +// │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE；当前编译器名 TSTORE；│ +// │ │ 二层实现 │ jcore/TStore.hpp 用 __vec__ 实现 │ // └──────────┴──────────────────┴──────────────────────────────────────────┘ // -// PTO ISA 文档签名 (Declared in include/pto/pto_instr.hpp): -// -// TLOAD: -// template -// PTO_INST RecordEvent TLOAD(TileData &dst, GlobalData &src, WaitEvents &... events); -// -// TCVT: -// template -// PTO_INST RecordEvent TCVT(TileDataD &dst, TileDataS &src, TmpTileData &tmp, -// RoundMode mode, SaturationMode satMode, -// WaitEvents &... events); -// -// TMAXS: -// template -// PTO_INST RecordEvent TMAXS(TileDataDst &dst, TileDataSrc &src, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TMINS: -// template -// PTO_INST RecordEvent TMINS(TileDataDst &dst, TileDataSrc &src, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TMUL: -// template -// PTO_INST RecordEvent TMUL(TileDataDst &dst, TileDataSrc0 &src0, -// TileDataSrc1 &src1, WaitEvents &... events); -// -// TMULS: -// template -// PTO_INST RecordEvent TMULS(TileDataDst &dst, TileDataSrc &src0, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TADDS: -// template -// PTO_INST RecordEvent TADDS(TileDataDst &dst, TileDataSrc &src0, -// typename TileDataSrc::DType scalar, -// WaitEvents &... events); -// -// TEXP: -// template -// PTO_INST RecordEvent TEXP(TileDataDst &dst, TileDataSrc &src, -// WaitEvents &... events); -// -// TRECIP: -// template -// PTO_INST RecordEvent TRECIP(TileDataDst &dst, TileDataSrc &src, -// WaitEvents &... events); -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); // ============================================================================ #include @@ -257,13 +196,13 @@ void gelu( auto gO = gOIter(0, i); // TLOAD: GM -> UB - // [当前编译器] 名为 TCOPYIN, jcore 为 __vec__ + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ TLOAD(inTile, gI); gelu_impl(inTile, outTile, tmpCvt); // TSTORE: UB -> GM - // [当前编译器] 名为 TCOPYOUT, jcore 为 __vec__ + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ TSTORE(gO, outTile); } if constexpr (rmd_M) { diff --git a/benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp b/benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp index 65bba29..57d5e45 100644 --- a/benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp +++ b/benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp @@ -109,7 +109,7 @@ void flash_attention_2d_unroll_pto(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, d // tMax[x]: [kTm, 1]，每行初值为 -inf 的近似值。 #pragma clang loop unroll(full) for(int x=0;x -// PTO_INST RecordEvent TLOAD(TileData &dst, GlobalData &src, -// WaitEvents &... events); -// -// MGATHER (Coalesce::Row 模式): -// template -// PTO_INST RecordEvent MGATHER(TileDst &dst, GlobalData &src, -// TileInd &indexes, WaitEvents &... events); -// -// Row 模式语义: dst[r,:] = table[idx[r], :] -// - idx 为行索引 (非字节偏移) -// - tablePtr + idx * tableRowStride 定位到行起始 -// - 读取 validCol 个元素到 dst 对应行 -// - 约束 (A2A3): TileIdx::ValidRow==1, TileIdx::ValidCol==TileDst::ValidRow -// - 约束 (A5): 同上, 或 [R,1] ColMajor; 且 staticShape[4]==ValidCol -// -// TSTORE: -// template -// PTO_INST RecordEvent TSTORE(GlobalData &dst, TileData &src, -// WaitEvents &... events); // ============================================================================ #include @@ -141,7 +113,7 @@ void gather( size_t n_base = i * tN; // TLOAD: 加载行索引 tile (1, tM) from GM - // [当前编译器] 名为 TCOPYIN + // [当前编译器] 名为 TLOAD TLOAD(inOffsetTile, gInOffset); // MGATHER: 按行索引从数据表取数 @@ -154,7 +126,7 @@ void gather( MGATHER(outTile, adjustedGm, inOffsetTile); // TSTORE: 写回输出 tile (tM, tN) to GM - // [当前编译器] 名为 TCOPYOUT + // [当前编译器] 名为 TSTORE TSTORE(gO, outTile); } diff --git a/benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp b/benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp index 8fe42ce..55cc6cf 100644 --- a/benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp +++ b/benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp @@ -31,7 +31,7 @@ void gen_ND2NN_offset_Impl(Gm &, Tile &, OffsetTile &, uint32_t, uint32_t); // GlobalTensor, \ // Stride<1,1,1,Cols,1>> _g(DumpBuf); \ -// TCOPYOUT(_g, TileVar); \ +// TSTORE(_g, TileVar); \ // printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ // for (int ri = 0; ri < Rows; ri++) { \ // printf(" row%2d: ", ri); \ @@ -50,7 +50,7 @@ using namespace pto; // TODO, move to utils.cpp template -void TCOPYOUT_ACC(GmOut &Gout, TileAcc &tAcc){ +void TSTORE_ACC(GmOut &Gout, TileAcc &tAcc){ using TileAccOut = Tile; TileAccOut tAccOut; if constexpr (TileAcc::Loc == Location::Acc) { @@ -58,16 +58,16 @@ void TCOPYOUT_ACC(GmOut &Gout, TileAcc &tAcc){ } else { TCVT(tAccOut, tAcc); } - TCOPYOUT(Gout, tAccOut); + TSTORE(Gout, tAccOut); } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - // static_assert(gM % tM == 0); + // static_assert(gM % tM == 0); // static_assert(gN % tN == 0); // static_assert(gK % tK == 0); static const uint32_t valid_row = (tM > gM) ? gM : tM; @@ -76,7 +76,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -87,7 +87,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; gm_shapeAMX gAMX(src0_mx); using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using itAMX = global_iterator; @@ -141,8 +141,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(k,j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); // if (src0_mx != nullptr && src1_mx != nullptr) { tile_shapeAMX tAMX; @@ -163,8 +163,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(Kb,j); tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_trows tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); @@ -180,7 +180,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -190,8 +190,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(k, Nb); tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, k); @@ -213,8 +213,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_trows tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); @@ -228,7 +228,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -243,8 +243,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcols tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); @@ -266,8 +266,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcorner tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); @@ -282,7 +282,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -295,8 +295,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcols tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); @@ -317,8 +317,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcorner tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); @@ -332,12 +332,12 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } -template void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { static_assert(typeb_wfactor == 1 ); @@ -347,7 +347,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -358,7 +358,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using itAMX = global_iterator; itAMX gAMXIter(src0_mx); @@ -412,10 +412,10 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -428,14 +428,14 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx auto gB = gBIter(Kb,j); tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, Kb); auto gBMX = gBMXIter(Kb, j); tile_shapeAMX_trows tAMX; tile_shapeBMX_tcols tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -443,7 +443,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -453,15 +453,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx auto gB = gBIter(k, Nb); tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, k); auto gBMX = gBMXIter(k, Nb); tile_shapeAMX tAMX; tile_shapeBMX_trows tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -476,15 +476,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, Kb); auto gBMX = gBMXIter(Kb, Nb); tile_shapeAMX_trows tAMX; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -492,7 +492,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -507,15 +507,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, k); auto gBMX = gBMXIter(k, j); tile_shapeAMX_tcols tAMX; tile_shapeBMX tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -530,15 +530,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, Kb); auto gBMX = gBMXIter(Kb, j); tile_shapeAMX_tcorner tAMX; tile_shapeBMX_tcols tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -546,7 +546,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -559,15 +559,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, k); auto gBMX = gBMXIter(k, Nb); tile_shapeAMX_tcols tAMX; tile_shapeBMX_trows tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -581,22 +581,22 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, Kb); auto gBMX = gBMXIter(Kb, Nb); tile_shapeAMX_tcorner tAMX; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); } else { MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -631,22 +631,22 @@ constexpr ResA find_reuseA(int Mb, int Kb, int MAX_TILE_NUM) { return {best_m, best_k, best_val}; } -template void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { static_assert(typeb_wfactor == 1 ); static const uint32_t valid_row = (tM > gM) ? gM : tM; static const uint32_t valid_col = (tN > gN) ? gN : tN; static const uint32_t MAX_TILE_NUM = 24; // TODO, check this value - + using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; - + using itA = global_iterator; using itB = global_iterator; using itC = global_iterator; @@ -655,8 +655,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; - using tile_shapeAMX = Tile; + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; using itAMX = global_iterator; itAMX gAMXIter(src0_mx); @@ -713,8 +713,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * // for(int k=0; k0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -785,7 +785,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } auto gC = gCIter(i*R.m+ii, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [m, rmd_N, k] @@ -798,8 +798,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[ii][k], tAMX[ii][k], tB, tBMX); }else{ @@ -817,10 +817,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(i*R.m+ii, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -836,11 +836,11 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeAMX_trows tAMX_tmp; tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -849,7 +849,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } auto gC = gCIter(i*R.m+ii, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -857,7 +857,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * if constexpr(rM>0){ tile_shapeA tA[rM][R.k]; tile_shapeAMX tAMX[rM][R.k]; - + #pragma clang loop unroll(full) for(int i=0; i0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -936,7 +936,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(i+dM*R.m, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rM, rmd_N, k] @@ -949,8 +949,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[i][k], tAMX[i][k], tB, tBMX); }else{ @@ -968,10 +968,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(i+dM*R.m, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -988,10 +988,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -999,7 +999,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(i+dM*R.m, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -1008,13 +1008,13 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * if constexpr (rmd_M) { tile_shapeA_tcols tA[R.k]; tile_shapeAMX_tcols tAMX[R.k]; - + #pragma clang loop unroll(full) for(int k=0; k0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -1077,7 +1077,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rmd_M, rmd_N, k] @@ -1090,8 +1090,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[k], tAMX[k], tB, tBMX); }else{ @@ -1109,10 +1109,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(Mb, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -1129,10 +1129,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -1140,25 +1140,25 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - static_assert(gM % tM == 0); + static_assert(gM % tM == 0); static_assert(gN % tN == 0); static_assert(gK % tK == 0); using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -1169,7 +1169,7 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; // gm_shapeAMX gAMX(src0_mx); // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 @@ -1206,10 +1206,10 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -1219,25 +1219,25 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src // MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - static_assert(gM % tM == 0); + static_assert(gM % tM == 0); static_assert(gN % tN == 0); static_assert(gK % tK == 0); using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -1248,7 +1248,7 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; // gm_shapeAMX gAMX(src0_mx); // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 @@ -1285,10 +1285,10 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMUL(tACC, tA, tB); @@ -1297,7 +1297,7 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -1354,7 +1354,7 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; // 伪量化固定float, group 大小128， 128个fp4共享一个scaling factor, 128的partial sum* scale - using gm_shape_scale = global_tensor>; + using gm_shape_scale = global_tensor>; using gm_shapeACC = global_tensor>; using tile_shapeA = TileLeft; using tile_shapeB = TileRight; @@ -1362,7 +1362,7 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using tile_shape_dequant = Tile; using tile_shapeACC = TileAcc; // copy of acc, input as vector - using tile_ACCin = Tile; + using tile_ACCin = Tile; using itA = global_iterator; using itB = global_iterator; @@ -1398,9 +1398,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using tile_shape_scale_tcols = Tile; using tile_shape_scale_tcorner = Tile; - using tile_ACCin_trows = Tile; - using tile_ACCin_tcols = Tile; - using tile_ACCin_tconer = Tile; + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; using tile_shape_dequant_trows = Tile; using tile_shape_dequant_tcols = Tile; @@ -1412,7 +1412,7 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeACC tACC; tile_ACCin tACCin; tile_shape_dequant tAdder[2]; - TEXPANDS(tAdder[0], 0.); + TEXPANDSCALAR(tAdder[0], 0.); int k=0; #pragma clang loop unroll(full) for(;k1 , 256 -> 2 scaling factor @@ -1444,26 +1444,26 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shape_scale_tcols ts; tile_shape_dequant tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); ACCCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); tAdder[(k+1)%2] = tC_dequant; } - TCOPYOUT(gACC, tAdder[(k+1)%2]); + TSTORE(gACC, tAdder[(k+1)%2]); } // if constexpr (rmd_N) // TODO } if constexpr (rmd_M) { for (int j = 0; j < Nb; ++j) { - auto gACC = gACCIter(Mb, j); + auto gACC = gACCIter(Mb, j); tile_shapeC_tcols tACC; tile_ACCin_tcols tACCin; tile_shape_dequant_tcols tAdder[2]; - TEXPANDS(tAdder[0], 0.); + TEXPANDSCALAR(tAdder[0], 0.); int k = 0; #pragma clang loop unroll(full) for (; k < Kb; ++k) { @@ -1475,9 +1475,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeB tB; tile_shape_scale ts; tile_shape_dequant_tcols tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); ACCCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); @@ -1493,15 +1493,15 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeB_tcols tB; tile_shape_scale_tcols ts; tile_shape_dequant tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); ACCCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); tAdder[(k+1)%2] = tC_dequant; } - TCOPYOUT(gACC, tAdder[(k+1)%2]); + TSTORE(gACC, tAdder[(k+1)%2]); } // todo // if constexpr (rmd_N) { @@ -1514,9 +1514,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcols tA; // tile_shapeB_trows tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); - // MATMUL(tACC, tA, tB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMUL(tACC, tA, tB); // } // #pragma clang loop unroll(full) // for (int k = 1; k < Kb; ++k) { @@ -1525,8 +1525,8 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcols tA; // tile_shapeB_trows tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); // MATMACC(tACC, tA, tB); // } // if constexpr (rmd_K) { @@ -1535,15 +1535,15 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcorner tA; // tile_shapeB_tcorner tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); // if constexpr(Kb>0){ // MATMACC(tACC, tA, tB); // } else { // MATMUL(tACC, tA, tB); // } // } - // TCOPYOUT_ACC(gC, tACC); + // TSTORE_ACC(gC, tACC); // } } } @@ -1614,7 +1614,7 @@ void matmul_mp_tileop(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr tile_shapeACC tACC; tile_ACCin tACCin; tile_shape_dequant tAdder[2]; - TEXPANDS(tAdder[0], 0.); + TEXPANDSCALAR(tAdder[0], 0.); int k=0; #pragma clang loop unroll(full) for(;k( - &input[i * 120 * 8], + &input[i * 120 * 8], &output[i * 8] ); } diff --git a/benchmark/one-level-arch/kernels/reduction/cumsum_colvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/cumsum_colvec_pto.hpp index a8219aa..9a5a03b 100644 --- a/benchmark/one-level-arch/kernels/reduction/cumsum_colvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/cumsum_colvec_pto.hpp @@ -23,7 +23,7 @@ void cumsum_col_rand( // 对每一列独立计算累积和 for (int j = 0; j < gIN; ++j) { ScalarTile running_sum; - TEXPANDS(running_sum, static_cast(0)); + TEXPANDSCALAR(running_sum, static_cast(0)); for (int i = 0; i < gIM; ++i) { auto gI = gIIter(i, j); diff --git a/benchmark/one-level-arch/kernels/reduction/cumsum_rowvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/cumsum_rowvec_pto.hpp index 9be3b11..478418a 100644 --- a/benchmark/one-level-arch/kernels/reduction/cumsum_rowvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/cumsum_rowvec_pto.hpp @@ -24,7 +24,7 @@ void cumsum_row_rand( // 对每一行独立计算累积和 for (int i = 0; i < gIM; ++i) { ScalarTile running_sum; - TEXPANDS(running_sum, static_cast(0)); + TEXPANDSCALAR(running_sum, static_cast(0)); for (int j = 0; j < gIN; ++j) { auto gI = gIIter(i, j); diff --git a/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp index 0c4dc7d..7b68067 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp @@ -47,7 +47,7 @@ void reducemax_col_rand( for (int j = 0; j < Nb; ++j) { auto gO = gOIter(0, j); - TEXPANDS(oldMaxTile, static_cast(0)); + TEXPANDSCALAR(oldMaxTile, static_cast(0)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); @@ -65,7 +65,7 @@ void reducemax_col_rand( } if constexpr (rmd_N > 0) { auto gO = gOIter(0, Nb); - TEXPANDS(oldMaxTile_row, static_cast(0)); + TEXPANDSCALAR(oldMaxTile_row, static_cast(0)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); diff --git a/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8_pto.hpp index 2b05fc2..0ba5129 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8_pto.hpp @@ -61,7 +61,7 @@ void reducemax_col_rand( itOut gOIter(out_ptr); auto gO = gOIter(0, 0); - TEXPANDS(oldMaxTile, static_cast(0)); + TEXPANDSCALAR(oldMaxTile, static_cast(0)); auto gI = gIIter(0, 0); TLOAD(dataTile, gI); diff --git a/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp index 7532d2f..8cde7d4 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp @@ -56,7 +56,7 @@ void reducemax_row_rand( for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); - TEXPANDS(oldMaxTile, static_cast(0)); + TEXPANDSCALAR(oldMaxTile, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(j, i); @@ -74,7 +74,7 @@ void reducemax_row_rand( } if constexpr (rmd_M > 0) { auto gO = gOIter(Mb, 0); - TEXPANDS(oldMaxTile_col, static_cast(0)); + TEXPANDSCALAR(oldMaxTile_col, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(Mb, i); diff --git a/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_single_tree_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_single_tree_pto.hpp index dac1150..70afe78 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_single_tree_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_single_tree_pto.hpp @@ -42,7 +42,7 @@ void reducemax_row_rand( itOut gOIter(out_ptr); auto gO = gOIter(0, 0); - TEXPANDS(oldtmpMaxTile, static_cast(0)); + TEXPANDSCALAR(oldtmpMaxTile, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(0, i); diff --git a/benchmark/one-level-arch/kernels/reduction/reduceprod_colvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reduceprod_colvec_pto.hpp index a5b1d4b..652ef47 100644 --- a/benchmark/one-level-arch/kernels/reduction/reduceprod_colvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reduceprod_colvec_pto.hpp @@ -47,7 +47,7 @@ void reduceprod_col_rand( for (int j = 0; j < Nb; ++j) { auto gO = gOIter(0, j); - TEXPANDS(oldProdTile, static_cast(1)); + TEXPANDSCALAR(oldProdTile, static_cast(1)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); @@ -65,7 +65,7 @@ void reduceprod_col_rand( } if constexpr (rmd_N > 0) { auto gO = gOIter(0, Nb); - TEXPANDS(oldProdTile_row, static_cast(1)); + TEXPANDSCALAR(oldProdTile_row, static_cast(1)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); diff --git a/benchmark/one-level-arch/kernels/reduction/reduceprod_rowvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reduceprod_rowvec_pto.hpp index c178b7f..d5f3469 100644 --- a/benchmark/one-level-arch/kernels/reduction/reduceprod_rowvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reduceprod_rowvec_pto.hpp @@ -56,7 +56,7 @@ void reduceprod_row_rand( for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); - TEXPANDS(oldProdTile, static_cast(1)); + TEXPANDSCALAR(oldProdTile, static_cast(1)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(j, i); @@ -74,7 +74,7 @@ void reduceprod_row_rand( } if constexpr (rmd_M > 0) { auto gO = gOIter(Mb, 0); - TEXPANDS(oldProdTile_col, static_cast(1)); + TEXPANDSCALAR(oldProdTile_col, static_cast(1)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(Mb, i); diff --git a/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp index a06c59e..71169f0 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp @@ -57,7 +57,7 @@ void reducesum_colsum_rand( for (int j = 0; j < Nb; ++j) { auto gO = gOIter(0, j); - TEXPANDS(oldSumTile, static_cast(0)); + TEXPANDSCALAR(oldSumTile, static_cast(0)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); @@ -75,7 +75,7 @@ void reducesum_colsum_rand( } if constexpr (rmd_N > 0) { auto gO = gOIter(0, Nb); - TEXPANDS(oldSumTile_row, static_cast(0)); + TEXPANDSCALAR(oldSumTile_row, static_cast(0)); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); diff --git a/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8_pto.hpp index 9831ea3..2025ed9 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8_pto.hpp @@ -70,7 +70,7 @@ void reducesum_colsum_rand( itOut gOIter(out_ptr); auto gO = gOIter(0, 0); - TEXPANDS(oldSumTile, static_cast(0)); + TEXPANDSCALAR(oldSumTile, static_cast(0)); auto gI = gIIter(0, 0); TLOAD(dataTile, gI); diff --git a/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp index e6396f8..713988e 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp @@ -56,7 +56,7 @@ void reducesum_trowsum_rand( for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); - TEXPANDS(oldSumTile, static_cast(0)); + TEXPANDSCALAR(oldSumTile, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(j, i); @@ -74,7 +74,7 @@ void reducesum_trowsum_rand( } if constexpr (rmd_M > 0) { auto gO = gOIter(Mb, 0); - TEXPANDS(oldSumTile_col, static_cast(0)); + TEXPANDSCALAR(oldSumTile_col, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(Mb, i); diff --git a/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_single_tree_pto.hpp b/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_single_tree_pto.hpp index ec28b88..534a139 100644 --- a/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_single_tree_pto.hpp +++ b/benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_single_tree_pto.hpp @@ -42,7 +42,7 @@ void reducesum_trowsum_rand( itOut gOIter(out_ptr); auto gO = gOIter(0, 0); - TEXPANDS(oldtmpSumTile, static_cast(0)); + TEXPANDSCALAR(oldtmpSumTile, static_cast(0)); for (int i = 0; i < Nb; ++i) { auto gI = gIIter(0, i); diff --git a/benchmark/one-level-arch/kernels/sort/topk_pto.hpp b/benchmark/one-level-arch/kernels/sort/topk_pto.hpp index 11be9ea..7d01283 100644 --- a/benchmark/one-level-arch/kernels/sort/topk_pto.hpp +++ b/benchmark/one-level-arch/kernels/sort/topk_pto.hpp @@ -38,37 +38,37 @@ template void ExtractHigh8Hist_PTO(tile_shape_out& dst, const uint16_t* src) { using DataTile = Tile; using HistTile = Tile; - + DataTile data_tile; HistTile hist_tile; - + // Initialize histogram to 0 - TEXPANDS(hist_tile, static_cast(0)); - + TEXPANDSCALAR(hist_tile, static_cast(0)); + // Process input in tiles for (int tile_idx = 0; tile_idx < kNumTiles; ++tile_idx) { // Load input tile TLOAD(data_tile, src + tile_idx * kTileSize); - + // Extract high8 bits: high8 = val >> 8 DataTile high8_tile; TSHRS(high8_tile, data_tile, static_cast(8)); - + // For each bucket, count matches // This is a simplified version - full implementation would need // to use TCMP and conditional accumulation for (int bucket = 0; bucket < kNumBuckets; ++bucket) { HistTile bucket_tile; - TEXPANDS(bucket_tile, static_cast(bucket)); - + TEXPANDSCALAR(bucket_tile, static_cast(bucket)); + // Compare high8 with bucket // Note: This requires more sophisticated tile operations // For now, we use a scalar fallback } } - + // Store result - TCOPYOUT(dst, hist_tile); + TSTORE(dst, hist_tile); } // ============================================================================ @@ -80,32 +80,32 @@ void ExtractLow8HistForKthBin_PTO(tile_shape_out& dst, const uint16_t* src, uint16_t kth_bin) { using DataTile = Tile; using HistTile = Tile; - + DataTile data_tile; HistTile hist_tile; - + // Initialize histogram to 0 - TEXPANDS(hist_tile, static_cast(0)); - + TEXPANDSCALAR(hist_tile, static_cast(0)); + // Process input in tiles for (int tile_idx = 0; tile_idx < kNumTiles; ++tile_idx) { // Load input tile TLOAD(data_tile, src + tile_idx * kTileSize); - + // Extract high8 bits DataTile high8_tile; TSHRS(high8_tile, data_tile, static_cast(8)); - + // Extract low8 bits: low8 = val & 0xFF DataTile low8_tile; TANDS(low8_tile, data_tile, static_cast(0xFF)); - + // Filter by kth_bin and accumulate // This requires conditional operations } - + // Store result - TCOPYOUT(dst, hist_tile); + TSTORE(dst, hist_tile); } // ============================================================================ diff --git a/benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp b/benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp index 11a3525..328c9e1 100644 --- a/benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp +++ b/benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp @@ -116,7 +116,7 @@ void tile_transpose_nd(DType *input, DType *output, std::uint32_t *input_shape, TCI(linear_index, output_base); // Step 2: 初始化 offset 累加器为 0 - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); // Step 3: 对每个维度计算坐标并累加 offset // 关键：这个循环在标量核心上运行，由于 Rank 是模板参数，编译期完全展开 @@ -206,7 +206,7 @@ void tile_transpose_nd(DType *input, DType *output, std::uint32_t *input_shape, // 区别仅在于 valid region 更小（kTailElements 而非 TileElements） // tile 操作以 valid region 为迭代域，因此自动只处理有效元素 TCI(linear_index, output_base); - TEXPANDS(offset_tile, static_cast(0)); + TEXPANDSCALAR(offset_tile, static_cast(0)); std::uint32_t input_stride = 1; for (int input_dim = Rank - 1; input_dim >= 0; --input_dim) { diff --git a/benchmark/one-level-arch/test/common/Makefile.common b/benchmark/one-level-arch/test/common/Makefile.common index 02d5f47..f6235d7 100644 --- a/benchmark/one-level-arch/test/common/Makefile.common +++ b/benchmark/one-level-arch/test/common/Makefile.common @@ -14,6 +14,8 @@ COMM_SRC_DIR := common/src COMM_OBJ += $(patsubst %.cpp, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.cpp))) COMM_OBJ += $(patsubst %.c, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.c))) COMM_OBJ += $(patsubst %.s, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.s))) +FREESTANDING_RUNTIME_SRC := $(TEST_ROOT)/$(COMM_SRC_DIR)/freestanding_runtime.c +FREESTANDING_RUNTIME_OBJ := $(patsubst %.c, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(FREESTANDING_RUNTIME_SRC))) OBJ := $(patsubst %.cpp, %.o, $(subst $(SRC_DIR), $(OBJ_DIR), $(SRC_FILE))) OBJ := $(patsubst %.c, %.o, $(OBJ)) @@ -24,6 +26,9 @@ LINK_SCRIPT = $(patsubst %.S, %, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(LINK_SCRI PLAT ?= linx baremetal ?= off +CC_OPT ?= default +LINX_SYSROOT ?= $(shell test -d "$(ROOT)/../../../../out/libc/musl/install/phase-b" && realpath "$(ROOT)/../../../../out/libc/musl/install/phase-b" || true) +LINX_SYSROOT_FLAGS := $(if $(LINX_SYSROOT),--sysroot=$(LINX_SYSROOT) -idirafter $(LINX_SYSROOT)/usr/include,) ifeq ($(PLAT), cpu) DEFINES += -D__cpu_sim__ @@ -51,30 +56,21 @@ CXX_VER ?= -std=c++20 else ifeq ($(PLAT), linx) DEFINES += -D__linx ifndef COMPILER_DIR -$(error COMPILER_DIR is not set. Export COMPILER_DIR pointing to the linx_blockisa_llvm_musl toolchain bin directory, e.g. export COMPILER_DIR=/path/to/linx_blockisa_llvm_musl/bin) +$(error COMPILER_DIR is not set. Point it to compiler/llvm/build-linxisa-clang/bin in the LinxISA superproject) endif AS = $(COMPILER_DIR)/clang -CC = $(COMPILER_DIR)/clang +CC = $(COMPILER_DIR)/clang CXX = $(COMPILER_DIR)/clang++ LINK = $(COMPILER_DIR)/clang++ DUMP = $(COMPILER_DIR)/llvm-objdump COPY = $(COMPILER_DIR)/llvm-objcopy -CC_O = -c -mlxbc -fenable-matrix -O2 -mllvm -enable-all-vector-as-tilereg=true $(CFLAGS) +CC_O = -c -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -fenable-matrix -O2 $(CFLAGS) CXX_VER ?= -std=c++20 - ifneq ($(CC_OPT), default) - CC_O += -mllvm -linxv5-enable-HL-Inst-Opt=true \ - -mllvm -linxv5-enable-dim-opt=true \ - -mllvm -linxv5-enable-ldst-bridge=false \ - -mllvm -linxv5-enable-continuous-mem-opt=true \ - -mllvm -linxv5-enable-tile-clock-hand=false \ - -mllvm -linxv5-enable-simt-clock-hand=true \ - -mllvm -enable-misched=false - endif ifeq ($(baremetal), on) - CC_LINK += -static -lm -nostartfiles -L $(OBJ_ROOT)/$(COMM_SRC_DIR) -T $(LINK_SCRIPT) + CC_LINK += $(LINX_SYSROOT_FLAGS) -static -lm -nostartfiles -L $(OBJ_ROOT)/$(COMM_SRC_DIR) -T $(LINK_SCRIPT) else - CC_LINK += -nostartfiles $(ROOT)/test/common/_start.s + CC_LINK += -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -nostdlib -Wl,-e,_start $(ROOT)/test/common/_start.s endif DEFINES += -DENABLE_TENSOR_INSTR @@ -92,16 +88,16 @@ CC_O += -fPIC CC_LINK += -shared endif -INCLUDE += -I$(ROOT)/include -I$(ROOT)/test/common -I$(ROOT)/test/common/src -I$(ROOT)/kernels -I$(ROOT)/models +INCLUDE += -I$(ROOT)/include -I$(ROOT)/../two-level-arch/include -I$(ROOT)/test/common -I$(ROOT)/test/common/src -I$(ROOT)/kernels -I$(ROOT)/models QEMU ?= /remote/lms60/c00622284/qemu/LinxBlockModel/build/qemu-linx CC_O_ALL = $(CC_O) $(CC_OPTS) CXX_O_ALL = $(CC_O) $(CXX_VER) $(CC_OPTS) ifneq ($(baremetal), on) -COMM_OBJ = -LINK_SCRIPT_SRC = -LINK_SCRIPT = +COMM_OBJ = +LINK_SCRIPT_SRC = +LINK_SCRIPT = else DEFINES += -D__baremetal endif @@ -109,7 +105,7 @@ endif ifeq ($(res_check), on) DEFINES += -DRES_CHECK -DENABLE_BINARY_OUTPUT DEFINES += -DCHK_DIR=\"$(ROOT)/compare/$(notdir $(basename $(TARGET)))\" -CC_LINK = +CC_LINK = endif .DEFAULT_GOAL := all @@ -147,18 +143,18 @@ $(OBJ_DIR)%.o: $(SRC_DIR)%.s $(OBJ_ROOT)/$(COMM_SRC_DIR)%.o: $(TEST_ROOT)/$(COMM_SRC_DIR)%.c @mkdir -p $(shell dirname $@) - $(CXX) -mlxbc -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ + $(CXX) -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ $(OBJ_ROOT)/$(COMM_SRC_DIR)%.o: $(TEST_ROOT)/$(COMM_SRC_DIR)%.s @mkdir -p $(shell dirname $@) - $(CXX) -mlxbc -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ + $(CXX) -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ $(OBJ_ROOT)/$(COMM_SRC_DIR)%.lds : $(TEST_ROOT)/$(COMM_SRC_DIR)%.lds.S $(CC) $(DEFINES) $(INCLUDE) -E -P -C -o $@ $< -$(TARGET): clean $(LINK_SCRIPT) $(COMM_OBJ) $(OBJ) $(EXTRA_OBJ_FILES) +$(TARGET): clean $(LINK_SCRIPT) $(COMM_OBJ) $(FREESTANDING_RUNTIME_OBJ) $(OBJ) $(EXTRA_OBJ_FILES) @mkdir -p $(shell dirname $@) - $(LINK) $(CC_LINK) $(OBJ) $(COMM_OBJ) $(EXTRA_OBJ_FILES) -o $@ + $(LINK) $(CC_LINK) $(OBJ) $(COMM_OBJ) $(FREESTANDING_RUNTIME_OBJ) $(EXTRA_OBJ_FILES) -o $@ pre_work: @mkdir -p $(OBJ_DIR) diff --git a/benchmark/one-level-arch/test/common/fileop.h b/benchmark/one-level-arch/test/common/fileop.h index 2643ae1..18b8ffa 100644 --- a/benchmark/one-level-arch/test/common/fileop.h +++ b/benchmark/one-level-arch/test/common/fileop.h @@ -1,4 +1,2 @@ -#include -#include #include "writeBinary.h" -#include "readBinary.h" \ No newline at end of file +#include "readBinary.h" diff --git a/benchmark/one-level-arch/test/common/multi_tile.hpp b/benchmark/one-level-arch/test/common/multi_tile.hpp index 6ca6a6c..e7ba914 100644 --- a/benchmark/one-level-arch/test/common/multi_tile.hpp +++ b/benchmark/one-level-arch/test/common/multi_tile.hpp @@ -129,19 +129,19 @@ void TCAST(tO &o, tA &a) { } template -void TCOPYIN(tile_shape &dst, itfn it) { +void TLOAD(tile_shape &dst, itfn it) { #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { auto gm = it(i); - TCOPYIN(dst.Tiles[i], gm); + TLOAD(dst.Tiles[i], gm); } } template -void TCOPYIN(tile_shape &dst, gm_shape &src) { +void TLOAD(tile_shape &dst, gm_shape &src) { #ifdef MULTI_REUSE typename tile_shape::TileType t; - TCOPYIN(t, src); + TLOAD(t, src); #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { dst.Tiles[i] = t; @@ -149,17 +149,17 @@ void TCOPYIN(tile_shape &dst, gm_shape &src) { #else #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { - TCOPYIN(dst.Tiles[i], src); + TLOAD(dst.Tiles[i], src); } #endif } template -void TCOPYOUT(itfn it, tile_shape &src) { +void TSTORE(itfn it, tile_shape &src) { #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { auto gm = it(i); - TCOPYOUT(gm, src.Tiles[i]); + TSTORE(gm, src.Tiles[i]); } } diff --git a/benchmark/one-level-arch/test/common/readBinary.h b/benchmark/one-level-arch/test/common/readBinary.h index be99c43..1df4324 100644 --- a/benchmark/one-level-arch/test/common/readBinary.h +++ b/benchmark/one-level-arch/test/common/readBinary.h @@ -6,8 +6,6 @@ #include #include #include -#include -#include // #define ENABLE_BINARY_OUTPUT @@ -34,4 +32,4 @@ bool readBinaryFile(const char * filename, uint8_t* data, size_t size) { #endif } -#endif \ No newline at end of file +#endif diff --git a/benchmark/one-level-arch/test/common/src/baremetal_linx.lds.S b/benchmark/one-level-arch/test/common/src/baremetal_linx.lds.S index 1891c2f..d893956 100644 --- a/benchmark/one-level-arch/test/common/src/baremetal_linx.lds.S +++ b/benchmark/one-level-arch/test/common/src/baremetal_linx.lds.S @@ -1,4 +1,4 @@ -OUTPUT_ARCH(linxv5) +OUTPUT_ARCH(linx) ENTRY(firmware_base) /* SEARCH_DIR("=/usr1/compiler_cpu/build/BiSheng_linx64be_elf/../../output/BiSheng_linx64be_elf/BiSheng_linx64be_elf/linx64-unknown-elf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); @@ -13,15 +13,15 @@ MEMORY { SECTIONS { - mem_base = MEM_BASE; + mem_base = MEM_BASE; .app_mem mem_base : { - KEEP(*(.app_text)) + KEEP(*(.app_text)) KEEP(*(.app_data)) } > RAM firmware_base = FIRMWARE_BASE; - .firmware firmware_base : + .firmware firmware_base : { KEEP(*benchmark_boot_linx.o(linxboot_text)) KEEP(*(boot_text)) @@ -75,7 +75,7 @@ SECTIONS *(.text .stub .text.* .gnu.linkonce.t.*) /* .gnu.warning sections are handled specially by elf.em. */ *(.gnu.warning) - } > RAM =0 + } > RAM =0 .init : { @@ -288,7 +288,7 @@ SECTIONS .heap (NOLOAD): { /* . = ALIGN(8);*/ - . = . + HEAP_SIZE; // 4GB + . = . + HEAP_SIZE; // 4GB } > HEAP heap_end = .; diff --git a/benchmark/one-level-arch/test/common/src/freestanding_runtime.c b/benchmark/one-level-arch/test/common/src/freestanding_runtime.c new file mode 100644 index 0000000..b2862ba --- /dev/null +++ b/benchmark/one-level-arch/test/common/src/freestanding_runtime.c @@ -0,0 +1,35 @@ +#ifdef __cplusplus +extern "C" { +#endif + +void *memset(void *dst, int value, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + for (unsigned long i = 0; i < size; ++i) + out[i] = (unsigned char)value; + return dst; +} + +void *memcpy(void *dst, const void *src, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + const volatile unsigned char *in = (const volatile unsigned char *)src; + for (unsigned long i = 0; i < size; ++i) + out[i] = in[i]; + return dst; +} + +void *memmove(void *dst, const void *src, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + const volatile unsigned char *in = (const volatile unsigned char *)src; + if (out < in) { + for (unsigned long i = 0; i < size; ++i) + out[i] = in[i]; + } else if (out > in) { + for (unsigned long i = size; i != 0; --i) + out[i - 1] = in[i - 1]; + } + return dst; +} + +#ifdef __cplusplus +} +#endif diff --git a/benchmark/one-level-arch/test/common/src/ldv5.lds.S b/benchmark/one-level-arch/test/common/src/ldv5.lds.S deleted file mode 100644 index 5a7322d..0000000 --- a/benchmark/one-level-arch/test/common/src/ldv5.lds.S +++ /dev/null @@ -1,259 +0,0 @@ -OUTPUT_ARCH(linxv5) -ENTRY(_start) -SEARCH_DIR("=/usr1/compiler_cpu/build/BiSheng_linx64be_elf/../../output/BiSheng_linx64be_elf/BiSheng_linx64be_elf/linx64-unknown-elf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); -SECTIONS -{ - /* Read-only sections, merged into text segment: */ - PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x10000)); . = SEGMENT_START("text-segment", 0x10000) + SIZEOF_HEADERS; - .interp : { *(.interp) } - .note.gnu.build-id : { *(.note.gnu.build-id) } - .hash : { *(.hash) } - .gnu.hash : { *(.gnu.hash) } - .dynsym : { *(.dynsym) } - .dynstr : { *(.dynstr) } - .gnu.version : { *(.gnu.version) } - .gnu.version_d : { *(.gnu.version_d) } - .gnu.version_r : { *(.gnu.version_r) } - .rela.dyn : - { - *(.rela.init) - *(.rela.text .rela.text.* .rela.gnu.linkonce.t.*) - *(.rela.fini) - *(.rela.rodata .rela.rodata.* .rela.gnu.linkonce.r.*) - *(.rela.data .rela.data.* .rela.gnu.linkonce.d.*) - *(.rela.tdata .rela.tdata.* .rela.gnu.linkonce.td.*) - *(.rela.tbss .rela.tbss.* .rela.gnu.linkonce.tb.*) - *(.rela.ctors) - *(.rela.dtors) - *(.rela.got) - *(.rela.sdata .rela.sdata.* .rela.gnu.linkonce.s.*) - *(.rela.sbss .rela.sbss.* .rela.gnu.linkonce.sb.*) - *(.rela.sdata2 .rela.sdata2.* .rela.gnu.linkonce.s2.*) - *(.rela.sbss2 .rela.sbss2.* .rela.gnu.linkonce.sb2.*) - *(.rela.bss .rela.bss.* .rela.gnu.linkonce.b.*) - PROVIDE_HIDDEN (__rela_iplt_start = .); - *(.rela.iplt) - PROVIDE_HIDDEN (__rela_iplt_end = .); - } - .rela.plt : - { - *(.rela.plt) - } - .init : - { - KEEP (*(SORT_NONE(.init))) - } - .plt : { *(.plt) } - .iplt : { *(.iplt) } - . = ALIGN(16); - .text : - { - *(.text.unlikely .text.*_unlikely .text.unlikely.*) - *(.text.exit .text.exit.*) - *(.text.startup .text.startup.*) - *(.text.hot .text.hot.*) - *(SORT(.text.sorted.*)) - *(.text .stub .text.* .gnu.linkonce.t.*) - /* .gnu.warning sections are handled specially by elf.em. */ - *(.gnu.warning) - } - .fini : - { - KEEP (*(SORT_NONE(.fini))) - } - PROVIDE (__etext = .); - PROVIDE (_etext = .); - PROVIDE (etext = .); - .rodata : { *(.rodata .rodata.* .gnu.linkonce.r.*) } - .rodata1 : { *(.rodata1) } - .sdata2 : - { - *(.sdata2 .sdata2.* .gnu.linkonce.s2.*) - } - .sbss2 : { *(.sbss2 .sbss2.* .gnu.linkonce.sb2.*) } - /* - .eh_frame : - { - __eh_frame_start = .; - KEEP(*(.eh_frame)) - __eh_frame_end = .; - } - */ - .eh_frame_hdr : - { - KEEP(*(.eh_frame_hdr)) - } - __eh_frame_hdr_start = SIZEOF(.eh_frame_hdr) > 0 ? ADDR(.eh_frame_hdr) : 0; - __eh_frame_hdr_end = SIZEOF(.eh_frame_hdr) > 0 ? . : 0; - .sframe : ONLY_IF_RO { *(.sframe) *(.sframe.*) } - .gcc_except_table : ONLY_IF_RO { *(.gcc_except_table .gcc_except_table.*) } - .gnu_extab : ONLY_IF_RO { *(.gnu_extab*) } - /* These sections are generated by the Sun/Oracle C++ compiler. */ - .exception_ranges : ONLY_IF_RO { *(.exception_ranges*) } - /* Adjust the address for the data segment. We want to adjust up to - the same address within the page on the next page up. */ - . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE)); - /* Exception handling */ - .eh_frame : ONLY_IF_RW { KEEP (*(.eh_frame)) *(.eh_frame.*) } - .sframe : ONLY_IF_RW { *(.sframe) *(.sframe.*) } - .gnu_extab : ONLY_IF_RW { *(.gnu_extab) } - .gcc_except_table : ONLY_IF_RW { *(.gcc_except_table .gcc_except_table.*) } - .exception_ranges : ONLY_IF_RW { *(.exception_ranges*) } - /* Thread Local Storage sections */ - .tdata : - { - PROVIDE_HIDDEN (__tdata_start = .); - *(.tdata .tdata.* .gnu.linkonce.td.*) - } - .tbss : { *(.tbss .tbss.* .gnu.linkonce.tb.*) *(.tcommon) } - .preinit_array : - { - PROVIDE_HIDDEN (__preinit_array_start = .); - KEEP (*(.preinit_array)) - PROVIDE_HIDDEN (__preinit_array_end = .); - } - .init_array : - { - PROVIDE_HIDDEN (__init_array_start = .); - KEEP (*(SORT_BY_INIT_PRIORITY(.init_array.*) SORT_BY_INIT_PRIORITY(.ctors.*))) - KEEP (*(.init_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .ctors)) - PROVIDE_HIDDEN (__init_array_end = .); - } - .fini_array : - { - PROVIDE_HIDDEN (__fini_array_start = .); - KEEP (*(SORT_BY_INIT_PRIORITY(.fini_array.*) SORT_BY_INIT_PRIORITY(.dtors.*))) - KEEP (*(.fini_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .dtors)) - PROVIDE_HIDDEN (__fini_array_end = .); - } - .ctors : - { - /* gcc uses crtbegin.o to find the start of - the constructors, so we make sure it is - first. Because this is a wildcard, it - doesn't matter if the user does not - actually link against crtbegin.o; the - linker won't look for a file to match a - wildcard. The wildcard also means that it - doesn't matter which directory crtbegin.o - is in. */ - KEEP (*crtbegin.o(.ctors)) - KEEP (*crtbegin?.o(.ctors)) - /* We don't want to include the .ctor section from - the crtend.o file until after the sorted ctors. - The .ctor section from the crtend file contains the - end of ctors marker and it must be last */ - KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors)) - KEEP (*(SORT(.ctors.*))) - KEEP (*(.ctors)) - } - .dtors : - { - KEEP (*crtbegin.o(.dtors)) - KEEP (*crtbegin?.o(.dtors)) - KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .dtors)) - KEEP (*(SORT(.dtors.*))) - KEEP (*(.dtors)) - } - .jcr : { KEEP (*(.jcr)) } - .data.rel.ro : { *(.data.rel.ro.local* .gnu.linkonce.d.rel.ro.local.*) *(.data.rel.ro .data.rel.ro.* .gnu.linkonce.d.rel.ro.*) } - .dynamic : { *(.dynamic) } - . = DATA_SEGMENT_RELRO_END (0, .); - .data : - { - __DATA_BEGIN__ = .; - *(.data .data.* .gnu.linkonce.d.*) - SORT(CONSTRUCTORS) - } - .data1 : { *(.data1) } - .got : { *(.got.plt) *(.igot.plt) *(.got) *(.igot) } - /* We want the small data sections together, so single-instruction offsets - can aaccelss them all, and initialized data all before uninitialized, so - we can shorten the on-disk segment size. */ - .sdata : - { - __SDATA_BEGIN__ = .; - *(.srodata.cst16) *(.srodata.cst8) *(.srodata.cst4) *(.srodata.cst2) *(.srodata .srodata.*) - *(.sdata .sdata.* .gnu.linkonce.s.*) - } - _edata = .; PROVIDE (edata = .); - . = .; - __bss_start = .; - .sbss : - { - *(.dynsbss) - *(.sbss .sbss.* .gnu.linkonce.sb.*) - *(.scommon) - } - .bss : - { - *(.dynbss) - *(.bss .bss.* .gnu.linkonce.b.*) - *(COMMON) - /* Align here to ensure that the .bss section occupies space up to - _end. Align after .bss to ensure correct alignment even if the - .bss section disappears because there are no input sections. - FIXME: Why do we need it? When there is no .bss section, we do not - pad the .data section. */ - . = ALIGN(. != 0 ? 64 / 8 : 1); - } - . = ALIGN(64 / 8); - . = SEGMENT_START("ldata-segment", .); - . = ALIGN(64 / 8); - __BSS_END__ = .; - __global_pointer$ = MIN(__SDATA_BEGIN__ + 0x800, - MAX(__DATA_BEGIN__ + 0x800, __BSS_END__ - 0x800)); - _end = .; PROVIDE (end = .); - . = DATA_SEGMENT_END (.); - /* Stabs debugging sections. */ - .stab 0 : { *(.stab) } - .stabstr 0 : { *(.stabstr) } - .stab.excl 0 : { *(.stab.excl) } - .stab.exclstr 0 : { *(.stab.exclstr) } - .stab.index 0 : { *(.stab.index) } - .stab.indexstr 0 : { *(.stab.indexstr) } - .comment 0 (INFO) : { *(.comment); LINKER_VERSION; } - .gnu.build.attributes : { *(.gnu.build.attributes .gnu.build.attributes.*) } - /* DWARF debug sections. - Symbols in the DWARF debugging sections are relative to the beginning - of the section so we begin them at 0. */ - /* DWARF 1. */ - .debug 0 : { *(.debug) } - .line 0 : { *(.line) } - /* GNU DWARF 1 extensions. */ - .debug_srcinfo 0 : { *(.debug_srcinfo) } - .debug_sfnames 0 : { *(.debug_sfnames) } - /* DWARF 1.1 and DWARF 2. */ - .debug_aranges 0 : { *(.debug_aranges) } - .debug_pubnames 0 : { *(.debug_pubnames) } - /* DWARF 2. */ - .debug_info 0 : { *(.debug_info .gnu.linkonce.wi.*) } - .debug_abbrev 0 : { *(.debug_abbrev) } - .debug_line 0 : { *(.debug_line .debug_line.* .debug_line_end) } - .debug_frame 0 : { *(.debug_frame) } - .debug_str 0 : { *(.debug_str) } - .debug_loc 0 : { *(.debug_loc) } - .debug_macinfo 0 : { *(.debug_macinfo) } - /* SGI/MIPS DWARF 2 extensions. */ - .debug_weaknames 0 : { *(.debug_weaknames) } - .debug_funcnames 0 : { *(.debug_funcnames) } - .debug_typenames 0 : { *(.debug_typenames) } - .debug_varnames 0 : { *(.debug_varnames) } - /* DWARF 3. */ - .debug_pubtypes 0 : { *(.debug_pubtypes) } - .debug_ranges 0 : { *(.debug_ranges) } - /* DWARF 5. */ - .debug_addr 0 : { *(.debug_addr) } - .debug_line_str 0 : { *(.debug_line_str) } - .debug_loclists 0 : { *(.debug_loclists) } - .debug_macro 0 : { *(.debug_macro) } - .debug_names 0 : { *(.debug_names) } - .debug_rnglists 0 : { *(.debug_rnglists) } - .debug_str_offsets 0 : { *(.debug_str_offsets) } - .debug_sup 0 : { *(.debug_sup) } - .gnu.attributes 0 : { KEEP (*(.gnu.attributes)) } - /DISCARD/ : { *(.note.GNU-stack) *(.gnu_debuglink) *(.gnu.lto_*) } -} - - -/*==================================================*/ diff --git a/benchmark/one-level-arch/test/common/template_asm.h b/benchmark/one-level-arch/test/common/template_asm.h index 2f60707..d1ad733 100644 --- a/benchmark/one-level-arch/test/common/template_asm.h +++ b/benchmark/one-level-arch/test/common/template_asm.h @@ -3,17 +3,252 @@ #include +#if !defined(__linx) + +namespace supernpubench_v057_test_fallback { +template +constexpr int tile_numel_v = T::Rows * T::Cols; + +template +inline void copy_tile(OutTile &dst, const InTile &src) { + constexpr int OutN = tile_numel_v; + constexpr int InN = tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src.data()[i]); +} + +template +inline void fill_tile(OutTile &dst, typename OutTile::DType value) { + for (int i = 0; i < tile_numel_v; ++i) + dst.data()[i] = value; +} + +template +inline void gather_bytes(OutTile &dst, GmShape &src, OffTile &offset) { + using DType = typename OutTile::DType; + constexpr int OutN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = OutN < OffN ? OutN : OffN; + const auto *base = src.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + dst.data()[i] = static_cast( + *(reinterpret_cast( + reinterpret_cast(base) + byte_off))); + } +} + +template +inline void scatter_bytes(GmShape &dst, InTile &src, OffTile &offset) { + using DType = typename InTile::DType; + constexpr int SrcN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = SrcN < OffN ? SrcN : OffN; + auto *base = dst.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + *(reinterpret_cast(reinterpret_cast(base) + byte_off)) = + src.data()[i]; + } +} +} // namespace supernpubench_v057_test_fallback + +template +void TXOR(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] ^ src1.data()[i]); +} + +template +void TSHLS(tile_shape &dst, tile_shape &src, unsigned shift) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src.data()[i] << shift); +} + +template +void TSHRS(tile_shape &dst, tile_shape &src, unsigned shift) { + using Unsigned = decltype(static_cast(src.data()[0])); + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast( + static_cast(src.data()[i]) >> shift); +} + +template +void TSEL(tile_shape &dst, tile_shape_index &cond, tile_shape &src) { + constexpr int DstN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int CondN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = DstN < CondN ? DstN : CondN; + for (int i = 0; i < N; ++i) + if (cond.data()[i]) + dst.data()[i] = src.data()[i]; +} + +template +void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { + supernpubench_v057_test_fallback::gather_bytes(dst, src, offset); +} + +template +void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { + supernpubench_v057_test_fallback::scatter_bytes(dst, src, offset); +} + +template +void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = src0.data()[i] > src1.data()[i] ? src0.data()[i] : src1.data()[i]; +} + +template +void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] * s); +} + +template +void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int row = 0; row < tile_shape_in::Rows && row < tile_shape_out::Rows; ++row) { + DType max_value{}; + bool init = false; + for (int col = 0; col < tile_shape_in::Cols; ++col) { + auto value = src.data()[row * tile_shape_in::Cols + col]; + if (!init || value > max_value) { + max_value = static_cast(value); + init = true; + } + } + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = max_value; + } +} + +template +void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] - src1.data()[i]); +} + +template +void TEXP_TEPL(tile_shape &dst, tile_shape &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] * src1.data()[i]); +} + +template +void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int row = 0; row < tile_shape_in::Rows && row < tile_shape_out::Rows; ++row) { + DType sum{}; + for (int col = 0; col < tile_shape_in::Cols; ++col) + sum = static_cast(sum + src.data()[row * tile_shape_in::Cols + col]); + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = sum; + } +} + +template +void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] + src1.data()[i]); +} + +template +void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { + supernpubench_v057_test_fallback::fill_tile(dst, s); +} + +template +void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { + for (int row = 0; row < tile_shape_out::Rows; ++row) { + auto value = src.data()[row < tile_shape_in::Rows ? row * tile_shape_in::Cols : 0]; + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = + static_cast(value); + } +} + +template +void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int col = 0; col < tile_shape_in::Cols && col < tile_shape_out::Cols; ++col) { + DType max_value{}; + bool init = false; + for (int row = 0; row < tile_shape_in::Rows; ++row) { + auto value = src.data()[row * tile_shape_in::Cols + col]; + if (!init || value > max_value) { + max_value = static_cast(value); + init = true; + } + } + for (int row = 0; row < tile_shape_out::Rows; ++row) + dst.data()[row * tile_shape_out::Cols + col] = max_value; + } +} + +template +void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int col = 0; col < tile_shape_in::Cols && col < tile_shape_out::Cols; ++col) { + DType sum{}; + for (int row = 0; row < tile_shape_in::Rows; ++row) + sum = static_cast(sum + src.data()[row * tile_shape_in::Cols + col]); + for (int row = 0; row < tile_shape_out::Rows; ++row) + dst.data()[row * tile_shape_out::Cols + col] = sum; + } +} + +template +void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, + tile_shape_in &src1) { + constexpr int OutN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int InN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] - src1.data()[i]); +} + +template +void TCOLEXPANDMUL_TEPL(tile_shape_out &dst, tile_shape_out &src0, + tile_shape_in &src1) { + constexpr int OutN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int InN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] * src1.data()[i]); +} + +#else + template void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { asm volatile( - "BSTART.TMA 4, %c[DataType]\n" + "BSTART.MGATHER %c[DataType]\n" "B.DIM zero, %c[VCOL], ->lb0\n" "B.DIM zero, %c[VROW], ->lb1\n" - "B.IOT [%[s1]], last, ->%[d0]<%c[TileSize]>\n" + "B.IOT %[s1], last, ->%q[d0]<%c[TileSize]>\n" "B.IOR [%[s0]], []\n" - : [d0]"=Tr"(dst.data()) + : [d0]"=r"(dst.data()) : [s0]"r"(src.data()), - [s1]"Tr"(offset.data()), + [s1]"r"(offset.data()), [DataType]"i"(type_traits::TypeCode), [TileSize]"i"(tile_type_traits::TilesizeCode), [VCOL]"i"(tile_shape_offset::ValidCol), [VROW]"i"(tile_shape_offset::ValidRow) @@ -23,14 +258,14 @@ void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { template void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { asm volatile( - "BSTART.TMA 5, %c[SrcType]\n" + "BSTART.MSCATTER %c[SrcType]\n" "B.DIM zero, %c[VCOL], ->lb0\n" "B.DIM zero, %c[VROW], ->lb1\n" - "B.IOT [%[s0], %[s1]], last\n" + "B.IOT %[s0], %[s1], last\n" "B.IOR [%[d0]], []\n" : - : [d0]"r"(dst.data()), [s0]"Tr"(src.data()), - [s1]"Tr"(offset.data()), + : [d0]"r"(dst.data()), [s0]"r"(src.data()), + [s1]"r"(offset.data()), [SrcType]"i"(type_traits::TypeCode), [VCOL]"i"(tile_shape_offset::ValidCol), [VROW]"i"(tile_shape_offset::ValidRow) ); @@ -39,20 +274,20 @@ void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { template void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 11, %c1\n" + "BSTART.TMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -60,20 +295,20 @@ void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) { asm volatile( - "BSTART.TEPL 0b0100010, %c1\n" + "BSTART.TMULS %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "B.IOR [%7],[]\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), + "r"(src0.data()), "i"(tile_type_traits::TilesizeCode), "r"(s) ); @@ -82,19 +317,19 @@ void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) template void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000001, %c1\n" + "BSTART.TROWMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -102,20 +337,20 @@ void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 1, %c1\n" + "BSTART.TSUB %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -123,19 +358,19 @@ void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TEXP_TEPL(tile_shape &dst, tile_shape &src) { asm volatile( - "BSTART.TEPL 18, %c1\n" + "BSTART.TEXP %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -143,20 +378,20 @@ void TEXP_TEPL(tile_shape &dst, tile_shape &src) { template void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 2, %c1\n" + "BSTART.TMUL %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -164,19 +399,19 @@ void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000000, %c1\n" + "BSTART.TROWSUM %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -184,20 +419,20 @@ void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 0, %c1\n" + "BSTART.TADD %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -205,19 +440,19 @@ void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { asm volatile( - "BSTART.TEPL 20, %c1\n" + "BSTART.TRECIP %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -225,19 +460,19 @@ void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { template void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 27, %c1\n" + "BSTART.TCVT %c1\n" "B.DATR %c2, RNONE\n" "C.B.DIMI %c3, ->LB0\n" "C.B.DIMI %c4, ->LB1\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -245,15 +480,15 @@ void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { asm volatile( - "BSTART.TEPL 0b0111011, %c1\n" + "BSTART.TEXPANDS %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [], last, ->%0<%c5>\n" + "B.IOT last, ->%q0<%c5>\n" "B.IOR [%6],[]\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), @@ -266,18 +501,18 @@ void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { template void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000100, %c1\n" + "BSTART.TROWEXPAND %c1\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -285,19 +520,19 @@ void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1010001, %c1\n" + "BSTART.TCOLMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -305,19 +540,19 @@ void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1010000, %c1\n" + "BSTART.TCOLSUM %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -325,20 +560,20 @@ void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in &src1) { asm volatile( - "BSTART.TEPL 0b1010110, %c1\n" + "BSTART.TCOLEXPANDSUB %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_out::ValidCol), "i"(tile_shape_out::ValidRow), "i"(tile_shape_out::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -346,22 +581,24 @@ void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in template void TCOLEXPANDMUL_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in &src1) { asm volatile( - "BSTART.TEPL 0b1010111, %c1\n" + "BSTART.TCOLEXPANDMUL %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_out::ValidCol), "i"(tile_shape_out::ValidRow), "i"(tile_shape_out::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } #endif + +#endif diff --git a/benchmark/one-level-arch/test/kernel/README.md b/benchmark/one-level-arch/test/kernel/README.md index 374bf5d..9660bd8 100644 --- a/benchmark/one-level-arch/test/kernel/README.md +++ b/benchmark/one-level-arch/test/kernel/README.md @@ -109,9 +109,8 @@ make clean_all # clean all ### Sort — `sort/topk`. -## Known Issues -- `fa_2d_unroll` `X=1,Y=1` / `X=2,Y=1` → `LinxV5 CallingConv Fail!` (Issue #6). -- `control` needs `-s core.singleTierMode=true` on gfsim; `.data` from `gen_data.py`. +## Runtime note +- `control` needs `-s core.singleTierMode=true` on gfsim; `.data` comes from `gen_data.py`. ## Adding a Test 1. Create `test/kernel//` with `src/`, `Makefile`, `compile.all`. diff --git a/benchmark/one-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py b/benchmark/one-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py index 139ef7e..5035af9 100644 --- a/benchmark/one-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py +++ b/benchmark/one-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py @@ -10,18 +10,18 @@ import torch import torch.nn as nn import numpy as np -import signal +import signal from concurrent.futures import ThreadPoolExecutor, as_completed #1 read elf list -#2 分析elf name: matmul MNK +#2 分析elf name: matmul MNK #3 产生随机src0, src1 -> golden_dst 放到 kernel/golden/kernel_matmul_MASK_M128_N1024_K128_tM32_tK32_tN32/src0.bin, src1.bin golden.bin #4 跑gfrun 读取src0, src1 产生dst -> /remote/lms01/c00622284/janus/JanusCoreBench/compare/kernel_matmul_MASK_M128_N1024_K128_tM32_tK32_tN32/res.bin #5 跑res_compare.py #6 输出一个报告. MAX_WORKERS = 20 #parallel thread num depend on your machine -gfrun = "/remote/lms01/l00948608/project/jcore_benchmark/BlockISA/BlockISA/bin/gfrun" +gfrun = os.environ.get("GFRUN") gfrun_args = " -t 1 -f " cmp_root = os.path.abspath(os.path.dirname(__file__)+"/../../../compare") @@ -72,7 +72,9 @@ def gen_input_and_golden(elf_name, path): def run_qemu(elf): print('Start to run gfrun----------') - try: + if args.plat != "cpu" and not gfrun: + raise RuntimeError("set GFRUN to the functional-model executable") + try: if os.path.exists(elf): if args.plat == "cpu": print('plat == "cpu"') @@ -88,12 +90,12 @@ def run_qemu(elf): else: print('elf not exist') return (elf, "not exist", "") - + except subprocess.TimeoutExpired: os.killpg(os.getpgid(proc.pid), signal.SIGKILL) proc.communicate() # 确保进程资源被正确清理 return (elf, "timeout", "Timeout expired") - + except Exception as e: os.killpg(os.getpgid(proc.pid), signal.SIGKILL) proc.communicate() # 确保进程资源被正确清理 @@ -103,7 +105,7 @@ def result_compare(cmp_data, golden_data): print("cmp data: ",cmp_data) if "accelerator_fusion_fa1" in cmp_data: res = np.fromfile(cmp_data, dtype=DType_np) - res_ref = np.fromfile(golden_data, dtype=DType_np) + res_ref = np.fromfile(golden_data, dtype=DType_np) else: res = np.fromfile(cmp_data, dtype=DType_np) res_ref = np.fromfile(golden_data, dtype=DType_np) @@ -179,4 +181,3 @@ def log_result(elf, status, chk_status, loss): f.write(f"{elf}\n") - diff --git a/benchmark/one-level-arch/test/kernel/control/compile.all b/benchmark/one-level-arch/test/kernel/control/compile.all index 6ec69f0..e195dd5 100755 --- a/benchmark/one-level-arch/test/kernel/control/compile.all +++ b/benchmark/one-level-arch/test/kernel/control/compile.all @@ -1,4 +1,6 @@ -#! /bin/bash +#!/usr/bin/env bash +set -euo pipefail + for debug in on off; do if [[ "$debug" == "on" ]]; then debug_define="" diff --git a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh index 2668212..38aa95a 100755 --- a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh +++ b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh @@ -1,5 +1,6 @@ -#!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms60/c00622284/janus/linxisa_compiler_v0.55/linx_blockisa_llvm_musl/bin}" +#!/usr/bin/env bash +set -euo pipefail +: "${COMPILER_DIR:?set COMPILER_DIR to compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -26,11 +27,11 @@ _binary_${name}_data_end: .equ _binary_${name}_data_size, .-_binary_${name}_data_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "inserted_slot" build_one "lookup_keys" build_one "lookup_values" -echo "Done building data object files" \ No newline at end of file +echo "Done building data object files" diff --git a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp index 7203abd..7dc3a59 100644 --- a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp +++ b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp @@ -1,6 +1,7 @@ #include #include "benchmark.h" #include "fileop.h" +#include "template_asm.h" #include "linx_print.h" #include "control/hashtable_lookup_simd.hpp" diff --git a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md index de90737..91c486f 100644 --- a/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md +++ b/benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md @@ -83,7 +83,7 @@ COMPILER_DIR= bash data_obj/build_data_obj.sh data_obj ../../ ```bash # 对每个 *.data 文件执行： -$COMPILER_DIR/clang++ -target linx64v5 -c *.s -o output/.../*.o +$COMPILER_DIR/clang++ -target linx64-linx-none-elf -c *.s -o output/.../*.o ``` 生成的 `.o` 文件： @@ -101,7 +101,7 @@ output/kernel/control/hashtable_lookup_simd/data_obj/lookup_values.o # 期望 # 实际执行（make 自动推导）： cd {WORKSPACE}/JanusCoreBench/test/kernel/control $COMPILER_DIR/clang++ \ - -c -mlxbc -fenable-matrix -O2 \ + -c -target linx64-linx-none-elf -fenable-matrix -O2 \ -std=c++20 \ -I{WORKSPACE}/JanusCoreBench/include \ -I{WORKSPACE}/JanusCoreBench/test/common \ @@ -156,7 +156,7 @@ cd {WORKSPACE}/LinxBlockModel 编译器路径通过环境变量 `COMPILER_DIR` 传入，不在 Makefile 或 compile.all 中写死： ```bash -export COMPILER_DIR=/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.18/bin +export COMPILER_DIR=${WORKSPACE}/compiler/llvm/build-linxisa-clang/bin cd {WORKSPACE}/JanusCoreBench/test/kernel/control ./compile.all ``` diff --git a/benchmark/one-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh b/benchmark/one-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh index ec4819c..07076a8 100755 --- a/benchmark/one-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh +++ b/benchmark/one-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh @@ -1,5 +1,10 @@ #!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.16/bin}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-superproject-working-tree 2>/dev/null || true)" +if [[ -z "$SUPERPROJECT_ROOT" ]]; then + SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)" +fi +COMPILER_DIR="${COMPILER_DIR:-${SUPERPROJECT_ROOT}/compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -28,7 +33,7 @@ _binary_${sym_name}_end: .equ _binary_${sym_name}_size, .-_binary_${sym_name}_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "buckets.bin" diff --git a/benchmark/one-level-arch/test/kernel/reduction/README.md b/benchmark/one-level-arch/test/kernel/reduction/README.md index 6292891..4ed2257 100644 --- a/benchmark/one-level-arch/test/kernel/reduction/README.md +++ b/benchmark/one-level-arch/test/kernel/reduction/README.md @@ -7,7 +7,7 @@ This directory contains benchmark implementations for various reduction operatio The reduction operators are categorized into three main types based on their reduction direction and optimization strategy: - **Column Reduction (col)**: Reduces along the column dimension (M-axis) -- **Row Reduction (row)**: Reduces along the row dimension (N-axis) +- **Row Reduction (row)**: Reduces along the row dimension (N-axis) - **3D Column Reduction (3dcol)**: Optimized for 3D tensor column reduction with unaligned dimensions ## Directory Structure @@ -158,7 +158,7 @@ All operators use 8-way unrolled tree reduction to maximize instruction-level pa ```cpp // Level 1: 8 pairwise operations sum_01 = a + b; sum_23 = c + d; sum_45 = e + f; sum_67 = g + h; -// Level 2: 4 pairwise operations +// Level 2: 4 pairwise operations sum_0123 = sum_01 + sum_23; sum_4567 = sum_45 + sum_67; // Level 3: Final reduction sum_tmp = sum_0123 + sum_4567; @@ -170,7 +170,7 @@ Uses the PTO (Parallel Tile Operations) framework: - `global_tensor`: Defines global memory layout - `Tile`: Defines tile shape and memory location (Vec/Scalar) - `global_iterator`: Iterates over tiles in global memory -- `TCOPYIN`/`TCOPYOUT`: DMA transfers between global and tile memory +- `TLOAD`/`TSTORE`: DMA transfers between global and tile memory ### 3. Corner Case Handling @@ -203,7 +203,7 @@ Uses built-in vector operations: // For reducesum_col (8192×1024 input) #define DType int32_t #define gIMs 8192 // Global M dimension -#define gINs 1024 // Global N dimension +#define gINs 1024 // Global N dimension #define tMs 32 // Tile M dimension #define tNs 128 // Tile N dimension ``` @@ -242,7 +242,7 @@ dtype output[381 * 1 * 8]; for(int i = 0; i < 381; i++) { reducesum_colsum_rand( - &input[i * 120 * 8], + &input[i * 120 * 8], &output[i * 8] ); } diff --git a/benchmark/one-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh b/benchmark/one-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh index 8128f1e..046b032 100755 --- a/benchmark/one-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh +++ b/benchmark/one-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh @@ -1,5 +1,10 @@ #!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.16/bin}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-superproject-working-tree 2>/dev/null || true)" +if [[ -z "$SUPERPROJECT_ROOT" ]]; then + SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)" +fi +COMPILER_DIR="${COMPILER_DIR:-${SUPERPROJECT_ROOT}/compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -25,10 +30,10 @@ _binary_${name}_data_end: .equ _binary_${name}_data_size, .-_binary_${name}_data_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "input_131072" build_one "top_2048_out" -echo "Done building data object files" \ No newline at end of file +echo "Done building data object files" diff --git a/benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp b/benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp index 1f579fd..9e36c15 100644 --- a/benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp +++ b/benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp @@ -53,7 +53,7 @@ int main() { using HistGT = GlobalTensor, Stride<1,1,1,16,1>>; uint32_t histResult[256]; HistGT histGlobal(histResult); - TCOPYOUT(histGlobal, high8HistTile); + TSTORE(histGlobal, high8HistTile); uint32_t global_high8_hist[256] = {0}; for (int b = 0; b < 256; b++) { @@ -92,7 +92,7 @@ int main() { uint32_t low8HistResult[256]; HistGT low8HistGlobal(low8HistResult); - TCOPYOUT(low8HistGlobal, low8HistTile); + TSTORE(low8HistGlobal, low8HistTile); uint32_t global_low8_hist_kth[256] = {0}; for (int b = 0; b < 256; b++) { diff --git a/benchmark/two-level-arch/compile_all.sh b/benchmark/two-level-arch/compile_all.sh index 92f5267..31d4f7f 100755 --- a/benchmark/two-level-arch/compile_all.sh +++ b/benchmark/two-level-arch/compile_all.sh @@ -1,7 +1,7 @@ -#!/bin/bash -# LinxISA compilation script for all kernel operators +#!/usr/bin/env bash +set -uo pipefail -# Don't use set -e as some operators may fail to compile +# LinxISA compilation script for all kernel operators SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" : "${COMPILER_DIR:?Set COMPILER_DIR to the in-repo Linx compiler bin directory}" @@ -38,6 +38,7 @@ compile_operator() { echo "✓ $operator_name compilation completed" else echo "✗ $operator_name compilation failed" + return 1 fi else echo "Warning: No compile.all found in $operator_path" @@ -46,20 +47,21 @@ compile_operator() { } # Compile all operators -compile_operator "$REPO_ROOT/test/tileop_api" "tileop_api" -compile_operator "$REPO_ROOT/test/kernel/matmul" "matmul" -compile_operator "$REPO_ROOT/test/kernel/broadcast" "broadcast" -compile_operator "$REPO_ROOT/test/kernel/concat" "concat" -compile_operator "$REPO_ROOT/test/kernel/gather" "gather" -compile_operator "$REPO_ROOT/test/kernel/transpose" "transpose" -compile_operator "$REPO_ROOT/test/kernel/element_wise/gelu" "gelu" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_col" "reducemax_col" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_row" "reducemax_row" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_col" "reducesum_col" -compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_row" "reducesum_row" -compile_operator "$REPO_ROOT/test/kernel/control" "control" -compile_operator "$REPO_ROOT/test/kernel/fa" "fa" -compile_operator "$REPO_ROOT/test/kernel/sort" "sort" +failures=() +compile_operator "$REPO_ROOT/test/tileop_api" "tileop_api" || failures+=("tileop_api") +compile_operator "$REPO_ROOT/test/kernel/matmul" "matmul" || failures+=("matmul") +compile_operator "$REPO_ROOT/test/kernel/broadcast" "broadcast" || failures+=("broadcast") +compile_operator "$REPO_ROOT/test/kernel/concat" "concat" || failures+=("concat") +compile_operator "$REPO_ROOT/test/kernel/gather" "gather" || failures+=("gather") +compile_operator "$REPO_ROOT/test/kernel/transpose" "transpose" || failures+=("transpose") +compile_operator "$REPO_ROOT/test/kernel/element_wise/gelu" "gelu" || failures+=("gelu") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_col" "reducemax_col" || failures+=("reducemax_col") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducemax_row" "reducemax_row" || failures+=("reducemax_row") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_col" "reducesum_col" || failures+=("reducesum_col") +compile_operator "$REPO_ROOT/test/kernel/reduction/reducesum_row" "reducesum_row" || failures+=("reducesum_row") +compile_operator "$REPO_ROOT/test/kernel/control" "control" || failures+=("control") +compile_operator "$REPO_ROOT/test/kernel/fa" "fa" || failures+=("fa") +compile_operator "$REPO_ROOT/test/kernel/sort" "sort" || failures+=("sort") echo "" echo "==========================================" @@ -69,3 +71,8 @@ echo "" echo "Generated ELF files:" find "$REPO_ROOT/output" -name "*.elf" -type f | wc -l echo "ELF files are located in: $REPO_ROOT/output/" + +if ((${#failures[@]})); then + echo "Failed operators: ${failures[*]}" >&2 + exit 1 +fi diff --git a/benchmark/two-level-arch/include/common/pto_tile.hpp b/benchmark/two-level-arch/include/common/pto_tile.hpp index cc608fa..a65886c 100644 --- a/benchmark/two-level-arch/include/common/pto_tile.hpp +++ b/benchmark/two-level-arch/include/common/pto_tile.hpp @@ -416,8 +416,11 @@ struct Tile { static_assert(SFractalSize_ == 512 || SFractalSize_ == 1024, "SFractalSize_ illegal"); -#if defined(__linx) && defined(SUPERNPUBENCH_LINX_TILE_SIZE) - using TileDType = DType tile_size(Rows *Cols / (sizeof(DType) * 8 / type_traits::bits)); +#if defined(__linx) + // Tile element interpretation is carried by PTO descriptors, not the C++ + // value type. Keep every architectural tile on the backend's canonical + // opaque v1024i32 carrier. + using TileDType = int32_t __attribute__((vector_size(4096))); #else using TileDType = DType[Rows * Cols]; #endif diff --git a/benchmark/two-level-arch/include/common/tileop_api.hpp b/benchmark/two-level-arch/include/common/tileop_api.hpp index aacf95f..19bbe14 100644 --- a/benchmark/two-level-arch/include/common/tileop_api.hpp +++ b/benchmark/two-level-arch/include/common/tileop_api.hpp @@ -48,15 +48,27 @@ template void TABS(tile_shape &dst, tile_shape &src) } template void TADD(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TADD_V057(dst, src0, src1); +#else TADD_Impl(dst, src0, src1); +#endif } template void TADDS(tile_shape &dst, tile_shape &src, typename tile_shape::DType s) { +#ifdef __linx + TADDS_V057(dst, src, s); +#else TADDS_Impl(dst, src, s); +#endif } template void TAND(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TAND_V057(dst, src0, src1); +#else TAND_Impl(dst, src0, src1); +#endif } template @@ -66,7 +78,11 @@ void TASSEMBLE(tile_shape_out &dst, tile_shape_in0 &src0, tile_shape_in1 &src1, } template void TCAST(tile_shape_out &dst, tile_shape_in &src) { - TCAST_Impl(dst, src); +#ifdef __linx + TCVT_V057(dst, src); +#else + TCVT_Impl(dst, src); +#endif } template void TCI(tile_shape &dst, T s) { @@ -74,23 +90,35 @@ void TCI(tile_shape &dst, T s) { } template void TCMP(tile_shape_out &dst, tile_shape_in &src0, tile_shape_in &src1, CmpMode cmpMode) { +#ifdef __linx + TCMP_V057(dst, src0, src1, cmpMode); +#else TCMP_Impl(dst, src0, src1, cmpMode); -} -template -void TCOPY(tile_shape &dst, tile_shape &src) { - TCOPY_Impl(dst, src); +#endif } template void TLOAD(tile_shape &dst, gm_shape &src) { +#ifdef __linx + TLOAD_V057(dst, src); +#else TLOAD_Impl(dst, src); +#endif } template void TSTORE(gm_shape &dst, tile_shape &src) { +#ifdef __linx + TSTORE_V057(dst, src); +#else TSTORE_Impl(dst, src); +#endif } template void TCVT(tile_shape_out &dst, tile_shape_in &src) { +#ifdef __linx + TCVT_V057(dst, src); +#else TCVT_Impl(dst, src); +#endif } template void TDIV(tile_shape &dst, tile_shape &src0, tile_shape &src1) { @@ -113,7 +141,11 @@ void TEXPANDROW(tile_shape_out &dst, tile_shape_in &src) { } template void TEXPANDSCALAR(tile_shape &dst, typename tile_shape::DType s) { +#ifdef __linx + TEXPANDS_V057(dst, s); +#else TEXPANDSCALAR_Impl(dst, s); +#endif } template void TEXTRACT(tile_shape_out &dst, tile_shape_in &src, uint16_t offset_i, @@ -148,15 +180,27 @@ void TMINS(tile_shape &dst, tile_shape &src, typename tile_shape::DType s) { } template void TMUL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TMUL_V057(dst, src0, src1); +#else TMUL_Impl(dst, src0, src1); +#endif } template void TMULS(tile_shape &dst, tile_shape &src, typename tile_shape::DType s) { +#ifdef __linx + TMULS_V057(dst, src, s); +#else TMULS_Impl(dst, src, s); +#endif } template void TOR(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TOR_V057(dst, src0, src1); +#else TOR_Impl(dst, src0, src1); +#endif } template void TPAD(tile_shape_out &dst, const tile_shape_in &src, T pad_value, @@ -169,7 +213,11 @@ void TRECIP(tile_shape &dst, tile_shape &src) { } template void TREM(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TREM_V057(dst, src0, src1); +#else TREM_Impl(dst, src0, src1); +#endif } template void TRESHAPE(tile_shape_out &tile_out, tile_shape_in &tile_in) { @@ -204,7 +252,11 @@ void TSCATTER(tile_shape_dst &dst, tile_shape_src &src, template void TSELECT(tile_shape &dst, tile_shape_index &cond, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TSELECT_V057(dst, cond, src0, src1); +#else TSELECT_Impl(dst, cond, src0, src1); +#endif } template void TSQRT(tile_shape &dst, tile_shape &src) { @@ -212,7 +264,11 @@ void TSQRT(tile_shape &dst, tile_shape &src) { } template void TSUB(tile_shape &dst, tile_shape &src0, tile_shape &src1) { +#ifdef __linx + TSUB_V057(dst, src0, src1); +#else TSUB_Impl(dst, src0, src1); +#endif } template void TSUBS(tile_shape &dst, tile_shape &src, typename tile_shape::DType s) { @@ -223,4 +279,4 @@ void TTRANS(tile_shape_out &dst, tile_shape_in &src) { TTRANS_Impl(dst, src); } -#endif \ No newline at end of file +#endif diff --git a/benchmark/two-level-arch/include/common/tileop_api_impl.hpp b/benchmark/two-level-arch/include/common/tileop_api_impl.hpp index fe8ad19..ee71fc5 100644 --- a/benchmark/two-level-arch/include/common/tileop_api_impl.hpp +++ b/benchmark/two-level-arch/include/common/tileop_api_impl.hpp @@ -10,7 +10,6 @@ #include "jcore/TAnd.hpp" #include "jcore/TCmp.hpp" #include "jcore/TCI.hpp" -#include "jcore/TCopy.hpp" #include "jcore/TLoad.hpp" #include "jcore/TStore.hpp" #include "jcore/TCvt.hpp" @@ -49,7 +48,6 @@ #include "aarch64/TCast.hpp" #include "aarch64/TCI.hpp" #include "aarch64/TCmp.hpp" -#include "aarch64/TCopy.hpp" #include "aarch64/TLoad.hpp" #include "aarch64/TStore.hpp" #include "aarch64/TDiv.hpp" @@ -92,7 +90,6 @@ #include "cpu_sim/TCast.hpp" #include "cpu_sim/TCI.hpp" #include "cpu_sim/TCmp.hpp" -#include "cpu_sim/TCopy.hpp" #include "cpu_sim/TLoad.hpp" #include "cpu_sim/TStore.hpp" #include "cpu_sim/TCvt.hpp" diff --git a/benchmark/two-level-arch/include/jcore/TCopy.hpp b/benchmark/two-level-arch/include/jcore/TCopy.hpp deleted file mode 100644 index 9e179d8..0000000 --- a/benchmark/two-level-arch/include/jcore/TCopy.hpp +++ /dev/null @@ -1,92 +0,0 @@ -#ifndef TCOPY_HPP -#define TCOPY_HPP - -#include "common/pto_tile.hpp" -#include "jcore/constants.hpp" -using namespace pto; - -#ifdef __linx -template -void TCOPY_Impl(tile_shape &dst, tile_shape &src) { - size_t rows = src.GetValidRow(); - size_t cols = src.GetValidCol(); - static_assert(tile_shape::Loc != Location::Acc, - "Unsupport ACC to be input or output here"); - static_assert(!tile_shape::isBoxedLayout, "TCOPY not support Boxed Layout!"); - - for (size_t row = 0; row < rows; ++row) { - for (size_t col = 0; col < cols; ++col) { - size_t index = tile_shape::isRowMajor - ? row * tile_shape::RowStride + col - : col * tile_shape::ColStride + row; - dst.data()[index] = src.data()[index]; - } - } -} -#else -template -void __vec__ TCopy_Vec_RowMajor( - typename tile_shape::TileDType __out__ dst, - const typename tile_shape::TileDType __in__ src) { - size_t i = blkv_get_index_x(); - size_t j = blkv_get_index_y(); - - size_t index = j * tile_shape::RowStride + i; - blkv_get_tile_ptr(dst)[index] = blkv_get_tile_ptr(src)[index]; -} - -template -void __vec__ TCopy_Vec_ColMajor( - typename tile_shape::TileDType __out__ dst, - const typename tile_shape::TileDType __in__ src) { - size_t i = blkv_get_index_x(); - size_t j = blkv_get_index_y(); - - size_t index = j * tile_shape::ColStride + i; - blkv_get_tile_ptr(dst)[index] = blkv_get_tile_ptr(src)[index]; -} - -//fractal blkc impl -template -void __vec__ TCopy_NzLayout_Impl( - typename tile_shape::TileDType __out__ dst, - const typename tile_shape::TileDType __in__ src) { - size_t i = blkv_get_index_x(); - size_t j = blkv_get_index_y(); - static constexpr int block_cols = tile_shape::Cols / tile_shape::InnerCols; - - __vbuf__ typename tile_shape::DType *d_ptr = blkv_get_tile_ptr(dst); - __vbuf__ typename tile_shape::DType *s_ptr = blkv_get_tile_ptr(src); - - #pragma clang loop unroll(full) - for (size_t k = 0; k < block_cols; ++k) { - size_t index = k * tile_shape::Rows * tile_shape::InnerCols + j * LaneNum + i; - d_ptr[index] = s_ptr[index]; - } -} - -template -void TCOPY_Impl(tile_shape &dst, tile_shape &src) { - static_assert(tile_shape::Loc != Location::Acc, "Unsupport ACC to be input or output here"); - size_t row = tile_shape::Rows; - size_t col = tile_shape::Cols; - size_t Y = row / (LaneNum / tile_shape::InnerCols); - - if constexpr (!tile_shape::isBoxedLayout) { - if constexpr (tile_shape::isRowMajor) { - TCopy_Vec_RowMajor<<>>(dst.data(), src.data()); - } else { - TCopy_Vec_ColMajor<<>>(dst.data(), src.data()); - } - } else { - if constexpr (is_Nz_layout::value) { - TCopy_NzLayout_Impl<<>>(dst.data(), src.data()); - } else { - static_assert(is_Nz_layout::value, - "Storage layout type not supported"); - } - } -} -#endif - -#endif diff --git a/benchmark/two-level-arch/include/jcore/template_asm.hpp b/benchmark/two-level-arch/include/jcore/template_asm.hpp index e0a3b4a..ed0a4d3 100644 --- a/benchmark/two-level-arch/include/jcore/template_asm.hpp +++ b/benchmark/two-level-arch/include/jcore/template_asm.hpp @@ -3,18 +3,357 @@ #include "common/pto_tile.hpp" +// This header defines the historical global PTO wrapper surface while its +// type traits live in namespace pto. Keep it self-contained for compile-only +// conformance tests instead of relying on include-order namespace imports. +using namespace pto; + +#if !defined(__linx) + +namespace supernpubench_v057_fallback { +template +constexpr int tile_numel_v = T::Rows * T::Cols; + +template +inline void copy_tile(OutTile &dst, const InTile &src) { + constexpr int OutN = tile_numel_v; + constexpr int InN = tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src.data()[i]); +} + +template +inline void fill_tile(OutTile &dst, typename OutTile::DType value) { + for (int i = 0; i < tile_numel_v; ++i) + dst.data()[i] = value; +} + +template +inline void gather_bytes(OutTile &dst, const auto &src, const InTile &offset) { + using DType = typename OutTile::DType; + constexpr int OutN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = OutN < OffN ? OutN : OffN; + const auto *base = src.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + dst.data()[i] = static_cast( + *(reinterpret_cast( + reinterpret_cast(base) + byte_off))); + } +} + +template +inline void scatter_bytes(auto &dst, const InTile &src, const OffTile &offset) { + using DType = typename InTile::DType; + constexpr int SrcN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = SrcN < OffN ? SrcN : OffN; + auto *base = dst.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + *(reinterpret_cast(reinterpret_cast(base) + byte_off)) = + src.data()[i]; + } +} + +template +inline void load_tile(Tile &dst, Gm &src) { + const int rows = dst.GetValidRow(); + const int cols = dst.GetValidCol(); + for (int row = 0; row < rows; ++row) { + for (int col = 0; col < cols; ++col) { + const int gm_index = + Gm::isRowMajor ? row * Gm::RowStride + col + : col * Gm::ColStride + row; + const int tile_index = + Tile::isRowMajor ? row * Tile::RowStride + col + : col * Tile::ColStride + row; + dst.data()[tile_index] = src.data()[gm_index]; + } + } +} + +template +inline void store_tile(Gm &dst, Tile &src) { + const int rows = src.GetValidRow(); + const int cols = src.GetValidCol(); + for (int row = 0; row < rows; ++row) { + for (int col = 0; col < cols; ++col) { + const int gm_index = + Gm::isRowMajor ? row * Gm::RowStride + col + : col * Gm::ColStride + row; + const int tile_index = + Tile::isRowMajor ? row * Tile::RowStride + col + : col * Tile::ColStride + row; + dst.data()[gm_index] = src.data()[tile_index]; + } + } +} +} // namespace supernpubench_v057_fallback + +enum class TmaPadValue : int { + Zero = 0, + Max = 1, + Min = 2, + Null = 3, +}; + +template +void ACCSCALE_T(tile_shape_out &dst, tile_shape_in &src, + typename tile_shape_in::DType s) { + (void)s; + supernpubench_v057_fallback::copy_tile(dst, src); +} + +template +void ACCSCALE_NZ2DN(tile_shape_out &dst, tile_shape_in &src, + typename tile_shape_in::DType s) { + ACCSCALE_T(dst, src, s); +} + +template +void ACCCVT_RMAX_SCALE_NZ2DN(tile_shape_max &row_max, tile_shape_out &dst, + tile_shape_in &src, + typename tile_shape_in::DType s) { + (void)s; + supernpubench_v057_fallback::copy_tile(dst, src); + supernpubench_v057_fallback::fill_tile(row_max, typename tile_shape_max::DType{}); +} + +template +void TMAX_T(tile_shape_out &dst, tile_shape_in0 &src0, tile_shape_in1 &src1) { + constexpr int N0 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N1 = supernpubench_v057_fallback::tile_numel_v; + constexpr int NO = supernpubench_v057_fallback::tile_numel_v; + constexpr int N = (NO < (N0 < N1 ? N0 : N1)) ? NO : (N0 < N1 ? N0 : N1); + for (int i = 0; i < N; ++i) + dst.data()[i] = src0.data()[i] > src1.data()[i] + ? static_cast(src0.data()[i]) + : static_cast(src1.data()[i]); +} + +template +void TSUB_EXP_EXPAND_T(tile_shape_out &dst, tile_shape_in0 &src0, + tile_shape_in1 &src1) { + constexpr int N0 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N1 = supernpubench_v057_fallback::tile_numel_v; + constexpr int NO = supernpubench_v057_fallback::tile_numel_v; + constexpr int N = (NO < (N0 < N1 ? N0 : N1)) ? NO : (N0 < N1 ? N0 : N1); + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] - src1.data()[i]); +} + +template +void TMUL_ADD_ROWSUM_T(tile_shape_out &dst, tile_shape_in0 &src0, + tile_shape_in1 &src1, tile_shape_in2 &src2) { + constexpr int N0 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N1 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N2 = supernpubench_v057_fallback::tile_numel_v; + constexpr int NO = supernpubench_v057_fallback::tile_numel_v; + constexpr int NI = (N0 < N1 ? N0 : N1) < N2 ? (N0 < N1 ? N0 : N1) : N2; + constexpr int N = NO < NI ? NO : NI; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] * src1.data()[i] + src2.data()[i]); +} + +template +void TADD_MUL_EXPAND_T(tile_shape_out &dst, tile_shape_in0 &src0, + tile_shape_in1 &src1, tile_shape_in2 &src2) { + constexpr int N0 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N1 = supernpubench_v057_fallback::tile_numel_v; + constexpr int N2 = supernpubench_v057_fallback::tile_numel_v; + constexpr int NO = supernpubench_v057_fallback::tile_numel_v; + constexpr int NI = (N0 < N1 ? N0 : N1) < N2 ? (N0 < N1 ? N0 : N1) : N2; + constexpr int N = NO < NI ? NO : NI; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast((src0.data()[i] + src1.data()[i]) * src2.data()[i]); +} + +template +void TCVT_T(tile_shape_out &dst, tile_shape_in &src) { + supernpubench_v057_fallback::copy_tile(dst, src); +} + +#define DEFINE_TMOV_LAYOUT(LAYOUT_NAME) \ +template \ +void TMOV_##LAYOUT_NAME(tile_shape_out &dst, tile_shape_in &src) { \ + supernpubench_v057_fallback::copy_tile(dst, src); \ +} + +DEFINE_TMOV_LAYOUT(ND2NZ) +DEFINE_TMOV_LAYOUT(NZ2ND) +DEFINE_TMOV_LAYOUT(ND2ZN) +DEFINE_TMOV_LAYOUT(DN2ZN) +DEFINE_TMOV_LAYOUT(DN2NZ) +DEFINE_TMOV_LAYOUT(NZ2DN) +DEFINE_TMOV_LAYOUT(NZ2ZN) +DEFINE_TMOV_LAYOUT(ZN2NZ) +DEFINE_TMOV_LAYOUT(NORM) +#undef DEFINE_TMOV_LAYOUT + +template +void TMOV_DN2NZ_DYN(tile_shape_out &dst, tile_shape_in &src) { + supernpubench_v057_fallback::copy_tile(dst, src); +} + +template +void THISTOGRAM(tile_shape_out &dst, tile_shape_in &src, tile_shape_in &Idx, + int ByteId) { + (void)src; + (void)Idx; + (void)ByteId; + supernpubench_v057_fallback::fill_tile(dst, typename tile_shape_out::DType{}); +} + +template +void TLOAD2_ND2NZ(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { + supernpubench_v057_fallback::load_tile(dst0, src); + supernpubench_v057_fallback::load_tile(dst1, src); +} + +template +void TLOAD2_ND2ZN(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { + TLOAD2_ND2NZ(dst1, dst0, src); +} + +template +void TLOAD2_DN2ZN(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { + TLOAD2_ND2NZ(dst1, dst0, src); +} + +template +void TSTORE2_DN2DN(gm_shape &dst, tile_shape &src1, tile_shape &src0) { + supernpubench_v057_fallback::store_tile(dst, src0); + (void)src1; +} + +template +void TLOAD4_ND2NZ(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, + tile_shape &dst0, gm_shape &src) { + supernpubench_v057_fallback::load_tile(dst0, src); + supernpubench_v057_fallback::load_tile(dst1, src); + supernpubench_v057_fallback::load_tile(dst2, src); + supernpubench_v057_fallback::load_tile(dst3, src); +} + +template +void TLOAD4_ND2ZN(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, + tile_shape &dst0, gm_shape &src) { + TLOAD4_ND2NZ(dst3, dst2, dst1, dst0, src); +} + +template +void TLOAD4_DN2ZN(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, + tile_shape &dst0, gm_shape &src) { + TLOAD4_ND2NZ(dst3, dst2, dst1, dst0, src); +} + +template +inline void MGATHER(tile_shape_out &dst, const gm_shape &src, + const tile_shape_offset &offset) { + (void)Pad; + supernpubench_v057_fallback::gather_bytes(dst, src, offset); +} + +template +inline void MSCATTER(gm_shape &dst, const tile_shape_in &src, + const tile_shape_offset &offset) { + supernpubench_v057_fallback::scatter_bytes(dst, src, offset); +} + +template +inline void TPREFETCH(const gm_shape &src) { + (void)sizeof(tile_shape); + (void)src; +} + +template +inline void MGATHER_MASK(tile_shape_out &dst, const gm_shape &src, + const tile_shape_offset &offset, + const tile_shape_mask &mask) { + (void)Pad; + supernpubench_v057_fallback::gather_bytes(dst, src, offset); + for (int i = 0; i < supernpubench_v057_fallback::tile_numel_v && + i < supernpubench_v057_fallback::tile_numel_v; ++i) + if (!mask.data()[i]) + dst.data()[i] = typename tile_shape_out::DType{}; +} + +template +inline void MGATHER_CAS(tile_shape_out &dst, const gm_shape &src, + const tile_shape_offset &offset, + const tile_shape_expected &expected, + const tile_shape_desired &desired) { + (void)Pad; + using DType = typename tile_shape_out::DType; + auto *base = const_cast(src.data()); + constexpr int DstN = supernpubench_v057_fallback::tile_numel_v; + constexpr int OffN = + supernpubench_v057_fallback::tile_numel_v; + constexpr int ExpN = + supernpubench_v057_fallback::tile_numel_v; + constexpr int DesN = + supernpubench_v057_fallback::tile_numel_v; + constexpr int N0 = DstN < OffN ? DstN : OffN; + constexpr int N1 = ExpN < DesN ? ExpN : DesN; + constexpr int N = N0 < N1 ? N0 : N1; + for (int i = 0; i < N; ++i) { + const long long byte_off = static_cast(offset.data()[i]); + auto *ptr = + reinterpret_cast(reinterpret_cast(base) + byte_off); + const DType old = *ptr; + if (old == expected.data()[i]) + *ptr = desired.data()[i]; + dst.data()[i] = old; + } +} + +template +inline void MSCATTER_MASK(gm_shape &dst, const tile_shape_in &src, + const tile_shape_offset &offset, + const tile_shape_mask &mask) { + using DType = typename tile_shape_in::DType; + auto *base = dst.data(); + constexpr int SrcN = supernpubench_v057_fallback::tile_numel_v; + constexpr int OffN = supernpubench_v057_fallback::tile_numel_v; + constexpr int MaskN = supernpubench_v057_fallback::tile_numel_v; + constexpr int N = (SrcN < OffN ? SrcN : OffN) < MaskN ? (SrcN < OffN ? SrcN : OffN) : MaskN; + for (int i = 0; i < N; ++i) { + if (!mask.data()[i]) + continue; + long long byte_off = static_cast(offset.data()[i]); + *(reinterpret_cast(reinterpret_cast(base) + byte_off)) = + src.data()[i]; + } +} + +#else + template void ACCSCALE_T(tile_shape_out &dst, tile_shape_in &src, typename tile_shape_in::DType s) { asm volatile( "BSTART.CUBE ACCCVT, %c1\n" "B.DATR NZ2ND.normal, %c1, Null\n" - "B.IOT [], last, ->%0<%c2>\n" + "B.IOT last, ->%q0<%c2>\n" "B.IOR [%3],[]\n" "C.B.DIMI %c4, ->lb0\n" "C.B.DIMI %c5, ->lb1\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "r"(s), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow) @@ -26,13 +365,13 @@ void ACCSCALE_NZ2DN(tile_shape_out &dst, tile_shape_in &src, typename tile_shape asm volatile( "BSTART.CUBE ACCCVT, %c1\n" "B.DATR NZ2DN.normal, %c1, Null\n" - "B.IOT [], last, ->%0<%c2>\n" + "B.IOT last, ->%q0<%c2>\n" "B.IOR [%3],[]\n" "C.B.DIMI %c4, ->lb0\n" "C.B.DIMI %c5, ->lb1\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "r"(s), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow) @@ -44,14 +383,14 @@ void ACCCVT_RMAX_SCALE_NZ2DN(tile_shape_max &row_max, tile_shape_out &dst, tile_ asm volatile( "BSTART.CUBE ACCCVT, %c[__pto_SrcType]\n" "B.DATR NZ2DN.normal, %c[__pto_DstType], Null\n" - "B.IOT [], ->%[__pto_dout]<%c[__pto_DstTileSize]>\n" - "B.IOT [], last, ->%[__pto_rmax]<%c[__pto_RmaxTileSize]>\n" + "B.IOT , ->%q[__pto_dout]<%c[__pto_DstTileSize]>\n" + "B.IOT last, ->%q[__pto_rmax]<%c[__pto_RmaxTileSize]>\n" "B.IOR [%[__pto_scale]],[]\n" "C.B.DIMI %c[__pto_VCOL], ->lb0\n" "C.B.DIMI %c[__pto_VROW], ->lb1\n" "" - : [__pto_dout]"=Tr"(dst.data()), [__pto_rmax]"=Tr"(row_max.data()) + : [__pto_dout]"=r"(dst.data()), [__pto_rmax]"=r"(row_max.data()) : [__pto_SrcType]"i"(type_traits::TypeCode), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_DstTileSize]"i"(tile_type_traits::TilesizeCode), @@ -64,13 +403,13 @@ template %0<%c4>\n" + "B.IOT %1,%2, last, ->%q0<%c4>\n" "C.B.DIMI %c5, ->lb0\n" "C.B.DIMI %c6, ->lb1\n" "" - : "=Tr"(dst.data()) - : "Tr"(src0.data()), "Tr"(src1.data()), \ + : "=r"(dst.data()) + : "r"(src0.data()), "r"(src1.data()), \ "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "i"(tile_shape_in0::ValidCol), "i"(tile_shape_in0::ValidRow) @@ -81,13 +420,13 @@ template %0<%c4>\n" + "B.IOT %1,%2, last, ->%q0<%c4>\n" "C.B.DIMI %c5, ->lb0\n" "C.B.DIMI %c6, ->lb1\n" "" - : "=Tr"(dst.data()) - : "Tr"(src0.data()), "Tr"(src1.data()), \ + : "=r"(dst.data()) + : "r"(src0.data()), "r"(src1.data()), \ "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "i"(tile_shape_in0::ValidCol), "i"(tile_shape_in0::ValidRow) @@ -98,14 +437,14 @@ template %0<%c5>\n" - "B.IOT [%3], 1\n" + "B.IOT %1,%2, last, ->%q0<%c5>\n" + "B.IOT %3, 1\n" "C.B.DIMI %c6, ->lb0\n" "C.B.DIMI %c7, ->lb1\n" "" - : "=Tr"(dst.data()) - : "Tr"(src0.data()), "Tr"(src1.data()), "Tr"(src2.data()), + : "=r"(dst.data()) + : "r"(src0.data()), "r"(src1.data()), "r"(src2.data()), "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "i"(tile_shape_in0::ValidCol), "i"(tile_shape_in0::ValidRow) @@ -116,14 +455,14 @@ template %0<%c5>\n" - "B.IOT [%3], 1\n" + "B.IOT %1,%2, last, ->%q0<%c5>\n" + "B.IOT %3, 1\n" "C.B.DIMI %c6, ->lb0\n" "C.B.DIMI %c7, ->lb1\n" "" - : "=Tr"(dst.data()) - : "Tr"(src0.data()), "Tr"(src1.data()), "Tr"(src2.data()), + : "=r"(dst.data()) + : "r"(src0.data()), "r"(src1.data()), "r"(src2.data()), "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "i"(tile_shape_in0::ValidCol), "i"(tile_shape_in0::ValidRow) @@ -133,15 +472,15 @@ void TADD_MUL_EXPAND_T(tile_shape_out &dst, tile_shape_in0 &src0, tile_shape_in1 template void TCVT_T(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 27, %c1\n" + "BSTART.TCVT %c1\n" "B.DATR %c2, RNone\n" - "B.IOT [%3], last, ->%0<%c4>\n" + "B.IOT %3, last, ->%q0<%c4>\n" "B.DIM zero, %c5, ->lb0\n" "B.DIM zero, %c6, ->lb1\n" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(type_traits::TypeCode), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow) @@ -152,13 +491,13 @@ void TCVT_T(tile_shape_out &dst, tile_shape_in &src) { template \ void TMOV_##LAYOUT_NAME(tile_shape_out &dst, tile_shape_in &src) { \ asm volatile( \ - "BSTART.TMA 2, %c2\n" \ + "BSTART.TMOV %c2\n" \ "B.DATR " #LAYOUT_NAME ".normal, Null\n" \ - "B.IOT [%1], last, ->%0<%c3>\n" \ + "B.IOT %1, last, ->%q0<%c3>\n" \ "B.DIM zero, %c4, ->lb0\n" \ "B.DIM zero, %c5, ->lb1\n" \ - : "=Tr"(dst.data()) \ - : "Tr"(src.data()), \ + : "=r"(dst.data()) \ + : "r"(src.data()), \ "i"(type_traits::TypeCode), \ "i"(tile_type_traits::TilesizeCode), \ "i"(tile_shape_in::ValidCol), \ @@ -179,14 +518,14 @@ DEFINE_TMOV_LAYOUT(NORM) template void TMOV_DN2NZ_DYN(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TMA 2, %c2\n" + "BSTART.TMOV %c2\n" "B.DATR DN2NZ.normal, Null\n" - "B.IOT [%1], last, ->%0<%c3>\n" + "B.IOT %1, last, ->%q0<%c3>\n" "B.DIM %4, 0, ->lb0\n" "B.DIM %5, 0, ->lb1\n" - : "=Tr"(dst.data()) - : "Tr"(src.data()), + : "=r"(dst.data()) + : "r"(src.data()), "i"(type_traits::TypeCode), "i"(tile_type_traits::TilesizeCode), "r"(src.GetValidCol()), @@ -198,21 +537,21 @@ template void THISTOGRAM(tile_shape_out &dst, tile_shape_in &src, tile_shape_in &Idx, int ByteId) { #define THISTOGRAM_ASM(BYTE_NAME) \ asm volatile( \ - "BSTART.TEPL 0b1101000, %c1\n" \ + "BSTART.THISTOGRAM %c1\n" \ "B.DATR %c2," BYTE_NAME ",Null\n" \ "B.DIM zero, %c3, ->LB0\n" \ "B.DIM zero, %c4, ->LB1\n" \ "B.DIM zero, %c4, ->LB2\n" \ - "B.IOT [%5, %6], last, ->%0<%c7>\n" \ + "B.IOT %5, %6, last, ->%q0<%c7>\n" \ "" \ - : "=Tr"(dst.data()) \ + : "=r"(dst.data()) \ : "i"(type_traits::TypeCode), \ "i"(type_traits::TypeCode), \ "i"(tile_shape_in::ValidCol), \ "i"(tile_shape_in::ValidRow), \ "i"(tile_shape_in::Cols), \ - "Tr"(src.data()), \ - "Tr"(Idx.data()), \ + "r"(src.data()), \ + "r"(Idx.data()), \ "i"(tile_type_traits::TilesizeCode)) switch (ByteId) { @@ -241,15 +580,15 @@ void TLOAD2_ND2NZ(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { static_assert(gm_shape::isRowMajor && is_Nz_layout::value, "GM_SHAPE should ND and TILE_SHAPE should be Nz "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR ND2NZ.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[__pto_d1]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -264,15 +603,15 @@ void TLOAD2_ND2ZN(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { static_assert(gm_shape::isRowMajor && is_Zn_layout::value, "GM_SHAPE should ND and TILE_SHAPE should be Zn "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR ND2ZN.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[__pto_d1]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -287,15 +626,15 @@ void TLOAD2_DN2ZN(tile_shape &dst1, tile_shape &dst0, gm_shape &src) { static_assert(!gm_shape::isRowMajor && is_Nz_layout::value, "GM_SHAPE should DN and TILE_SHAPE should be Zn "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR DN2ZN.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[__pto_d1]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -310,15 +649,15 @@ void TSTORE2_DN2DN(gm_shape &dst, tile_shape &src1, tile_shape &src0) { static_assert(!gm_shape::isRowMajor && !tile_shape::isRowMajor, "GM_SHAPE should DN and TILE_SHAPE should be DN"); asm volatile( - "BSTART.TMA TSTORE, %c[__pto_SrcType]\n" + "BSTART.TSTORE %c[__pto_SrcType]\n" "B.DATR NORM.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [%[__pto_s0], %[s1]], last\n" + "B.IOT %[__pto_s0], %[s1], last\n" "B.IOR [%[__pto_d0],%[__pto_GmStride]], []\n" : - : [__pto_d0]"r"(dst.data()), [__pto_s0]"Tr"(src0.data()), [s1]"Tr"(src1.data()), + : [__pto_d0]"r"(dst.data()), [__pto_s0]"r"(src0.data()), [s1]"r"(src1.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), [__pto_VCOL]"i"(tile_shape::ValidRow*2), [__pto_VROW]"i"(tile_shape::ValidCol), [__pto_COL]"i"(tile_shape::Rows*2), @@ -331,17 +670,17 @@ void TLOAD4_ND2NZ(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, tile_sha static_assert(gm_shape::isRowMajor && is_Nz_layout::value, "GM_SHAPE should ND and TILE_SHAPE should be Nz "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR ND2NZ.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[__pto_d1]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[d2]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[d3]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[d2]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[d3]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()),[d2]"=Tr"(dst2.data()),[d3]"=Tr"(dst3.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()),[d2]"=r"(dst2.data()),[d3]"=r"(dst3.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -356,17 +695,17 @@ void TLOAD4_ND2ZN(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, tile_sha static_assert(gm_shape::isRowMajor && is_Zn_layout::value, "GM_SHAPE should ND and TILE_SHAPE should be Nz "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR ND2ZN.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[__pto_d1]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[d2]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[d3]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[d2]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[d3]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()),[d2]"=Tr"(dst2.data()),[d3]"=Tr"(dst3.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()),[d2]"=r"(dst2.data()),[d3]"=r"(dst3.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -381,17 +720,17 @@ void TLOAD4_DN2ZN(tile_shape &dst3, tile_shape &dst2, tile_shape &dst1, tile_sha static_assert(!gm_shape::isRowMajor && is_Zn_layout::value, "GM_SHAPE should DN and TILE_SHAPE should be Zn "); asm volatile( - "BSTART.TMA TLOAD, %c[__pto_SrcType]\n" + "BSTART.TLOAD %c[__pto_SrcType]\n" "B.DATR DN2ZN.normal, %c[__pto_DstType], Null\n" "B.DIM zero, %[__pto_VCOL], ->lb0\n" "B.DIM zero, %[__pto_VROW], ->lb1\n" "B.DIM zero, %[__pto_COL], ->lb2\n" - "B.IOT [], ->%[__pto_d0]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[__pto_d1]<%c[__pto_TileSize]>\n" - "B.IOT [], ->%[d2]<%c[__pto_TileSize]>\n" - "B.IOT [], last, ->%[d3]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d0]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[__pto_d1]<%c[__pto_TileSize]>\n" + "B.IOT , ->%q[d2]<%c[__pto_TileSize]>\n" + "B.IOT last, ->%q[d3]<%c[__pto_TileSize]>\n" "B.IOR [%[__pto_s0],%[__pto_GmStride]], []\n" - : [__pto_d0]"=Tr"(dst0.data()),[__pto_d1]"=Tr"(dst1.data()),[d2]"=Tr"(dst2.data()),[d3]"=Tr"(dst3.data()) + : [__pto_d0]"=r"(dst0.data()),[__pto_d1]"=r"(dst1.data()),[d2]"=r"(dst2.data()),[d3]"=r"(dst3.data()) : [__pto_s0]"r"(src.data()), [__pto_DstType]"i"(type_traits::TypeCode), [__pto_SrcType]"i"(type_traits::TypeCode), @@ -414,15 +753,15 @@ inline void MGATHER(tile_shape_out &dst, const gm_shape &src, const tile_shape_offset &offset) { static_assert(tile_shape_offset::ValidCol <= tile_shape_offset::Cols, ""); asm volatile( - "BSTART.TMA MGATHER, %c[DataType]\n" + "BSTART.MGATHER %c[DataType]\n" "B.DATR %c[PadValue]\n" "B.DIM zero, %c[ValidCol], ->LB0\n" "B.DIM zero, %c[ValidRow], ->LB1\n" "B.DIM zero, %c[Col], ->LB2\n" - "B.IOT [%[off]], last, ->%[dst]<%c[TileSize]>\n" + "B.IOT %[off], last, ->%q[dst]<%c[TileSize]>\n" "B.IOR [%[base]], []\n" - : [dst] "=Tr"(dst.data()) - : [base] "r"(src.data()), [off] "Tr"(offset.data()), + : [dst] "=r"(dst.data()) + : [base] "r"(src.data()), [off] "r"(offset.data()), [DataType] "i"(type_traits::TypeCode), [PadValue] "i"(static_cast(Pad)), [TileSize] "i"( @@ -438,15 +777,15 @@ inline void MSCATTER(gm_shape &dst, const tile_shape_in &src, const tile_shape_offset &offset) { static_assert(tile_shape_offset::ValidCol <= tile_shape_offset::Cols, ""); asm volatile( - "BSTART.TMA MSCATTER, %c[DataType]\n" + "BSTART.MSCATTER %c[DataType]\n" "B.DIM zero, %c[ValidCol], ->LB0\n" "B.DIM zero, %c[ValidRow], ->LB1\n" "B.DIM zero, %c[Col], ->LB2\n" - "B.IOT [%[src], %[off]], last\n" + "B.IOT %[src], %[off], last\n" "B.IOR [%[base]], []\n" : - : [base] "r"(dst.data()), [src] "Tr"(src.data()), - [off] "Tr"(offset.data()), + : [base] "r"(dst.data()), [src] "r"(src.data()), + [off] "r"(offset.data()), [DataType] "i"(type_traits::TypeCode), [ValidCol] "i"(tile_shape_offset::ValidCol), [ValidRow] "i"(tile_shape_offset::ValidRow), @@ -454,6 +793,24 @@ inline void MSCATTER(gm_shape &dst, const tile_shape_in &src, : "memory"); } +template +inline void TPREFETCH(const gm_shape &src) { + asm volatile( + "BSTART.TPREFETCH %c[DataType]\n" + "B.DATR NORM.normal, %c[DataType], Null\n" + "B.DIM zero, %c[ValidCol], ->LB0\n" + "B.DIM zero, %c[ValidRow], ->LB1\n" + "B.DIM zero, %c[Col], ->LB2\n" + "B.IOR [%[base]], []\n" + : + : [base] "r"(src.data()), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), + [Col] "i"(tile_shape::Cols) + : "memory"); +} + template @@ -461,16 +818,16 @@ inline void MGATHER_MASK(tile_shape_out &dst, const gm_shape &src, const tile_shape_offset &offset, const tile_shape_mask &mask) { asm volatile( - "BSTART.TMA MGATHER.MASK, %c[DataType]\n" + "BSTART.MGATHER.MASK %c[DataType]\n" "B.DATR %c[PadValue]\n" "B.DIM zero, %c[Col], ->LB0\n" "B.DIM zero, %c[Row], ->LB1\n" - "B.IOT [%[off]], ->%[dst]<%c[TileSize]>\n" - "B.IOT [%[mask]], last\n" + "B.IOT %[off], ->%q[dst]<%c[TileSize]>\n" + "B.IOT %[mask], last\n" "B.IOR [%[base]], []\n" - : [dst] "=Tr"(dst.data()) - : [base] "r"(src.data()), [off] "Tr"(offset.data()), - [mask] "Tr"(mask.data()), + : [dst] "=r"(dst.data()) + : [base] "r"(src.data()), [off] "r"(offset.data()), + [mask] "r"(mask.data()), [DataType] "i"(type_traits::TypeCode), [PadValue] "i"(static_cast(Pad)), [TileSize] "i"( @@ -479,26 +836,316 @@ inline void MGATHER_MASK(tile_shape_out &dst, const gm_shape &src, : "memory"); } +template +inline void MGATHER_CAS(tile_shape_out &dst, const gm_shape &src, + const tile_shape_offset &offset, + const tile_shape_expected &expected, + const tile_shape_desired &desired) { + static_assert(tile_shape_offset::ValidCol <= tile_shape_offset::Cols, ""); + static_assert(tile_shape_expected::Rows == tile_shape_desired::Rows, ""); + static_assert(tile_shape_expected::Cols == tile_shape_desired::Cols, ""); + asm volatile( + "BSTART.MGATHER.CAS %c[DataType]\n" + "B.DATR %c[PadValue]\n" + "B.DIM zero, %c[ValidCol], ->LB0\n" + "B.DIM zero, %c[ValidRow], ->LB1\n" + "B.DIM zero, %c[Col], ->LB2\n" + "B.IOT %[off], %[expected]\n" + "B.IOT %[desired], last, ->%q[dst]<%c[TileSize]>\n" + "B.IOR [%[base]], []\n" + : [dst] "=r"(dst.data()) + : [base] "r"(src.data()), [off] "r"(offset.data()), + [expected] "r"(expected.data()), [desired] "r"(desired.data()), + [DataType] "i"(type_traits::TypeCode), + [PadValue] "i"(static_cast(Pad)), + [TileSize] "i"( + tile_type_traits::TilesizeCode), + [ValidCol] "i"(tile_shape_offset::ValidCol), + [ValidRow] "i"(tile_shape_offset::ValidRow), + [Col] "i"(tile_shape_offset::Cols) + : "memory"); +} + template inline void MSCATTER_MASK(gm_shape &dst, const tile_shape_in &src, const tile_shape_offset &offset, const tile_shape_mask &mask) { asm volatile( - "BSTART.TMA MSCATTER.MASK, %c[DataType]\n" + "BSTART.MSCATTER.MASK %c[DataType]\n" "B.DIM zero, %c[Col], ->LB0\n" "B.DIM zero, %c[Row], ->LB1\n" - "B.IOT [%[src], %[off]]\n" - "B.IOT [%[mask]], last\n" + "B.IOT %[src], %[off]\n" + "B.IOT %[mask], last\n" "B.IOR [%[base]], []\n" : - : [base] "r"(dst.data()), [src] "Tr"(src.data()), - [off] "Tr"(offset.data()), [mask] "Tr"(mask.data()), + : [base] "r"(dst.data()), [src] "r"(src.data()), + [off] "r"(offset.data()), [mask] "r"(mask.data()), [DataType] "i"(type_traits::TypeCode), [Col] "i"(tile_shape_offset::Cols), [Row] "i"(tile_shape_offset::Rows) : "memory"); } +template +inline void TXOR(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + asm volatile( + "BSTART.TXOR %c[DataType]\n" + "B.DATR Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[src0], %[src1], last, ->%q[dst]<%c[TileSize]>\n" + : [dst] "=r"(dst.data()) + : [src0] "r"(src0.data()), [src1] "r"(src1.data()), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +template +inline void TSHLS(tile_shape &dst, tile_shape &src, unsigned shift) { + asm volatile( + "BSTART.TSHLS %c[DataType]\n" + "B.DATR Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[src], last, ->%q[dst]<%c[TileSize]>\n" + "B.IOR [%[shift]], []\n" + : [dst] "=r"(dst.data()) + : [src] "r"(src.data()), [shift] "r"(shift), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +template +inline void TSHRS(tile_shape &dst, tile_shape &src, unsigned shift) { + asm volatile( + "BSTART.TSHRS %c[DataType]\n" + "B.DATR Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[src], last, ->%q[dst]<%c[TileSize]>\n" + "B.IOR [%[shift]], []\n" + : [dst] "=r"(dst.data()) + : [src] "r"(src.data()), [shift] "r"(shift), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +template +inline void TSELECT_V057(tile_shape &dst, tile_shape_index &cond, + tile_shape &src0, tile_shape &src1) { + typename tile_shape::TileDType tmp; + asm volatile( + "BSTART.TSEL %c[DataType]\n" + "B.DATR Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[cond], %[src0], ->%q[dst]<%c[TileSize]>\n" + "B.IOT %[src1], last, ->%q[tmp]<%c[TileSize]>\n" + : [dst] "=r"(dst.data()), [tmp] "=r"(tmp) + : [cond] "r"(cond.data()), [src0] "r"(src0.data()), + [src1] "r"(src1.data()), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +// v0.57 named TEPL/TMA surface. These wrappers replace the retired vendor +// tile_size-based per-element implementation: tile values stay opaque and are +// produced/consumed only by architectural tile blocks. +#define SUPERNPUBENCH_V057_TEPL_BINARY(FUNCTION, WRAPPER) \ + template \ + inline void WRAPPER(tile_shape &dst, tile_shape &src0, tile_shape &src1) { \ + asm volatile( \ + "BSTART." #FUNCTION " %c[DataType]\n" \ + "B.DATR Null\n" \ + "C.B.DIMI %c[ValidCol], ->LB0\n" \ + "C.B.DIMI %c[ValidRow], ->LB1\n" \ + "C.B.DIMI %c[Cols], ->LB2\n" \ + "B.IOT %[src0], %[src1], last, ->%q[dst]<%c[TileSize]>\n" \ + : [dst] "=r"(dst.data()) \ + : [src0] "r"(src0.data()), [src1] "r"(src1.data()), \ + [DataType] "i"(type_traits::TypeCode), \ + [ValidCol] "i"(tile_shape::ValidCol), \ + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), \ + [TileSize] "i"( \ + tile_type_traits::TilesizeCode));\ + } + +SUPERNPUBENCH_V057_TEPL_BINARY(TADD, TADD_V057) +SUPERNPUBENCH_V057_TEPL_BINARY(TSUB, TSUB_V057) +SUPERNPUBENCH_V057_TEPL_BINARY(TMUL, TMUL_V057) +SUPERNPUBENCH_V057_TEPL_BINARY(TAND, TAND_V057) +SUPERNPUBENCH_V057_TEPL_BINARY(TOR, TOR_V057) +SUPERNPUBENCH_V057_TEPL_BINARY(TREM, TREM_V057) +#undef SUPERNPUBENCH_V057_TEPL_BINARY + +#define SUPERNPUBENCH_V057_TEPL_SCALAR(FUNCTION, WRAPPER) \ + template \ + inline void WRAPPER(tile_shape &dst, tile_shape &src, \ + typename tile_shape::DType scalar) { \ + asm volatile( \ + "BSTART." #FUNCTION " %c[DataType]\n" \ + "B.DATR Null\n" \ + "C.B.DIMI %c[ValidCol], ->LB0\n" \ + "C.B.DIMI %c[ValidRow], ->LB1\n" \ + "C.B.DIMI %c[Cols], ->LB2\n" \ + "B.IOT %[src], last, ->%q[dst]<%c[TileSize]>\n" \ + "B.IOR [%[scalar]], []\n" \ + : [dst] "=r"(dst.data()) \ + : [src] "r"(src.data()), [scalar] "r"(scalar), \ + [DataType] "i"(type_traits::TypeCode), \ + [ValidCol] "i"(tile_shape::ValidCol), \ + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), \ + [TileSize] "i"( \ + tile_type_traits::TilesizeCode));\ + } + +SUPERNPUBENCH_V057_TEPL_SCALAR(TADDS, TADDS_V057) +SUPERNPUBENCH_V057_TEPL_SCALAR(TMULS, TMULS_V057) +#undef SUPERNPUBENCH_V057_TEPL_SCALAR + +template +inline void TEXPANDS_V057(tile_shape &dst, + typename tile_shape::DType scalar) { + asm volatile( + "BSTART.TEXPANDS %c[DataType]\n" + "B.DATR Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT last, ->%q[dst]<%c[TileSize]>\n" + "B.IOR [%[scalar]], []\n" + : [dst] "=r"(dst.data()) + : [scalar] "r"(scalar), + [DataType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +template +inline void TCVT_V057(tile_shape_out &dst, tile_shape_in &src) { + asm volatile( + "BSTART.TCVT %c[SrcType]\n" + "B.DATR %c[DstType], RNONE\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[src], last, ->%q[dst]<%c[TileSize]>\n" + : [dst] "=r"(dst.data()) + : [src] "r"(src.data()), + [SrcType] "i"(type_traits::TypeCode), + [DstType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape_in::ValidCol), + [ValidRow] "i"(tile_shape_in::ValidRow), + [Cols] "i"(tile_shape_in::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode)); +} + +#define SUPERNPUBENCH_V057_TCMP_CASE(MODE, ASM_MODE, LEFT, RIGHT) \ + case CmpMode::MODE: \ + asm volatile( \ + "BSTART.TCMP %c[DataType]\n" \ + "B.DATR Null, " #ASM_MODE "\n" \ + "C.B.DIMI %c[ValidCol], ->LB0\n" \ + "C.B.DIMI %c[ValidRow], ->LB1\n" \ + "C.B.DIMI %c[Cols], ->LB2\n" \ + "B.IOT %[left], %[right], last, ->%q[dst]<%c[TileSize]>\n" \ + : [dst] "=r"(dst.data()) \ + : [left] "r"(LEFT.data()), [right] "r"(RIGHT.data()), \ + [DataType] "i"(type_traits::TypeCode),\ + [ValidCol] "i"(tile_shape_in::ValidCol), \ + [ValidRow] "i"(tile_shape_in::ValidRow), \ + [Cols] "i"(tile_shape_in::Cols), \ + [TileSize] "i"( \ + tile_type_traits::TilesizeCode));\ + return + +template +inline void TCMP_V057(tile_shape_out &dst, tile_shape_in &src0, + tile_shape_in &src1, CmpMode mode) { + switch (mode) { + SUPERNPUBENCH_V057_TCMP_CASE(EQ, EQ, src0, src1); + SUPERNPUBENCH_V057_TCMP_CASE(NE, NE, src0, src1); + SUPERNPUBENCH_V057_TCMP_CASE(LT, LT, src0, src1); + SUPERNPUBENCH_V057_TCMP_CASE(GE, GE, src0, src1); + SUPERNPUBENCH_V057_TCMP_CASE(GT, LT, src1, src0); + SUPERNPUBENCH_V057_TCMP_CASE(LE, GE, src1, src0); + } +} +#undef SUPERNPUBENCH_V057_TCMP_CASE + +template +inline void TLOAD_V057(tile_shape &dst, gm_shape &src) { + static_assert(tile_shape::isRowMajor && gm_shape::isRowMajor, + "v0.57 direct TLOAD currently requires row-major tensors"); + asm volatile( + "BSTART.TLOAD %c[SrcType]\n" + "B.DATR NORM.normal, %c[DstType], Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT last, ->%q[dst]<%c[TileSize]>\n" + "B.IOR [%[base], %[stride]], []\n" + : [dst] "=r"(dst.data()) + : [base] "r"(src.data()), + [stride] "r"(gm_shape::RowStride * sizeof(typename gm_shape::DType)), + [SrcType] "i"(type_traits::TypeCode), + [DstType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode) + : "memory"); +} + +template +inline void TSTORE_V057(gm_shape &dst, tile_shape &src) { + static_assert(tile_shape::isRowMajor && gm_shape::isRowMajor, + "v0.57 direct TSTORE currently requires row-major tensors"); + asm volatile( + "BSTART.TSTORE %c[SrcType]\n" + "B.DATR NORM.normal, %c[DstType], Null\n" + "C.B.DIMI %c[ValidCol], ->LB0\n" + "C.B.DIMI %c[ValidRow], ->LB1\n" + "C.B.DIMI %c[Cols], ->LB2\n" + "B.IOT %[src], last, ->%q[src]<%c[TileSize]>\n" + "B.IOR [%[base], %[stride]], []\n" + : + : [base] "r"(dst.data()), [src] "r"(src.data()), + [stride] "r"(gm_shape::RowStride * sizeof(typename gm_shape::DType)), + [SrcType] "i"(type_traits::TypeCode), + [DstType] "i"(type_traits::TypeCode), + [ValidCol] "i"(tile_shape::ValidCol), + [ValidRow] "i"(tile_shape::ValidRow), [Cols] "i"(tile_shape::Cols), + [TileSize] "i"( + tile_type_traits::TilesizeCode) + : "memory"); +} + +#endif + +#if !defined(__linx) + #ifndef LINX_CVT_INLINE #define LINX_CVT_INLINE __attribute__((always_inline)) inline #endif @@ -507,6 +1154,38 @@ inline void MSCATTER_MASK(gm_shape &dst, const tile_shape_in &src, #define SIMPLE_STORAGE(d) (d) #endif +// v0.57 exposes only architectural GPR inline-assembly operands. The prior +// toolchain supplied private *_STORAGE accessors alongside its vector-register +// constraints; keep the public value types usable by mapping those accessors to +// their scalar or packed value representation. +#define __tf32_STORAGE(d) (d) +#define __hf32_STORAGE(d) (d) +#define __blkc_bf16_STORAGE(d) (d) +#define __hif8_STORAGE(d) (d) +#define __fp8_e4m3_STORAGE(d) (d) +#define __fp8_e5m2_STORAGE(d) (d) +#define __fp6_e3m2_STORAGE(d) (d) +#define __fp6_e2m3_STORAGE(d) (d) +#define __fp8_e8m0_STORAGE(d) (d) +#define __fp8_e6m2_STORAGE(d) (d) +#define __fp4_e2m1x2_STORAGE(d) (d) +#define __fp4_e1m2x2_STORAGE(d) (d) +#define __fp4_hif4x2_STORAGE(d) (d) +#define __fp8_e6m2x2_STORAGE(d) (d) +#define __fp8_e4m3x2_STORAGE(d) (d) +#define __fp8_e5m2x2_STORAGE(d) (d) +#define __fp16x2_STORAGE(d) (d) +#define __bf16x2_STORAGE(d) (d) +#define __blkc_bf16x2_STORAGE(d) (d) +#define __uint4x2_STORAGE(d) (d) +#define __uint16x2_STORAGE(d) (d) +#define __int4x2_STORAGE(d) (d) +#define __int16x2_STORAGE(d) (d) +#define __fp8_e4m3x4_STORAGE(d) (d) +#define __fp8_e5m2x4_STORAGE(d) (d) +#define __uint8x4_STORAGE(d) (d) +#define __int8x4_STORAGE(d) (d) + // 默认生成： // v.cvt.xxx2yyy %1.src, ->%0.dst, RMode, sat // @@ -563,17 +1242,17 @@ struct linx_valid_sat { RMODE_STR, SAT_STR) \ asm volatile(LINX_CVT_ASM(SRC_TYPE, DST_TYPE, SRC_REG, DST_REG, \ RMODE_STR, SAT_STR) \ - : "=vr"(DST_STORAGE(DST)) \ - : "vr"(SRC_STORAGE(SRC))) + : "=r"(DST_STORAGE(DST)) \ + : "r"(SRC_STORAGE(SRC))) #define LINX_CVT_EMIT_PACKED(SRC_TYPE, DST_TYPE, SRC_REG, DST_REG, \ DST_STORAGE, SRC_STORAGE, DST, SRC0, SRC1, \ RMODE_STR, SAT_STR) \ asm volatile(LINX_CVT_PACKED_ASM(SRC_TYPE, DST_TYPE, SRC_REG, DST_REG, \ RMODE_STR, SAT_STR) \ - : "=vr"(DST_STORAGE(DST)) \ - : "vr"(SRC_STORAGE(SRC0)), \ - "vr"(SRC_STORAGE(SRC1))) + : "=r"(DST_STORAGE(DST)) \ + : "r"(SRC_STORAGE(SRC0)), \ + "r"(SRC_STORAGE(SRC1))) #define LINX_CVT_DISPATCH_NORMAL(RMODE, SAT, SRC_TYPE, DST_TYPE, SRC_REG, \ DST_REG, DST_STORAGE, SRC_STORAGE, DST, SRC) \ @@ -1242,30 +1921,30 @@ enum BlkvBf16Sat { #define BLKV_BF16_EMIT_UNARY(OP, DST, SRC, RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bf" BLKV_BF16_DST_PREFIX ", " RMODE_STR \ ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16_STORAGE(DST)) \ - : "vr"(BLKV_BF16_STORAGE(SRC))) + : "=r"(BLKV_BF16_STORAGE(DST)) \ + : "r"(BLKV_BF16_STORAGE(SRC))) #define BLKV_BF16_EMIT_BINARY(OP, DST, SRC_L, SRC_R, RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bf, %2.bf" BLKV_BF16_DST_PREFIX ", " RMODE_STR \ ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16_STORAGE(DST)) \ - : "vr"(BLKV_BF16_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16_STORAGE(SRC_R))) + : "=r"(BLKV_BF16_STORAGE(DST)) \ + : "r"(BLKV_BF16_STORAGE(SRC_L)), \ + "r"(BLKV_BF16_STORAGE(SRC_R))) #define BLKV_BF16_EMIT_TERNARY(OP, DST, SRC_L, SRC_R, SRC_A, \ RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bf, %2.bf, %3.bf" BLKV_BF16_DST_PREFIX ", " \ RMODE_STR ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16_STORAGE(DST)) \ - : "vr"(BLKV_BF16_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16_STORAGE(SRC_R)), \ - "vr"(BLKV_BF16_STORAGE(SRC_A))) + : "=r"(BLKV_BF16_STORAGE(DST)) \ + : "r"(BLKV_BF16_STORAGE(SRC_L)), \ + "r"(BLKV_BF16_STORAGE(SRC_R)), \ + "r"(BLKV_BF16_STORAGE(SRC_A))) #define BLKV_BF16_EMIT_FMAX(DST, SRC_L, SRC_R) \ asm volatile("v.fmax %1.bf, %2.bf" BLKV_BF16_DST_PREFIX "\n" \ - : "=vr"(BLKV_BF16_STORAGE(DST)) \ - : "vr"(BLKV_BF16_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16_STORAGE(SRC_R))) + : "=r"(BLKV_BF16_STORAGE(DST)) \ + : "r"(BLKV_BF16_STORAGE(SRC_L)), \ + "r"(BLKV_BF16_STORAGE(SRC_R))) #define BLKV_BF16_DISPATCH_RMODE_SAT(RMODE, SAT, EMIT, ...) \ do { \ @@ -1488,30 +2167,30 @@ BLKV_BF16_INLINE __bf16 blkv_bf16_max(const __bf16 &src_l, #define BLKV_BF16X2_EMIT_UNARY(OP, DST, SRC, RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bfx2" BLKV_BF16X2_DST_PREFIX ", " RMODE_STR \ ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16X2_STORAGE(DST)) \ - : "vr"(BLKV_BF16X2_STORAGE(SRC))) + : "=r"(BLKV_BF16X2_STORAGE(DST)) \ + : "r"(BLKV_BF16X2_STORAGE(SRC))) #define BLKV_BF16X2_EMIT_BINARY(OP, DST, SRC_L, SRC_R, RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bfx2, %2.bfx2" BLKV_BF16X2_DST_PREFIX ", " \ RMODE_STR ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16X2_STORAGE(DST)) \ - : "vr"(BLKV_BF16X2_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16X2_STORAGE(SRC_R))) + : "=r"(BLKV_BF16X2_STORAGE(DST)) \ + : "r"(BLKV_BF16X2_STORAGE(SRC_L)), \ + "r"(BLKV_BF16X2_STORAGE(SRC_R))) #define BLKV_BF16X2_EMIT_TERNARY(OP, DST, SRC_L, SRC_R, SRC_A, \ RMODE_STR, SAT_STR) \ asm volatile("v." OP " %1.bfx2, %2.bfx2, %3.bfx2" \ BLKV_BF16X2_DST_PREFIX ", " RMODE_STR ", " SAT_STR "\n" \ - : "=vr"(BLKV_BF16X2_STORAGE(DST)) \ - : "vr"(BLKV_BF16X2_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16X2_STORAGE(SRC_R)), \ - "vr"(BLKV_BF16X2_STORAGE(SRC_A))) + : "=r"(BLKV_BF16X2_STORAGE(DST)) \ + : "r"(BLKV_BF16X2_STORAGE(SRC_L)), \ + "r"(BLKV_BF16X2_STORAGE(SRC_R)), \ + "r"(BLKV_BF16X2_STORAGE(SRC_A))) #define BLKV_BF16X2_EMIT_FMAX(DST, SRC_L, SRC_R) \ asm volatile("v.fmax %1.bfx2, %2.bfx2" BLKV_BF16X2_DST_PREFIX "\n" \ - : "=vr"(BLKV_BF16X2_STORAGE(DST)) \ - : "vr"(BLKV_BF16X2_STORAGE(SRC_L)), \ - "vr"(BLKV_BF16X2_STORAGE(SRC_R))) + : "=r"(BLKV_BF16X2_STORAGE(DST)) \ + : "r"(BLKV_BF16X2_STORAGE(SRC_L)), \ + "r"(BLKV_BF16X2_STORAGE(SRC_R))) #define BLKV_DEFINE_BF16X2_UNARY(NAME, OP) \ template \ @@ -1674,4 +2353,5 @@ blkv_bf16x2_max(const BLKV_BF16X2_TYPE &src_l, #endif // BLKV_BF16_OPS_HPP -#endif \ No newline at end of file +#endif +#endif diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast.hpp index 5276b2b..0b6756a 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -162,7 +162,7 @@ void gen_offset_impl( const size_t total_elements) { static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, "Only static shape supported"); - + #if MAX_DIMs >= 1 size_t in_shape0 = in_shape[0]; size_t out_shape0 = out_shape[0]; @@ -236,11 +236,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -276,18 +276,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -296,7 +296,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_019.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_019.hpp index b6916f2..c1e9b28 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_019.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_019.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -268,11 +268,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -308,18 +308,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -328,7 +328,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; // MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_039.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_039.hpp index b3fc5e5..1faf050 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_039.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_039.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -317,11 +317,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -357,18 +357,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -377,7 +377,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; // MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_07.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_07.hpp index 4b3b7b1..d38101c 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_07.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_07.hpp @@ -11,7 +11,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -268,11 +268,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -308,18 +308,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -328,7 +328,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_07_simple.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_07_simple.hpp index 75e03d0..1e39c97 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_07_simple.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_07_simple.hpp @@ -40,11 +40,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -66,7 +66,7 @@ void broadcast( gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile, inGm, offsetTile); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { auto gO = gOIter(0, Mb); @@ -74,6 +74,6 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } \ No newline at end of file diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_Hunyuan.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_Hunyuan.hpp index 80b5b1c..191ab31 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_Hunyuan.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_Hunyuan.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -273,11 +273,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -313,18 +313,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -333,7 +333,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_mscatter.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_mscatter.hpp index f38161b..958da18 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_mscatter.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_mscatter.hpp @@ -167,7 +167,7 @@ void broadcast_mscatter( // ====================== for (int i = 0; i < input_tiles; ++i) { auto gIn = gInIter(0, i); - TCOPYIN(inDataTile, gIn); + TLOAD(inDataTile, gIn); // 重置广播步进 memset(bcast_step, 0, sizeof(bcast_step)); @@ -185,7 +185,7 @@ void broadcast_mscatter( // 散射写入 MSCATTER(outGm, inDataTile, offsetTile); auto gOffset = gOffsetIter(0, offset_idx); - TCOPYOUT(gOffset, offsetTile); + TSTORE(gOffset, offsetTile); offset_idx ++; // 下一组广播坐标 @@ -200,7 +200,7 @@ void broadcast_mscatter( if constexpr (rmd_input > 0) { auto gIn = gInIter(0, input_tiles); total_elements = rmd_input; - TCOPYIN(inDataTile_rmd, gIn); + TLOAD(inDataTile_rmd, gIn); memset(bcast_step, 0, sizeof(bcast_step)); done = false; @@ -212,7 +212,7 @@ void broadcast_mscatter( ); MSCATTER(outGm, inDataTile_rmd, offsetTile_rmd); auto gOffset = gOffsetIter(0, offset_idx); - TCOPYOUT(gOffset, offsetTile_rmd); + TSTORE(gOffset, offsetTile_rmd); offset_idx ++; done = !next_broadcast_step(); diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_nocopyout.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_nocopyout.hpp index 5340d09..0f9aa7d 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_nocopyout.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_nocopyout.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -162,7 +162,7 @@ void gen_offset_impl( const size_t total_elements) { static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, "Only static shape supported"); - + #if MAX_DIMs >= 1 size_t in_shape0 = in_shape[0]; size_t out_shape0 = out_shape[0]; @@ -236,11 +236,11 @@ void broadcast_nocopyout( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -273,10 +273,10 @@ void broadcast_nocopyout( auto gO = gOIter(0, i); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + MGATHER(outTile, inGm, offsetTile); - // TCOPYOUT(gO, outTile); + // TSTORE(gO, outTile); } if constexpr (rmd_M) { auto gO = gOIter(0, Mb); @@ -284,7 +284,7 @@ void broadcast_nocopyout( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - // TCOPYOUT(gO, outTile_rmd); + // TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_nomg.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_nomg.hpp index 16c5332..c619c0f 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_nomg.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_nomg.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -162,7 +162,7 @@ void gen_offset_impl( const size_t total_elements) { static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, "Only static shape supported"); - + #if MAX_DIMs >= 1 size_t in_shape0 = in_shape[0]; size_t out_shape0 = out_shape[0]; @@ -236,11 +236,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -276,18 +276,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); // MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, offsetTile); + TSTORE(gO, offsetTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -296,7 +296,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; // MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, offsetTile_rmd); + TSTORE(gO, offsetTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_simple.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_simple.hpp index c8ce29f..43e135e 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_simple.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_simple.hpp @@ -85,7 +85,7 @@ void gen_offset_impl( const size_t total_elements) { static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, "Only static shape supported"); - + #if MAX_DIMs >= 1 size_t in_shape0 = in_shape[0]; size_t out_shape0 = out_shape[0]; @@ -159,11 +159,11 @@ void broadcast( const size_t *out_shape ) { const size_t Mb = gOM / tM; - const size_t rmd_M = gOM % tM; + const size_t rmd_M = gOM % tM; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - using tile_shapeData = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; using tile_shapeData_rmd = Tile; using tile_shapeOffset_rmd = Tile; @@ -199,18 +199,18 @@ void broadcast( // printf("total_elements = %d\n", total_elements); // printf("in_shape[0] = %d\n", in_shape[0]); // printf("inGm = %ld\n", inGm); - - // TCOPYIN(inTile, gI); + + // TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_inTile, 1, tM); gen_offset_impl(offsetTile, in_shape, out_shape, base, total_elements); base += total_elements; - + // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(outTile, inGm, offsetTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { // printf("rmd_M = %d\n", rmd_M); @@ -219,7 +219,7 @@ void broadcast( gen_offset_impl(offsetTile_rmd, in_shape, out_shape, base, total_elements); base += total_elements; MGATHER(outTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_019.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_019.hpp index e58991b..d03fdbb 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_019.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_019.hpp @@ -5,7 +5,7 @@ using namespace pto; // ===================================================================== -// Broadcast (B,1,K) -> (B,N,K) via TCOPYIN + __vec__ broadcast + TCOPYOUT +// Broadcast (B,1,K) -> (B,N,K) via TLOAD + __vec__ broadcast + TSTORE // // Optimized for: (1280,1,49) -> (1280,8,49), dtype=half // @@ -17,7 +17,7 @@ using namespace pto; // Processing strategy: // Divide B batches into tiles of kTileBatch batches each. // Per tile: -// 1. TCOPYIN (kTileBatch, K) from GlobalMem -> TileReg +// 1. TLOAD (kTileBatch, K) from GlobalMem -> TileReg // Reads kTileBatch * K contiguous elements. // 2. __vec__ broadcast within TileReg: // Launch <<>> threads: @@ -27,7 +27,7 @@ using namespace pto; // batch_idx = y & (kTileBatch - 1) (0..kTileBatch-1, bitwise) // Read src[batch_idx * RowStride + x] // Write dst[batch_idx * RowStride + copy * K + x] -// 3. TCOPYOUT (kTileBatch, N*K) from TileReg -> GlobalMem +// 3. TSTORE (kTileBatch, N*K) from TileReg -> GlobalMem // // TileReg layout: // Physical tile cols = 512 (padded for 512B alignment). @@ -111,23 +111,23 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, for (size_t i = 0; i < Nb; i++) { gm_in gsrc(in_ptr + i * kTileBatch * kInner); - TCOPYIN(inTile, gsrc); + TLOAD(inTile, gsrc); vec_broadcast_3d <<>>(outTile.data(), inTile.data()); gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); - TCOPYOUT(gdst, outTile); + TSTORE(gdst, outTile); } if constexpr (rmd > 0) { gm_in gsrc(in_ptr + Nb * kTileBatch * kInner); - TCOPYIN(inTile_rmd, gsrc); + TLOAD(inTile_rmd, gsrc); vec_broadcast_3d <<>>(outTile_rmd.data(), inTile_rmd.data()); gm_out gdst(out_ptr + Nb * kTileBatch * kBCast * kInner); - TCOPYOUT(gdst, outTile_rmd); + TSTORE(gdst, outTile_rmd); } } \ No newline at end of file diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_039.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_039.hpp index 72cccbd..58cd521 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_039.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_039.hpp @@ -5,7 +5,7 @@ using namespace pto; // ===================================================================== -// Broadcast (B,1,K) -> (B,N,K) via TCOPYIN + __vec__ broadcast + TCOPYOUT +// Broadcast (B,1,K) -> (B,N,K) via TLOAD + __vec__ broadcast + TSTORE // // Optimized for: (8192,1,16) -> (8192,8,16), dtype=half // @@ -17,7 +17,7 @@ using namespace pto; // Processing strategy: // Divide B batches into tiles of kTileBatch batches each. // Per tile: -// 1. TCOPYIN (kTileBatch, K) from GlobalMem -> TileReg +// 1. TLOAD (kTileBatch, K) from GlobalMem -> TileReg // Reads kTileBatch * K contiguous elements. // 2. __vec__ broadcast within TileReg: // For each batch, replicate its K elements N times (row-wise). @@ -28,7 +28,7 @@ using namespace pto; // col = x % K (inner column 0..K-1) // Read src[y * RowStride + col] // Write dst[y * RowStride + x] -// 3. TCOPYOUT (kTileBatch, N*K) from TileReg -> GlobalMem +// 3. TSTORE (kTileBatch, N*K) from TileReg -> GlobalMem // // TileReg layout: // Physical tile cols = 256 (padded for 512B alignment). @@ -113,23 +113,23 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, for (size_t i = 0; i < Nb; i++) { gm_in gsrc(in_ptr + i * kTileBatch * kInner); - TCOPYIN(inTile, gsrc); + TLOAD(inTile, gsrc); vec_broadcast_3d <<>>(outTile.data(), inTile.data()); gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); - TCOPYOUT(gdst, outTile); + TSTORE(gdst, outTile); } if constexpr (rmd > 0) { gm_in gsrc(in_ptr + Nb * kTileBatch * kInner); - TCOPYIN(inTile_rmd, gsrc); + TLOAD(inTile_rmd, gsrc); vec_broadcast_3d <<>>(outTile_rmd.data(), inTile_rmd.data()); gm_out gdst(out_ptr + Nb * kTileBatch * kBCast * kInner); - TCOPYOUT(gdst, outTile_rmd); + TSTORE(gdst, outTile_rmd); } } \ No newline at end of file diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07.hpp index 5934187..2129afb 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07.hpp @@ -5,20 +5,20 @@ using namespace pto; // ===================================================================== -// Broadcast (N,1) -> (N,C) via TCOPYIN + __vec__ broadcast + TCOPYOUT +// Broadcast (N,1) -> (N,C) via TLOAD + __vec__ broadcast + TSTORE // // Optimized for: (1443,1) -> (1443,129), dtype=half // // Processing strategy: // Divide N rows into tiles of kTileRows rows each. // Per tile: -// 1. TCOPYIN (kTileRows, 1) from GlobalMem -> TileReg +// 1. TLOAD (kTileRows, 1) from GlobalMem -> TileReg // 2. __vec__ broadcast (kTileRows, 1) -> (kTileRows, C) within TileReg // Launch <<>> threads: // x = column index (0..kC-1), y = row index (0..kTileRows-1) // Each thread reads src[j*src_RowStride] (col 0 of row j) // and writes to dst[i + j*dst_RowStride] (col i of row j) -// 3. TCOPYOUT (kTileRows, C) from TileReg -> GlobalMem +// 3. TSTORE (kTileRows, C) from TileReg -> GlobalMem // // TileReg layout: // Physical tile cols padded to 256 for 512B alignment. @@ -77,13 +77,13 @@ void broadcast(dtype *in_ptr, dtype *out_ptr, for (size_t i = 0; i < Nb; i++) { gm_in gsrc(in_ptr + i * kTileRows); - TCOPYIN(inTile, gsrc); + TLOAD(inTile, gsrc); vec_broadcast_rowmajor <<>>(outTile.data(), inTile.data()); gm_out gdst(out_ptr + i * kTileRows * kC); - TCOPYOUT(gdst, outTile); + TSTORE(gdst, outTile); } using tile_in_r = Tile 0) { gm_in gsrc(in_ptr + Nb * kTileRows); - TCOPYIN(inTile_rmd, gsrc); + TLOAD(inTile_rmd, gsrc); vec_broadcast_rowmajor <<>>(outTile_rmd.data(), inTile_rmd.data()); gm_out gdst(out_ptr + Nb * kTileRows * kC); - TCOPYOUT(gdst, outTile_rmd); + TSTORE(gdst, outTile_rmd); } } \ No newline at end of file diff --git a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07_handwrite.hpp b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07_handwrite.hpp index 88cccf2..b6d3a6c 100644 --- a/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07_handwrite.hpp +++ b/benchmark/two-level-arch/kernels/broadcast/broadcast_vec_07_handwrite.hpp @@ -12,7 +12,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -58,7 +58,7 @@ void broadcast( const size_t rmd_N = rmd_M; const size_t vld_N = N * 129; // 实际一次写回数量 const size_t rmd_vld_N = rmd_M * 129; // 尾块，实际一次写回数量 - + Assert(tO0 > 129); Assert(tO0 % 128 == 0); @@ -86,26 +86,26 @@ void broadcast( auto gI = gIIter(0, i); t_in_offset = 0; t_out_offset = 0; - TCOPYIN(inTile, gI); + TLOAD(inTile, gI); for (int j = 0; j < N; ++i) { vec_broadcast<<<129, 1, 1>>>(inTile, outTile, t_in_offset, t_out_offset); t_in_offset += 1; t_out_offset += 129; } - TCOPYOUT(outTile, out_ptr); + TSTORE(outTile, out_ptr); out_ptr += sizeof(dtype) * vld_N; } if constexpr (rmd_M) { auto gI = gIIter(0, Mb); t_in_offset = 0; t_out_offset = 0; - TCOPYIN(inTile, gI); + TLOAD(inTile, gI); for (int j = 0; j < rmd_N; ++i) { vec_broadcast<<<129, 1, 1>>>(inTile_rmd, outTile_rmd, t_in_offset, t_out_offset); t_in_offset += 1; t_out_offset += 129; } - TCOPYOUT(outTile, out_ptr); + TSTORE(outTile, out_ptr); out_ptr += sizeof(dtype) * rmd_vld_N; } } diff --git a/benchmark/two-level-arch/kernels/concat/concat_gather.hpp b/benchmark/two-level-arch/kernels/concat/concat_gather.hpp index 85a69fd..404f7ae 100644 --- a/benchmark/two-level-arch/kernels/concat/concat_gather.hpp +++ b/benchmark/two-level-arch/kernels/concat/concat_gather.hpp @@ -16,7 +16,7 @@ using namespace pto; GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -36,7 +36,7 @@ void __vec__ gen_offset_concat( typename tile_shape::TileDType __out__ out, typename tile_Inshape::TileDType __in__ in_shape, typename tile_Outshape::TileDType __in__ out_shape, -// const size_t in_dim, +// const size_t in_dim, const size_t base, const size_t total_elements ) { @@ -64,14 +64,14 @@ void __vec__ gen_offset_concat( // 输出一维索引 → 输出坐标 size_t out_coord[MAX_DIM] = {0}; // size_t tmp = idx; // - + #pragma clang loop unroll(full) for (int d = DATA_DIM - 1; d >= 0; d--) { out_coord[d] = tmp % out_shape_ptr[d]; tmp /= out_shape_ptr[d]; } - size_t n = out_coord[CONCAT_DIM] / in_shape_ptr[CONCAT_DIM]; + size_t n = out_coord[CONCAT_DIM] / in_shape_ptr[CONCAT_DIM]; size_t offset = out_coord[CONCAT_DIM] % in_shape_ptr[CONCAT_DIM]; out_coord[CONCAT_DIM] = offset; @@ -88,10 +88,10 @@ void __vec__ gen_offset_concat( } } */ -// uint16_t in_offset = 0; - uint32_t in_offset = 0; +// uint16_t in_offset = 0; + uint32_t in_offset = 0; - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for (int i = 0; i < DATA_DIM; i++) { in_offset += out_coord[i] * stride[i] * sizeof(dtype); } @@ -109,7 +109,7 @@ void gen_offset_Impl( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0, +// const size_t transpose_dim0, const size_t base, const size_t total_elements ) @@ -130,31 +130,31 @@ void concat_gather( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0 -) +// const size_t transpose_dim0 +) { const int Mb = gOM / tM; - + const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_shapeOut = global_tensor>; - using gm_InDataShape = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_InDataShape = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_OutDataShape = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec using tile_Inshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec + using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec gm_shapeIn inGm(in_ptr); - gm_InDataShape InShapeGm(in_shape); - gm_OutDataShape OutShapeGm(out_shape); + gm_InDataShape InShapeGm(in_shape); + gm_OutDataShape OutShapeGm(out_shape); tile_shapeData dataTile; tile_shapeOffset offsetTile; @@ -175,25 +175,25 @@ void concat_gather( for (int i = 0; i < Mb; ++i) { auto gO = gOIter(0, i); - TCOPYIN(InshapeTile, InShapeGm); - TCOPYIN(OutshapeTile, OutShapeGm); + TLOAD(InshapeTile, InShapeGm); + TLOAD(OutshapeTile, OutShapeGm); gen_offset_Impl(offsetTile, InshapeTile, OutshapeTile, base, total_elements); // printf("end genoffset\n"); base += total_elements; // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(dataTile, inGm, offsetTile); -// printf("end mgather\n"); - TCOPYOUT(gO, dataTile); +// printf("end mgather\n"); + TSTORE(gO, dataTile); } - if constexpr (rmd_M) { + if constexpr (rmd_M) { auto gO = gOIter(0, Mb); - TCOPYIN(InshapeTile, InShapeGm); - TCOPYIN(OutshapeTile, OutShapeGm); + TLOAD(InshapeTile, InShapeGm); + TLOAD(OutshapeTile, OutShapeGm); total_elements = rmd_M;//尾片的大小。 gen_offset_Impl(offsetTile, InshapeTile, OutshapeTile, base, total_elements); base += total_elements; MGATHER(dataTile, inGm, offsetTile); - TCOPYOUT(gO, dataTile); + TSTORE(gO, dataTile); } } diff --git a/benchmark/two-level-arch/kernels/concat/concat_scatter.hpp b/benchmark/two-level-arch/kernels/concat/concat_scatter.hpp index e0f14b8..13b6587 100644 --- a/benchmark/two-level-arch/kernels/concat/concat_scatter.hpp +++ b/benchmark/two-level-arch/kernels/concat/concat_scatter.hpp @@ -16,7 +16,7 @@ using namespace pto; GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -36,15 +36,15 @@ void __vec__ gen_offset_concat( typename tile_shape::TileDType __out__ out, typename tile_Inshape::TileDType __in__ in_shape, typename tile_Outshape::TileDType __in__ out_shape, -// const size_t in_dim, +// const size_t in_dim, const size_t base, const size_t total_elements ) { size_t index = blkv_get_index_x(); size_t idx = blkv_get_index_x(); - __vbuf__ typename tile_Inshape::DType *in_shape_ptr = blkv_get_tile_ptr(in_shape); - __vbuf__ typename tile_Outshape::DType *out_shape_ptr = blkv_get_tile_ptr(out_shape); + __vbuf__ typename tile_Inshape::DType *in_shape_ptr = blkv_get_tile_ptr(in_shape); + __vbuf__ typename tile_Outshape::DType *out_shape_ptr = blkv_get_tile_ptr(out_shape); if (index >= total_elements) return; idx = idx + base; // todo idx是个向量，base是个标量，获得所有的基地址或者说基offset @@ -64,7 +64,7 @@ void __vec__ gen_offset_concat( // 输出一维索引 → 输出坐标 size_t in_coord[MAX_DIM] = {0}; // size_t tmp = idx; // - + #pragma clang loop unroll(full) for (int d = DATA_DIM - 1; d >= 0; d--) { in_coord[d] = tmp % in_shape_ptr[d]; @@ -73,7 +73,7 @@ void __vec__ gen_offset_concat( size_t n = tmp; in_coord[CONCAT_DIM] = n * in_shape_ptr[CONCAT_DIM] + in_coord[CONCAT_DIM]; -// size_t n = out_coord[CONCAT_DIM] / in_shape_ptr[CONCAT_DIM]; +// size_t n = out_coord[CONCAT_DIM] / in_shape_ptr[CONCAT_DIM]; // size_t offset = out_coord[CONCAT_DIM] % in_shape_ptr[CONCAT_DIM]; // out_coord[CONCAT_DIM] = offset; @@ -90,11 +90,11 @@ void __vec__ gen_offset_concat( } } */ -// uint16_t in_offset = 0; -// uint32_t out_offset = 0; - uint16_t out_offset = 0; +// uint16_t in_offset = 0; +// uint32_t out_offset = 0; + uint16_t out_offset = 0; - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for (int i = 0; i < DATA_DIM; i++) { out_offset += in_coord[i] * stride[i] * sizeof(dtype); } @@ -111,7 +111,7 @@ void gen_offset_Impl( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0, +// const size_t transpose_dim0, const size_t base, const size_t total_elements ) @@ -132,33 +132,33 @@ void concat_scatter( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0 -) +// const size_t transpose_dim0 +) { const int Mb = gOM / tM; - + const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_shapeOut = global_tensor>; - using gm_InDataShape = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_InDataShape = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_OutDataShape = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec + using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec using tile_Inshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec + using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); - gm_InDataShape InShapeGm(in_shape); - gm_OutDataShape OutShapeGm(out_shape); + gm_InDataShape InShapeGm(in_shape); + gm_OutDataShape OutShapeGm(out_shape); tile_shapeData dataTile; tile_shapeOffset offsetTile; @@ -181,24 +181,24 @@ void concat_scatter( for (int i = 0; i < Mb; ++i) { - auto gI = gIIter(0, i); - TCOPYIN(InshapeTile, InShapeGm); - TCOPYIN(OutshapeTile, OutShapeGm); + auto gI = gIIter(0, i); + TLOAD(InshapeTile, InShapeGm); + TLOAD(OutshapeTile, OutShapeGm); gen_offset_Impl(offsetTile, InshapeTile, OutshapeTile, base, total_elements); // printf("end genoffset\n"); base += total_elements; // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); - TCOPYIN(dataTile, gI); + TLOAD(dataTile, gI); MSCATTER(outGm, dataTile, offsetTile); } - if constexpr (rmd_M) { - auto gI = gIIter(0, Mb); - TCOPYIN(InshapeTile, InShapeGm); - TCOPYIN(OutshapeTile, OutShapeGm); + if constexpr (rmd_M) { + auto gI = gIIter(0, Mb); + TLOAD(InshapeTile, InShapeGm); + TLOAD(OutshapeTile, OutShapeGm); total_elements = rmd_M;//尾片的大小。 gen_offset_Impl(offsetTile, InshapeTile, OutshapeTile, base, total_elements); base += total_elements; - TCOPYIN(dataTile, gI); + TLOAD(dataTile, gI); MSCATTER(outGm, dataTile, offsetTile); } } diff --git a/benchmark/two-level-arch/kernels/control/hashtable_lookup_simd.hpp b/benchmark/two-level-arch/kernels/control/hashtable_lookup_simd.hpp index 19659f3..d858b65 100644 --- a/benchmark/two-level-arch/kernels/control/hashtable_lookup_simd.hpp +++ b/benchmark/two-level-arch/kernels/control/hashtable_lookup_simd.hpp @@ -9,13 +9,13 @@ // Pure-SIMD (tile-level) hashtable lookup. // // Everything below is expressed with whole-tile operations from the tile-op API -// (TMULS/TADDS/TXOR/TSLL/TSRL/TOR/TAND/TREM for the hash, MGATHER/TCMP/TSELECT for +// (TMULS/TADDS/TXOR/TSHLS/TSHRS/TOR/TAND/TREM for the hash, MGATHER/TCMP/TSELECT for // the probe). There are NO __vec__ SIMT kernels, no <<<>>> launches, and no per-lane // control flow (blkv_get_index / blkv_rdadd / data-dependent branches). The probe loop // is a host loop bounded by kMaxProbe with NO data-dependent early break: a tile-level // early exit would need to reduce the "still-searching" mask to a host scalar, and the // tile-op API has no tile->scalar-register reduction (only TROWSUM->tile and -// TCOPYOUT->memory). Since TSELECT keeps already-found lanes immune to later probes, +// TSTORE->memory). Since TSELECT keeps already-found lanes immune to later probes, // running all kMaxProbe iterations is correct; the loop is pure tile ops + a host counter. // ============================================================================ @@ -45,15 +45,15 @@ struct HashFindTypes { // dst = rotl32(x, r) = (x << r) | (x >>(logical) (32 - r)) (in-place safe) template inline void tileRotl32(TileU32& dst, TileU32& x, unsigned r, TileU32& tA, TileU32& tB) { - TSLL(tA, x, r); - TSRL(tB, x, 32u - r); + TSHLS(tA, x, r); + TSHRS(tB, x, 32u - r); TOR(dst, tA, tB); } // One MurmurHash3 mixing round for a 32-bit block: folds `block` into `h`. template inline void murmurRound(TileU32& h, TileU32& block, TileU32& k, TileU32& tA, TileU32& tB) { - TCOPY(k, block); + TCVT(k, block); TMULS(k, k, c1); tileRotl32(k, k, rot_c1, tA, tB); TMULS(k, k, c2); @@ -69,8 +69,9 @@ inline void murmurRound(TileU32& h, TileU32& block, TileU32& k, TileU32& tA, Til // (h>>1) < 2^31 so it is a non-negative int32; 2*(m-1)+1 < 2^31 so the second // operand stays non-negative -> signed TREM yields the correct unsigned result. template -void uModU32(typename HashFindTypes::TileI32& dst, - typename HashFindTypes::TileU32& h) +__attribute__((always_inline)) inline void +uModU32(typename HashFindTypes::TileI32& dst, + typename HashFindTypes::TileU32& h) { using Types = HashFindTypes; using TileU32 = typename Types::TileU32; @@ -79,7 +80,7 @@ void uModU32(typename HashFindTypes::TileI32& dst, TileU32 halfU, bitU, oneU; TileI32 halfI, bitI, capI, tmp; - TSRL(halfU, h, 1u); // h >> 1 (0 .. 2^31-1) + TSHRS(halfU, h, 1u); // h >> 1 (0 .. 2^31-1) TEXPANDSCALAR(oneU, 1u); TAND(bitU, h, oneU); // h & 1 @@ -95,8 +96,10 @@ void uModU32(typename HashFindTypes::TileI32& dst, // Compute per-key probe byte-offset = (MurmurHash3(key) % kCap) * sizeof(TableEntry) template -void computeProbeOffsets(typename HashFindTypes::TileI32& probeOff, - typename HashFindTypes::TileI64& queryKeys) +__attribute__((always_inline)) inline void +computeProbeOffsets( + typename HashFindTypes::TileI32& probeOff, + typename HashFindTypes::TileI64& queryKeys) { using Types = HashFindTypes; using TileU32 = typename Types::TileU32; @@ -108,24 +111,24 @@ void computeProbeOffsets(typename HashFindTypes::TileI32& // block0 = low 32 bits, block1 = high 32 bits of each 8-byte key TileU32 block0, block1; TCAST(block0, queryKeys); // static_cast truncates to low 32 - TSRL(keyHi, queryKeys, 32u); // logical >> 32 (unsigned), high 32 -> low + TSHRS(keyHi, queryKeys, 32u); // logical >> 32 (unsigned), high 32 -> low TCAST(block1, keyHi); TEXPANDSCALAR(h, DEFAULT_HASH_SEED); - TCOPY(blk, block0); + TCVT(blk, block0); murmurRound(h, blk, k, tA, tB); - TCOPY(blk, block1); + TCVT(blk, block1); murmurRound(h, blk, k, tA, tB); // finalize: h ^= len(=8); h ^= h>>16; h *= ..; h ^= h>>13; h *= ..; h ^= h>>16 TileU32 lenTile; TEXPANDSCALAR(lenTile, 8u); TXOR(h, h, lenTile); - TSRL(tmp, h, 16u); TXOR(h, h, tmp); + TSHRS(tmp, h, 16u); TXOR(h, h, tmp); TMULS(h, h, 0x85ebca6bu); - TSRL(tmp, h, 13u); TXOR(h, h, tmp); + TSHRS(tmp, h, 13u); TXOR(h, h, tmp); TMULS(h, h, 0xc2b2ae35u); - TSRL(tmp, h, 16u); TXOR(h, h, tmp); + TSHRS(tmp, h, 16u); TXOR(h, h, tmp); // slot = h % kCap (unsigned) ; byte offset = slot * sizeof(TableEntry) uModU32(kCap)>(probeOff, h); @@ -160,7 +163,7 @@ void runHashFind(int32_t __out__ *out, TileI32 outTile; KeyGT key_gt(queries); - TCOPYIN(queryKeyTile, key_gt); + TLOAD(queryKeyTile, key_gt); computeProbeOffsets(probeOffTile, queryKeyTile); @@ -171,7 +174,7 @@ void runHashFind(int32_t __out__ *out, { TileI64 qHi; TCAST(queryLo, queryKeyTile); // low 32 bits - TSRL(qHi, queryKeyTile, 32u); + TSHRS(qHi, queryKeyTile, 32u); TCAST(queryHi, qHi); // high 32 bits } @@ -188,15 +191,15 @@ void runHashFind(int32_t __out__ *out, MGATHER(tableValTile, tableValGlobal, probeOffTile); // 64-bit key equality = (low32 == low32) && (high32 == high32) TCAST(tableLo, tableKeyTile); - TSRL(tableKeyHi, tableKeyTile, 32u); + TSHRS(tableKeyHi, tableKeyTile, 32u); TCAST(tableHi, tableKeyHi); TCMP(matchLo, queryLo, tableLo, CmpMode::EQ); TCMP(matchHi, queryHi, tableHi, CmpMode::EQ); TAND(matchMask, matchLo, matchHi); - TSELECT(outTile, matchMask, tableValTile, outTile); // match ? value : keep + TSELECT(outTile, matchMask, tableValTile, outTile); // match ? value : keep // NOTE: no early break. A data-dependent early exit would require reducing the - // "still-searching" mask to a host scalar (TROWSUMEXPAND -> TCOPYOUT -> scalar + // "still-searching" mask to a host scalar (TROWSUMEXPAND -> TSTORE -> scalar // load), i.e. a tile->memory->scalar round-trip every iteration. The tile-op API // has no tile->scalar-register reduction, so that round-trip is the only option // and it is exactly the fragile path we avoid. TSELECT already makes already-found @@ -209,7 +212,7 @@ void runHashFind(int32_t __out__ *out, } OutGT outGlobal(out); - TCOPYOUT(outGlobal, outTile); + TSTORE(outGlobal, outTile); } template diff --git a/benchmark/two-level-arch/kernels/element_wise/gelu.hpp b/benchmark/two-level-arch/kernels/element_wise/gelu.hpp index b2e3043..8bfd074 100644 --- a/benchmark/two-level-arch/kernels/element_wise/gelu.hpp +++ b/benchmark/two-level-arch/kernels/element_wise/gelu.hpp @@ -20,7 +20,7 @@ void __vec__ gelu_simd( // 数据格式转换 V.FCVT float x = static_cast(indata); - + constexpr uint32_t TOTAL_COUNT = 24*8*1024; constexpr float SCALAR_A5 = -3.5123395303315874e-09f; constexpr float SCALAR_A4 = 2.6452661927578447e-07f; @@ -30,7 +30,7 @@ void __vec__ gelu_simd( constexpr float SCALAR_A0 = -7.2666168212890625e-02f; constexpr float SCALAR_AM1 = -1.5957698822021484e+00f; constexpr float FP32_MAX = 5.75f; - + float t = blkv_max(x, -FP32_MAX); t = blkv_min(t, FP32_MAX); float t2 = t * t; @@ -44,7 +44,7 @@ void __vec__ gelu_simd( float exp_val = blkv_fexp(t * p); float y = x / (1.0f + exp_val); - + BLKC_ASSIGN_CAST(out, index, y); // blkv_get_tile_ptr(out)[index] = static_cast(result); } @@ -64,7 +64,7 @@ void gelu_impl( GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -82,7 +82,7 @@ void gelu( bool approximate = false // false:none, true:tanh ) { const int Mb = gM / tM; - + const int rmd_M = gM % tM; using gm_shape = global_tensor>; @@ -111,18 +111,18 @@ void gelu( // printf("iter i %d\n",i); auto gI = gIIter(0, i); auto gO = gOIter(0, i); - TCOPYIN(inTile, gI); + TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_intTile, 1, tM); gelu_impl(inTile, outTile); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { auto gI = gIIter(0, Mb); auto gO = gOIter(0, Mb); - TCOPYIN(inTile_rmd, gI); + TLOAD(inTile_rmd, gI); gelu_impl(inTile_rmd, outTile_rmd); // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/element_wise/gelu_origin.hpp b/benchmark/two-level-arch/kernels/element_wise/gelu_origin.hpp index 8f9b75f..5560416 100644 --- a/benchmark/two-level-arch/kernels/element_wise/gelu_origin.hpp +++ b/benchmark/two-level-arch/kernels/element_wise/gelu_origin.hpp @@ -90,7 +90,7 @@ void __vec__ gelu_simd( float result; // 数据格式转换 V.FCVT float x = static_cast(indata); - + // GELU(x)=x∗Φ(x), Φ(x)=负无穷~x积分 φ(exp(-0.5f*x*x) / sqrt(2π)) // 等价于GELU(x)=0.5⋅x⋅(1+erf(x/sqrt(2)) if (!approximate) { @@ -125,7 +125,7 @@ void gelu_impl( GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -143,7 +143,7 @@ void gelu( bool approximate = false // false:none, true:tanh ) { const int Mb = gM / tM; - + const int rmd_M = gM % tM; using gm_shape = global_tensor>; @@ -172,19 +172,19 @@ void gelu( // printf("iter i %d\n",i); auto gI = gIIter(0, i); auto gO = gOIter(0, i); - TCOPYIN(inTile, gI); + TLOAD(inTile, gI); // DUMP_TILE("inTile", inTile, g_dump_intTile, 1, tM); gelu_impl(inTile, outTile, approximate); // DUMP_TILE("outTile", outTile, g_dump_outTile, 1, tM); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_M) { auto gI = gIIter(0, Mb); auto gO = gOIter(0, Mb); - TCOPYIN(inTile_rmd, gI); + TLOAD(inTile_rmd, gI); gelu_impl(inTile_rmd, outTile_rmd, approximate); // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); - TCOPYOUT(gO, outTile_rmd); + TSTORE(gO, outTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/fa/fa_2d_unroll.hpp b/benchmark/two-level-arch/kernels/fa/fa_2d_unroll.hpp index 296e6e9..82e16a6 100644 --- a/benchmark/two-level-arch/kernels/fa/fa_2d_unroll.hpp +++ b/benchmark/two-level-arch/kernels/fa/fa_2d_unroll.hpp @@ -52,9 +52,9 @@ void __vec__ new_max_1src( #ifndef RES_CHECK upd_max = upd_max * src_scale; #endif - new_max_ptr[max_idx] = upd_max; + new_max_ptr[max_idx] = upd_max; - scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); + scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); } template @@ -103,7 +103,7 @@ void __vec__ new_sum_1src( typename tileSrc::DType exp_src_3 = src_ptr[src_idx_3]; typename tileSrc::DType exp_src_01 = exp_src_0 + exp_src_1; typename tileSrc::DType exp_src_23 = exp_src_2 + exp_src_3; - typename tileSrc::DType exp_src_0123 = exp_src_01 + exp_src_23; + typename tileSrc::DType exp_src_0123 = exp_src_01 + exp_src_23; upd_sum += exp_src_0123; } blkv_get_tile_ptr(new_sum)[sum_idx] = upd_sum; @@ -335,9 +335,9 @@ void __vec__ new_max_4src( upd_max = blkv_max(upd_max, s3_max_0123); } upd_max = upd_max * src_scale; - new_max_ptr[max_idx] = upd_max; + new_max_ptr[max_idx] = upd_max; - scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); + scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); } template @@ -410,7 +410,7 @@ void __vec__ src_exp_2src_with_local_sum( BLKC_ASSIGN_CAST(src_exp1, idx_2, src1_exp2); BLKC_ASSIGN_CAST(src_exp1, idx_3, src1_exp3); typename tileSum::DType src1_exp_sum = src1_exp0 + src1_exp1 + src1_exp2 + src1_exp3; - + upd_sum += src0_exp_sum + src1_exp_sum; } size_t idx_sum = i * tileSum::RowStride; @@ -453,7 +453,7 @@ void __vec__ new_sum_4src( typename tileSrc::DType s0_exp_src_3 = src0_ptr[src_idx_3]; typename tileSrc::DType s0_exp_src_01 = s0_exp_src_0 + s0_exp_src_1; typename tileSrc::DType s0_exp_src_23 = s0_exp_src_2 + s0_exp_src_3; - typename tileSrc::DType s0_exp_src_0123 = s0_exp_src_01 + s0_exp_src_23; + typename tileSrc::DType s0_exp_src_0123 = s0_exp_src_01 + s0_exp_src_23; typename tileSrc::DType s1_exp_src_0 = src1_ptr[src_idx_0]; typename tileSrc::DType s1_exp_src_1 = src1_ptr[src_idx_1]; @@ -461,7 +461,7 @@ void __vec__ new_sum_4src( typename tileSrc::DType s1_exp_src_3 = src1_ptr[src_idx_3]; typename tileSrc::DType s1_exp_src_01 = s1_exp_src_0 + s1_exp_src_1; typename tileSrc::DType s1_exp_src_23 = s1_exp_src_2 + s1_exp_src_3; - typename tileSrc::DType s1_exp_src_0123 = s1_exp_src_01 + s1_exp_src_23; + typename tileSrc::DType s1_exp_src_0123 = s1_exp_src_01 + s1_exp_src_23; typename tileSrc::DType s2_exp_src_0 = src2_ptr[src_idx_0]; typename tileSrc::DType s2_exp_src_1 = src2_ptr[src_idx_1]; @@ -469,7 +469,7 @@ void __vec__ new_sum_4src( typename tileSrc::DType s2_exp_src_3 = src2_ptr[src_idx_3]; typename tileSrc::DType s2_exp_src_01 = s2_exp_src_0 + s2_exp_src_1; typename tileSrc::DType s2_exp_src_23 = s2_exp_src_2 + s2_exp_src_3; - typename tileSrc::DType s2_exp_src_0123 = s2_exp_src_01 + s2_exp_src_23; + typename tileSrc::DType s2_exp_src_0123 = s2_exp_src_01 + s2_exp_src_23; typename tileSrc::DType s3_exp_src_0 = src3_ptr[src_idx_0]; typename tileSrc::DType s3_exp_src_1 = src3_ptr[src_idx_1]; @@ -535,7 +535,7 @@ void __vec__ local_max_4src( upd_max = blkv_max(upd_max, s0123_max); } upd_max = upd_max * src_scale; - local_max_ptr[max_idx] = upd_max; + local_max_ptr[max_idx] = upd_max; } template @@ -571,7 +571,7 @@ void __vec__ local_sum_4src( typename tileSrc::DType s0_exp_src_3 = src0_ptr[src_idx_3]; typename tileSrc::DType s0_exp_src_01 = s0_exp_src_0 + s0_exp_src_1; typename tileSrc::DType s0_exp_src_23 = s0_exp_src_2 + s0_exp_src_3; - typename tileSrc::DType s0_exp_src_0123 = s0_exp_src_01 + s0_exp_src_23; + typename tileSrc::DType s0_exp_src_0123 = s0_exp_src_01 + s0_exp_src_23; typename tileSrc::DType s1_exp_src_0 = src1_ptr[src_idx_0]; typename tileSrc::DType s1_exp_src_1 = src1_ptr[src_idx_1]; @@ -579,7 +579,7 @@ void __vec__ local_sum_4src( typename tileSrc::DType s1_exp_src_3 = src1_ptr[src_idx_3]; typename tileSrc::DType s1_exp_src_01 = s1_exp_src_0 + s1_exp_src_1; typename tileSrc::DType s1_exp_src_23 = s1_exp_src_2 + s1_exp_src_3; - typename tileSrc::DType s1_exp_src_0123 = s1_exp_src_01 + s1_exp_src_23; + typename tileSrc::DType s1_exp_src_0123 = s1_exp_src_01 + s1_exp_src_23; typename tileSrc::DType s2_exp_src_0 = src2_ptr[src_idx_0]; typename tileSrc::DType s2_exp_src_1 = src2_ptr[src_idx_1]; @@ -587,7 +587,7 @@ void __vec__ local_sum_4src( typename tileSrc::DType s2_exp_src_3 = src2_ptr[src_idx_3]; typename tileSrc::DType s2_exp_src_01 = s2_exp_src_0 + s2_exp_src_1; typename tileSrc::DType s2_exp_src_23 = s2_exp_src_2 + s2_exp_src_3; - typename tileSrc::DType s2_exp_src_0123 = s2_exp_src_01 + s2_exp_src_23; + typename tileSrc::DType s2_exp_src_0123 = s2_exp_src_01 + s2_exp_src_23; typename tileSrc::DType s3_exp_src_0 = src3_ptr[src_idx_0]; typename tileSrc::DType s3_exp_src_1 = src3_ptr[src_idx_1]; @@ -625,7 +625,7 @@ void __vec__ new_max_of_2_loc_max( typename tileMax::DType local_max_01 = blkv_max(local_max_0_ptr[max_idx], local_max_1_ptr[max_idx]); upd_max = blkv_max(upd_max, local_max_01); new_max_ptr[max_idx] = upd_max; - scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); + scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); } template void __vec__ new_sum_of_2_loc_sum( @@ -680,7 +680,7 @@ void __vec__ new_max_of_4_loc_max( typename tileMax::DType local_max_0123 = blkv_max(local_max_01, local_max_23); upd_max = blkv_max(upd_max, local_max_0123); new_max_ptr[max_idx] = upd_max; - scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); + scale_ptr[max_idx] = blkv_fexp(old_max_val - upd_max); } template void __vec__ new_sum_of_4_loc_sum( @@ -705,7 +705,7 @@ void __vec__ new_sum_of_4_loc_sum( size_t sum_idx = i*tileSum::RowStride; - new_sum_ptr[sum_idx] = old_sum_ptr[sum_idx] * scale_ptr[sum_idx] + + new_sum_ptr[sum_idx] = old_sum_ptr[sum_idx] * scale_ptr[sum_idx] + local_sum_0_ptr[sum_idx] + local_sum_1_ptr[sum_idx] + local_sum_2_ptr[sum_idx] + local_sum_3_ptr[sum_idx]; } @@ -734,7 +734,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype using tileW_out = TileAcc; // [kTm×kTk] using tileW = Tile; using tileW_cast = Tile::DType, kTm, kTk, BLayout::ColMajor>; - using tileW_left = TileLeft; + using tileW_left = TileLeft; using tileO_out = TileAcc; using tileO = Tile; // [kTm×vD] @@ -771,7 +771,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype tileQ tQ[Xdim]; - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data(), scale); @@ -891,7 +891,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tNewMax[x].data(), // scale); - + src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -909,7 +909,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -929,7 +929,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -969,7 +969,7 @@ void flash_attention_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype #pragma clang loop unroll(full) for(int y=0;y; // [kTm×kTk] using tileW = Tile; using tileW_cast = Tile::DType, kTm, kTk, BLayout::ColMajor>; - using tileW_left = TileLeft; + using tileW_left = TileLeft; using tileO_out = TileAcc; using tileO = Tile; // [kTm×vD] @@ -50,7 +50,7 @@ void flash_attention_2d_unroll_pto(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, d #pragma clang loop unroll(full) for(int x=0;x Nz @@ -233,7 +233,7 @@ void flash_attention_2d_unroll_pto(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, d TCOLEXPANDMUL_TEPL(tO[x], tO[x], tInvSum[x]); TCAST_TEPL(tO_cast[x], tO[x]); auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]); + TSTORE(dstO, tO_cast[x]); } } } diff --git a/benchmark/two-level-arch/kernels/fa/fa_dcore.hpp b/benchmark/two-level-arch/kernels/fa/fa_dcore.hpp index 80322da..1ed8436 100644 --- a/benchmark/two-level-arch/kernels/fa/fa_dcore.hpp +++ b/benchmark/two-level-arch/kernels/fa/fa_dcore.hpp @@ -124,7 +124,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ using tileQ = MultiTile>; // [kTm×qD] using tileK = MultiTile>; // [vD×kTk] // using tileW_out = TileAcc; // [kTm×kTk] - using tileW = MultiTile>; + using tileW = MultiTile>; using tileW_cast = MultiTile>; using tileW_left = MultiTile>; @@ -151,7 +151,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ const float scale = 1.0f / sqrt((float)qD); const int Qb = (Sq + kTm - 1) / kTm; const int Kb = (Skv + kTk - 1) / kTk; - + // 对每个 Q-block (i) for (int i = 0; i < Qb; i += MULTI) { // 加载当前Q块 (仅一次) @@ -161,7 +161,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ auto gQ = gIterQ(i,0); TLOAD2_ND2NZ(tQ.Tiles[1], tQ.Tiles[0], gQ); #else - TCOPYIN(tQ, [&](int t) { return gIterQ(i + t, 0); }); + TLOAD(tQ, [&](int t) { return gIterQ(i + t, 0); }); #endif // 初始化状态: 最大值/指数和/输出累加 @@ -178,7 +178,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ // 加载K_j和V_j auto gK = gIterK(0, j); tileK tK; - TCOPYIN(tK, gK); + TLOAD(tK, gK); // 计算注意力分数块 tileW tW; @@ -207,7 +207,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ // 计算当前块的加权输出: O_j = W * V auto gV = gIterV(j, 0); tileV tV; - TCOPYIN(tV, gV); + TLOAD(tV, gV); MATMUL(tPV, tW_left, tV); if(j==0){ @@ -233,7 +233,7 @@ void flash_attention_dcore(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ auto gO = gIterO(i, 0); TSTORE2_DN2DN(gO, tO_cast.Tiles[1], tO_cast.Tiles[0]); #else - TCOPYOUT([&](int t) { return gIterO(i + t, 0); }, tO_cast); + TSTORE([&](int t) { return gIterO(i + t, 0); }, tO_cast); #endif } } diff --git a/benchmark/two-level-arch/kernels/fa/fa_hif4.hpp b/benchmark/two-level-arch/kernels/fa/fa_hif4.hpp index 805ca04..a5df569 100644 --- a/benchmark/two-level-arch/kernels/fa/fa_hif4.hpp +++ b/benchmark/two-level-arch/kernels/fa/fa_hif4.hpp @@ -271,7 +271,7 @@ void __vec__ pkg_rowmax( __bf16x2 upd_max; __bf16 old_max_bf160, old_max_bf161; linx_cvt(old_max_bf160, old_max_ptr[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); + linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); linx_cvt_package(upd_max, old_max_bf160, old_max_bf161); // calc tile rowmax @@ -322,7 +322,7 @@ void __vec__ pkg_rowmax( // recalculate scale of softmax __bf16x2 scale, old_max_bf16x2; linx_cvt_package(old_max_bf16x2, old_max_bf160, old_max_bf161); - blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); + blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); blkv_bf16x2_fexp(scale, old_max_bf16x2); // opt1 union { __bf16x2 vec; uint32_t u32; } scale_u; @@ -364,7 +364,7 @@ void __vec__ pkg_rowmax_4src( __bf16x2 upd_max; __bf16 old_max_bf160, old_max_bf161; linx_cvt(old_max_bf160, old_max_ptr[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); + linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); linx_cvt_package(upd_max, old_max_bf160, old_max_bf161); // calc tile rowmax @@ -378,7 +378,7 @@ void __vec__ pkg_rowmax_4src( uint32_t src_idx_21 = (2*i + 1) * tileSrc::RowStride + (j + 2) * tileSrc::ColStride; uint32_t src_idx_30 = (2*i) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; uint32_t src_idx_31 = (2*i + 1) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; - + __bf16x2 s0_0, s0_1, s0_2, s0_3; linx_cvt_package(s0_0, src0_ptr[src_idx_00], src0_ptr[src_idx_01]); linx_cvt_package(s0_1, src0_ptr[src_idx_10], src0_ptr[src_idx_11]); @@ -434,7 +434,7 @@ void __vec__ pkg_rowmax_4src( // recalculate scale of softmax __bf16x2 scale, old_max_bf16x2; linx_cvt_package(old_max_bf16x2, old_max_bf160, old_max_bf161); - blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); + blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); blkv_bf16x2_fexp(scale, old_max_bf16x2); union { __bf16x2 vec; uint32_t u32; } scale_u; @@ -479,7 +479,7 @@ void __vec__ pkg_rowmax_4srcx2( __bf16x2 upd_max; __bf16 old_max_bf160, old_max_bf161; linx_cvt(old_max_bf160, old_max_ptr[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); + linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); linx_cvt_package(upd_max, old_max_bf160, old_max_bf161); // calc tile rowmax @@ -490,7 +490,7 @@ void __vec__ pkg_rowmax_4srcx2( uint32_t src_idx_10 = (2*i) * tileSrc::RowStride + (j + 1) * tileSrc::ColStride; uint32_t src_idx_20 = (2*i) * tileSrc::RowStride + (j + 2) * tileSrc::ColStride; uint32_t src_idx_30 = (2*i) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; - + __bf16x2 s0_0, s0_1; blkv_bf16x2_fmax(s0_0, src0_x2_ptr[src_idx_00], src0_x2_ptr[src_idx_10]); blkv_bf16x2_fmax(s0_1, src0_x2_ptr[src_idx_20], src0_x2_ptr[src_idx_30]); @@ -530,7 +530,7 @@ void __vec__ pkg_rowmax_4srcx2( // recalculate scale of softmax __bf16x2 scale, old_max_bf16x2; linx_cvt_package(old_max_bf16x2, old_max_bf160, old_max_bf161); - blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); + blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); blkv_bf16x2_fexp(scale, old_max_bf16x2); union { __bf16x2 vec; uint32_t u32; } scale_u; @@ -563,12 +563,12 @@ void __vec__ rowsum_2src_with_local_sum( linx_cvt_package(src_scale, 1.0f / sqrt((float)qD), 1.0f / sqrt((float)qD)); __bf16x2 upd_sum, new_max_val; __bf16 new_max_bf16_0, new_max_bf16_1; - + // Initialize local sum to 0 linx_cvt_package(upd_sum, 0.0f, 0.0f); linx_cvt(new_max_bf16_0, blkv_get_tile_ptr(new_max)[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); + linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); linx_cvt_package(new_max_val, new_max_bf16_0, new_max_bf16_1); #pragma clang loop unroll(full) @@ -581,7 +581,7 @@ void __vec__ rowsum_2src_with_local_sum( uint32_t src_idx_21 = (2*i + 1) * tileSrc::RowStride + (j + 2) * tileSrc::ColStride; uint32_t src_idx_30 = (2*i) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; uint32_t src_idx_31 = (2*i + 1) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; - + // Process src0 __bf16x2 s0_0, s0_1, s0_2, s0_3; __bf16x2 sum01_0, sum23_0, sum0123_0; @@ -589,7 +589,7 @@ void __vec__ rowsum_2src_with_local_sum( linx_cvt_package(s0_1, src0_ptr[src_idx_10], src0_ptr[src_idx_11]); linx_cvt_package(s0_2, src0_ptr[src_idx_20], src0_ptr[src_idx_21]); linx_cvt_package(s0_3, src0_ptr[src_idx_30], src0_ptr[src_idx_31]); - + blkv_bf16x2_fmsub(s0_0, s0_0, src_scale, new_max_val); blkv_bf16x2_fmsub(s0_1, s0_1, src_scale, new_max_val); blkv_bf16x2_fmsub(s0_2, s0_2, src_scale, new_max_val); @@ -602,7 +602,7 @@ void __vec__ rowsum_2src_with_local_sum( blkv_bf16x2_fadd(sum23_0, s0_2, s0_3); blkv_bf16x2_fadd(sum0123_0, sum01_0, sum23_0); blkv_bf16x2_fadd(upd_sum, upd_sum, sum0123_0); - + BLKC_ASSIGN_CAST(src_exp0, src_idx_00, s0_0); BLKC_ASSIGN_CAST(src_exp0, src_idx_10, s0_1); BLKC_ASSIGN_CAST(src_exp0, src_idx_20, s0_2); @@ -615,7 +615,7 @@ void __vec__ rowsum_2src_with_local_sum( linx_cvt_package(s1_1, src1_ptr[src_idx_10], src1_ptr[src_idx_11]); linx_cvt_package(s1_2, src1_ptr[src_idx_20], src1_ptr[src_idx_21]); linx_cvt_package(s1_3, src1_ptr[src_idx_30], src1_ptr[src_idx_31]); - + blkv_bf16x2_fmsub(s1_0, s1_0, src_scale, new_max_val); blkv_bf16x2_fmsub(s1_1, s1_1, src_scale, new_max_val); blkv_bf16x2_fmsub(s1_2, s1_2, src_scale, new_max_val); @@ -628,7 +628,7 @@ void __vec__ rowsum_2src_with_local_sum( blkv_bf16x2_fadd(sum23_1, s1_2, s1_3); blkv_bf16x2_fadd(sum0123_1, sum01_1, sum23_1); blkv_bf16x2_fadd(upd_sum, upd_sum, sum0123_1); - + BLKC_ASSIGN_CAST(src_exp1, src_idx_00, s1_0); BLKC_ASSIGN_CAST(src_exp1, src_idx_10, s1_1); BLKC_ASSIGN_CAST(src_exp1, src_idx_20, s1_2); @@ -639,7 +639,7 @@ void __vec__ rowsum_2src_with_local_sum( sum_u.vec = upd_sum; __bf16 sum0 = (sum_u.u32 >> 16) & 0xffff; __bf16 sum1 = (sum_u.u32 & 0xffff); - + local_sum_ptr[(i*2)*tileSum::RowStride] = sum0; local_sum_ptr[(i*2+1)*tileSum::RowStride] = sum1; } @@ -669,12 +669,12 @@ void __vec__ rowsum_2src_with_local_sumx2( linx_cvt_package(src_scale, 1.0f / sqrt((float)qD), 1.0f / sqrt((float)qD)); __bf16x2 upd_sum, new_max_val; __bf16 new_max_bf16_0, new_max_bf16_1; - + // Initialize local sum to 0 linx_cvt_package(upd_sum, 0.0f, 0.0f); linx_cvt(new_max_bf16_0, blkv_get_tile_ptr(new_max)[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); + linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); linx_cvt_package(new_max_val, new_max_bf16_0, new_max_bf16_1); // row sum @@ -685,17 +685,17 @@ void __vec__ rowsum_2src_with_local_sumx2( uint32_t src_idx_10 = (2*i) * tileSrc::RowStride + (j + 1) * tileSrc::ColStride; uint32_t src_idx_20 = (2*i) * tileSrc::RowStride + (j + 2) * tileSrc::ColStride; uint32_t src_idx_30 = (2*i) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; - + // Process src0 __bf16x2 s0_0, s0_1, s0_2, s0_3; __bf16x2 sum01_0, sum23_0, sum0123_0; - + // 直接将内存读取作为 fmsub 的输入操作数 blkv_bf16x2_fmsub(s0_0, src0_x2_ptr[src_idx_00], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_1, src0_x2_ptr[src_idx_10], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_2, src0_x2_ptr[src_idx_20], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_3, src0_x2_ptr[src_idx_30], src_scale, new_max_val); - + blkv_bf16x2_fexp(s0_0, s0_0); blkv_bf16x2_fexp(s0_1, s0_1); blkv_bf16x2_fexp(s0_2, s0_2); @@ -712,12 +712,12 @@ void __vec__ rowsum_2src_with_local_sumx2( // Process src1 __bf16x2 s1_0, s1_1, s1_2, s1_3; __bf16x2 sum01_1, sum23_1, sum0123_1; - + blkv_bf16x2_fmsub(s1_0, src1_x2_ptr[src_idx_00], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_1, src1_x2_ptr[src_idx_10], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_2, src1_x2_ptr[src_idx_20], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_3, src1_x2_ptr[src_idx_30], src_scale, new_max_val); - + blkv_bf16x2_fexp(s1_0, s1_0); blkv_bf16x2_fexp(s1_1, s1_1); blkv_bf16x2_fexp(s1_2, s1_2); @@ -726,7 +726,7 @@ void __vec__ rowsum_2src_with_local_sumx2( blkv_bf16x2_fadd(sum23_1, s1_2, s1_3); blkv_bf16x2_fadd(sum0123_1, sum01_1, sum23_1); blkv_bf16x2_fadd(upd_sum, upd_sum, sum0123_1); - + blkc_assign_elem(src_exp1_x2_ptr, src_idx_00, s1_0); blkc_assign_elem(src_exp1_x2_ptr, src_idx_10, s1_1); blkc_assign_elem(src_exp1_x2_ptr, src_idx_20, s1_2); @@ -737,7 +737,7 @@ void __vec__ rowsum_2src_with_local_sumx2( sum_u.vec = upd_sum; __bf16 sum0 = (sum_u.u32 >> 16) & 0xffff; __bf16 sum1 = (sum_u.u32 & 0xffff); - + local_sum_ptr[(i*2)*tileSum::RowStride] = sum0; local_sum_ptr[(i*2+1)*tileSum::RowStride] = sum1; } @@ -767,12 +767,12 @@ void __vec__ rowsum_2src_with_local_expx2( linx_cvt_package(src_scale, 1.0f / sqrt((float)qD), 1.0f / sqrt((float)qD)); __bf16x2 upd_sum, new_max_val; __bf16 new_max_bf16_0, new_max_bf16_1; - + // Initialize local sum to 0 linx_cvt_package(upd_sum, 0.0f, 0.0f); linx_cvt(new_max_bf16_0, blkv_get_tile_ptr(new_max)[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); + linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); linx_cvt_package(new_max_val, new_max_bf16_0, new_max_bf16_1); // row sum @@ -783,17 +783,17 @@ void __vec__ rowsum_2src_with_local_expx2( uint32_t src_idx_10 = (2*i) * tileSrc::RowStride + (j + 1) * tileSrc::ColStride; uint32_t src_idx_20 = (2*i) * tileSrc::RowStride + (j + 2) * tileSrc::ColStride; uint32_t src_idx_30 = (2*i) * tileSrc::RowStride + (j + 3) * tileSrc::ColStride; - + // Process src0 __bf16x2 s0_0, s0_1, s0_2, s0_3; __bf16x2 sum01_0, sum23_0, sum0123_0; - + // 直接将内存读取作为 fmsub 的输入操作数 blkv_bf16x2_fmsub(s0_0, src0_x2_ptr[src_idx_00], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_1, src0_x2_ptr[src_idx_10], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_2, src0_x2_ptr[src_idx_20], src_scale, new_max_val); blkv_bf16x2_fmsub(s0_3, src0_x2_ptr[src_idx_30], src_scale, new_max_val); - + blkv_bf16x2_fexp(s0_0, s0_0); blkv_bf16x2_fexp(s0_1, s0_1); blkv_bf16x2_fexp(s0_2, s0_2); @@ -810,12 +810,12 @@ void __vec__ rowsum_2src_with_local_expx2( // Process src1 __bf16x2 s1_0, s1_1, s1_2, s1_3; __bf16x2 sum01_1, sum23_1, sum0123_1; - + blkv_bf16x2_fmsub(s1_0, src1_x2_ptr[src_idx_00], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_1, src1_x2_ptr[src_idx_10], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_2, src1_x2_ptr[src_idx_20], src_scale, new_max_val); blkv_bf16x2_fmsub(s1_3, src1_x2_ptr[src_idx_30], src_scale, new_max_val); - + blkv_bf16x2_fexp(s1_0, s1_0); blkv_bf16x2_fexp(s1_1, s1_1); blkv_bf16x2_fexp(s1_2, s1_2); @@ -824,7 +824,7 @@ void __vec__ rowsum_2src_with_local_expx2( // blkv_bf16x2_fadd(sum23_1, s1_2, s1_3); // blkv_bf16x2_fadd(sum0123_1, sum01_1, sum23_1); // blkv_bf16x2_fadd(upd_sum, upd_sum, sum0123_1); - + blkc_assign_elem(src_exp1_x2_ptr, src_idx_00, s1_0); blkc_assign_elem(src_exp1_x2_ptr, src_idx_10, s1_1); blkc_assign_elem(src_exp1_x2_ptr, src_idx_20, s1_2); @@ -835,7 +835,7 @@ void __vec__ rowsum_2src_with_local_expx2( // sum_u.vec = upd_sum; // __bf16 sum0 = (sum_u.u32 >> 16) & 0xffff; // __bf16 sum1 = (sum_u.u32 & 0xffff); - + // local_sum_ptr[(i*2)*tileSum::RowStride] = sum0; // local_sum_ptr[(i*2+1)*tileSum::RowStride] = sum1; } @@ -889,14 +889,14 @@ void __vec__ rowsum( __bf16 new_max_bf16_0, new_max_bf16_1; // old_sum * rescale + new_sum linx_cvt(old_sum_bf16_0, old_sum_ptr[i*2*tileSum::RowStride]); //float->bf16 - linx_cvt(old_sum_bf16_1, old_sum_ptr[(i*2+1)*tileSum::RowStride]); + linx_cvt(old_sum_bf16_1, old_sum_ptr[(i*2+1)*tileSum::RowStride]); linx_cvt_package(upd_sum, old_sum_bf16_0, old_sum_bf16_1); linx_cvt(scale_bf16_0, scale_ptr[i*2*tileSum::RowStride]); //float->bf16 - linx_cvt(scale_bf16_1, scale_ptr[(i*2+1)*tileSum::RowStride]); + linx_cvt(scale_bf16_1, scale_ptr[(i*2+1)*tileSum::RowStride]); linx_cvt_package(scale, scale_bf16_0, scale_bf16_1); - blkv_bf16x2_fmul(upd_sum, upd_sum, scale); + blkv_bf16x2_fmul(upd_sum, upd_sum, scale); linx_cvt(new_max_bf16_0, blkv_get_tile_ptr(new_max)[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); + linx_cvt(new_max_bf16_1, blkv_get_tile_ptr(new_max)[(i*2+1)*tileMax::RowStride]); linx_cvt_package(new_max_val, new_max_bf16_0, new_max_bf16_1); // calculate row sum of softmax, l_i @@ -951,7 +951,7 @@ void __vec__ rowsum( sum_u.vec = upd_sum; __bf16 sum0 = (sum_u.u32 >> 16) & 0xffff; __bf16 sum1 = (sum_u.u32 & 0xffff); - + new_sum_ptr[(i*2)*tileSum::RowStride] = sum0; new_sum_ptr[(i*2+1)*tileSum::RowStride] = sum1; } @@ -986,7 +986,7 @@ void __vec__ flashsoftmax_dn_mout_cast_kernel_bf16x2( __bf16x2 upd_max; __bf16 old_max_bf160, old_max_bf161; linx_cvt(old_max_bf160, old_max_ptr[i*2*tileMax::RowStride]); //float->bf16 - linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); + linx_cvt(old_max_bf161, old_max_ptr[(i*2+1)*tileMax::RowStride]); linx_cvt_package(upd_max, old_max_bf160, old_max_bf161); // calc tile rowmax @@ -1043,7 +1043,7 @@ void __vec__ flashsoftmax_dn_mout_cast_kernel_bf16x2( // recalculate scale of softmax __bf16x2 scale, old_max_bf16x2; linx_cvt_package(old_max_bf16x2, old_max_bf160, old_max_bf161); - blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); + blkv_bf16x2_fsub(old_max_bf16x2, old_max_bf16x2, upd_max); blkv_bf16x2_fexp(scale, old_max_bf16x2); uint32_t scale_idx00 = i*2*tileScale::RowStride; uint32_t scale_idx01 = (i*2+1)*tileScale::RowStride; @@ -1063,7 +1063,7 @@ void __vec__ flashsoftmax_dn_mout_cast_kernel_bf16x2( __bf16x2 upd_sum; __bf16 old_sum_bf16_0, old_sum_bf16_1; linx_cvt(old_sum_bf16_0, old_sum_ptr[i*2*tileSum::RowStride]); //float->bf16 - linx_cvt(old_sum_bf16_1, old_sum_ptr[(i*2+1)*tileSum::RowStride]); + linx_cvt(old_sum_bf16_1, old_sum_ptr[(i*2+1)*tileSum::RowStride]); linx_cvt_package(upd_sum, old_sum_bf16_0, old_sum_bf16_1); blkv_bf16x2_fmul(upd_sum, upd_sum, scale); // *** TODO @@ -1207,9 +1207,9 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -1277,7 +1277,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); MGATHER(tQMX[x], gQMX, nd2zz_offset); - // TCOPYIN(tQMX[x], gQMX); + // TLOAD(tQMX[x], gQMX); } #endif @@ -1329,10 +1329,10 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - MGATHER(tKMX[y], gKMX, nd2nn_offset); - // TCOPYIN(tKMX[y], gKM); + MGATHER(tKMX[y], gKMX, nd2nn_offset); + // TLOAD(tKMX[y], gKM); } #endif @@ -1371,7 +1371,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -1382,7 +1382,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4_bf16x2<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); @@ -1421,8 +1421,8 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); @@ -1431,7 +1431,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // // tNewMax[x].data(), // // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -1449,7 +1449,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -1469,7 +1469,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -1511,10 +1511,10 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - MGATHER(tVMX[y], gVMX, nd2nn_offset); - // TCOPYIN(tVMX[y], gVMX); + MGATHER(tVMX[y], gVMX, nd2nn_offset); + // TLOAD(tVMX[y], gVMX); } #endif @@ -1585,7 +1585,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif @@ -1600,9 +1600,9 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // using gmK = global_tensor>; // K: [qD×S] // using gmV = global_tensor>; // V: [S×vD] // using gmO = global_tensor>; // O: [SxvD] -// using gm_QMX = global_tensor>; -// using gm_KMX = global_tensor>; -// using gm_VMX = global_tensor>; +// using gm_QMX = global_tensor>; +// using gm_KMX = global_tensor>; +// using gm_VMX = global_tensor>; // // tile 寄存器形状 // using tileQ = TileLeft; // [kTm×qD] // using tileK = TileRight; // [vD×kTk] @@ -1670,7 +1670,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // tileQ tQ[Xdim]; // tile_QMX tQMX[Xdim]; // // load tile Q, TODO: add ND2ZZ transform for QMX -// #ifdef MULTI_LDST // don't use, no need for multi tload/tstore +// #ifdef MULTI_LDST // don't use, no need for multi tload/tstore // #pragma clang loop unroll(full) // for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); +// TLOAD(tQ[x], gQ); +// gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); -// // TCOPYIN(tQMX[x], gQMX); +// // TLOAD(tQMX[x], gQMX); // } // #endif @@ -1722,10 +1722,10 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); -// MGATHER(tKMX[y], gKMX, nd2nn_offset); -// // TCOPYIN(tKMX[y], gKM); +// MGATHER(tKMX[y], gKMX, nd2nn_offset); +// // TLOAD(tKMX[y], gKM); // } // #endif @@ -1764,7 +1764,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // tNewMax[x].data(), // tNewSum[x].data(), // tExpW[x][0].data(), -// tW[x][0].data(), // +// tW[x][0].data(), // // tMax[x].data(), // tSum[x].data()); // tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -1775,7 +1775,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // // tNewMax[x].data(), // // tNewSum[x].data(), // // tExpW[x][0].data(), -// // tW[x][0].data(), // +// // tW[x][0].data(), // // // tMax[x].data(), // // tSum[x].data()); @@ -1820,8 +1820,8 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // #pragma clang loop unroll(full) // for(int x=0;x<<>>( -// tScale[x].data(), -// tNewMax[x].data(), +// tScale[x].data(), +// tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); @@ -1830,7 +1830,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // // tNewMax[x].data(), // // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -1848,7 +1848,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // tileSum tLocalSum[Xdim][4]; // #pragma clang loop unroll(full) -// for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -1868,7 +1868,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // tileSum tLocalSum[Xdim][4]; // #pragma clang loop unroll(full) -// for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); // } @@ -1910,10 +1910,10 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); -// MGATHER(tVMX[y], gVMX, nd2nn_offset); -// // TCOPYIN(tVMX[y], gVMX); +// MGATHER(tVMX[y], gVMX, nd2nn_offset); +// // TLOAD(tVMX[y], gVMX); // } // #endif @@ -1984,7 +1984,7 @@ void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, // #pragma clang loop unroll(full) // for (int x = 0; x < Xdim; ++x) { // auto dstO = gIterO(i+x, 0); -// TCOPYOUT(dstO, tO_cast[x]);//TMOV +// TSTORE(dstO, tO_cast[x]);//TMOV // } // #endif @@ -2000,9 +2000,9 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -2070,7 +2070,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + // gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); - TCOPYIN(tQMX[x], gQMX); + TLOAD(tQMX[x], gQMX); } #endif @@ -2122,10 +2122,10 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - // MGATHER(tKMX[y], gKMX, nd2nn_offset); - TCOPYIN(tKMX[y], gKM); + // MGATHER(tKMX[y], gKMX, nd2nn_offset); + TLOAD(tKMX[y], gKM); } #endif @@ -2163,7 +2163,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -2175,7 +2175,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4_bf16x2<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); @@ -2214,8 +2214,8 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); @@ -2224,7 +2224,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype // // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // // tNewMax[x].data(), // // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -2242,7 +2242,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -2262,7 +2262,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -2304,10 +2304,10 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - // MGATHER(tVMX[y], gVMX, nd2nn_offset); + // MGATHER(tVMX[y], gVMX, nd2nn_offset); } #endif @@ -2379,7 +2379,7 @@ void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif @@ -2394,9 +2394,9 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -2465,7 +2465,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + // gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); - TCOPYIN(tQMX[x], gQMX); + TLOAD(tQMX[x], gQMX); } #endif @@ -2517,10 +2517,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - // MGATHER(tKMX[y], gKMX, nd2nn_offset); - TCOPYIN(tKMX[y], gKM); + // MGATHER(tKMX[y], gKMX, nd2nn_offset); + TLOAD(tKMX[y], gKM); } #endif @@ -2559,7 +2559,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -2571,7 +2571,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty // tNewMax[x].data(), // tNewSum[x].data(), // tExpW[x][0].data(), - // tW[x][0].data(), // + // tW[x][0].data(), // // tMax[x].data(), // tSum[x].data()); // bf16tobf16x2(); @@ -2605,13 +2605,13 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -2623,8 +2623,8 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data()); @@ -2634,7 +2634,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty rowsum_2src_with_local_sum<<>>( tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), tW[x][2].data(), tW[x][3].data(), tNewMax[x].data()); - + new_sum_of_2_loc_sum_bf16x2<<>>( tNewSum[x].data(), tLocalSum[x][0].data(), tLocalSum[x][1].data(), tSum[x].data(), tScale[x].data()); @@ -2648,7 +2648,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -2668,7 +2668,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -2710,10 +2710,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - // MGATHER(tVMX[y], gVMX, nd2nn_offset); + // MGATHER(tVMX[y], gVMX, nd2nn_offset); } #endif @@ -2785,7 +2785,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, dty #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif @@ -2800,9 +2800,9 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -2871,7 +2871,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + // gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); - TCOPYIN(tQMX[x], gQMX); + TLOAD(tQMX[x], gQMX); } #endif @@ -2923,10 +2923,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - // MGATHER(tKMX[y], gKMX, nd2nn_offset); - TCOPYIN(tKMX[y], gKM); + // MGATHER(tKMX[y], gKMX, nd2nn_offset); + TLOAD(tKMX[y], gKM); } #endif @@ -2965,7 +2965,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -2977,7 +2977,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p // tNewMax[x].data(), // tNewSum[x].data(), // tExpW[x][0].data(), - // tW[x][0].data(), // + // tW[x][0].data(), // // tMax[x].data(), // tSum[x].data()); // bf16tobf16x2(); @@ -3011,13 +3011,13 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -3029,8 +3029,8 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data()); @@ -3040,7 +3040,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p rowsum_2src_with_local_sumx2<<>>( tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), tW[x][2].data(), tW[x][3].data(), tNewMax[x].data()); - + new_sum_of_2_loc_sum_bf16x2<<>>( tNewSum[x].data(), tLocalSum[x][0].data(), tLocalSum[x][1].data(), tSum[x].data(), tScale[x].data()); @@ -3054,7 +3054,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -3074,7 +3074,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -3116,10 +3116,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - // MGATHER(tVMX[y], gVMX, nd2nn_offset); + // MGATHER(tVMX[y], gVMX, nd2nn_offset); } #endif @@ -3191,7 +3191,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_p #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif @@ -3224,9 +3224,9 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -3302,7 +3302,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + // gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); - TCOPYIN(tQMX[x], gQMX); + TLOAD(tQMX[x], gQMX); } #endif @@ -3354,10 +3354,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - // MGATHER(tKMX[y], gKMX, nd2nn_offset); - TCOPYIN(tKMX[y], gKM); + // MGATHER(tKMX[y], gKMX, nd2nn_offset); + TLOAD(tKMX[y], gKM); } #endif @@ -3399,7 +3399,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -3411,7 +3411,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype // tNewMax[x].data(), // tNewSum[x].data(), // tExpW[x][0].data(), - // tW[x][0].data(), // + // tW[x][0].data(), // // tMax[x].data(), // tSum[x].data()); // bf16tobf16x2(); @@ -3445,13 +3445,13 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -3464,12 +3464,12 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data()); - + rowsum_2src_with_local_expx2<<>>( tExpW[x][0].data(), tExpW[x][1].data(), tW[x][0].data(), tW[x][1].data(), tNewMax[x].data()); @@ -3502,7 +3502,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -3522,7 +3522,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -3564,10 +3564,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - // MGATHER(tVMX[y], gVMX, nd2nn_offset); + // MGATHER(tVMX[y], gVMX, nd2nn_offset); } #endif @@ -3639,7 +3639,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif } @@ -3654,9 +3654,9 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp using gmK = global_tensor>; // K: [qD×S] using gmV = global_tensor>; // V: [S×vD] using gmO = global_tensor>; // O: [SxvD] - using gm_QMX = global_tensor>; - using gm_KMX = global_tensor>; - using gm_VMX = global_tensor>; + using gm_QMX = global_tensor>; + using gm_KMX = global_tensor>; + using gm_VMX = global_tensor>; // tile 寄存器形状 using tileQ = TileLeft; // [kTm×qD] using tileK = TileRight; // [vD×kTk] @@ -3739,7 +3739,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp tileQ tQ[Xdim]; tile_QMX tQMX[Xdim]; // load tile Q, TODO: add ND2ZZ transform for QMX - #ifdef MULTI_LDST // don't use, no need for multi tload/tstore + #ifdef MULTI_LDST // don't use, no need for multi tload/tstore #pragma clang loop unroll(full) for(int x=0;x(gQMX, tQMX[x], nd2zz_offset, i+x, 0); + TLOAD(tQ[x], gQ); + // gen_ND2ZZ_offset_Impl(gQMX, tQMX[x], nd2zz_offset, i+x, 0); // MGATHER(tQMX[x], gQMX, nd2zz_offset); - TCOPYIN(tQMX[x], gQMX); + TLOAD(tQMX[x], gQMX); } #endif @@ -3791,10 +3791,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp for(int y=0;y(gKMX, tKMX[y], nd2nn_offset, 0, j+y); - // MGATHER(tKMX[y], gKMX, nd2nn_offset); - TCOPYIN(tKMX[y], gKM); + // MGATHER(tKMX[y], gKMX, nd2nn_offset); + TLOAD(tKMX[y], gKM); } #endif @@ -3836,7 +3836,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp tNewMax[x].data(), tNewSum[x].data(), tExpW[x][0].data(), - tW[x][0].data(), // + tW[x][0].data(), // tMax[x].data(), tSum[x].data()); tohif4<<>>(tP_hif4[x][0].data(), tP_scale[x][0].data(), tExpW[x][0].data()); // 64 = 1 group, tExpW(Zn, bf16), tP_scale(ZZ, E6M2 with zero E1_8 && E1_16) @@ -3848,7 +3848,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp // tNewMax[x].data(), // tNewSum[x].data(), // tExpW[x][0].data(), - // tW[x][0].data(), // + // tW[x][0].data(), // // tMax[x].data(), // tSum[x].data()); // bf16tobf16x2(); @@ -3882,13 +3882,13 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp // #pragma clang loop unroll(full) // for(int x=0;x<<>>( - // tScale[x].data(), - // tNewMax[x].data(), + // tScale[x].data(), + // tNewMax[x].data(), // tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), // tMax[x].data(), // scale); - + // src_exp_2src_with_local_sum<<>>(tLocalSum[x][0].data(), tExpW[x][0].data(), tExpW[x][1].data(), // tW[x][0].data(), tW[x][1].data(), tNewMax[x].data(), scale); // src_exp_2src_with_local_sum<<>>(tLocalSum[x][1].data(), tExpW[x][2].data(), tExpW[x][3].data(), @@ -3901,12 +3901,12 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data()); - + rowsum_2src_with_local_expx2<<>>( tExpW[x][0].data(), tExpW[x][1].data(), tW[x][0].data(), tW[x][1].data(), tNewMax[x].data()); @@ -3939,7 +3939,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); @@ -3959,7 +3959,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp tileSum tLocalSum[Xdim][4]; #pragma clang loop unroll(full) - for(int x=0;x<<>>(tLocalMax[x][k].data(), tW[x][4*k].data(), tW[x][4*k+1].data(), tW[x][4*k+2].data(), tW[x][4*k+3].data(), scale); } @@ -4001,10 +4001,10 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp for(int y=0;y(gVMX, tVMX[y], nd2nn_offset, j+y, 0); - // MGATHER(tVMX[y], gVMX, nd2nn_offset); + // MGATHER(tVMX[y], gVMX, nd2nn_offset); } #endif @@ -4084,7 +4084,7 @@ void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtyp #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]);//TMOV + TSTORE(dstO, tO_cast[x]);//TMOV } #endif } diff --git a/benchmark/two-level-arch/kernels/fa/fa_unalign_2d_unroll.hpp b/benchmark/two-level-arch/kernels/fa/fa_unalign_2d_unroll.hpp index c0250ca..f451c58 100644 --- a/benchmark/two-level-arch/kernels/fa/fa_unalign_2d_unroll.hpp +++ b/benchmark/two-level-arch/kernels/fa/fa_unalign_2d_unroll.hpp @@ -41,7 +41,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt using tileW_out = TileAcc; // [kTm×kTk] using tileW = Tile; using tileW_cast = Tile; - using tileW_left = TileLeft; + using tileW_left = TileLeft; using tileO_out = TileAcc; using tileO = Tile; // [kTm×vD] @@ -100,7 +100,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt #pragma clang loop unroll(full) for(int x=0;x<<>>( - tScale[x].data(), - tNewMax[x].data(), + tScale[x].data(), + tNewMax[x].data(), tW[x][0].data(), tW[x][1].data(), tW[x][2].data(), tW[x][3].data(), tMax[x].data(), scale); @@ -216,7 +216,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt #pragma clang loop unroll(full) for(int y=0;y Nz @@ -261,7 +261,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt tileK_tcols tK; auto gK = gIterK(0, Kb); - TCOPYIN(tK, gK); + TLOAD(tK, gK); tileW_tcols tW[Xdim]; #pragma clang loop unroll(full) @@ -291,7 +291,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt tileV_trows tV; auto gV = gIterV(Kb, 0); - TCOPYIN(tV, gV); + TLOAD(tV, gV); // 计算当前块的加权输出: O_j = W * V tileW_left_tcols tW_left[Xdim]; @@ -325,7 +325,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt #pragma clang loop unroll(full) for (int x = 0; x < Xdim; ++x) { auto dstO = gIterO(i+x, 0); - TCOPYOUT(dstO, tO_cast[x]); + TSTORE(dstO, tO_cast[x]); } } @@ -334,7 +334,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt tileQ_trows tQ; auto gQ = gIterQ(Qb,0); - TCOPYIN(tQ, gQ); + TLOAD(tQ, gQ); tileMax_trows tMax; TEXPANDSCALAR(tMax, -1e30f); @@ -354,7 +354,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt #pragma clang loop unroll(full) for(int y=0;y<<>>( - tScale.data(), - tNewMax.data(), + tScale.data(), + tNewMax.data(), tW[0].data(), tW[1].data(), tW[2].data(), tW[3].data(), tMax.data(), scale); @@ -432,7 +432,7 @@ void flash_attention_unalign_2d_unroll(dtype* out_ptr, dtype* q_ptr, dtype* k_pt #pragma clang loop unroll(full) for(int y=0;y<<>>(tO_cast.data(), tO.data(), tSum.data()); auto dstO = gIterO(Qb, 0); - TCOPYOUT(dstO, tO_cast); + TSTORE(dstO, tO_cast); } } \ No newline at end of file diff --git a/benchmark/two-level-arch/kernels/gather/gather.hpp b/benchmark/two-level-arch/kernels/gather/gather.hpp index 0222852..03a1069 100644 --- a/benchmark/two-level-arch/kernels/gather/gather.hpp +++ b/benchmark/two-level-arch/kernels/gather/gather.hpp @@ -10,7 +10,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -27,7 +27,7 @@ GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -40,7 +40,7 @@ template void __vec__ gen_offset( typename tile_shapeInOffset::TileDType __in__ in, // inOffset - typename tile_shapeOffset::TileDType __out__ out, // + typename tile_shapeOffset::TileDType __out__ out, // const size_t n_base ) { size_t data_width = sizeof(dtype); @@ -64,7 +64,7 @@ void gen_offset_impl( ) { static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, "Only static shape supported"); - + gen_offset<<>>( in_offset.data(), @@ -86,25 +86,25 @@ void gather( using gm_shapeInOffset = global_tensor>; using gm_shapeIn = global_tensor>; using gm_shapeOut = global_tensor>; - - using tile_shapeInData = Tile; + + using tile_shapeInData = Tile; using itIn = global_iterator; tile_shapeInData inTile; itIn gInIter(in_data_ptr); - using tile_shapeInOffset = Tile; - using tile_shapeData = Tile; + using tile_shapeInOffset = Tile; + using tile_shapeData = Tile; using tile_shapeOffset = Tile; - using tile_shapeInOffset_rmd_n = Tile; + using tile_shapeInOffset_rmd_n = Tile; using tile_shapeData_rmd_n = Tile; using tile_shapeOffset_rmd_n = Tile; - using tile_shapeInOffset_rmd_mn = Tile; + using tile_shapeInOffset_rmd_mn = Tile; using tile_shapeData_rmd_mn = Tile; using tile_shapeOffset_rmd_mn = Tile; - using tile_shapeInOffset_rmd_m = Tile; + using tile_shapeInOffset_rmd_m = Tile; using tile_shapeData_rmd_m = Tile; using tile_shapeOffset_rmd_m = Tile; @@ -117,11 +117,11 @@ void gather( tile_shapeInOffset_rmd_n inOffsetTile_rmd_n; tile_shapeData_rmd_n outTile_rmd_n; tile_shapeOffset_rmd_n offsetTile_rmd_n; - + tile_shapeInOffset_rmd_mn inOffsetTile_rmd_mn; tile_shapeData_rmd_mn outTile_rmd_mn; tile_shapeOffset_rmd_mn offsetTile_rmd_mn; - + tile_shapeInOffset_rmd_m inOffsetTile_rmd_m; tile_shapeData_rmd_m outTile_rmd_m; tile_shapeOffset_rmd_m offsetTile_rmd_m; @@ -140,24 +140,24 @@ void gather( // /////////////////////////////////////// size_t n_base = 0; - + // #pragma clang loop unroll(full) for (int j = 0; j < Mb; ++j) { printf("j = %d\n", j); for (int i = 0; i < Nb; ++i) { auto gInOffset = gInOffsetIter(0, j); auto gO = gOIter(j, i); - TCOPYIN(inOffsetTile, gInOffset); + TLOAD(inOffsetTile, gInOffset); // test // auto gIn = gInIter(j, i); - // TCOPYIN(inTile, gIn); + // TLOAD(inTile, gIn); n_base = i * tN; // printf("j = %d\n", j); // printf("i = %d\n", i); // printf("base = %d\n", base); // printf("in_shape[0] = %d\n", in_shape[0]); gen_offset_impl(inOffsetTile, offsetTile, n_base); - + MGATHER(outTile, inGm, offsetTile); // printf("inGm = %d\n", inGm); @@ -165,19 +165,19 @@ void gather( // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tN); // DUMP_TILE_FLOAT("inTile", inTile, g_dump_outdata, 1, tN); // DUMP_TILE_FLOAT("outTile", outTile, g_dump_outdata, 1, tN); - TCOPYOUT(gO, outTile); + TSTORE(gO, outTile); } if constexpr (rmd_N) { auto gInOffset = gInOffsetIter(0, j); auto gO = gOIter(j, Nb); n_base = Nb * tN; - TCOPYIN(inOffsetTile_rmd_n, gInOffset); + TLOAD(inOffsetTile_rmd_n, gInOffset); gen_offset_impl(inOffsetTile_rmd_n, offsetTile_rmd_n, n_base); MGATHER(outTile_rmd_n, inGm, offsetTile_rmd_n); // DUMP_TILE("inOffsetTile_rmd_n", inOffsetTile_rmd_n, g_dump_inoffset, 1, rmd_N); // DUMP_TILE_FLOAT("outTile_rmd_n", outTile_rmd_n, g_dump_outdata, 20, rmd_N); // DUMP_TILE("offsetTile_rmd_n", offsetTile_rmd_n, g_dump, 20, rmd_N); - TCOPYOUT(gO, outTile_rmd_n); + TSTORE(gO, outTile_rmd_n); } } if constexpr (rmd_M) { @@ -185,19 +185,19 @@ void gather( auto gInOffset = gInOffsetIter(0, Mb); auto gO = gOIter(Mb, i); n_base = i * tN; - TCOPYIN(inOffsetTile_rmd_m, gInOffset); + TLOAD(inOffsetTile_rmd_m, gInOffset); gen_offset_impl(inOffsetTile_rmd_m, offsetTile_rmd_m, n_base); MGATHER(outTile_rmd_m, inGm, offsetTile_rmd_m); - TCOPYOUT(gO, outTile_rmd_m); + TSTORE(gO, outTile_rmd_m); } if constexpr (rmd_N) { auto gInOffset = gInOffsetIter(0, Mb); auto gO = gOIter(Mb, Nb); n_base = Nb * tN; - TCOPYIN(inOffsetTile_rmd_mn, gInOffset); + TLOAD(inOffsetTile_rmd_mn, gInOffset); gen_offset_impl(inOffsetTile_rmd_mn, offsetTile_rmd_mn, n_base); MGATHER(outTile_rmd_mn, inGm, offsetTile_rmd_mn); - TCOPYOUT(gO, outTile_rmd_mn); + TSTORE(gO, outTile_rmd_mn); } } diff --git a/benchmark/two-level-arch/kernels/matmul/matmul.hpp b/benchmark/two-level-arch/kernels/matmul/matmul.hpp index a3671c7..afa6384 100644 --- a/benchmark/two-level-arch/kernels/matmul/matmul.hpp +++ b/benchmark/two-level-arch/kernels/matmul/matmul.hpp @@ -10,19 +10,19 @@ using namespace pto; template -void TCOPYOUT_ACC(GmOut &Gout, TileAcc &tAcc){ +void TSTORE_ACC(GmOut &Gout, TileAcc &tAcc){ using TileAccOut = Tile; TileAccOut tAccOut; TCVT(tAccOut, tAcc); - TCOPYOUT(Gout, tAccOut); + TSTORE(Gout, tAccOut); } template -void TCOPYOUT_ACC_DYNAMIC(GmOut &Gout, TileAcc &tAcc, size_t valid_row, size_t valid_col){ +void TSTORE_ACC_DYNAMIC(GmOut &Gout, TileAcc &tAcc, size_t valid_row, size_t valid_col){ using TileAccOut = Tile; TileAccOut tAccOut(valid_row, valid_col); TCVT(tAccOut, tAcc); - TCOPYOUT(Gout, tAccOut); + TSTORE(Gout, tAccOut); } // A * B -> C with any shape @@ -74,16 +74,16 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeACC tACC; // tile_shapecast tcast; - + if constexpr(Kb>0){ auto gA = gAIter(i, 0); auto gB = gBIter(0, j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD(tA, gA); + TLOAD(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -92,8 +92,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } @@ -103,8 +103,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { @@ -112,8 +112,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { } } // TCVT(tCast, tACC); - // TCOPYOUT(gC, tCast); - TCOPYOUT_ACC(gC, tACC); + // TSTORE(gC, tCast); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -125,9 +125,9 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD(tA, gA); + TLOAD(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -136,8 +136,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -146,15 +146,15 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -168,9 +168,9 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD(tA, gA); + TLOAD(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -179,8 +179,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -189,15 +189,15 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -209,9 +209,9 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD(tA, gA); + TLOAD(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -220,8 +220,8 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -230,15 +230,15 @@ void matmul_mask(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -275,8 +275,8 @@ void matmul_frac(float* dst, dtype* src0, dtype* src1){ tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) @@ -285,11 +285,11 @@ void matmul_frac(float* dst, dtype* src0, dtype* src1){ auto gB = gBIter(k,j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -406,7 +406,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ // #pragma clang loop unroll(full) // for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -460,7 +460,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } auto gC = gIterC(i*R.m+ii,j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [m, rmd_N, k] @@ -471,7 +471,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ for(int k=0;k Kb); if constexpr(R.k < Kb){ - + for(int k=R.k;k0){ MATMACC(tACC, tA, tB); } else { @@ -510,7 +510,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } auto gC = gIterC(i*R.m+ii,Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -518,7 +518,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ if constexpr(rM>0){ tile_shapeA tA[rM][R.k]; - + #pragma clang loop unroll(full) for(int i=0;i0){ MATMACC(tACC, tA, tB); } else { @@ -574,7 +574,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(i+dM*R.m,j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rM, rmd_N, k] @@ -585,7 +585,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -622,7 +622,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(i+dM*R.m,Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -630,11 +630,11 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ // [rmd_M, n, k] if constexpr (rmd_M) { tile_shapeA_tcols tA[R.k]; - + #pragma clang loop unroll(full) for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -682,7 +682,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(Mb,j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rmd_M, rmd_N, k] @@ -693,7 +693,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -730,7 +730,7 @@ void matmul_mask_reuseA(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(Mb,Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } }// Batch @@ -802,7 +802,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < R.k; k++) { auto gA = gIterA(row, k); - TCOPYIN(tA_phase0[k], gA); + TLOAD(tA_phase0[k], gA); } // --- N 主列 --- @@ -813,12 +813,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB tB; auto gB = gIterB(k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // --- N 余列 (rmd_N) --- @@ -828,12 +828,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB_trows tB; auto gB = gIterB(k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // Phase B-1: 剩余 K 轴 Full chunks (每块 MAX_TILE_NUM 个 k tile) @@ -847,7 +847,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < MAX_TILE_NUM; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA_chunk[k], gA); + TLOAD(tA_chunk[k], gA); } // --- N 主列 --- @@ -858,12 +858,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // --- N 余列 --- @@ -873,12 +873,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -891,7 +891,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < K2_rem; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA_tail[k], gA); + TLOAD(tA_tail[k], gA); } // --- N 主列 --- @@ -902,12 +902,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // --- N 余列 --- @@ -917,12 +917,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -930,7 +930,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ if constexpr (rmd_K) { tile_shapeA_trows tA_rmdK; auto gA = gIterA(row, Kb); - TCOPYIN(tA_rmdK, gA); + TLOAD(tA_rmdK, gA); // --- N 主列 --- #pragma clang loop unroll(full) @@ -938,11 +938,11 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ tile_shapeACC tACC; tile_shapeB_tcols tB; auto gB = gIterB(Kb, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // --- N 余列 --- @@ -950,11 +950,11 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ tile_shapeC_trows tACC; tile_shapeB_tcorner tB; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -974,7 +974,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < R.k; k++) { auto gA = gIterA(row, k); - TCOPYIN(tA_phase0[k], gA); + TLOAD(tA_phase0[k], gA); } #pragma clang loop unroll(full) @@ -984,12 +984,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB tB; auto gB = gIterB(k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -998,12 +998,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB_trows tB; auto gB = gIterB(k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // Phase B-1: Full chunks @@ -1016,7 +1016,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < MAX_TILE_NUM; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA_chunk[k], gA); + TLOAD(tA_chunk[k], gA); } #pragma clang loop unroll(full) @@ -1026,12 +1026,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -1040,12 +1040,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -1058,7 +1058,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < K2_rem; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA_tail[k], gA); + TLOAD(tA_tail[k], gA); } #pragma clang loop unroll(full) @@ -1068,12 +1068,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -1082,12 +1082,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -1095,29 +1095,29 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ if constexpr (rmd_K) { tile_shapeA_trows tA_rmdK; auto gA = gIterA(row, Kb); - TCOPYIN(tA_rmdK, gA); + TLOAD(tA_rmdK, gA); #pragma clang loop unroll(full) for (int j = 0; j < Nb; j++) { tile_shapeACC tACC; tile_shapeB_tcols tB; auto gB = gIterB(Kb, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(row, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { tile_shapeC_trows tACC; tile_shapeB_tcorner tB; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(row, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -1134,7 +1134,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < R.k; k++) { auto gA = gIterA(Mb, k); - TCOPYIN(tA_phase0[k], gA); + TLOAD(tA_phase0[k], gA); } #pragma clang loop unroll(full) @@ -1144,12 +1144,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB tB; auto gB = gIterB(k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -1158,12 +1158,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < R.k; k++) { tile_shapeB_trows tB; auto gB = gIterB(k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_phase0[k], tB); else MATMACC(tACC, tA_phase0[k], tB); } auto gC = gIterC(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // Phase B-1: Full chunks (rmd_M 行，A 类型为 tcols) @@ -1176,7 +1176,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < MAX_TILE_NUM; k++) { auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA_chunk[k], gA); + TLOAD(tA_chunk[k], gA); } #pragma clang loop unroll(full) @@ -1186,12 +1186,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -1200,12 +1200,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < MAX_TILE_NUM; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_chunk[k], tB); else MATMACC(tACC, tA_chunk[k], tB); } auto gC = gIterC(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -1218,7 +1218,7 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < K2_rem; k++) { auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA_tail[k], gA); + TLOAD(tA_tail[k], gA); } #pragma clang loop unroll(full) @@ -1228,12 +1228,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { @@ -1242,12 +1242,12 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < K2_rem; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA_tail[k], tB); else MATMACC(tACC, tA_tail[k], tB); } auto gC = gIterC(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -1255,29 +1255,29 @@ void matmul_mask_reuseA_OPT(float *dst, dtype *src0, dtype *src1){ if constexpr (rmd_K) { tile_shapeA_tcorner tA_rmdK; auto gA = gIterA(Mb, Kb); - TCOPYIN(tA_rmdK, gA); + TLOAD(tA_rmdK, gA); #pragma clang loop unroll(full) for (int j = 0; j < Nb; j++) { tile_shapeC_tcols tACC; tile_shapeB_tcols tB; auto gB = gIterB(Kb, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { tile_shapeC_tcorner tACC; tile_shapeB_tcorner tB; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if constexpr (Kb > 0) MATMACC(tACC, tA_rmdK, tB); else MATMUL (tACC, tA_rmdK, tB); auto gC = gIterC(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } // rmd_M @@ -1380,7 +1380,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1390,7 +1390,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1403,7 +1403,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1417,7 +1417,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1427,7 +1427,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1440,7 +1440,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1463,7 +1463,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1473,7 +1473,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1488,7 +1488,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1503,7 +1503,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1513,7 +1513,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1528,7 +1528,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1556,7 +1556,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(row, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1566,7 +1566,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1585,7 +1585,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1604,7 +1604,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA[k], gA); + TLOAD(tA[k], gA); } #pragma clang loop unroll(full) @@ -1614,7 +1614,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB tB; auto gB = gIterB(k_base + k, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1633,7 +1633,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeB_trows tB; auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if (k == 0) MATMUL (tACC, tA[k], tB); else MATMACC(tACC, tA[k], tB); } @@ -1659,14 +1659,14 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ for (int row = 0; row < Mb; row++) { tile_shapeA_trows tA_rmdK; auto gA = gIterA(row, Kb); - TCOPYIN(tA_rmdK, gA); + TLOAD(tA_rmdK, gA); #pragma clang loop unroll(full) for (int j = 0; j < Nb; j++) { tile_shapeACC tACC; tile_shapeB_tcols tB; auto gB = gIterB(Kb, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); MATMUL(tACC, tA_rmdK, tB); if constexpr (is_first) { @@ -1682,7 +1682,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ tile_shapeACC_trows tACC; tile_shapeB_tcorner tB; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); MATMUL(tACC, tA_rmdK, tB); if constexpr (is_first) { @@ -1698,14 +1698,14 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ if constexpr (rmd_M) { tile_shapeA_tcorner tA_rmdK; auto gA = gIterA(Mb, Kb); - TCOPYIN(tA_rmdK, gA); + TLOAD(tA_rmdK, gA); #pragma clang loop unroll(full) for (int j = 0; j < Nb; j++) { tile_shapeACC_tcols tACC; tile_shapeB_tcols tB; auto gB = gIterB(Kb, j); - TCOPYIN(tB, gB); + TLOAD(tB, gB); MATMUL(tACC, tA_rmdK, tB); if constexpr (is_first) { @@ -1721,7 +1721,7 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ tile_shapeACC_tcorner tACC; tile_shapeB_tcorner tB; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB, gB); + TLOAD(tB, gB); MATMUL(tACC, tA_rmdK, tB); if constexpr (is_first) { @@ -1745,15 +1745,15 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ tile_C_bf16 tC_b; // TMOV_NZ2DN(tC_b, tC_main[m][n]); auto gC = gIterC(m, n); - // TCOPYOUT(gC, tC_b); - TCOPYOUT(gC, tC_main[m][n]); + // TSTORE(gC, tC_b); + TSTORE(gC, tC_main[m][n]); } if constexpr (rmd_N) { tile_C_bf16_trows tC_b; // TMOV_NZ2DN(tC_b, tC_rcol[m]); auto gC = gIterC(m, Nb); - TCOPYOUT(gC, tC_rcol[m]); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_rcol[m]); + // TSTORE(gC, tC_b); } } if constexpr (rmd_M) { @@ -1762,15 +1762,15 @@ void matmul_mask_reuseA_OPT2(float *dst, dtype *src0, dtype *src1){ tile_C_bf16_tcols tC_b; // TMOV_NZ2DN(tC_b, tC_rrow[n]); auto gC = gIterC(Mb, n); - TCOPYOUT(gC, tC_rrow[n]); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_rrow[n]); + // TSTORE(gC, tC_b); } if constexpr (rmd_N) { tile_C_bf16_tcorner tC_b; // TMOV_NZ2DN(tC_b, tC_corner); auto gC = gIterC(Mb, Nb); - TCOPYOUT(gC, tC_corner); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_corner); + // TSTORE(gC, tC_b); } } @@ -1873,7 +1873,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, col); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -1883,7 +1883,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1896,7 +1896,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1910,7 +1910,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -1920,7 +1920,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1933,7 +1933,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1956,7 +1956,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, col); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -1966,7 +1966,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1981,7 +1981,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -1996,7 +1996,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -2006,7 +2006,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2021,7 +2021,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2049,7 +2049,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, col); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -2059,7 +2059,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2078,7 +2078,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2097,7 +2097,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for (int k = 0; k < LEN; k++) { auto gB = gIterB(k_base + k, Nb); - TCOPYIN(tB[k], gB); + TLOAD(tB[k], gB); } #pragma clang loop unroll(full) @@ -2107,7 +2107,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA tA; auto gA = gIterA(row, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2126,7 +2126,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int k = 0; k < LEN; k++) { tile_shapeA_tcols tA; auto gA = gIterA(Mb, k_base + k); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if (k == 0) MATMUL (tACC, tA, tB[k]); else MATMACC(tACC, tA, tB[k]); } @@ -2152,14 +2152,14 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ for (int col = 0; col < Nb; col++) { tile_shapeB_tcols tB_rmdK; auto gB = gIterB(Kb, col); - TCOPYIN(tB_rmdK, gB); + TLOAD(tB_rmdK, gB); #pragma clang loop unroll(full) for (int row = 0; row < Mb; row++) { tile_shapeACC tACC; tile_shapeA_trows tA; auto gA = gIterA(row, Kb); - TCOPYIN(tA, gA); + TLOAD(tA, gA); MATMUL(tACC, tA, tB_rmdK); if constexpr (is_first) { @@ -2175,7 +2175,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ tile_shapeACC_tcols tACC; tile_shapeA_tcorner tA; auto gA = gIterA(Mb, Kb); - TCOPYIN(tA, gA); + TLOAD(tA, gA); MATMUL(tACC, tA, tB_rmdK); if constexpr (is_first) { @@ -2191,14 +2191,14 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ if constexpr (rmd_N) { tile_shapeB_tcorner tB_rmdK; auto gB = gIterB(Kb, Nb); - TCOPYIN(tB_rmdK, gB); + TLOAD(tB_rmdK, gB); #pragma clang loop unroll(full) for (int row = 0; row < Mb; row++) { tile_shapeACC_trows tACC; tile_shapeA_trows tA; auto gA = gIterA(row, Kb); - TCOPYIN(tA, gA); + TLOAD(tA, gA); MATMUL(tACC, tA, tB_rmdK); if constexpr (is_first) { @@ -2214,7 +2214,7 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ tile_shapeACC_tcorner tACC; tile_shapeA_tcorner tA; auto gA = gIterA(Mb, Kb); - TCOPYIN(tA, gA); + TLOAD(tA, gA); MATMUL(tACC, tA, tB_rmdK); if constexpr (is_first) { @@ -2238,15 +2238,15 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ tile_C_bf16 tC_b; // TMOV_NZ2DN(tC_b, tC_main[m][n]); auto gC = gIterC(m, n); - // TCOPYOUT(gC, tC_b); - TCOPYOUT(gC, tC_main[m][n]); + // TSTORE(gC, tC_b); + TSTORE(gC, tC_main[m][n]); } if constexpr (rmd_N) { tile_C_bf16_trows tC_b; // TMOV_NZ2DN(tC_b, tC_rcol[m]); auto gC = gIterC(m, Nb); - TCOPYOUT(gC, tC_rcol[m]); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_rcol[m]); + // TSTORE(gC, tC_b); } } if constexpr (rmd_M) { @@ -2255,15 +2255,15 @@ void matmul_mask_reuseB_OPT2(float *dst, dtype *src0, dtype *src1){ tile_C_bf16_tcols tC_b; // TMOV_NZ2DN(tC_b, tC_rrow[n]); auto gC = gIterC(Mb, n); - TCOPYOUT(gC, tC_rrow[n]); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_rrow[n]); + // TSTORE(gC, tC_b); } if constexpr (rmd_N) { tile_C_bf16_tcorner tC_b; // TMOV_NZ2DN(tC_b, tC_corner); auto gC = gIterC(Mb, Nb); - TCOPYOUT(gC, tC_corner); - // TCOPYOUT(gC, tC_b); + TSTORE(gC, tC_corner); + // TSTORE(gC, tC_b); } } @@ -2331,7 +2331,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ // #pragma clang loop unroll(full) // for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -2387,7 +2387,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } auto gC = gIterC(j, i*R.n+ii); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [n, rmd_M, k] @@ -2398,7 +2398,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -2438,7 +2438,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } auto gC = gIterC(Mb, i*R.n+ii); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -2447,7 +2447,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ // [rN, m, k] if constexpr(rN>0){ tile_shapeB tB[R.k][rN]; - + #pragma clang loop unroll(full) for(int i=0;i0){ MATMACC(tACC, tA, tB); } else { @@ -2508,7 +2508,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(j, i+dN*R.n); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rN, rmd_M, k] @@ -2519,7 +2519,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -2560,7 +2560,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(Mb, i+dN*R.n); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -2568,11 +2568,11 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ // [rmd_N, m, k] if constexpr (rmd_N) { tile_shapeB_trows tB[R.k]; - + #pragma clang loop unroll(full) for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -2624,7 +2624,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(j, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rmd_N, rmd_M, k] @@ -2635,7 +2635,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ for(int k=0;k0){ MATMACC(tACC, tA, tB); } else { @@ -2676,7 +2676,7 @@ void matmul_mask_reuseB(float *dst, dtype *src0, dtype *src1){ } } auto gC = gIterC(Mb,Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } @@ -2760,10 +2760,10 @@ void matmul_mask_reuseAB(float *dst, dtype *src0, dtype *src1){ #pragma clang loop unroll(full) for(int k=0;k tK ? tK : gK - k; tile_shapeA tA(dyn_m, dyn_k); tile_shapeB tB(dyn_k, dyn_n); - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if(k==0){ MATMUL(tACC, tA, tB); }else{ MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + TSTORE_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); } } } @@ -3150,15 +3150,15 @@ __attribute__((noinline)) void matmul_dynamic(float* dst, dtype* src0, dtype* sr int dyn_k = (k+1) * tK > gK ? rem_k:tK; tile_shapeA tA(dyn_m, dyn_k); tile_shapeB tB(dyn_k, dyn_n); - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if(k==0){ MATMUL(tACC, tA, tB); }else{ MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + TSTORE_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); } } } @@ -3201,7 +3201,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt int rem_k = gK % tK; ResA R = find_reuseA_dynamic(Mb, Kb, MAX_TILE_NUM); - + int dM = R.m == 0? 0 : Mb / R.m; int rM = R.m == 0? 0 : Mb % R.m; @@ -3223,7 +3223,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt tile_shapeA tA[m_step][R.k]; for (int mm=0;mm gM ){ tA[mm][kk]= tile_shapeA(rem_m, tK); }else{ @@ -3236,7 +3236,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt for(int k=0;k gM? rem_m:tM; @@ -3249,7 +3249,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt size_t offset_B = k * gN * tile_shapeB::Rows + j * tile_shapeB::Cols; gm_shapeB gB(src1 + offset_B, gK, gN); tile_shapeB tB(tK, dyn_n); - TCOPYIN(tB, gB); + TLOAD(tB, gB); if(k==0){ MATMUL(tACC, tA[ii][k], tB); }else{ @@ -3268,8 +3268,8 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt tile_shapeA tA(dyn_m, dyn_k); tile_shapeB tB(dyn_k, dyn_n); - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if(k==0){ MATMUL(tACC, tA, tB); }else{ @@ -3280,7 +3280,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseA(float* dst, dtype* src0, dt size_t offset_C = (i+ii) * gN * tile_shapeACC::Rows + j * tile_shapeACC::Cols; gm_shapeC gC(dst + offset_C, gM, gN); - TCOPYOUT_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + TSTORE_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); } } @@ -3313,7 +3313,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt R.n = Ra.m; R.k = Ra.k; R.val = Ra.val; - + int dN = R.n == 0? 0 : Nb / R.n; int rN = R.n == 0? 0 : Nb % R.n; @@ -3333,7 +3333,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt tile_shapeB tB[R.k][n_step]; for (int nn=0;nn gN ){ tB[kk][nn]= tile_shapeB(tK, rem_n); }else{ @@ -3346,7 +3346,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt for(int k=0;k gN? rem_n:tN; @@ -3359,7 +3359,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt size_t offset_A = j * gK * tile_shapeA::Rows + k * tile_shapeA::Cols; gm_shapeA gA(src0 + offset_A, gM, gK); tile_shapeA tA(dyn_m, tK); - TCOPYIN(tA, gA); + TLOAD(tA, gA); if(k==0){ MATMUL(tACC, tA, tB[k][ii]); }else{ @@ -3378,8 +3378,8 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt tile_shapeA tA(dyn_m, dyn_k); tile_shapeB tB(dyn_k, dyn_n); - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); if(k==0){ MATMUL(tACC, tA, tB); }else{ @@ -3390,7 +3390,7 @@ __attribute__((noinline)) void matmul_dynamic_reuseB(float* dst, dtype* src0, dt size_t offset_C = j * gN * tile_shapeACC::Rows + (i+ii) * tile_shapeACC::Cols; gm_shapeC gC(dst + offset_C, gM, gN); - TCOPYOUT_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + TSTORE_ACC_DYNAMIC(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); } } @@ -3445,8 +3445,8 @@ void matmul_mx(float *dst, dtype *src0, dtype *src1, uint8_t *src0_mx, uint8_t * tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); blk_tload(tAMX.GetValidCol(), tAMX.GetValidRow(), tile_shapeAMX::Cols, type_traits::TypeCode, @@ -3470,7 +3470,7 @@ void matmul_mx(float *dst, dtype *src0, dtype *src1, uint8_t *src0_mx, uint8_t * MATMACCMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -3519,16 +3519,16 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { auto gC = gCIter(i, j); tile_shapeACC tACC; - + if constexpr(Kb>0){ auto gA = gAIter(i, 0); auto gB = gBIter(0, j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -3537,8 +3537,8 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); MATMACC(tACC, tA, tB); } @@ -3548,15 +3548,15 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -3568,9 +3568,9 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -3579,8 +3579,8 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -3589,15 +3589,15 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -3611,9 +3611,9 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -3622,8 +3622,8 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -3632,15 +3632,15 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -3652,9 +3652,9 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); - MATMUL(tACC, tA, tB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); + MATMUL(tACC, tA, tB); } #pragma clang loop unroll(full) for (int k = 1; k < Kb; ++k) { @@ -3663,8 +3663,8 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); MATMACC(tACC, tA, tB); } if constexpr (rmd_K) { @@ -3673,15 +3673,15 @@ void matmul_mask_2lvl(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN_2LVL(tA, gA); - TCOPYIN_2LVL(tB, gB); + TLOAD_2LVL(tA, gA); + TLOAD_2LVL(tB, gB); if constexpr(Kb>0){ MATMACC(tACC, tA, tB); } else { MATMUL(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -3717,11 +3717,11 @@ void matmul_vec(float* dst, float* src0, float* src1){ auto gB = gBIter(k,j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); MATMACC(tACC, tA, tB); } - TCOPYOUT(gC, tACC); + TSTORE(gC, tACC); } } } @@ -3744,10 +3744,10 @@ void matmul_tile_vec(float* dst, float* src0, float* src1) { tile_shape_B d1; tile_shape_C d2; - TCOPYIN(d0, s0); - TCOPYIN(d1, s1); + TLOAD(d0, s0); + TLOAD(d1, s1); MATMUL(d2, d0, d1); - TCOPYOUT(res, d2); + TSTORE(res, d2); } template @@ -3768,10 +3768,10 @@ void matmul_tile_frac(float* dst, float* src0, float* src1) { tile_shape_B d1; tile_shape_C d2; - TCOPYIN(d0, s0); - TCOPYIN(d1, s1); + TLOAD(d0, s0); + TLOAD(d1, s1); MATMUL(d2, d0, d1); - TCOPYOUT_ACC(res, d2); + TSTORE_ACC(res, d2); } diff --git a/benchmark/two-level-arch/kernels/matmul/matmul_mx.hpp b/benchmark/two-level-arch/kernels/matmul/matmul_mx.hpp index b26e87f..35e8f3d 100644 --- a/benchmark/two-level-arch/kernels/matmul/matmul_mx.hpp +++ b/benchmark/two-level-arch/kernels/matmul/matmul_mx.hpp @@ -13,7 +13,7 @@ // GlobalTensor, \ // Stride<1,1,1,Cols,1>> _g(DumpBuf); \ -// TCOPYOUT(_g, TileVar); \ +// TSTORE(_g, TileVar); \ // printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ // for (int ri = 0; ri < Rows; ri++) { \ // printf(" row%2d: ", ri); \ @@ -32,20 +32,20 @@ using namespace pto; // TODO, move to utils.cpp template -void TCOPYOUT_ACC(GmOut &Gout, TileAcc &tAcc){ +void TSTORE_ACC(GmOut &Gout, TileAcc &tAcc){ using TileAccOut = Tile; TileAccOut tAccOut; TCVT(tAccOut, tAcc); - TCOPYOUT(Gout, tAccOut); + TSTORE(Gout, tAccOut); } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - // static_assert(gM % tM == 0); + // static_assert(gM % tM == 0); // static_assert(gN % tN == 0); // static_assert(gK % tK == 0); static const uint32_t valid_row = (tM > gM) ? gM : tM; @@ -54,7 +54,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -65,7 +65,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; gm_shapeAMX gAMX(src0_mx); using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using itAMX = global_iterator; @@ -119,8 +119,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(k,j); tile_shapeA tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); // if (src0_mx != nullptr && src1_mx != nullptr) { tile_shapeAMX tAMX; @@ -141,8 +141,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(Kb,j); tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_trows tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); @@ -158,7 +158,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -168,8 +168,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 auto gB = gBIter(k, Nb); tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, k); @@ -191,8 +191,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_trows tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); @@ -206,7 +206,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -221,8 +221,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcols tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); @@ -244,8 +244,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcorner tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); @@ -260,7 +260,7 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -273,8 +273,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcols tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); @@ -295,8 +295,8 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); tile_shapeAMX_tcorner tAMX; gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); @@ -310,12 +310,12 @@ void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8 MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } -template void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { static_assert(typeb_wfactor == 1 ); @@ -325,7 +325,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -336,7 +336,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using itAMX = global_iterator; itAMX gAMXIter(src0_mx); @@ -390,10 +390,10 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -406,14 +406,14 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx auto gB = gBIter(Kb,j); tile_shapeA_trows tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, Kb); auto gBMX = gBMXIter(Kb, j); tile_shapeAMX_trows tAMX; tile_shapeBMX_tcols tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -421,7 +421,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(i, Nb); @@ -431,15 +431,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx auto gB = gBIter(k, Nb); tile_shapeA tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, k); auto gBMX = gBMXIter(k, Nb); tile_shapeAMX tAMX; tile_shapeBMX_trows tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -454,15 +454,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_trows tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(i, Kb); auto gBMX = gBMXIter(Kb, Nb); tile_shapeAMX_trows tAMX; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -470,7 +470,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } if constexpr (rmd_M) { @@ -485,15 +485,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcols tA; tile_shapeB tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, k); auto gBMX = gBMXIter(k, j); tile_shapeAMX_tcols tAMX; tile_shapeBMX tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -508,15 +508,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcorner tA; tile_shapeB_tcols tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, Kb); auto gBMX = gBMXIter(Kb, j); tile_shapeAMX_tcorner tAMX; tile_shapeBMX_tcols tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); @@ -524,7 +524,7 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } if constexpr (rmd_N) { auto gC = gCIter(Mb, Nb); @@ -537,15 +537,15 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcols tA; tile_shapeB_trows tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, k); auto gBMX = gBMXIter(k, Nb); tile_shapeAMX_tcols tAMX; tile_shapeBMX_trows tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -559,22 +559,22 @@ void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx tile_shapeA_tcorner tA; tile_shapeB_tcorner tB; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); + TLOAD(tA, gA); + TLOAD(tB, gB); auto gAMX = gAMXIter(Mb, Kb); auto gBMX = gBMXIter(Kb, Nb); tile_shapeAMX_tcorner tAMX; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA, tAMX, tB, tBMX); } else { MATMULMX(tACC, tA, tAMX, tB, tBMX); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -609,22 +609,22 @@ constexpr ResA find_reuseA(int Mb, int Kb, int MAX_TILE_NUM) { return {best_m, best_k, best_val}; } -template void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { static_assert(typeb_wfactor == 1 ); static const uint32_t valid_row = (tM > gM) ? gM : tM; static const uint32_t valid_col = (tN > gN) ? gN : tN; static const uint32_t MAX_TILE_NUM = 24; // TODO, check this value - + using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; - + using itA = global_iterator; using itB = global_iterator; using itC = global_iterator; @@ -633,8 +633,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; - using tile_shapeAMX = Tile; + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; using itAMX = global_iterator; itAMX gAMXIter(src0_mx); @@ -691,8 +691,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * // for(int k=0; k0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -763,7 +763,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } auto gC = gCIter(i*R.m+ii, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [m, rmd_N, k] @@ -776,8 +776,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[ii][k], tAMX[ii][k], tB, tBMX); }else{ @@ -795,10 +795,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(i*R.m+ii, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -814,11 +814,11 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeAMX_trows tAMX_tmp; tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -827,7 +827,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } auto gC = gCIter(i*R.m+ii, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -835,7 +835,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * if constexpr(rM>0){ tile_shapeA tA[rM][R.k]; tile_shapeAMX tAMX[rM][R.k]; - + #pragma clang loop unroll(full) for(int i=0; i0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -914,7 +914,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(i+dM*R.m, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rM, rmd_N, k] @@ -927,8 +927,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[i][k], tAMX[i][k], tB, tBMX); }else{ @@ -946,10 +946,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(i+dM*R.m, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -966,10 +966,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -977,7 +977,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(i+dM*R.m, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -986,13 +986,13 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * if constexpr (rmd_M) { tile_shapeA_tcols tA[R.k]; tile_shapeAMX_tcols tAMX[R.k]; - + #pragma clang loop unroll(full) for(int k=0; k0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -1055,7 +1055,7 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(Mb, j); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } // [rmd_M, rmd_N, k] @@ -1068,8 +1068,8 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gBMX = gBMXIter(k, Nb); tile_shapeB_trows tB; tile_shapeBMX_trows tBMX; - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA[k], tAMX[k], tB, tBMX); }else{ @@ -1087,10 +1087,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * auto gAMX = gAMXIter(Mb, k); auto gB = gBIter(k, Nb); auto gBMX = gBMXIter(k, Nb); - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } } @@ -1107,10 +1107,10 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * tile_shapeB_tcorner tB; tile_shapeBMX_tcorner tBMX; - TCOPYIN(tA_tmp, gA); - TCOPYIN(tAMX_tmp, gAMX); - TCOPYIN(tB, gB); - TCOPYIN(tBMX, gBMX); + TLOAD(tA_tmp, gA); + TLOAD(tAMX_tmp, gAMX); + TLOAD(tB, gB); + TLOAD(tBMX, gBMX); if constexpr(Kb>0){ MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); } else { @@ -1118,25 +1118,25 @@ void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t * } } auto gC = gCIter(Mb, Nb); - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - static_assert(gM % tM == 0); + static_assert(gM % tM == 0); static_assert(gN % tN == 0); static_assert(gK % tK == 0); using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -1147,7 +1147,7 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; // gm_shapeAMX gAMX(src0_mx); // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 @@ -1184,10 +1184,10 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMULMX(tACC, tA, tAMX, tB, tBMX); @@ -1197,25 +1197,25 @@ void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src // MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， // smatrix_wfactor : scaling matrix 与计算matrix位宽比 -template void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { // only support regular shape now for this operator! - static_assert(gM % tM == 0); + static_assert(gM % tM == 0); static_assert(gN % tN == 0); static_assert(gK % tK == 0); using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; using gm_shapeC = global_tensor>; - using tile_shapeA = TileLeft; + using tile_shapeA = TileLeft; using tile_shapeB = TileRight; using tile_shapeACC = TileAcc; using itA = global_iterator; @@ -1226,7 +1226,7 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, itB gBIter(src1); itC gCIter(dst); - using gm_shapeAMX = global_tensor>; + using gm_shapeAMX = global_tensor>; // gm_shapeAMX gAMX(src0_mx); // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 @@ -1263,10 +1263,10 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, tile_shapeB tB; tile_shapeAMX tAMX; tile_shapeBMX tBMX; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(tAMX, gAMX); - TCOPYIN(tBMX, gBMX); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); if(k==0){ MATMUL(tACC, tA, tB); @@ -1275,7 +1275,7 @@ void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, MATMACC(tACC, tA, tB); } } - TCOPYOUT_ACC(gC, tACC); + TSTORE_ACC(gC, tACC); } } } @@ -1323,7 +1323,7 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using gm_shapeA = global_tensor>; using gm_shapeB = global_tensor>; // 伪量化固定float, group 大小128， 128个fp4共享一个scaling factor, 128的partial sum* scale - using gm_shape_scale = global_tensor>; + using gm_shape_scale = global_tensor>; using gm_shapeACC = global_tensor>; using tile_shapeA = TileLeft; using tile_shapeB = TileRight; @@ -1331,7 +1331,7 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using tile_shape_dequant = Tile; using tile_shapeACC = TileAcc; // copy of acc, input as vector - using tile_ACCin = Tile; + using tile_ACCin = Tile; using itA = global_iterator; using itB = global_iterator; @@ -1367,9 +1367,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { using tile_shape_scale_tcols = Tile; using tile_shape_scale_tcorner = Tile; - using tile_ACCin_trows = Tile; - using tile_ACCin_tcols = Tile; - using tile_ACCin_tconer = Tile; + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; using tile_shape_dequant_trows = Tile; using tile_shape_dequant_tcols = Tile; @@ -1392,9 +1392,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeB tB; tile_shape_scale ts; tile_shape_dequant tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); // [1, tN] + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); // [1, tN] MATMUL(tACC, tA, tB); TCVT(tACCin, tACC);//[tM, tN] 256->1 , 256 -> 2 scaling factor @@ -1413,22 +1413,22 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shape_scale_tcols ts; tile_shape_dequant tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); TCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); tAdder[(k+1)%2] = tC_dequant; } - TCOPYOUT(gACC, tAdder[(k+1)%2]); + TSTORE(gACC, tAdder[(k+1)%2]); } // if constexpr (rmd_N) // TODO } if constexpr (rmd_M) { for (int j = 0; j < Nb; ++j) { - auto gACC = gACCIter(Mb, j); + auto gACC = gACCIter(Mb, j); tile_shapeC_tcols tACC; tile_ACCin_tcols tACCin; tile_shape_dequant_tcols tAdder[2]; @@ -1444,9 +1444,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeB tB; tile_shape_scale ts; tile_shape_dequant_tcols tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); TCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); @@ -1462,15 +1462,15 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { tile_shapeB_tcols tB; tile_shape_scale_tcols ts; tile_shape_dequant tC_dequant; - TCOPYIN(tA, gA); - TCOPYIN(tB, gB); - TCOPYIN(ts, gS); + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); MATMUL(tACC, tA, tB); TCVT(tACCin, tACC); dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); tAdder[(k+1)%2] = tC_dequant; } - TCOPYOUT(gACC, tAdder[(k+1)%2]); + TSTORE(gACC, tAdder[(k+1)%2]); } // todo // if constexpr (rmd_N) { @@ -1483,9 +1483,9 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcols tA; // tile_shapeB_trows tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); - // MATMUL(tACC, tA, tB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMUL(tACC, tA, tB); // } // #pragma clang loop unroll(full) // for (int k = 1; k < Kb; ++k) { @@ -1494,8 +1494,8 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcols tA; // tile_shapeB_trows tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); // MATMACC(tACC, tA, tB); // } // if constexpr (rmd_K) { @@ -1504,15 +1504,15 @@ void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { // tile_shapeA_tcorner tA; // tile_shapeB_tcorner tB; - // TCOPYIN(tA, gA); - // TCOPYIN(tB, gB); + // TLOAD(tA, gA); + // TLOAD(tB, gB); // if constexpr(Kb>0){ // MATMACC(tACC, tA, tB); // } else { // MATMUL(tACC, tA, tB); // } // } - // TCOPYOUT_ACC(gC, tACC); + // TSTORE_ACC(gC, tACC); // } } } diff --git a/benchmark/two-level-arch/kernels/reduction/README.md b/benchmark/two-level-arch/kernels/reduction/README.md index 6292891..4ed2257 100644 --- a/benchmark/two-level-arch/kernels/reduction/README.md +++ b/benchmark/two-level-arch/kernels/reduction/README.md @@ -7,7 +7,7 @@ This directory contains benchmark implementations for various reduction operatio The reduction operators are categorized into three main types based on their reduction direction and optimization strategy: - **Column Reduction (col)**: Reduces along the column dimension (M-axis) -- **Row Reduction (row)**: Reduces along the row dimension (N-axis) +- **Row Reduction (row)**: Reduces along the row dimension (N-axis) - **3D Column Reduction (3dcol)**: Optimized for 3D tensor column reduction with unaligned dimensions ## Directory Structure @@ -158,7 +158,7 @@ All operators use 8-way unrolled tree reduction to maximize instruction-level pa ```cpp // Level 1: 8 pairwise operations sum_01 = a + b; sum_23 = c + d; sum_45 = e + f; sum_67 = g + h; -// Level 2: 4 pairwise operations +// Level 2: 4 pairwise operations sum_0123 = sum_01 + sum_23; sum_4567 = sum_45 + sum_67; // Level 3: Final reduction sum_tmp = sum_0123 + sum_4567; @@ -170,7 +170,7 @@ Uses the PTO (Parallel Tile Operations) framework: - `global_tensor`: Defines global memory layout - `Tile`: Defines tile shape and memory location (Vec/Scalar) - `global_iterator`: Iterates over tiles in global memory -- `TCOPYIN`/`TCOPYOUT`: DMA transfers between global and tile memory +- `TLOAD`/`TSTORE`: DMA transfers between global and tile memory ### 3. Corner Case Handling @@ -203,7 +203,7 @@ Uses built-in vector operations: // For reducesum_col (8192×1024 input) #define DType int32_t #define gIMs 8192 // Global M dimension -#define gINs 1024 // Global N dimension +#define gINs 1024 // Global N dimension #define tMs 32 // Tile M dimension #define tNs 128 // Tile N dimension ``` @@ -242,7 +242,7 @@ dtype output[381 * 1 * 8]; for(int i = 0; i < 381; i++) { reducesum_colsum_rand( - &input[i * 120 * 8], + &input[i * 120 * 8], &output[i * 8] ); } diff --git a/benchmark/two-level-arch/kernels/reduction/cumsum_colvec.hpp b/benchmark/two-level-arch/kernels/reduction/cumsum_colvec.hpp index e7ca443..02a8345 100644 --- a/benchmark/two-level-arch/kernels/reduction/cumsum_colvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/cumsum_colvec.hpp @@ -23,41 +23,41 @@ void __vec__ cumsum_col_kernel( typename tileSum::TileDType __out__ new_sum, typename tileData::TileDType __out__ out, const typename tileData::TileDType __in__ src, - const typename tileSum::TileDType __in__ old_sum + const typename tileSum::TileDType __in__ old_sum ) { - size_t i = blkv_get_index_x(); - size_t sum_idx = i * tileSum::RowStride; + size_t i = blkv_get_index_x(); + size_t sum_idx = i * tileSum::RowStride; __vbuf__ typename tileSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); - __vbuf__ typename tileData::DType *out_ptr = blkv_get_tile_ptr(out); + __vbuf__ typename tileData::DType *out_ptr = blkv_get_tile_ptr(out); __vbuf__ typename tileData::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); typename tileSum::DType upd_sum = old_sum_ptr[i]; // printf("upd_sum = %d",upd_sum); -// typename tileData::DType upd_out = old_sum_ptr[i]; -/* +// typename tileData::DType upd_out = old_sum_ptr[i]; +/* for(size_t j=0;j(sum_out); - } - new_sum_ptr[i] = upd_sum; + } + new_sum_ptr[i] = upd_sum; } @@ -66,35 +66,35 @@ void __vec__ cumsum_col_kernel( template void cumsum_col_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { const int Mb = gIM / tM; - const int Nb = gIN / tN; + const int Nb = gIN / tN; - const int rmd_M = gIM % tM; - const int rmd_N = gIN % tN; + const int rmd_M = gIM % tM; + const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // - using tile_shapeData_col = Tile; // - using tile_shapeSum = Tile; // + using tile_shapeData_col = Tile; // + using tile_shapeSum = Tile; // - using tile_shapeData_row = Tile; // - using tile_shapeData_cor = Tile; // + using tile_shapeData_row = Tile; // + using tile_shapeData_cor = Tile; // using tile_shapeSum_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); tile_shapeData dataTile; tile_shapeData OutTile; @@ -105,19 +105,19 @@ void cumsum_col_rand( tile_shapeData_row dataTile_row; tile_shapeData_row OutTile_row; - tile_shapeData_cor dataTile_cor; - tile_shapeData_cor OutTile_cor; - + tile_shapeData_cor dataTile_cor; + tile_shapeData_cor OutTile_cor; + tile_shapeSum_row SumTile_row; - tile_shapeSum_row oldSumTile_row; + tile_shapeSum_row oldSumTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; -// using itZero = global_iterator; + using itIn = global_iterator; +// using itZero = global_iterator; using itOut = global_iterator; -// using itSum = global_iterator; +// using itSum = global_iterator; itIn gIIter(in_ptr); itOut gOIter(out_ptr); @@ -128,44 +128,44 @@ void cumsum_col_rand( TEXPANDSCALAR(oldSumTile, 0);//初始化为0 for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); - auto gO = gOIter(i, j); - TCOPYIN(dataTile, gI); -// printf("in0 : %d, %d\n",in_ptr[i*tM], i*tM); + auto gO = gOIter(i, j); + TLOAD(dataTile, gI); +// printf("in0 : %d, %d\n",in_ptr[i*tM], i*tM); cumsum_col_kernel<<>>(SumTile.data(), OutTile.data(), dataTile.data(), oldSumTile.data()); oldSumTile = SumTile; - TCOPYOUT(gO, OutTile); -// printf("out0 : %d,%d\n", out_ptr[i*tM],i*tM); + TSTORE(gO, OutTile); +// printf("out0 : %d,%d\n", out_ptr[i*tM],i*tM); } - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, j); auto gO = gOIter(Mb, j); - TCOPYIN(dataTile_col, gI); + TLOAD(dataTile_col, gI); cumsum_col_kernel<<>>(SumTile.data(), OutTile_col.data(), dataTile_col.data(), oldSumTile.data()); oldSumTile = SumTile; - TCOPYOUT(gO, OutTile_col); + TSTORE(gO, OutTile_col); } -// TCOPYOUT(gO, SumTile); +// TSTORE(gO, SumTile); } if constexpr (rmd_N > 0){ -// auto gZero = gZeroIter(0, Nb); +// auto gZero = gZeroIter(0, Nb); // auto gO = gOIter(0, Nb); - TEXPANDSCALAR(oldSumTile_row, 0);//初始化为0 -// TCOPYIN(oldSumTile_row, gZero);//初始化为0 - for (int i = 0; i < Mb; ++i) { + TEXPANDSCALAR(oldSumTile_row, 0);//初始化为0 +// TLOAD(oldSumTile_row, gZero);//初始化为0 + for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); auto gO = gOIter(i, Nb); - TCOPYIN(dataTile_row, gI); + TLOAD(dataTile_row, gI); cumsum_col_kernel<<>>(SumTile_row.data(), OutTile_row.data(), dataTile_row.data(), oldSumTile_row.data()); oldSumTile_row = SumTile_row; - TCOPYOUT(gO, OutTile_row); + TSTORE(gO, OutTile_row); } - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, Nb); auto gO = gOIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); cumsum_col_kernel<<>>(SumTile_row.data(), OutTile_cor.data(), dataTile_cor.data(), oldSumTile_row.data()); oldSumTile_row = SumTile_row; - TCOPYOUT(gO, OutTile_cor); + TSTORE(gO, OutTile_cor); } } /* diff --git a/benchmark/two-level-arch/kernels/reduction/cumsum_rowvec.hpp b/benchmark/two-level-arch/kernels/reduction/cumsum_rowvec.hpp index ec998c1..eeedc93 100644 --- a/benchmark/two-level-arch/kernels/reduction/cumsum_rowvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/cumsum_rowvec.hpp @@ -16,42 +16,42 @@ using namespace pto; template void __vec__ cumsum_row_kernel( typename tileSum::TileDType __out__ new_sum, - const typename tileData::TileDType __out__ out, + const typename tileData::TileDType __out__ out, const typename tileData::TileDType __in__ src, - const typename tileSum::TileDType __in__ old_sum + const typename tileSum::TileDType __in__ old_sum ) { -// size_t i = blkv_get_index_x(); - size_t j = blkv_get_index_y(); - size_t sum_idx = j * tileSum::RowStride; +// size_t i = blkv_get_index_x(); + size_t j = blkv_get_index_y(); + size_t sum_idx = j * tileSum::RowStride; __vbuf__ typename tileSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); - __vbuf__ typename tileData::DType *out_ptr = blkv_get_tile_ptr(out); + __vbuf__ typename tileData::DType *out_ptr = blkv_get_tile_ptr(out); __vbuf__ typename tileData::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); typename tileSum::DType upd_sum = old_sum_ptr[sum_idx]; -/* +/* for(size_t j=0;j(sum_out); - } - new_sum_ptr[sum_idx] = upd_sum; + upd_sum = sum_out; + out_ptr[idx] = static_cast(sum_out); + } + new_sum_ptr[sum_idx] = upd_sum; } @@ -60,105 +60,105 @@ template>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_row = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeSum = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_cor = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_shapeSum_col = Tile; + using tile_shapeSum_col = Tile; - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); - tile_shapeData dataTile; + tile_shapeData dataTile; tile_shapeData_row dataTile_row; tile_shapeData_col dataTile_col; - tile_shapeData_cor dataTile_cor; + tile_shapeData_cor dataTile_cor; - tile_shapeData OutTile; + tile_shapeData OutTile; tile_shapeData_row OutTile_row; tile_shapeData_col OutTile_col; - tile_shapeData_cor OutTile_cor; - + tile_shapeData_cor OutTile_cor; + tile_shapeSum SumTile; tile_shapeSum oldSumTile; tile_shapeSum_col SumTile_col; - tile_shapeSum_col oldSumTile_col; + tile_shapeSum_col oldSumTile_col; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itOut gOIter(out_ptr); // printf("tile_shapeSum::ValidCol = %d\n", tile_shapeSum::ValidCol); -// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); +// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); for (int j = 0; j < Mb; ++j) { // auto gO = gOIter(j, 0); TEXPANDSCALAR(oldSumTile, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(j, i); - auto gO = gOIter(j, i); - TCOPYIN(dataTile, gI); + auto gI = gIIter(j, i); + auto gO = gOIter(j, i); + TLOAD(dataTile, gI); cumsum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), OutTile.data(), dataTile.data(), oldSumTile.data()); // reducesum_row_kernel<<>>(SumTile.data(), dataTile.data(), oldSumTile.data()); oldSumTile = SumTile; - TCOPYOUT(gO, OutTile); + TSTORE(gO, OutTile); } // printf("end for%d\n",j); //for row corner if constexpr (rmd_N > 0){ auto gI = gIIter(j, Nb); auto gO = gOIter(j, Nb); - TCOPYIN(dataTile_row, gI); - cumsum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), OutTile_row.data(), dataTile_row.data(), oldSumTile.data()); + TLOAD(dataTile_row, gI); + cumsum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), OutTile_row.data(), dataTile_row.data(), oldSumTile.data()); // reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); oldSumTile = SumTile; - TCOPYOUT(gO, OutTile_row); + TSTORE(gO, OutTile_row); } } //for col cor if constexpr (rmd_M > 0){ TEXPANDSCALAR(oldSumTile_col, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(Mb, i); - auto gO = gOIter(Mb, i); - TCOPYIN(dataTile_col, gI); + auto gI = gIIter(Mb, i); + auto gO = gOIter(Mb, i); + TLOAD(dataTile_col, gI); cumsum_row_kernel<<<1, tile_shapeSum_col::ValidRow, 1>>>(SumTile_col.data(), OutTile_col.data(), dataTile_col.data(), oldSumTile_col.data()); oldSumTile_col = SumTile_col; - TCOPYOUT(gO, OutTile_col); + TSTORE(gO, OutTile_col); } if constexpr (rmd_N > 0){ auto gI = gIIter(Mb, Nb); auto gO = gOIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); cumsum_row_kernel<<<1, tile_shapeSum_col::ValidRow, 1>>>(SumTile_col.data(), OutTile_cor.data(), dataTile_cor.data(), oldSumTile_col.data()); oldSumTile_col = SumTile_col; - TCOPYOUT(gO, OutTile_cor); - } + TSTORE(gO, OutTile_cor); + } } /* for(int i = 0; i < gIM; i++){ diff --git a/benchmark/two-level-arch/kernels/reduction/reducemax_colvec.hpp b/benchmark/two-level-arch/kernels/reduction/reducemax_colvec.hpp index 8265eb2..c1fbedc 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducemax_colvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducemax_colvec.hpp @@ -30,33 +30,33 @@ template void __vec__ reducemax_col_kernel( typename tileMax::TileDType __out__ new_max, const typename tileSrc::TileDType __in__ src, - const typename tileMax::TileDType __in__ old_max + const typename tileMax::TileDType __in__ old_max ) { // 当前 lane 索引：每个 lane 负责一列的最大值 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileMax::DType *new_max_ptr = blkv_get_tile_ptr(new_max); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); + __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); // 取历史最大值作为初值 typename tileMax::DType upd_max = old_max_ptr[i]; -/* +/* for(size_t j=0;j void reducemax_col_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { const int Mb = gIM / tM; - const int Nb = gIN / tN; + const int Nb = gIN / tN; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; // -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; // +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // - using tile_shapeData_col = Tile; // - using tile_shapeMax = Tile; // + using tile_shapeData_col = Tile; // + using tile_shapeMax = Tile; // - using tile_shapeData_row = Tile; // - using tile_shapeData_cor = Tile; // - using tile_shapeMax_row = Tile; // + using tile_shapeData_row = Tile; // + using tile_shapeData_cor = Tile; // + using tile_shapeMax_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); tile_shapeData dataTile; - tile_shapeData_col dataTile_col; + tile_shapeData_col dataTile_col; tile_shapeMax MaxTile; tile_shapeMax oldMaxTile; tile_shapeData_row dataTile_row; - tile_shapeData_cor dataTile_cor; + tile_shapeData_cor dataTile_cor; tile_shapeMax_row MaxTile_row; - tile_shapeMax_row oldMaxTile_row; + tile_shapeMax_row oldMaxTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itIn_row = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itIn_row gIIter_rmd_row(in_ptr); -// itZero gZeroIter(inzero_ptr); +// itZero gZeroIter(inzero_ptr); itOut gOIter(out_ptr); // dtype zero = 0; @@ -153,44 +153,44 @@ void reducemax_col_rand( // auto gZero = gZeroIter(0, j); auto gO = gOIter(0, j); TEXPANDSCALAR(oldMaxTile, 0);//初始化为0 -// TCOPYIN(oldSumTile, gZero);//初始化为0 - //初始化old_sum的tile - //need +// TLOAD(oldSumTile, gZero);//初始化为0 + //初始化old_sum的tile + //need // 沿 M 方向逐 tile 拷入并取最大 for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); - TCOPYIN(dataTile, gI); + TLOAD(dataTile, gI); reducemax_col_kernel<<>>(MaxTile.data(), dataTile.data(), oldMaxTile.data()); oldMaxTile = MaxTile; } // M 方向尾块 - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, j); - TCOPYIN(dataTile_col, gI); + TLOAD(dataTile_col, gI); reducemax_col_kernel<<>>(MaxTile.data(), dataTile_col.data(), oldMaxTile.data()); oldMaxTile = MaxTile; } - TCOPYOUT(gO, MaxTile); + TSTORE(gO, MaxTile); } // ---- N 方向尾块区域 ---- if constexpr (rmd_N > 0){ -// auto gZero = gZeroIter(0, Nb); +// auto gZero = gZeroIter(0, Nb); auto gO = gOIter(0, Nb); - TEXPANDSCALAR(oldMaxTile_row, 0);//初始化为0 -// TCOPYIN(oldSumTile_row, gZero);//初始化为0 - for (int i = 0; i < Mb; ++i) { + TEXPANDSCALAR(oldMaxTile_row, 0);//初始化为0 +// TLOAD(oldSumTile_row, gZero);//初始化为0 + for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); - TCOPYIN(dataTile_row, gI); + TLOAD(dataTile_row, gI); reducemax_col_kernel<<>>(MaxTile_row.data(), dataTile_row.data(), oldMaxTile_row.data()); oldMaxTile_row = MaxTile_row; } - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reducemax_col_kernel<<>>(MaxTile_row.data(), dataTile_cor.data(), oldMaxTile_row.data()); oldMaxTile_row = MaxTile_row; } - TCOPYOUT(gO, MaxTile_row); + TSTORE(gO, MaxTile_row); } } diff --git a/benchmark/two-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8.hpp b/benchmark/two-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8.hpp index b706b75..452c947 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducemax_colvec_unalign_120_8.hpp @@ -22,7 +22,7 @@ using namespace pto; // 关键点： // 1) 布局变换 (gIM×gIN) -> (gIM/8 × gIN*8)，凑齐 8 元向量； // 2) 用补零让非 valid 区参与 8 路树形取 max（0 不影响 max 结果前提是数据非负， -// 实际依赖 TCOPYIN 默认补 0）； +// 实际依赖 TLOAD 默认补 0）； // 3) 两阶段：reducemax_col_tmp() 每 tile 取临时最大；reducemax_col_final() 合并。 // ============================================================ @@ -33,21 +33,21 @@ using namespace pto; template void __vec__ reducemax_col_tmp( typename tileTmp::TileDType __out__ tmp_max, - const typename tileSrc::TileDType __in__ src + const typename tileSrc::TileDType __in__ src ) { // 当前 lane 负责一列 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileTmp::DType *tmp_max_ptr = blkv_get_tile_ptr(tmp_max); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); -// __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); +// __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); // 局部最大值初值 typename tileTmp::DType upd_tmp_max = 0; - + // 用 Rows（非 valid 补零区也参与）凑齐 8 元树形取 max - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t j=0;j 树形合并 - typename tileTmp::DType max_01 = blkv_max(src_ptr[src_idx_0], src_ptr[src_idx_1]); + typename tileTmp::DType max_01 = blkv_max(src_ptr[src_idx_0], src_ptr[src_idx_1]); typename tileTmp::DType max_23 = blkv_max(src_ptr[src_idx_2], src_ptr[src_idx_3]); - typename tileTmp::DType max_45 = blkv_max(src_ptr[src_idx_4], src_ptr[src_idx_5]); - typename tileTmp::DType max_67 = blkv_max(src_ptr[src_idx_6], src_ptr[src_idx_7]); - typename tileTmp::DType max_0123 = blkv_max(max_01, max_23); + typename tileTmp::DType max_45 = blkv_max(src_ptr[src_idx_4], src_ptr[src_idx_5]); + typename tileTmp::DType max_67 = blkv_max(src_ptr[src_idx_6], src_ptr[src_idx_7]); + typename tileTmp::DType max_0123 = blkv_max(max_01, max_23); typename tileTmp::DType max_4567 = blkv_max(max_45, max_67); - typename tileTmp::DType max_tmp = blkv_max(max_0123, max_4567); - upd_tmp_max = blkv_max(upd_tmp_max, max_tmp); + typename tileTmp::DType max_tmp = blkv_max(max_0123, max_4567); + upd_tmp_max = blkv_max(upd_tmp_max, max_tmp); } - tmp_max_ptr[i] = upd_tmp_max; + tmp_max_ptr[i] = upd_tmp_max; } // ---- 阶段2：把临时最大值与历史累加器 old_max 合并，写出最终列最大 ---- template void __vec__ reducemax_col_final( typename tileMax::TileDType __out__ new_max, - const typename tileTmp::TileDType __in__ src, - const typename tileMax::TileDType __in__ old_max + const typename tileTmp::TileDType __in__ src, + const typename tileMax::TileDType __in__ old_max ) { - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileMax::DType *new_max_ptr = blkv_get_tile_ptr(new_max); __vbuf__ typename tileTmp::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); + __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); // 取历史最大值 typename tileMax::DType upd_max = old_max_ptr[i]; - + // 对临时 tile 中 8 个切片结果做 8 路树形取 max size_t src_idx_0 = i * tileMax::ColStride + 0 * tileMax::ValidCol; size_t src_idx_1 = i * tileMax::ColStride + 1 * tileMax::ValidCol; size_t src_idx_2 = i * tileMax::ColStride + 2 * tileMax::ValidCol; - size_t src_idx_3 = i * tileMax::ColStride + 3 * tileMax::ValidCol; + size_t src_idx_3 = i * tileMax::ColStride + 3 * tileMax::ValidCol; size_t src_idx_4 = i * tileMax::ColStride + 4 * tileMax::ValidCol; size_t src_idx_5 = i * tileMax::ColStride + 5 * tileMax::ValidCol; size_t src_idx_6 = i * tileMax::ColStride + 6 * tileMax::ValidCol; - size_t src_idx_7 = i * tileMax::ColStride + 7 * tileMax::ValidCol; - typename tileMax::DType max_01 = blkv_max(src_ptr[src_idx_0], src_ptr[src_idx_1]); - typename tileMax::DType max_23 = blkv_max(src_ptr[src_idx_2], src_ptr[src_idx_3]); - typename tileMax::DType max_45 = blkv_max(src_ptr[src_idx_4], src_ptr[src_idx_5]); - typename tileMax::DType max_67 = blkv_max(src_ptr[src_idx_6], src_ptr[src_idx_7]); - typename tileMax::DType max_0123 = blkv_max(max_01, max_23); - typename tileMax::DType max_4567 = blkv_max(max_45, max_67); - typename tileMax::DType max_all = blkv_max(max_0123, max_4567); - -// upd_max = upd_max + max_tmp; + size_t src_idx_7 = i * tileMax::ColStride + 7 * tileMax::ValidCol; + typename tileMax::DType max_01 = blkv_max(src_ptr[src_idx_0], src_ptr[src_idx_1]); + typename tileMax::DType max_23 = blkv_max(src_ptr[src_idx_2], src_ptr[src_idx_3]); + typename tileMax::DType max_45 = blkv_max(src_ptr[src_idx_4], src_ptr[src_idx_5]); + typename tileMax::DType max_67 = blkv_max(src_ptr[src_idx_6], src_ptr[src_idx_7]); + typename tileMax::DType max_0123 = blkv_max(max_01, max_23); + typename tileMax::DType max_4567 = blkv_max(max_45, max_67); + typename tileMax::DType max_all = blkv_max(max_0123, max_4567); + +// upd_max = upd_max + max_tmp; // 合并临时结果与历史最大值 - new_max_ptr[i] = blkv_max(max_all, upd_max); + new_max_ptr[i] = blkv_max(max_all, upd_max); } @@ -124,60 +124,60 @@ void __vec__ reducemax_col_final( template void reducemax_col_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { -// const int Mb = (gIM/8) / tM; +// const int Mb = (gIM/8) / tM; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ // 布局变换：(gIM×gIN) -> (gIM/8 × gIN*8) - using gm_shapeIn = global_tensor>; // -// using gm_shapeMax = global_tensor>; + using gm_shapeIn = global_tensor>; // +// using gm_shapeMax = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // // using tile_shapeData_col = Tile; // - using tile_shapeTmp = Tile; // - using tile_shapeMax = Tile; // + using tile_shapeTmp = Tile; // + using tile_shapeMax = Tile; // -// using tile_shapeData_row = Tile; // -// using tile_shapeData_cor = Tile; // -// using tile_shapeMax_row = Tile; // +// using tile_shapeData_row = Tile; // +// using tile_shapeData_cor = Tile; // +// using tile_shapeMax_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeMax olcMaxGm(old_max_ptr); +// gm_shapeMax olcMaxGm(old_max_ptr); tile_shapeData dataTile; // tile_shapeData_col dataTile_col; - tile_shapeTmp TmpTile; + tile_shapeTmp TmpTile; tile_shapeMax MaxTile; tile_shapeMax oldMaxTile; // tile_shapeData_row dataTile_row; -// tile_shapeData_cor dataTile_cor; +// tile_shapeData_cor dataTile_cor; // tile_shapeMax_row MaxTile_row; -// tile_shapeMax_row oldMaxTile_row; +// tile_shapeMax_row oldMaxTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itIn_row = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itIn_row gIIter_rmd_row(in_ptr); -// itZero gZeroIter(inzero_ptr); +// itZero gZeroIter(inzero_ptr); itOut gOIter(out_ptr); @@ -185,11 +185,11 @@ void reducemax_col_rand( auto gO = gOIter(0, 0); TEXPANDSCALAR(oldMaxTile, 0);//初始化为0 auto gI = gIIter(0, 0); - TCOPYIN(dataTile, gI);//补0的TLOAD + TLOAD(dataTile, gI);//补0的TLOAD reducemax_col_tmp<<>>(TmpTile.data(), dataTile.data()); reducemax_col_final<<>>(MaxTile.data(), TmpTile.data(), oldMaxTile.data()); oldMaxTile = MaxTile; - TCOPYOUT(gO, MaxTile); + TSTORE(gO, MaxTile); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec.hpp b/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec.hpp index fe865ff..75b2914 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec.hpp @@ -28,18 +28,18 @@ template void __vec__ reducemax_row_kernel( typename tileMax::TileDType __out__ new_max, const typename tileSrc::TileDType __in__ src, - const typename tileMax::TileDType __in__ old_max + const typename tileMax::TileDType __in__ old_max ) { -// size_t i = blkv_get_index_x(); +// size_t i = blkv_get_index_x(); // j 为行号（lane 索引），每个 lane 独立取自己那一行的最大值 - size_t j = blkv_get_index_x(); + size_t j = blkv_get_index_x(); // size_t j = blkv_get_index_y(); - size_t idx = j * tileMax::RowStride; + size_t idx = j * tileMax::RowStride; __vbuf__ typename tileMax::DType *new_max_ptr = blkv_get_tile_ptr(new_max); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); + __vbuf__ typename tileMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); // 取上一 tile 的行最大作为初值，保证跨 tile 连续 @@ -52,35 +52,35 @@ void __vec__ reducemax_row_kernel( size_t src_idx0 = i * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx1 = (i+1) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx2 = (i+2) * tileSrc::ColStride + j * tileSrc::RowStride; - size_t src_idx3 = (i+3) * tileSrc::ColStride + j * tileSrc::RowStride; + size_t src_idx3 = (i+3) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx4 = (i+4) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx5 = (i+5) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx6 = (i+6) * tileSrc::ColStride + j * tileSrc::RowStride; - size_t src_idx7 = (i+7) * tileSrc::ColStride + j * tileSrc::RowStride; + size_t src_idx7 = (i+7) * tileSrc::ColStride + j * tileSrc::RowStride; typename tileMax::DType max_01 = blkv_max(src_ptr[src_idx0], src_ptr[src_idx1]); - typename tileMax::DType max_23 = blkv_max(src_ptr[src_idx2], src_ptr[src_idx3]); - typename tileMax::DType max_45 = blkv_max(src_ptr[src_idx4], src_ptr[src_idx5]); - typename tileMax::DType max_67 = blkv_max(src_ptr[src_idx6], src_ptr[src_idx7]); + typename tileMax::DType max_23 = blkv_max(src_ptr[src_idx2], src_ptr[src_idx3]); + typename tileMax::DType max_45 = blkv_max(src_ptr[src_idx4], src_ptr[src_idx5]); + typename tileMax::DType max_67 = blkv_max(src_ptr[src_idx6], src_ptr[src_idx7]); typename tileMax::DType max_0123 = blkv_max(max_01, max_23); - typename tileMax::DType max_4567 = blkv_max(max_45, max_67); + typename tileMax::DType max_4567 = blkv_max(max_45, max_67); typename tileMax::DType max_tmp = blkv_max(max_0123, max_4567); // 与历史最大值比较更新 - upd_max = blkv_max(upd_max, max_tmp); - } + upd_max = blkv_max(upd_max, max_tmp); + } /* #pragma clang loop unroll(full) for(size_t i=0;i>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeMax = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeMax = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_row = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeMax = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_cor = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_shapeMax_col = Tile; + using tile_shapeMax_col = Tile; - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); - tile_shapeData dataTile; + tile_shapeData dataTile; tile_shapeData_row dataTile_row; tile_shapeData_col dataTile_col; - tile_shapeData_cor dataTile_cor; - + tile_shapeData_cor dataTile_cor; + tile_shapeMax MaxTile; tile_shapeMax oldMaxTile; tile_shapeMax_col MaxTile_col; - tile_shapeMax_col oldMaxTile_col; + tile_shapeMax_col oldMaxTile_col; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itOut gOIter(out_ptr); // printf("tile_shapeSum::ValidCol = %d\n", tile_shapeSum::ValidCol); -// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); +// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); // printf("before for\n"); // ---- 主区域：M 方向对齐的行块 ---- for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); TEXPANDSCALAR(oldMaxTile, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile // 沿 N 方向逐 tile 拷入并取最大 for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(j, i); -// printf("before copy in , %d\n", i); - TCOPYIN(dataTile, gI); + auto gI = gIIter(j, i); +// printf("before copy in , %d\n", i); + TLOAD(dataTile, gI); reducemax_row_kernel<<>>(MaxTile.data(), dataTile.data(), oldMaxTile.data()); // reducesum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), dataTile.data(), oldSumTile.data()); // printf("kernel , %d\n", i); @@ -162,41 +162,41 @@ void reducemax_row_rand( // N 方向尾块 if constexpr (rmd_N > 0){ auto gI = gIIter(j, Nb); - TCOPYIN(dataTile_row, gI); - reducemax_row_kernel<<>>(MaxTile.data(), dataTile_row.data(), oldMaxTile.data()); + TLOAD(dataTile_row, gI); + reducemax_row_kernel<<>>(MaxTile.data(), dataTile_row.data(), oldMaxTile.data()); // reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); oldMaxTile = MaxTile; } -// printf("before tcopyout\n"); - TCOPYOUT(gO, MaxTile); -// printf("end tcopyout\n"); +// printf("before tstore\n"); + TSTORE(gO, MaxTile); +// printf("end tstore\n"); } //for col cor // ---- M 方向尾块（rmd_M>0）---- if constexpr (rmd_M > 0){ auto gO = gOIter(Mb, 0); TEXPANDSCALAR(oldMaxTile_col, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile for (int i = 0; i < Nb; ++i) { auto gI = gIIter(Mb, i); - TCOPYIN(dataTile_col, gI); + TLOAD(dataTile_col, gI); reducemax_row_kernel<<>>(MaxTile_col.data(), dataTile_col.data(), oldMaxTile_col.data()); oldMaxTile_col = MaxTile_col; } if constexpr (rmd_N > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reducemax_row_kernel<<>>(MaxTile_col.data(), dataTile_cor.data(), oldMaxTile_col.data()); oldMaxTile_col = MaxTile_col; } - TCOPYOUT(gO, MaxTile_col); + TSTORE(gO, MaxTile_col); } /* for(int i = 0; i < gIM; i++){ printf("out%d = %d\n", i, out_ptr[i]); } */ -// printf("end program\n"); +// printf("end program\n"); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec_single_tree.hpp b/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec_single_tree.hpp index 5343ae2..e12de1c 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec_single_tree.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducemax_rowvec_single_tree.hpp @@ -33,28 +33,28 @@ void __vec__ reducemax_row_kernel( const typename tileSrc::TileDType __in__ src, const typename tileSrcCol::TileDType __in__ src_col, const typename tileTmpMax::TileDType __in__ old_max, - const size_t tile_idx + const size_t tile_idx ) { // j 为行 lane，z 为 4 路并行的段索引 - size_t j = blkv_get_index_x(); - size_t z = blkv_get_index_y(); + size_t j = blkv_get_index_x(); + size_t z = blkv_get_index_y(); // 本段在 src / src_col 中的基地址偏移 size_t stride_src = z * (tileSrc::ValidCol/4) * tileSrc::ColStride; - size_t stride_src_col = z * (tileSrcCol::ValidCol/4) * tileSrcCol::ColStride; - + size_t stride_src_col = z * (tileSrcCol::ValidCol/4) * tileSrcCol::ColStride; + __vbuf__ typename tileTmpMax::DType *new_max_ptr = blkv_get_tile_ptr(new_max); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSrc::DType *src_col_ptr = blkv_get_tile_ptr(src_col); - __vbuf__ typename tileTmpMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); + __vbuf__ typename tileSrc::DType *src_col_ptr = blkv_get_tile_ptr(src_col); + __vbuf__ typename tileTmpMax::DType *old_max_ptr = blkv_get_tile_ptr(old_max); /* - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t i=0;i>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeMax = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeMax = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 // 列主序中间缓冲：tN/8 列 - using tile_shapeDataCol = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeDataCol = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeMax = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // 中间 tile：列主序，64 列、有效列=Nb*4（4 路并行 × Nb 个 tile） using tile_shapeTmpMax = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeMax olcMaxGm(old_max_ptr); +// gm_shapeMax olcMaxGm(old_max_ptr); - tile_shapeData dataTile; - tile_shapeDataCol dataTile_col; + tile_shapeData dataTile; + tile_shapeDataCol dataTile_col; tile_shapeMax MaxTile; tile_shapeTmpMax oldtmpMaxTile; tile_shapeTmpMax tmpMaxTile; @@ -236,7 +236,7 @@ void reducemax_row_rand( // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); @@ -244,23 +244,23 @@ void reducemax_row_rand( auto gO = gOIter(0, 0); - TEXPANDSCALAR(oldtmpMaxTile, 0);//初始化为0 - TEXPANDSCALAR(dataTile_col, 0);//初始化为0 + TEXPANDSCALAR(oldtmpMaxTile, 0);//初始化为0 + TEXPANDSCALAR(dataTile_col, 0);//初始化为0 // 阶段1：逐 tile 拷入并做 4 路并行多级树形取最大，写入中间 tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(0, i); - TCOPYIN(dataTile, gI); - reducemax_row_kernel<<>>(tmpMaxTile.data(), - dataTile.data(), - dataTile_col.data(), + auto gI = gIIter(0, i); + TLOAD(dataTile, gI); + reducemax_row_kernel<<>>(tmpMaxTile.data(), + dataTile.data(), + dataTile_col.data(), oldtmpMaxTile.data(), i); oldtmpMaxTile = tmpMaxTile; } // 阶段2：最终归约并写出 - reducemax_row_final_kernel<<>>(MaxTile.data(), - tmpMaxTile.data()); - TCOPYOUT(gO, MaxTile); + reducemax_row_final_kernel<<>>(MaxTile.data(), + tmpMaxTile.data()); + TSTORE(gO, MaxTile); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reduceprod_colvec.hpp b/benchmark/two-level-arch/kernels/reduction/reduceprod_colvec.hpp index 43387a6..8458724 100644 --- a/benchmark/two-level-arch/kernels/reduction/reduceprod_colvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reduceprod_colvec.hpp @@ -31,15 +31,15 @@ template void __vec__ reduceprod_col_kernel( typename timeProd::TileDType __out__ new_prod, const typename tileSrc::TileDType __in__ src, - const typename timeProd::TileDType __in__ old_prod + const typename timeProd::TileDType __in__ old_prod ) { // 当前 lane 索引：每个 lane 负责一列的连乘 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename timeProd::DType *new_prod_ptr = blkv_get_tile_ptr(new_prod); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename timeProd::DType *old_prod_ptr = blkv_get_tile_ptr(old_prod); + __vbuf__ typename timeProd::DType *old_prod_ptr = blkv_get_tile_ptr(old_prod); // 取上一 tile 的列积作为初值，保证跨 tile 连乘连续 @@ -49,9 +49,9 @@ void __vec__ reduceprod_col_kernel( #pragma clang loop unroll(full) for(size_t j=0;j void reduceprod_col_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { const int Mb = gIM / tM; - const int Nb = gIN / tN; + const int Nb = gIN / tN; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; // -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; // +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // - using tile_shapeData_col = Tile; // - using tile_shapeProd = Tile; // + using tile_shapeData_col = Tile; // + using tile_shapeProd = Tile; // - using tile_shapeData_row = Tile; // - using tile_shapeData_cor = Tile; // - using tile_shapeProd_row = Tile; // + using tile_shapeData_row = Tile; // + using tile_shapeData_cor = Tile; // + using tile_shapeProd_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); tile_shapeData dataTile; - tile_shapeData_col dataTile_col; + tile_shapeData_col dataTile_col; tile_shapeProd ProdTile; tile_shapeProd oldProdTile; tile_shapeData_row dataTile_row; - tile_shapeData_cor dataTile_cor; + tile_shapeData_cor dataTile_cor; tile_shapeProd_row ProdTile_row; - tile_shapeProd_row oldProdTile_row; + tile_shapeProd_row oldProdTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itIn_row = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itIn_row gIIter_rmd_row(in_ptr); -// itZero gZeroIter(inzero_ptr); +// itZero gZeroIter(inzero_ptr); itOut gOIter(out_ptr); // dtype zero = 0; @@ -124,44 +124,44 @@ void reduceprod_col_rand( // auto gZero = gZeroIter(0, j); auto gO = gOIter(0, j); TEXPANDSCALAR(oldProdTile, 0);//初始化为0 -// TCOPYIN(oldSumTile, gZero);//初始化为0 - //初始化old_sum的tile - //need +// TLOAD(oldSumTile, gZero);//初始化为0 + //初始化old_sum的tile + //need // 沿 M 方向逐 tile 拷入并连乘 for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); - TCOPYIN(dataTile, gI); + TLOAD(dataTile, gI); reduceprod_col_kernel<<>>(ProdTile.data(), dataTile.data(), oldProdTile.data()); oldProdTile = ProdTile; } // M 方向尾块 - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, j); - TCOPYIN(dataTile_col, gI); + TLOAD(dataTile_col, gI); reduceprod_col_kernel<<>>(ProdTile.data(), dataTile_col.data(), oldProdTile.data()); oldProdTile = ProdTile; } - TCOPYOUT(gO, ProdTile); + TSTORE(gO, ProdTile); } // ---- N 方向尾块区域 ---- if constexpr (rmd_N > 0){ -// auto gZero = gZeroIter(0, Nb); +// auto gZero = gZeroIter(0, Nb); auto gO = gOIter(0, Nb); - TEXPANDSCALAR(oldProdTile_row, 0);//初始化为0 -// TCOPYIN(oldSumTile_row, gZero);//初始化为0 - for (int i = 0; i < Mb; ++i) { + TEXPANDSCALAR(oldProdTile_row, 0);//初始化为0 +// TLOAD(oldSumTile_row, gZero);//初始化为0 + for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); - TCOPYIN(dataTile_row, gI); + TLOAD(dataTile_row, gI); reduceprod_col_kernel<<>>(ProdTile_row.data(), dataTile_row.data(), oldProdTile_row.data()); oldProdTile_row = ProdTile_row; } - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reduceprod_col_kernel<<>>(ProdTile_row.data(), dataTile_cor.data(), oldProdTile_row.data()); oldProdTile_row = ProdTile_row; } - TCOPYOUT(gO, ProdTile_row); + TSTORE(gO, ProdTile_row); } } diff --git a/benchmark/two-level-arch/kernels/reduction/reduceprod_rowvec.hpp b/benchmark/two-level-arch/kernels/reduction/reduceprod_rowvec.hpp index db46eaf..92f98d5 100644 --- a/benchmark/two-level-arch/kernels/reduction/reduceprod_rowvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reduceprod_rowvec.hpp @@ -28,18 +28,18 @@ template void __vec__ reduceprod_row_kernel( typename tileProd::TileDType __out__ new_prod, const typename tileSrc::TileDType __in__ src, - const typename tileProd::TileDType __in__ old_prod + const typename tileProd::TileDType __in__ old_prod ) { -// size_t i = blkv_get_index_x(); +// size_t i = blkv_get_index_x(); // j 为行号（lane 索引），每个 lane 独立连乘自己那一行 - size_t j = blkv_get_index_x(); + size_t j = blkv_get_index_x(); // size_t j = blkv_get_index_y(); - size_t idx = j * tileProd::RowStride; + size_t idx = j * tileProd::RowStride; __vbuf__ typename tileProd::DType *new_prod_ptr = blkv_get_tile_ptr(new_prod); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileProd::DType *old_prod_ptr = blkv_get_tile_ptr(old_prod); + __vbuf__ typename tileProd::DType *old_prod_ptr = blkv_get_tile_ptr(old_prod); // 取上一 tile 的行积作为初值，保证跨 tile 连乘连续 @@ -49,9 +49,9 @@ void __vec__ reduceprod_row_kernel( #pragma clang loop unroll(full) for(size_t i=0;i>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_row = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeProd = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_cor = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_shapeProd_col = Tile; + using tile_shapeProd_col = Tile; - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); - tile_shapeData dataTile; + tile_shapeData dataTile; tile_shapeData_row dataTile_row; tile_shapeData_col dataTile_col; - tile_shapeData_cor dataTile_cor; - + tile_shapeData_cor dataTile_cor; + tile_shapeProd ProdTile; tile_shapeProd oldProdTile; tile_shapeProd_col ProdTile_col; - tile_shapeProd_col oldProdTile_col; + tile_shapeProd_col oldProdTile_col; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itOut gOIter(out_ptr); // printf("tile_shapeSum::ValidCol = %d\n", tile_shapeSum::ValidCol); -// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); +// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); // printf("before for\n"); // ---- 主区域：M 方向对齐的行块 ---- for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); TEXPANDSCALAR(oldProdTile, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile // 沿 N 方向逐 tile 拷入并连乘 for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(j, i); -// printf("before copy in , %d\n", i); - TCOPYIN(dataTile, gI); + auto gI = gIIter(j, i); +// printf("before copy in , %d\n", i); + TLOAD(dataTile, gI); reduceprod_row_kernel<<>>(ProdTile.data(), dataTile.data(), oldProdTile.data()); // reducesum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), dataTile.data(), oldSumTile.data()); // printf("kernel , %d\n", i); @@ -133,41 +133,41 @@ void reduceprod_row_rand( // N 方向尾块 if constexpr (rmd_N > 0){ auto gI = gIIter(j, Nb); - TCOPYIN(dataTile_row, gI); - reduceprod_row_kernel<<>>(ProdTile.data(), dataTile_row.data(), oldProdTile.data()); + TLOAD(dataTile_row, gI); + reduceprod_row_kernel<<>>(ProdTile.data(), dataTile_row.data(), oldProdTile.data()); // reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); oldProdTile = ProdTile; } -// printf("before tcopyout\n"); - TCOPYOUT(gO, ProdTile); -// printf("end tcopyout\n"); +// printf("before tstore\n"); + TSTORE(gO, ProdTile); +// printf("end tstore\n"); } //for col cor // ---- M 方向尾块（rmd_M>0）---- if constexpr (rmd_M > 0){ auto gO = gOIter(Mb, 0); TEXPANDSCALAR(oldProdTile_col, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(Mb, i); - TCOPYIN(dataTile_col, gI); + auto gI = gIIter(Mb, i); + TLOAD(dataTile_col, gI); reduceprod_row_kernel<<>>(ProdTile_col.data(), dataTile_col.data(), oldProdTile_col.data()); oldProdTile_col = ProdTile_col; } if constexpr (rmd_N > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reduceprod_row_kernel<<>>(ProdTile_col.data(), dataTile_cor.data(), oldProdTile_col.data()); oldProdTile_col = ProdTile_col; } - TCOPYOUT(gO, ProdTile_col); + TSTORE(gO, ProdTile_col); } /* for(int i = 0; i < gIM; i++){ printf("out%d = %d\n", i, out_ptr[i]); } */ -// printf("end program\n"); +// printf("end program\n"); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec.hpp b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec.hpp index 3fe9b3f..76d2ca7 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec.hpp @@ -34,24 +34,24 @@ template void __vec__ reducesum_col_kernel( typename tileSum::TileDType __out__ new_sum, const typename tileSrc::TileDType __in__ src, - const typename tileSum::TileDType __in__ old_sum + const typename tileSum::TileDType __in__ old_sum ) { // 取当前向量单元在 x 方向的 lane 索引：每个 lane 负责一列的累加 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); // 获取各 tile 在向量内存中的基地址指针 __vbuf__ typename tileSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); // typename tileSum::DType upd_sum = old_sum_ptr[i]; // 本 tile 内的局部累加器，先清零（与历史 old_sum 在最后再合并） typename tileSum::DType upd_sum = 0; - + // 8 路展开的树形归约：每次同时取 8 个相邻行做两两求和，再树形合并 - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t j=0;j void reducesum_colsum_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { // 主块数量与尾块余数（用于非对齐维度的边角处理） const int Mb = gIM / tM; - const int Nb = gIN / tN; + const int Nb = gIN / tN; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ // 全局输入/输出张量形状定义 - using gm_shapeIn = global_tensor>; // -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; // +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; // 各类分块形状：主块、列尾块（M 不对齐）、行尾块（N 不对齐）、角块（M、N 都不对齐） using tile_shapeData = Tile; // - using tile_shapeData_col = Tile; // - using tile_shapeSum = Tile; // + using tile_shapeData_col = Tile; // + using tile_shapeSum = Tile; // - using tile_shapeData_row = Tile; // - using tile_shapeData_cor = Tile; // - using tile_shapeSum_row = Tile; // + using tile_shapeData_row = Tile; // + using tile_shapeData_cor = Tile; // + using tile_shapeSum_row = Tile; // //need tM = 1; // 实例化全局张量与本地 tile - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); tile_shapeData dataTile; - tile_shapeData_col dataTile_col; + tile_shapeData_col dataTile_col; tile_shapeSum SumTile; tile_shapeSum oldSumTile; tile_shapeData_row dataTile_row; - tile_shapeData_cor dataTile_cor; + tile_shapeData_cor dataTile_cor; tile_shapeSum_row SumTile_row; - tile_shapeSum_row oldSumTile_row; + tile_shapeSum_row oldSumTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 // 全局迭代器：按 tile 粒度遍历全局内存 - using itIn = global_iterator; + using itIn = global_iterator; using itIn_row = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itIn_row gIIter_rmd_row(in_ptr); -// itZero gZeroIter(inzero_ptr); +// itZero gZeroIter(inzero_ptr); itOut gOIter(out_ptr); // dtype zero = 0; @@ -165,44 +165,44 @@ void reducesum_colsum_rand( // auto gZero = gZeroIter(0, j); auto gO = gOIter(0, j); TEXPANDSCALAR(oldSumTile, 0);//初始化为0 -// TCOPYIN(oldSumTile, gZero);//初始化为0 - //初始化old_sum的tile - //need +// TLOAD(oldSumTile, gZero);//初始化为0 + //初始化old_sum的tile + //need // 沿 M 方向逐 tile 拷入并累加 for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, j); - TCOPYIN(dataTile, gI); + TLOAD(dataTile, gI); reducesum_col_kernel<<>>(SumTile.data(), dataTile.data(), oldSumTile.data()); oldSumTile = SumTile; } // 处理 M 方向尾块（rmd_M>0 时使用列尾块形状） - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, j); - TCOPYIN(dataTile_col, gI); + TLOAD(dataTile_col, gI); reducesum_col_kernel<<>>(SumTile.data(), dataTile_col.data(), oldSumTile.data()); oldSumTile = SumTile; } - TCOPYOUT(gO, SumTile); + TSTORE(gO, SumTile); } // ---- N 方向尾块区域（rmd_N>0）：使用行尾块/角块形状 ---- if constexpr (rmd_N > 0){ -// auto gZero = gZeroIter(0, Nb); +// auto gZero = gZeroIter(0, Nb); auto gO = gOIter(0, Nb); - TEXPANDSCALAR(oldSumTile_row, 0);//初始化为0 -// TCOPYIN(oldSumTile_row, gZero);//初始化为0 - for (int i = 0; i < Mb; ++i) { + TEXPANDSCALAR(oldSumTile_row, 0);//初始化为0 +// TLOAD(oldSumTile_row, gZero);//初始化为0 + for (int i = 0; i < Mb; ++i) { auto gI = gIIter(i, Nb); - TCOPYIN(dataTile_row, gI); + TLOAD(dataTile_row, gI); reducesum_col_kernel<<>>(SumTile_row.data(), dataTile_row.data(), oldSumTile_row.data()); oldSumTile_row = SumTile_row; } - if constexpr (rmd_M > 0){ + if constexpr (rmd_M > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reducesum_col_kernel<<>>(SumTile_row.data(), dataTile_cor.data(), oldSumTile_row.data()); oldSumTile_row = SumTile_row; } - TCOPYOUT(gO, SumTile_row); + TSTORE(gO, SumTile_row); } } diff --git a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_single_tree.hpp b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_single_tree.hpp index 48c3b54..8115250 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_single_tree.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_single_tree.hpp @@ -39,43 +39,43 @@ void __vec__ reducesum_col_kernel( typename tileTmpSum::TileDType __out__ new_sum, const typename tileSrc::TileDType __in__ src, const typename tileTmpSum::TileDType __in__ old_sum, - const size_t tile_idx + const size_t tile_idx ) { // 当前 lane 索引：每个 lane 负责一列 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileTmpSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileTmpSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileTmpSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); // 先把 old_sum 的内容拷到 new_sum（保留历史值，后续合并） - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t j=0;j void reducesum_colsum_rand( - dtype *in_ptr, + dtype *in_ptr, dtype *out_ptr -) +) { const int Mb = gIM / tM; - const int Nb = gIN / tN; + const int Nb = gIN / tN; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; // + using gm_shapeIn = global_tensor>; // using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // - using tile_shapeData_col = Tile; // - using tile_shapeSum = Tile; // + using tile_shapeData_col = Tile; // + using tile_shapeSum = Tile; // // 中间 tile：行数=Mb（每个输入 tile 占一行槽位），用于两阶段合并 - using tile_shapeTmpSum = Tile; // -// using tile_shapeTmpSum_l2 = Tile; // + using tile_shapeTmpSum = Tile; // +// using tile_shapeTmpSum_l2 = Tile; // -// using tile_shapeData_row = Tile; // -// using tile_shapeData_cor = Tile; // -// using tile_shapeSum_row = Tile; // +// using tile_shapeData_row = Tile; // +// using tile_shapeData_cor = Tile; // +// using tile_shapeSum_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); - gm_shapeOut outGm(out_ptr); + gm_shapeIn inGm(in_ptr); + gm_shapeOut outGm(out_ptr); tile_shapeData dataTile; - tile_shapeData_col dataTile_col; + tile_shapeData_col dataTile_col; tile_shapeSum SumTile; tile_shapeTmpSum oldtmpSumTile; tile_shapeTmpSum tmpSumTile; // tile_shapeTmpSum_l2 tmpSumTile_l2; // tile_shapeData_row dataTile_row; -// tile_shapeData_cor dataTile_cor; +// tile_shapeData_cor dataTile_cor; // tile_shapeSum_row SumTile_row; -// tile_shapeSum_row oldSumTile_row; +// tile_shapeSum_row oldSumTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 @@ -232,7 +232,7 @@ void reducesum_colsum_rand( using itIn = global_iterator; using itOut = global_iterator; - itIn gIIter(in_ptr); + itIn gIIter(in_ptr); itOut gOIter(out_ptr); // dtype zero = 0; @@ -242,21 +242,21 @@ void reducesum_colsum_rand( auto gO = gOIter(0, 0); TEXPANDSCALAR(oldtmpSumTile, 0);//初始化为0 // TEXPANDSCALAR(tmpSumTile, 0);//初始化为0 -// TEXPANDSCALAR(tmpSumTile_l2, 0);//初始化为0 +// TEXPANDSCALAR(tmpSumTile_l2, 0);//初始化为0 // 阶段1：逐 tile 拷入并做单 tile 树形归约，结果写入中间 tile 的对应槽位 for (size_t i = 0; i < Mb; ++i){ auto gI = gIIter(i, 0); - TCOPYIN(dataTile, gI); - reducesum_col_kernel<<>>(tmpSumTile.data(), + TLOAD(dataTile, gI); + reducesum_col_kernel<<>>(tmpSumTile.data(), dataTile.data(), - oldtmpSumTile.data(), + oldtmpSumTile.data(), i); oldtmpSumTile = tmpSumTile; } // 阶段2：对中间 tile 做最终归约，写出结果 - reducesum_col_final_kernel<<>>(SumTile.data(), + reducesum_col_final_kernel<<>>(SumTile.data(), tmpSumTile.data()); - TCOPYOUT(gO, SumTile); + TSTORE(gO, SumTile); } diff --git a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8.hpp b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8.hpp index 88897ed..2ed31ca 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducesum_colvec_unalign_120_8.hpp @@ -22,7 +22,7 @@ using namespace pto; // 1) 专为批量 3D 张量（Nums×gIM×gIN）设计，外层循环处理 Nums 个切片； // 2) 对非二次幂行数（如 120 行）做布局变换：把 (gIM×gIN) 重排为 // (gIM/8 × gIN*8)，以便凑齐 8 元向量树形累加； -// 3) 利用补零（TLOAD/TCOPYIN 默认补 0）让非 valid 区也参与 8 路归约， +// 3) 利用补零（TLOAD/TLOAD 默认补 0）让非 valid 区也参与 8 路归约， // 不影响结果且无需特殊尾块处理。 // 两阶段结构： // reducesum_col_tmp() : 每 tile 列归约到临时结果； @@ -37,22 +37,22 @@ using namespace pto; template void __vec__ reducesum_col_tmp( typename tileTmp::TileDType __out__ tmp_sum, - const typename tileSrc::TileDType __in__ src + const typename tileSrc::TileDType __in__ src ) { // 当前 lane 负责一列 - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileTmp::DType *tmp_sum_ptr = blkv_get_tile_ptr(tmp_sum); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); -// __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); +// __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); // 局部累加器清零 typename tileTmp::DType upd_tmp_sum = 0; - + // 注意：此处用 tileSrc::Rows（而非 ValidRow），让非 valid 的补零区也参与计算， // 从而凑出完整的 8 元树形累加，省去尾块特判 - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t j=0;j 树形合并 - typename tileTmp::DType sum_01 = src_ptr[src_idx_0] + src_ptr[src_idx_1]; + typename tileTmp::DType sum_01 = src_ptr[src_idx_0] + src_ptr[src_idx_1]; typename tileTmp::DType sum_23 = src_ptr[src_idx_2] + src_ptr[src_idx_3]; - typename tileTmp::DType sum_45 = src_ptr[src_idx_4] + src_ptr[src_idx_5]; - typename tileTmp::DType sum_67 = src_ptr[src_idx_6] + src_ptr[src_idx_7]; - typename tileTmp::DType sum_0123 = sum_01 + sum_23; + typename tileTmp::DType sum_45 = src_ptr[src_idx_4] + src_ptr[src_idx_5]; + typename tileTmp::DType sum_67 = src_ptr[src_idx_6] + src_ptr[src_idx_7]; + typename tileTmp::DType sum_0123 = sum_01 + sum_23; typename tileTmp::DType sum_4567 = sum_45 + sum_67; - typename tileTmp::DType sum_tmp = sum_0123 + sum_4567; - upd_tmp_sum = upd_tmp_sum + sum_tmp; + typename tileTmp::DType sum_tmp = sum_0123 + sum_4567; + upd_tmp_sum = upd_tmp_sum + sum_tmp; } - tmp_sum_ptr[i] = upd_tmp_sum; + tmp_sum_ptr[i] = upd_tmp_sum; } // ---- 阶段2：把临时和与历史累加器 old_sum 合并，写出最终列和 ---- template void __vec__ reducesum_col_final( typename tileSum::TileDType __out__ new_sum, - const typename tileTmp::TileDType __in__ src, - const typename tileSum::TileDType __in__ old_sum + const typename tileTmp::TileDType __in__ src, + const typename tileSum::TileDType __in__ old_sum ) { - size_t i = blkv_get_index_x(); + size_t i = blkv_get_index_x(); __vbuf__ typename tileSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); __vbuf__ typename tileTmp::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); // 取历史累加值（跨切片/跨 tile 的累积） typename tileSum::DType upd_sum = old_sum_ptr[i]; - + // 对临时 tile 中 8 个切片结果再做一次 8 路树形合并 size_t src_idx_0 = i * tileSum::ColStride + 0 * tileSum::ValidCol; size_t src_idx_1 = i * tileSum::ColStride + 1 * tileSum::ValidCol; size_t src_idx_2 = i * tileSum::ColStride + 2 * tileSum::ValidCol; - size_t src_idx_3 = i * tileSum::ColStride + 3 * tileSum::ValidCol; + size_t src_idx_3 = i * tileSum::ColStride + 3 * tileSum::ValidCol; size_t src_idx_4 = i * tileSum::ColStride + 4 * tileSum::ValidCol; size_t src_idx_5 = i * tileSum::ColStride + 5 * tileSum::ValidCol; size_t src_idx_6 = i * tileSum::ColStride + 6 * tileSum::ValidCol; - size_t src_idx_7 = i * tileSum::ColStride + 7 * tileSum::ValidCol; - typename tileSum::DType sum_01 = src_ptr[src_idx_0] + src_ptr[src_idx_1]; - typename tileSum::DType sum_23 = src_ptr[src_idx_2] + src_ptr[src_idx_3]; - typename tileSum::DType sum_45 = src_ptr[src_idx_4] + src_ptr[src_idx_5]; - typename tileSum::DType sum_67 = src_ptr[src_idx_6] + src_ptr[src_idx_7]; - typename tileSum::DType sum_0123 = sum_01 + sum_23; - typename tileSum::DType sum_4567 = sum_45 + sum_67; - typename tileSum::DType sum_all = sum_0123 + sum_4567; - -// upd_sum = upd_sum + sum_tmp; + size_t src_idx_7 = i * tileSum::ColStride + 7 * tileSum::ValidCol; + typename tileSum::DType sum_01 = src_ptr[src_idx_0] + src_ptr[src_idx_1]; + typename tileSum::DType sum_23 = src_ptr[src_idx_2] + src_ptr[src_idx_3]; + typename tileSum::DType sum_45 = src_ptr[src_idx_4] + src_ptr[src_idx_5]; + typename tileSum::DType sum_67 = src_ptr[src_idx_6] + src_ptr[src_idx_7]; + typename tileSum::DType sum_0123 = sum_01 + sum_23; + typename tileSum::DType sum_4567 = sum_45 + sum_67; + typename tileSum::DType sum_all = sum_0123 + sum_4567; + +// upd_sum = upd_sum + sum_tmp; /* #pragma clang loop unroll(full) for(size_t j=0;j void reducesum_colsum_rand( dtype *in_ptr, -// dtype *inzero_ptr, +// dtype *inzero_ptr, dtype *out_ptr -) +) { -// const int Mb = (gIM/8) / tM; +// const int Mb = (gIM/8) / tM; const int rmd_M = gIM % tM; const int rmd_N = gIN % tN; // const int rmd_M = gOM % tM; // todo 尾块怎么处理？ // 关键布局变换：(gIM×gIN) -> (gIM/8 × gIN*8)，使每 8 行拍平到列上做向量化 - using gm_shapeIn = global_tensor>; // -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; // +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; // tile 行维度也除以 8、列维度乘 8；tM_VLD/8 为有效行（用于非对齐补零） using tile_shapeData = Tile; // // using tile_shapeData_col = Tile; // - using tile_shapeTmp = Tile; // - using tile_shapeSum = Tile; // + using tile_shapeTmp = Tile; // + using tile_shapeSum = Tile; // -// using tile_shapeData_row = Tile; // -// using tile_shapeData_cor = Tile; // -// using tile_shapeSum_row = Tile; // +// using tile_shapeData_row = Tile; // +// using tile_shapeData_cor = Tile; // +// using tile_shapeSum_row = Tile; // //need tM = 1; - gm_shapeIn inGm(in_ptr); -// gm_shapeOut ZeroGm(inzero_ptr); + gm_shapeIn inGm(in_ptr); +// gm_shapeOut ZeroGm(inzero_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); tile_shapeData dataTile; // tile_shapeData_col dataTile_col; - tile_shapeTmp TmpTile; + tile_shapeTmp TmpTile; tile_shapeSum SumTile; tile_shapeSum oldSumTile; // tile_shapeData_row dataTile_row; -// tile_shapeData_cor dataTile_cor; +// tile_shapeData_cor dataTile_cor; // tile_shapeSum_row SumTile_row; -// tile_shapeSum_row oldSumTile_row; +// tile_shapeSum_row oldSumTile_row; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itIn_row = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itIn_row gIIter_rmd_row(in_ptr); -// itZero gZeroIter(inzero_ptr); +// itZero gZeroIter(inzero_ptr); itOut gOIter(out_ptr); @@ -199,11 +199,11 @@ void reducesum_colsum_rand( auto gO = gOIter(0, 0); TEXPANDSCALAR(oldSumTile, 0);//初始化为0 auto gI = gIIter(0, 0); - TCOPYIN(dataTile, gI);//TLOAD应补0，目前gfrun默认补0，需要接口去弄 + TLOAD(dataTile, gI);//TLOAD应补0，目前gfrun默认补0，需要接口去弄 reducesum_col_tmp<<>>(TmpTile.data(), dataTile.data()); reducesum_col_final<<>>(SumTile.data(), TmpTile.data(), oldSumTile.data()); oldSumTile = SumTile; - TCOPYOUT(gO, SumTile); + TSTORE(gO, SumTile); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec.hpp b/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec.hpp index bc1ed21..8da5cc4 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec.hpp @@ -29,18 +29,18 @@ template void __vec__ reducesum_row_kernel( typename tileSum::TileDType __out__ new_sum, const typename tileSrc::TileDType __in__ src, - const typename tileSum::TileDType __in__ old_sum + const typename tileSum::TileDType __in__ old_sum ) { -// size_t i = blkv_get_index_x(); +// size_t i = blkv_get_index_x(); // j 为行号（lane 索引），每个 lane 独立累加自己那一行 - size_t j = blkv_get_index_x(); + size_t j = blkv_get_index_x(); // size_t j = blkv_get_index_y(); - size_t idx = j * tileSum::RowStride; + size_t idx = j * tileSum::RowStride; __vbuf__ typename tileSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); // 取上一 tile 的行和作为初值，保证跨 tile 累加连续 @@ -52,32 +52,32 @@ void __vec__ reducesum_row_kernel( size_t src_idx0 = i * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx1 = (i+1) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx2 = (i+2) * tileSrc::ColStride + j * tileSrc::RowStride; - size_t src_idx3 = (i+3) * tileSrc::ColStride + j * tileSrc::RowStride; + size_t src_idx3 = (i+3) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx4 = (i+4) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx5 = (i+5) * tileSrc::ColStride + j * tileSrc::RowStride; size_t src_idx6 = (i+6) * tileSrc::ColStride + j * tileSrc::RowStride; - size_t src_idx7 = (i+7) * tileSrc::ColStride + j * tileSrc::RowStride; + size_t src_idx7 = (i+7) * tileSrc::ColStride + j * tileSrc::RowStride; typename tileSum::DType sum_01 = src_ptr[src_idx0] + src_ptr[src_idx1]; - typename tileSum::DType sum_23 = src_ptr[src_idx2] + src_ptr[src_idx3]; - typename tileSum::DType sum_45 = src_ptr[src_idx4] + src_ptr[src_idx5]; - typename tileSum::DType sum_67 = src_ptr[src_idx6] + src_ptr[src_idx7]; + typename tileSum::DType sum_23 = src_ptr[src_idx2] + src_ptr[src_idx3]; + typename tileSum::DType sum_45 = src_ptr[src_idx4] + src_ptr[src_idx5]; + typename tileSum::DType sum_67 = src_ptr[src_idx6] + src_ptr[src_idx7]; typename tileSum::DType sum_0123 = sum_01 + sum_23; - typename tileSum::DType sum_4567 = sum_45 + sum_67; + typename tileSum::DType sum_4567 = sum_45 + sum_67; typename tileSum::DType sum_tmp = sum_0123 + sum_4567; - upd_sum = upd_sum + sum_tmp; - } + upd_sum = upd_sum + sum_tmp; + } -/* +/* for(size_t i=0;i>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_row = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeSum = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeData_col = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeData_cor = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_shapeSum_col = Tile; + using tile_shapeSum_col = Tile; - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); - tile_shapeData dataTile; + tile_shapeData dataTile; tile_shapeData_row dataTile_row; tile_shapeData_col dataTile_col; - tile_shapeData_cor dataTile_cor; - + tile_shapeData_cor dataTile_cor; + tile_shapeSum SumTile; tile_shapeSum oldSumTile; tile_shapeSum_col SumTile_col; - tile_shapeSum_col oldSumTile_col; + tile_shapeSum_col oldSumTile_col; // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); itOut gOIter(out_ptr); // printf("tile_shapeSum::ValidCol = %d\n", tile_shapeSum::ValidCol); -// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); +// printf("tile_shapeSum::ValidRow = %d\n", tile_shapeSum::ValidRow); // printf("before for\n"); // ---- 主区域：M 方向对齐的行块 ---- for (int j = 0; j < Mb; ++j) { auto gO = gOIter(j, 0); TEXPANDSCALAR(oldSumTile, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile // 沿 N 方向逐 tile 拷入并累加 for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(j, i); -// printf("before copy in , %d\n", i); - TCOPYIN(dataTile, gI); + auto gI = gIIter(j, i); +// printf("before copy in , %d\n", i); + TLOAD(dataTile, gI); reducesum_row_kernel<<>>(SumTile.data(), dataTile.data(), oldSumTile.data()); // reducesum_row_kernel<<<1, tile_shapeSum::ValidRow, 1>>>(SumTile.data(), dataTile.data(), oldSumTile.data()); // printf("kernel , %d\n", i); @@ -161,41 +161,41 @@ void reducesum_trowsum_rand( // N 方向尾块 if constexpr (rmd_N > 0){ auto gI = gIIter(j, Nb); - TCOPYIN(dataTile_row, gI); - reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); + TLOAD(dataTile_row, gI); + reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); // reducesum_row_kernel<<>>(SumTile.data(), dataTile_row.data(), oldSumTile.data()); oldSumTile = SumTile; } -// printf("before tcopyout\n"); - TCOPYOUT(gO, SumTile); -// printf("end tcopyout\n"); +// printf("before tstore\n"); + TSTORE(gO, SumTile); +// printf("end tstore\n"); } //for col cor // ---- M 方向尾块（rmd_M>0）：使用列尾/角块形状 ---- if constexpr (rmd_M > 0){ auto gO = gOIter(Mb, 0); TEXPANDSCALAR(oldSumTile_col, 0);//初始化为0 - //初始化old_sum的tile + //初始化old_sum的tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(Mb, i); - TCOPYIN(dataTile_col, gI); + auto gI = gIIter(Mb, i); + TLOAD(dataTile_col, gI); reducesum_row_kernel<<>>(SumTile_col.data(), dataTile_col.data(), oldSumTile_col.data()); oldSumTile_col = SumTile_col; } if constexpr (rmd_N > 0){ auto gI = gIIter(Mb, Nb); - TCOPYIN(dataTile_cor, gI); + TLOAD(dataTile_cor, gI); reducesum_row_kernel<<>>(SumTile_col.data(), dataTile_cor.data(), oldSumTile_col.data()); oldSumTile_col = SumTile_col; } - TCOPYOUT(gO, SumTile_col); + TSTORE(gO, SumTile_col); } /* for(int i = 0; i < gIM; i++){ printf("out%d = %d\n", i, out_ptr[i]); } */ -// printf("end program\n"); +// printf("end program\n"); } #endif diff --git a/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec_single_tree.hpp b/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec_single_tree.hpp index 73ac5d1..66b416d 100644 --- a/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec_single_tree.hpp +++ b/benchmark/two-level-arch/kernels/reduction/reducesum_rowvec_single_tree.hpp @@ -34,28 +34,28 @@ void __vec__ reducesum_row_kernel( const typename tileSrc::TileDType __in__ src, const typename tileSrcCol::TileDType __in__ src_col, const typename tileTmpSum::TileDType __in__ old_sum, - const size_t tile_idx + const size_t tile_idx ) { // j 为行 lane，z 为 4 路并行的段索引 - size_t j = blkv_get_index_x(); - size_t z = blkv_get_index_y(); + size_t j = blkv_get_index_x(); + size_t z = blkv_get_index_y(); // 本段在 src / src_col 中的基地址偏移（按 ValidCol/4 划分 4 段） size_t stride_src = z * (tileSrc::ValidCol/4) * tileSrc::ColStride; - size_t stride_src_col = z * (tileSrcCol::ValidCol/4) * tileSrcCol::ColStride; - + size_t stride_src_col = z * (tileSrcCol::ValidCol/4) * tileSrcCol::ColStride; + __vbuf__ typename tileTmpSum::DType *new_sum_ptr = blkv_get_tile_ptr(new_sum); __vbuf__ typename tileSrc::DType *src_ptr = blkv_get_tile_ptr(src); - __vbuf__ typename tileSrc::DType *src_col_ptr = blkv_get_tile_ptr(src_col); - __vbuf__ typename tileTmpSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); + __vbuf__ typename tileSrc::DType *src_col_ptr = blkv_get_tile_ptr(src_col); + __vbuf__ typename tileTmpSum::DType *old_sum_ptr = blkv_get_tile_ptr(old_sum); /* - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for(size_t i=0;i>; //将gm中的Tensor先声明为一维数据 -// using gm_shapeSum = global_tensor>; + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 +// using gm_shapeSum = global_tensor>; using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 // 列主序中间缓冲：tN/8 列，便于 8 路归约后存储 - using tile_shapeDataCol = Tile; // todo 尾块怎么处理？是否要作为参数写在这 + using tile_shapeDataCol = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeSum = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // 中间 tile：列主序，64 列、有效列=Nb*4（4 路并行 × Nb 个 tile） using tile_shapeTmpSum = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - gm_shapeIn inGm(in_ptr); + gm_shapeIn inGm(in_ptr); gm_shapeOut outGm(out_ptr); -// gm_shapeSum olcSumGm(old_sum_ptr); +// gm_shapeSum olcSumGm(old_sum_ptr); - tile_shapeData dataTile; - tile_shapeDataCol dataTile_col; + tile_shapeData dataTile; + tile_shapeDataCol dataTile_col; tile_shapeSum SumTile; tile_shapeTmpSum oldtmpSumTile; tile_shapeTmpSum tmpSumTile; @@ -237,7 +237,7 @@ void reducesum_trowsum_rand( // int base = 0;// todo 生成一个标量 // int all_num = gOM; // 总元素数量 - using itIn = global_iterator; + using itIn = global_iterator; using itOut = global_iterator; itIn gIIter(in_ptr); @@ -245,23 +245,23 @@ void reducesum_trowsum_rand( auto gO = gOIter(0, 0); - TEXPANDSCALAR(oldtmpSumTile, 0);//初始化为0 - TEXPANDSCALAR(dataTile_col, 0);//初始化为0 + TEXPANDSCALAR(oldtmpSumTile, 0);//初始化为0 + TEXPANDSCALAR(dataTile_col, 0);//初始化为0 // 阶段1：逐 tile 拷入并做 4 路并行多级树形归约，写入中间 tile for (int i = 0; i < Nb; ++i) { - auto gI = gIIter(0, i); - TCOPYIN(dataTile, gI); - reducesum_row_kernel<<>>(tmpSumTile.data(), - dataTile.data(), - dataTile_col.data(), + auto gI = gIIter(0, i); + TLOAD(dataTile, gI); + reducesum_row_kernel<<>>(tmpSumTile.data(), + dataTile.data(), + dataTile_col.data(), oldtmpSumTile.data(), i); oldtmpSumTile = tmpSumTile; } // 阶段2：最终归约并写出 - reducesum_row_final_kernel<<>>(SumTile.data(), - tmpSumTile.data()); - TCOPYOUT(gO, SumTile); + reducesum_row_final_kernel<<>>(SumTile.data(), + tmpSumTile.data()); + TSTORE(gO, SumTile); } #endif diff --git a/benchmark/two-level-arch/kernels/transpose/transpose.hpp b/benchmark/two-level-arch/kernels/transpose/transpose.hpp index 2747040..ed17cce 100644 --- a/benchmark/two-level-arch/kernels/transpose/transpose.hpp +++ b/benchmark/two-level-arch/kernels/transpose/transpose.hpp @@ -16,7 +16,7 @@ using namespace pto; GlobalTensor, \ Stride<1,1,1,Cols,1>> _g(DumpBuf); \ - TCOPYOUT(_g, TileVar); \ + TSTORE(_g, TileVar); \ printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ for (int ri = 0; ri < Rows; ri++) { \ printf(" row%2d: ", ri); \ @@ -38,16 +38,16 @@ void __vec__ gen_offset_trans( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0, +// const size_t transpose_dim0, const size_t base, const size_t total_elements ) { size_t index = blkv_get_index_x(); size_t idx = blkv_get_index_x(); - + __vbuf__ typename tile_Inshape::DType *in_shape_ptr = blkv_get_tile_ptr(in_shape); __vbuf__ typename tile_Outshape::DType *out_shape_ptr = blkv_get_tile_ptr(out_shape); - + // if (index >= total_elements) return; idx = idx + base; // todo idx是个向量，base是个标量，获得所有的基地址或者说基offset @@ -66,7 +66,7 @@ void __vec__ gen_offset_trans( // 输出一维索引 → 输出坐标 size_t out_coord[MAX_DIM] = {0}; // size_t tmp = idx; // - + #pragma clang loop unroll(full) for (int d = OUT_DIM - 1; d >= 0; d--) { out_coord[d] = tmp % out_shape_ptr[d]; @@ -85,10 +85,10 @@ void __vec__ gen_offset_trans( } } */ -// uint16_t in_offset = 0; - uint32_t in_offset = 0; +// uint16_t in_offset = 0; + uint32_t in_offset = 0; - #pragma clang loop unroll(full) + #pragma clang loop unroll(full) for (int i = 0; i < IN_DIM; i++) { in_offset += out_coord[i] * stride[i] * sizeof(dtype); } @@ -105,7 +105,7 @@ void gen_offset_Impl( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0, +// const size_t transpose_dim0, const size_t base, const size_t total_elements ) @@ -126,21 +126,21 @@ void transpose( // const size_t in_dim, // const size_t out_dim, // const size_t transpose_dim1, -// const size_t transpose_dim0 -) +// const size_t transpose_dim0 +) { const int Mb = gOM / tM; - + const int rmd_M = gOM % tM; // todo 尾块怎么处理？ - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_shapeOut = global_tensor>; - - using gm_InMatShape = global_tensor>; //将gm中的Tensor先声明为一维数据 + + using gm_InMatShape = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_OutMatShape = global_tensor>; - + using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec // using tile_shapeOffset = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec @@ -150,19 +150,19 @@ void transpose( using tile_shapeOffset_rmd = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec using tile_Inshape = Tile; // todo 尾块怎么处理？是否要作为参数写在这 - using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec - + using tile_Outshape = Tile; // todo 这里的location，一定要是Vec吗？哪怕没有传入Vec + gm_shapeIn inGm(in_ptr); - + gm_InMatShape InShapeGm(in_shape); gm_OutMatShape OutShapeGm(out_shape); tile_shapeData dataTile; tile_shapeOffset offsetTile; - + tile_shapeData_rmd dataTile_rmd; tile_shapeOffset_rmd offsetTile_rmd; - + tile_Inshape InshapeTile; tile_Outshape OutshapeTile; @@ -172,8 +172,8 @@ void transpose( using itOut = global_iterator; itOut gOIter(out_ptr); - TCOPYIN(InshapeTile, InShapeGm); - TCOPYIN(OutshapeTile, OutShapeGm); + TLOAD(InshapeTile, InShapeGm); + TLOAD(OutshapeTile, OutShapeGm); int total_elements = tM; @@ -184,9 +184,9 @@ void transpose( base += total_elements; // DUMP_TILE("offsetTile", offsetTile, g_dump, 1, tM); MGATHER(dataTile, inGm, offsetTile); -// printf("end mgather\n"); - TCOPYOUT(gO, dataTile); -// TCOPYOUT(gO, dataTile); +// printf("end mgather\n"); + TSTORE(gO, dataTile); +// TSTORE(gO, dataTile); } if constexpr (rmd_M) { auto gO = gOIter(0, Mb); @@ -194,7 +194,7 @@ void transpose( gen_offset_Impl(offsetTile_rmd, InshapeTile, OutshapeTile, base, total_elements); base += total_elements; MGATHER(dataTile_rmd, inGm, offsetTile_rmd); - TCOPYOUT(gO, dataTile_rmd); + TSTORE(gO, dataTile_rmd); } } diff --git a/benchmark/two-level-arch/kernels/transpose/transpose_vector_007.hpp b/benchmark/two-level-arch/kernels/transpose/transpose_vector_007.hpp index c9fbe07..4093211 100644 --- a/benchmark/two-level-arch/kernels/transpose/transpose_vector_007.hpp +++ b/benchmark/two-level-arch/kernels/transpose/transpose_vector_007.hpp @@ -4,11 +4,11 @@ using namespace pto; //AI/IA = A, placeholder -//like Ttrans +//like Ttrans template void __vec__ transpose_007_impl( typename tileOutData::TileDType __out__ out, - const typename tileInData::TileDType __in__ in + const typename tileInData::TileDType __in__ in ) { size_t i = blkv_get_index_x(); // 4096 @@ -27,13 +27,13 @@ void __vec__ transpose_007_impl( // in1[DimIn1, 1] in2 [DimIn2, 1] bias[DimOut, 1] weight [DimOut, DimIn1, DimIn2] template void transpose_007( - dtype *out_ptr, + dtype *out_ptr, dtype *in_ptr ) { - const int Mb = 4096 / 4096; + const int Mb = 4096 / 4096; - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_shapeOut = global_tensor>; using tile_shapeInData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using tile_shapeOutData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 @@ -42,18 +42,18 @@ void transpose_007( using itOut = global_iterator; tile_shapeInData InDataTile; - tile_shapeOutData OutDataTile; + tile_shapeOutData OutDataTile; itIn gIIter(in_ptr); - itOut gOIter(out_ptr); + itOut gOIter(out_ptr); for (int i = 0; i < Mb; ++i) { auto gI = gIIter(0, i); auto gO = gOIter(0, i); - TCOPYIN(InDataTile, gI); + TLOAD(InDataTile, gI); transpose_007_impl<<>>(OutDataTile.data(), InDataTile.data()); - TCOPYOUT(gO, OutDataTile); - } + TSTORE(gO, OutDataTile); + } } diff --git a/benchmark/two-level-arch/kernels/transpose/transpose_vector_050.hpp b/benchmark/two-level-arch/kernels/transpose/transpose_vector_050.hpp index d280dd3..d9c2053 100644 --- a/benchmark/two-level-arch/kernels/transpose/transpose_vector_050.hpp +++ b/benchmark/two-level-arch/kernels/transpose/transpose_vector_050.hpp @@ -7,7 +7,7 @@ using namespace pto; template void __vec__ transpose_050_impl( typename tileData::TileDType __out__ out, - const typename tileData::TileDType __in__ in + const typename tileData::TileDType __in__ in ) { size_t i = blkv_get_index_x(); // y @@ -26,33 +26,33 @@ void __vec__ transpose_050_impl( template void transpose_050( - dtype *out_ptr, + dtype *out_ptr, dtype *in_ptr ) -{ +{ - using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 + using gm_shapeIn = global_tensor>; //将gm中的Tensor先声明为一维数据 using gm_shapeOut = global_tensor>; using tile_shapeData = Tile; // todo 尾块怎么处理？是否要作为参数写在这 using itIn = global_iterator; - using itOut = global_iterator; + using itOut = global_iterator; tile_shapeData InDataTile; tile_shapeData OutDataTile; itIn gIIter(in_ptr); - itOut gOIter(out_ptr); + itOut gOIter(out_ptr); auto gI = gIIter(0, 0); auto gO = gOIter(0, 0); - TCOPYIN(InDataTile, gI); + TLOAD(InDataTile, gI); transpose_050_impl<<>>(OutDataTile.data(), InDataTile.data()); - TCOPYOUT(gO, OutDataTile); - - + TSTORE(gO, OutDataTile); + + } diff --git a/benchmark/two-level-arch/test/common/Makefile.common b/benchmark/two-level-arch/test/common/Makefile.common index 4df6a81..60a6c51 100644 --- a/benchmark/two-level-arch/test/common/Makefile.common +++ b/benchmark/two-level-arch/test/common/Makefile.common @@ -14,6 +14,8 @@ COMM_SRC_DIR := common/src COMM_OBJ += $(patsubst %.cpp, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.cpp))) COMM_OBJ += $(patsubst %.c, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.c))) COMM_OBJ += $(patsubst %.s, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(wildcard $(TEST_ROOT)/$(COMM_SRC_DIR)/*.s))) +FREESTANDING_RUNTIME_SRC := $(TEST_ROOT)/$(COMM_SRC_DIR)/freestanding_runtime.c +FREESTANDING_RUNTIME_OBJ := $(patsubst %.c, %.o, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(FREESTANDING_RUNTIME_SRC))) OBJ := $(patsubst %.cpp, %.o, $(subst $(SRC_DIR), $(OBJ_DIR), $(SRC_FILE))) OBJ := $(patsubst %.c, %.o, $(OBJ)) @@ -24,6 +26,8 @@ LINK_SCRIPT = $(patsubst %.S, %, $(subst $(TEST_ROOT), $(OBJ_ROOT), $(LINK_SCRI PLAT ?= linx baremetal ?= off +LINX_SYSROOT ?= $(shell test -d "$(ROOT)/../../../../out/libc/musl/install/phase-b" && realpath "$(ROOT)/../../../../out/libc/musl/install/phase-b" || true) +LINX_SYSROOT_FLAGS := $(if $(LINX_SYSROOT),--sysroot=$(LINX_SYSROOT) -idirafter $(LINX_SYSROOT)/usr/include,) ifeq ($(PLAT), cpu) DEFINES += -D__cpu_sim__ @@ -51,7 +55,7 @@ CXX_VER ?= -std=c++20 else ifeq ($(PLAT), linx) DEFINES += -D__linx ifndef COMPILER_DIR -$(error COMPILER_DIR is not set. Export COMPILER_DIR pointing to the linx_blockisa_llvm_musl toolchain bin directory, e.g. export COMPILER_DIR=/path/to/linx_blockisa_llvm_musl/bin) +$(error COMPILER_DIR is not set. Point it to compiler/llvm/build-linxisa-clang/bin in the LinxISA superproject) endif AS = $(COMPILER_DIR)/clang CC = $(COMPILER_DIR)/clang @@ -59,12 +63,12 @@ CXX = $(COMPILER_DIR)/clang++ LINK = $(COMPILER_DIR)/clang++ DUMP = $(COMPILER_DIR)/llvm-objdump COPY = $(COMPILER_DIR)/llvm-objcopy -CC_O = -c -target linx64-linx-none-elf -fenable-matrix -O2 $(CFLAGS) +CC_O = -c -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -fenable-matrix -O2 $(CFLAGS) CXX_VER ?= -std=c++20 ifeq ($(baremetal), on) - CC_LINK += -static -lm -nostartfiles -L $(OBJ_ROOT)/$(COMM_SRC_DIR) -T $(LINK_SCRIPT) + CC_LINK += $(LINX_SYSROOT_FLAGS) -static -lm -nostartfiles -L $(OBJ_ROOT)/$(COMM_SRC_DIR) -T $(LINK_SCRIPT) else - CC_LINK += -target linx64-linx-none-elf -nostdlib -Wl,-e,_start $(ROOT)/test/common/_start.s + CC_LINK += -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -nostdlib -Wl,-e,_start $(ROOT)/test/common/_start.s endif DEFINES += -DENABLE_TENSOR_INSTR @@ -137,18 +141,18 @@ $(OBJ_DIR)%.o: $(SRC_DIR)%.s $(OBJ_ROOT)/$(COMM_SRC_DIR)%.o: $(TEST_ROOT)/$(COMM_SRC_DIR)%.c @mkdir -p $(shell dirname $@) - $(CXX) -target linx64-linx-none-elf -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ + $(CXX) -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ $(OBJ_ROOT)/$(COMM_SRC_DIR)%.o: $(TEST_ROOT)/$(COMM_SRC_DIR)%.s @mkdir -p $(shell dirname $@) - $(CXX) -target linx64-linx-none-elf -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ + $(CXX) -target linx64-linx-none-elf $(LINX_SYSROOT_FLAGS) -c -O2 -Wno-return-type -Wno-deprecated -Wno-unused-command-line-argument -Wno-invalid-noreturn $(INCLUDE) $(DEFINES) $< -o $@ $(OBJ_ROOT)/$(COMM_SRC_DIR)%.lds : $(TEST_ROOT)/$(COMM_SRC_DIR)%.lds.S $(CC) $(DEFINES) $(INCLUDE) -E -P -C -o $@ $< -$(TARGET): clean $(LINK_SCRIPT) $(COMM_OBJ) $(OBJ) $(EXTRA_OBJ_FILES) +$(TARGET): clean $(LINK_SCRIPT) $(COMM_OBJ) $(FREESTANDING_RUNTIME_OBJ) $(OBJ) $(EXTRA_OBJ_FILES) @mkdir -p $(shell dirname $@) - $(LINK) $(CC_LINK) $(OBJ) $(COMM_OBJ) $(EXTRA_OBJ_FILES) -o $@ + $(LINK) $(CC_LINK) $(OBJ) $(COMM_OBJ) $(FREESTANDING_RUNTIME_OBJ) $(EXTRA_OBJ_FILES) -o $@ pre_work: @mkdir -p $(OBJ_DIR) diff --git a/benchmark/two-level-arch/test/common/fileop.h b/benchmark/two-level-arch/test/common/fileop.h index 2643ae1..18b8ffa 100644 --- a/benchmark/two-level-arch/test/common/fileop.h +++ b/benchmark/two-level-arch/test/common/fileop.h @@ -1,4 +1,2 @@ -#include -#include #include "writeBinary.h" -#include "readBinary.h" \ No newline at end of file +#include "readBinary.h" diff --git a/benchmark/two-level-arch/test/common/multi_tile.hpp b/benchmark/two-level-arch/test/common/multi_tile.hpp index 6ca6a6c..e7ba914 100644 --- a/benchmark/two-level-arch/test/common/multi_tile.hpp +++ b/benchmark/two-level-arch/test/common/multi_tile.hpp @@ -129,19 +129,19 @@ void TCAST(tO &o, tA &a) { } template -void TCOPYIN(tile_shape &dst, itfn it) { +void TLOAD(tile_shape &dst, itfn it) { #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { auto gm = it(i); - TCOPYIN(dst.Tiles[i], gm); + TLOAD(dst.Tiles[i], gm); } } template -void TCOPYIN(tile_shape &dst, gm_shape &src) { +void TLOAD(tile_shape &dst, gm_shape &src) { #ifdef MULTI_REUSE typename tile_shape::TileType t; - TCOPYIN(t, src); + TLOAD(t, src); #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { dst.Tiles[i] = t; @@ -149,17 +149,17 @@ void TCOPYIN(tile_shape &dst, gm_shape &src) { #else #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { - TCOPYIN(dst.Tiles[i], src); + TLOAD(dst.Tiles[i], src); } #endif } template -void TCOPYOUT(itfn it, tile_shape &src) { +void TSTORE(itfn it, tile_shape &src) { #pragma clang loop unroll(full) for (int i = 0; i < tile_shape::NumTiles; ++i) { auto gm = it(i); - TCOPYOUT(gm, src.Tiles[i]); + TSTORE(gm, src.Tiles[i]); } } diff --git a/benchmark/two-level-arch/test/common/readBinary.h b/benchmark/two-level-arch/test/common/readBinary.h index be99c43..1df4324 100644 --- a/benchmark/two-level-arch/test/common/readBinary.h +++ b/benchmark/two-level-arch/test/common/readBinary.h @@ -6,8 +6,6 @@ #include #include #include -#include -#include // #define ENABLE_BINARY_OUTPUT @@ -34,4 +32,4 @@ bool readBinaryFile(const char * filename, uint8_t* data, size_t size) { #endif } -#endif \ No newline at end of file +#endif diff --git a/benchmark/two-level-arch/test/common/src/baremetal_linx.lds.S b/benchmark/two-level-arch/test/common/src/baremetal_linx.lds.S index 1891c2f..d893956 100644 --- a/benchmark/two-level-arch/test/common/src/baremetal_linx.lds.S +++ b/benchmark/two-level-arch/test/common/src/baremetal_linx.lds.S @@ -1,4 +1,4 @@ -OUTPUT_ARCH(linxv5) +OUTPUT_ARCH(linx) ENTRY(firmware_base) /* SEARCH_DIR("=/usr1/compiler_cpu/build/BiSheng_linx64be_elf/../../output/BiSheng_linx64be_elf/BiSheng_linx64be_elf/linx64-unknown-elf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); @@ -13,15 +13,15 @@ MEMORY { SECTIONS { - mem_base = MEM_BASE; + mem_base = MEM_BASE; .app_mem mem_base : { - KEEP(*(.app_text)) + KEEP(*(.app_text)) KEEP(*(.app_data)) } > RAM firmware_base = FIRMWARE_BASE; - .firmware firmware_base : + .firmware firmware_base : { KEEP(*benchmark_boot_linx.o(linxboot_text)) KEEP(*(boot_text)) @@ -75,7 +75,7 @@ SECTIONS *(.text .stub .text.* .gnu.linkonce.t.*) /* .gnu.warning sections are handled specially by elf.em. */ *(.gnu.warning) - } > RAM =0 + } > RAM =0 .init : { @@ -288,7 +288,7 @@ SECTIONS .heap (NOLOAD): { /* . = ALIGN(8);*/ - . = . + HEAP_SIZE; // 4GB + . = . + HEAP_SIZE; // 4GB } > HEAP heap_end = .; diff --git a/benchmark/two-level-arch/test/common/src/freestanding_runtime.c b/benchmark/two-level-arch/test/common/src/freestanding_runtime.c new file mode 100644 index 0000000..b2862ba --- /dev/null +++ b/benchmark/two-level-arch/test/common/src/freestanding_runtime.c @@ -0,0 +1,35 @@ +#ifdef __cplusplus +extern "C" { +#endif + +void *memset(void *dst, int value, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + for (unsigned long i = 0; i < size; ++i) + out[i] = (unsigned char)value; + return dst; +} + +void *memcpy(void *dst, const void *src, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + const volatile unsigned char *in = (const volatile unsigned char *)src; + for (unsigned long i = 0; i < size; ++i) + out[i] = in[i]; + return dst; +} + +void *memmove(void *dst, const void *src, unsigned long size) { + volatile unsigned char *out = (volatile unsigned char *)dst; + const volatile unsigned char *in = (const volatile unsigned char *)src; + if (out < in) { + for (unsigned long i = 0; i < size; ++i) + out[i] = in[i]; + } else if (out > in) { + for (unsigned long i = size; i != 0; --i) + out[i - 1] = in[i - 1]; + } + return dst; +} + +#ifdef __cplusplus +} +#endif diff --git a/benchmark/two-level-arch/test/common/src/ldv5.lds.S b/benchmark/two-level-arch/test/common/src/ldv5.lds.S deleted file mode 100644 index 5a7322d..0000000 --- a/benchmark/two-level-arch/test/common/src/ldv5.lds.S +++ /dev/null @@ -1,259 +0,0 @@ -OUTPUT_ARCH(linxv5) -ENTRY(_start) -SEARCH_DIR("=/usr1/compiler_cpu/build/BiSheng_linx64be_elf/../../output/BiSheng_linx64be_elf/BiSheng_linx64be_elf/linx64-unknown-elf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); -SECTIONS -{ - /* Read-only sections, merged into text segment: */ - PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x10000)); . = SEGMENT_START("text-segment", 0x10000) + SIZEOF_HEADERS; - .interp : { *(.interp) } - .note.gnu.build-id : { *(.note.gnu.build-id) } - .hash : { *(.hash) } - .gnu.hash : { *(.gnu.hash) } - .dynsym : { *(.dynsym) } - .dynstr : { *(.dynstr) } - .gnu.version : { *(.gnu.version) } - .gnu.version_d : { *(.gnu.version_d) } - .gnu.version_r : { *(.gnu.version_r) } - .rela.dyn : - { - *(.rela.init) - *(.rela.text .rela.text.* .rela.gnu.linkonce.t.*) - *(.rela.fini) - *(.rela.rodata .rela.rodata.* .rela.gnu.linkonce.r.*) - *(.rela.data .rela.data.* .rela.gnu.linkonce.d.*) - *(.rela.tdata .rela.tdata.* .rela.gnu.linkonce.td.*) - *(.rela.tbss .rela.tbss.* .rela.gnu.linkonce.tb.*) - *(.rela.ctors) - *(.rela.dtors) - *(.rela.got) - *(.rela.sdata .rela.sdata.* .rela.gnu.linkonce.s.*) - *(.rela.sbss .rela.sbss.* .rela.gnu.linkonce.sb.*) - *(.rela.sdata2 .rela.sdata2.* .rela.gnu.linkonce.s2.*) - *(.rela.sbss2 .rela.sbss2.* .rela.gnu.linkonce.sb2.*) - *(.rela.bss .rela.bss.* .rela.gnu.linkonce.b.*) - PROVIDE_HIDDEN (__rela_iplt_start = .); - *(.rela.iplt) - PROVIDE_HIDDEN (__rela_iplt_end = .); - } - .rela.plt : - { - *(.rela.plt) - } - .init : - { - KEEP (*(SORT_NONE(.init))) - } - .plt : { *(.plt) } - .iplt : { *(.iplt) } - . = ALIGN(16); - .text : - { - *(.text.unlikely .text.*_unlikely .text.unlikely.*) - *(.text.exit .text.exit.*) - *(.text.startup .text.startup.*) - *(.text.hot .text.hot.*) - *(SORT(.text.sorted.*)) - *(.text .stub .text.* .gnu.linkonce.t.*) - /* .gnu.warning sections are handled specially by elf.em. */ - *(.gnu.warning) - } - .fini : - { - KEEP (*(SORT_NONE(.fini))) - } - PROVIDE (__etext = .); - PROVIDE (_etext = .); - PROVIDE (etext = .); - .rodata : { *(.rodata .rodata.* .gnu.linkonce.r.*) } - .rodata1 : { *(.rodata1) } - .sdata2 : - { - *(.sdata2 .sdata2.* .gnu.linkonce.s2.*) - } - .sbss2 : { *(.sbss2 .sbss2.* .gnu.linkonce.sb2.*) } - /* - .eh_frame : - { - __eh_frame_start = .; - KEEP(*(.eh_frame)) - __eh_frame_end = .; - } - */ - .eh_frame_hdr : - { - KEEP(*(.eh_frame_hdr)) - } - __eh_frame_hdr_start = SIZEOF(.eh_frame_hdr) > 0 ? ADDR(.eh_frame_hdr) : 0; - __eh_frame_hdr_end = SIZEOF(.eh_frame_hdr) > 0 ? . : 0; - .sframe : ONLY_IF_RO { *(.sframe) *(.sframe.*) } - .gcc_except_table : ONLY_IF_RO { *(.gcc_except_table .gcc_except_table.*) } - .gnu_extab : ONLY_IF_RO { *(.gnu_extab*) } - /* These sections are generated by the Sun/Oracle C++ compiler. */ - .exception_ranges : ONLY_IF_RO { *(.exception_ranges*) } - /* Adjust the address for the data segment. We want to adjust up to - the same address within the page on the next page up. */ - . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE)); - /* Exception handling */ - .eh_frame : ONLY_IF_RW { KEEP (*(.eh_frame)) *(.eh_frame.*) } - .sframe : ONLY_IF_RW { *(.sframe) *(.sframe.*) } - .gnu_extab : ONLY_IF_RW { *(.gnu_extab) } - .gcc_except_table : ONLY_IF_RW { *(.gcc_except_table .gcc_except_table.*) } - .exception_ranges : ONLY_IF_RW { *(.exception_ranges*) } - /* Thread Local Storage sections */ - .tdata : - { - PROVIDE_HIDDEN (__tdata_start = .); - *(.tdata .tdata.* .gnu.linkonce.td.*) - } - .tbss : { *(.tbss .tbss.* .gnu.linkonce.tb.*) *(.tcommon) } - .preinit_array : - { - PROVIDE_HIDDEN (__preinit_array_start = .); - KEEP (*(.preinit_array)) - PROVIDE_HIDDEN (__preinit_array_end = .); - } - .init_array : - { - PROVIDE_HIDDEN (__init_array_start = .); - KEEP (*(SORT_BY_INIT_PRIORITY(.init_array.*) SORT_BY_INIT_PRIORITY(.ctors.*))) - KEEP (*(.init_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .ctors)) - PROVIDE_HIDDEN (__init_array_end = .); - } - .fini_array : - { - PROVIDE_HIDDEN (__fini_array_start = .); - KEEP (*(SORT_BY_INIT_PRIORITY(.fini_array.*) SORT_BY_INIT_PRIORITY(.dtors.*))) - KEEP (*(.fini_array EXCLUDE_FILE (*crtbegin.o *crtbegin?.o *crtend.o *crtend?.o ) .dtors)) - PROVIDE_HIDDEN (__fini_array_end = .); - } - .ctors : - { - /* gcc uses crtbegin.o to find the start of - the constructors, so we make sure it is - first. Because this is a wildcard, it - doesn't matter if the user does not - actually link against crtbegin.o; the - linker won't look for a file to match a - wildcard. The wildcard also means that it - doesn't matter which directory crtbegin.o - is in. */ - KEEP (*crtbegin.o(.ctors)) - KEEP (*crtbegin?.o(.ctors)) - /* We don't want to include the .ctor section from - the crtend.o file until after the sorted ctors. - The .ctor section from the crtend file contains the - end of ctors marker and it must be last */ - KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors)) - KEEP (*(SORT(.ctors.*))) - KEEP (*(.ctors)) - } - .dtors : - { - KEEP (*crtbegin.o(.dtors)) - KEEP (*crtbegin?.o(.dtors)) - KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .dtors)) - KEEP (*(SORT(.dtors.*))) - KEEP (*(.dtors)) - } - .jcr : { KEEP (*(.jcr)) } - .data.rel.ro : { *(.data.rel.ro.local* .gnu.linkonce.d.rel.ro.local.*) *(.data.rel.ro .data.rel.ro.* .gnu.linkonce.d.rel.ro.*) } - .dynamic : { *(.dynamic) } - . = DATA_SEGMENT_RELRO_END (0, .); - .data : - { - __DATA_BEGIN__ = .; - *(.data .data.* .gnu.linkonce.d.*) - SORT(CONSTRUCTORS) - } - .data1 : { *(.data1) } - .got : { *(.got.plt) *(.igot.plt) *(.got) *(.igot) } - /* We want the small data sections together, so single-instruction offsets - can aaccelss them all, and initialized data all before uninitialized, so - we can shorten the on-disk segment size. */ - .sdata : - { - __SDATA_BEGIN__ = .; - *(.srodata.cst16) *(.srodata.cst8) *(.srodata.cst4) *(.srodata.cst2) *(.srodata .srodata.*) - *(.sdata .sdata.* .gnu.linkonce.s.*) - } - _edata = .; PROVIDE (edata = .); - . = .; - __bss_start = .; - .sbss : - { - *(.dynsbss) - *(.sbss .sbss.* .gnu.linkonce.sb.*) - *(.scommon) - } - .bss : - { - *(.dynbss) - *(.bss .bss.* .gnu.linkonce.b.*) - *(COMMON) - /* Align here to ensure that the .bss section occupies space up to - _end. Align after .bss to ensure correct alignment even if the - .bss section disappears because there are no input sections. - FIXME: Why do we need it? When there is no .bss section, we do not - pad the .data section. */ - . = ALIGN(. != 0 ? 64 / 8 : 1); - } - . = ALIGN(64 / 8); - . = SEGMENT_START("ldata-segment", .); - . = ALIGN(64 / 8); - __BSS_END__ = .; - __global_pointer$ = MIN(__SDATA_BEGIN__ + 0x800, - MAX(__DATA_BEGIN__ + 0x800, __BSS_END__ - 0x800)); - _end = .; PROVIDE (end = .); - . = DATA_SEGMENT_END (.); - /* Stabs debugging sections. */ - .stab 0 : { *(.stab) } - .stabstr 0 : { *(.stabstr) } - .stab.excl 0 : { *(.stab.excl) } - .stab.exclstr 0 : { *(.stab.exclstr) } - .stab.index 0 : { *(.stab.index) } - .stab.indexstr 0 : { *(.stab.indexstr) } - .comment 0 (INFO) : { *(.comment); LINKER_VERSION; } - .gnu.build.attributes : { *(.gnu.build.attributes .gnu.build.attributes.*) } - /* DWARF debug sections. - Symbols in the DWARF debugging sections are relative to the beginning - of the section so we begin them at 0. */ - /* DWARF 1. */ - .debug 0 : { *(.debug) } - .line 0 : { *(.line) } - /* GNU DWARF 1 extensions. */ - .debug_srcinfo 0 : { *(.debug_srcinfo) } - .debug_sfnames 0 : { *(.debug_sfnames) } - /* DWARF 1.1 and DWARF 2. */ - .debug_aranges 0 : { *(.debug_aranges) } - .debug_pubnames 0 : { *(.debug_pubnames) } - /* DWARF 2. */ - .debug_info 0 : { *(.debug_info .gnu.linkonce.wi.*) } - .debug_abbrev 0 : { *(.debug_abbrev) } - .debug_line 0 : { *(.debug_line .debug_line.* .debug_line_end) } - .debug_frame 0 : { *(.debug_frame) } - .debug_str 0 : { *(.debug_str) } - .debug_loc 0 : { *(.debug_loc) } - .debug_macinfo 0 : { *(.debug_macinfo) } - /* SGI/MIPS DWARF 2 extensions. */ - .debug_weaknames 0 : { *(.debug_weaknames) } - .debug_funcnames 0 : { *(.debug_funcnames) } - .debug_typenames 0 : { *(.debug_typenames) } - .debug_varnames 0 : { *(.debug_varnames) } - /* DWARF 3. */ - .debug_pubtypes 0 : { *(.debug_pubtypes) } - .debug_ranges 0 : { *(.debug_ranges) } - /* DWARF 5. */ - .debug_addr 0 : { *(.debug_addr) } - .debug_line_str 0 : { *(.debug_line_str) } - .debug_loclists 0 : { *(.debug_loclists) } - .debug_macro 0 : { *(.debug_macro) } - .debug_names 0 : { *(.debug_names) } - .debug_rnglists 0 : { *(.debug_rnglists) } - .debug_str_offsets 0 : { *(.debug_str_offsets) } - .debug_sup 0 : { *(.debug_sup) } - .gnu.attributes 0 : { KEEP (*(.gnu.attributes)) } - /DISCARD/ : { *(.note.GNU-stack) *(.gnu_debuglink) *(.gnu.lto_*) } -} - - -/*==================================================*/ diff --git a/benchmark/two-level-arch/test/common/src/v057_tma_compile.cpp b/benchmark/two-level-arch/test/common/src/v057_tma_compile.cpp new file mode 100644 index 0000000..ba487c9 --- /dev/null +++ b/benchmark/two-level-arch/test/common/src/v057_tma_compile.cpp @@ -0,0 +1,30 @@ +#include "common/pto_tile.hpp" +#include "jcore/template_asm.hpp" + +using namespace pto; + +namespace { + +using TestShape = Shape<1, 1, 1, 8, 8>; +using TestStride = Stride<1, 1, 64, 8, 1>; +using TestGlobal = GlobalTensor; +using TestTile = Tile; + +} // namespace + +extern "C" void compile_v057_tma_wrappers(int32_t *base) { + TestGlobal global(base); + TestTile dst; + TestTile data; + TestTile offset; + TestTile mask; + TestTile expected; + TestTile desired; + + TPREFETCH(global); + MGATHER(dst, global, offset); + MSCATTER(global, data, offset); + MGATHER_MASK(dst, global, offset, mask); + MSCATTER_MASK(global, data, offset, mask); + MGATHER_CAS(dst, global, offset, expected, desired); +} diff --git a/benchmark/two-level-arch/test/common/template_asm.h b/benchmark/two-level-arch/test/common/template_asm.h index 2f60707..d1ad733 100644 --- a/benchmark/two-level-arch/test/common/template_asm.h +++ b/benchmark/two-level-arch/test/common/template_asm.h @@ -3,17 +3,252 @@ #include +#if !defined(__linx) + +namespace supernpubench_v057_test_fallback { +template +constexpr int tile_numel_v = T::Rows * T::Cols; + +template +inline void copy_tile(OutTile &dst, const InTile &src) { + constexpr int OutN = tile_numel_v; + constexpr int InN = tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src.data()[i]); +} + +template +inline void fill_tile(OutTile &dst, typename OutTile::DType value) { + for (int i = 0; i < tile_numel_v; ++i) + dst.data()[i] = value; +} + +template +inline void gather_bytes(OutTile &dst, GmShape &src, OffTile &offset) { + using DType = typename OutTile::DType; + constexpr int OutN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = OutN < OffN ? OutN : OffN; + const auto *base = src.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + dst.data()[i] = static_cast( + *(reinterpret_cast( + reinterpret_cast(base) + byte_off))); + } +} + +template +inline void scatter_bytes(GmShape &dst, InTile &src, OffTile &offset) { + using DType = typename InTile::DType; + constexpr int SrcN = tile_numel_v; + constexpr int OffN = tile_numel_v; + constexpr int N = SrcN < OffN ? SrcN : OffN; + auto *base = dst.data(); + for (int i = 0; i < N; ++i) { + long long byte_off = static_cast(offset.data()[i]); + *(reinterpret_cast(reinterpret_cast(base) + byte_off)) = + src.data()[i]; + } +} +} // namespace supernpubench_v057_test_fallback + +template +void TXOR(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] ^ src1.data()[i]); +} + +template +void TSHLS(tile_shape &dst, tile_shape &src, unsigned shift) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src.data()[i] << shift); +} + +template +void TSHRS(tile_shape &dst, tile_shape &src, unsigned shift) { + using Unsigned = decltype(static_cast(src.data()[0])); + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast( + static_cast(src.data()[i]) >> shift); +} + +template +void TSEL(tile_shape &dst, tile_shape_index &cond, tile_shape &src) { + constexpr int DstN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int CondN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = DstN < CondN ? DstN : CondN; + for (int i = 0; i < N; ++i) + if (cond.data()[i]) + dst.data()[i] = src.data()[i]; +} + +template +void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { + supernpubench_v057_test_fallback::gather_bytes(dst, src, offset); +} + +template +void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { + supernpubench_v057_test_fallback::scatter_bytes(dst, src, offset); +} + +template +void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = src0.data()[i] > src1.data()[i] ? src0.data()[i] : src1.data()[i]; +} + +template +void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] * s); +} + +template +void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int row = 0; row < tile_shape_in::Rows && row < tile_shape_out::Rows; ++row) { + DType max_value{}; + bool init = false; + for (int col = 0; col < tile_shape_in::Cols; ++col) { + auto value = src.data()[row * tile_shape_in::Cols + col]; + if (!init || value > max_value) { + max_value = static_cast(value); + init = true; + } + } + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = max_value; + } +} + +template +void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] - src1.data()[i]); +} + +template +void TEXP_TEPL(tile_shape &dst, tile_shape &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] * src1.data()[i]); +} + +template +void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int row = 0; row < tile_shape_in::Rows && row < tile_shape_out::Rows; ++row) { + DType sum{}; + for (int col = 0; col < tile_shape_in::Cols; ++col) + sum = static_cast(sum + src.data()[row * tile_shape_in::Cols + col]); + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = sum; + } +} + +template +void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { + for (int i = 0; i < supernpubench_v057_test_fallback::tile_numel_v; ++i) + dst.data()[i] = static_cast(src0.data()[i] + src1.data()[i]); +} + +template +void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { + supernpubench_v057_test_fallback::copy_tile(dst, src); +} + +template +void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { + supernpubench_v057_test_fallback::fill_tile(dst, s); +} + +template +void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { + for (int row = 0; row < tile_shape_out::Rows; ++row) { + auto value = src.data()[row < tile_shape_in::Rows ? row * tile_shape_in::Cols : 0]; + for (int col = 0; col < tile_shape_out::Cols; ++col) + dst.data()[row * tile_shape_out::Cols + col] = + static_cast(value); + } +} + +template +void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int col = 0; col < tile_shape_in::Cols && col < tile_shape_out::Cols; ++col) { + DType max_value{}; + bool init = false; + for (int row = 0; row < tile_shape_in::Rows; ++row) { + auto value = src.data()[row * tile_shape_in::Cols + col]; + if (!init || value > max_value) { + max_value = static_cast(value); + init = true; + } + } + for (int row = 0; row < tile_shape_out::Rows; ++row) + dst.data()[row * tile_shape_out::Cols + col] = max_value; + } +} + +template +void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { + using DType = typename tile_shape_out::DType; + for (int col = 0; col < tile_shape_in::Cols && col < tile_shape_out::Cols; ++col) { + DType sum{}; + for (int row = 0; row < tile_shape_in::Rows; ++row) + sum = static_cast(sum + src.data()[row * tile_shape_in::Cols + col]); + for (int row = 0; row < tile_shape_out::Rows; ++row) + dst.data()[row * tile_shape_out::Cols + col] = sum; + } +} + +template +void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, + tile_shape_in &src1) { + constexpr int OutN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int InN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] - src1.data()[i]); +} + +template +void TCOLEXPANDMUL_TEPL(tile_shape_out &dst, tile_shape_out &src0, + tile_shape_in &src1) { + constexpr int OutN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int InN = supernpubench_v057_test_fallback::tile_numel_v; + constexpr int N = OutN < InN ? OutN : InN; + for (int i = 0; i < N; ++i) + dst.data()[i] = static_cast(src0.data()[i] * src1.data()[i]); +} + +#else + template void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { asm volatile( - "BSTART.TMA 4, %c[DataType]\n" + "BSTART.MGATHER %c[DataType]\n" "B.DIM zero, %c[VCOL], ->lb0\n" "B.DIM zero, %c[VROW], ->lb1\n" - "B.IOT [%[s1]], last, ->%[d0]<%c[TileSize]>\n" + "B.IOT %[s1], last, ->%q[d0]<%c[TileSize]>\n" "B.IOR [%[s0]], []\n" - : [d0]"=Tr"(dst.data()) + : [d0]"=r"(dst.data()) : [s0]"r"(src.data()), - [s1]"Tr"(offset.data()), + [s1]"r"(offset.data()), [DataType]"i"(type_traits::TypeCode), [TileSize]"i"(tile_type_traits::TilesizeCode), [VCOL]"i"(tile_shape_offset::ValidCol), [VROW]"i"(tile_shape_offset::ValidRow) @@ -23,14 +258,14 @@ void MGATHER(tile_shape_out &dst, gm_shape &src, tile_shape_offset &offset) { template void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { asm volatile( - "BSTART.TMA 5, %c[SrcType]\n" + "BSTART.MSCATTER %c[SrcType]\n" "B.DIM zero, %c[VCOL], ->lb0\n" "B.DIM zero, %c[VROW], ->lb1\n" - "B.IOT [%[s0], %[s1]], last\n" + "B.IOT %[s0], %[s1], last\n" "B.IOR [%[d0]], []\n" : - : [d0]"r"(dst.data()), [s0]"Tr"(src.data()), - [s1]"Tr"(offset.data()), + : [d0]"r"(dst.data()), [s0]"r"(src.data()), + [s1]"r"(offset.data()), [SrcType]"i"(type_traits::TypeCode), [VCOL]"i"(tile_shape_offset::ValidCol), [VROW]"i"(tile_shape_offset::ValidRow) ); @@ -39,20 +274,20 @@ void MSCATTER(gm_shape &dst, tile_shape_in &src, tile_shape_offset &offset) { template void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 11, %c1\n" + "BSTART.TMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -60,20 +295,20 @@ void TMAX_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) { asm volatile( - "BSTART.TEPL 0b0100010, %c1\n" + "BSTART.TMULS %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "B.IOR [%7],[]\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), + "r"(src0.data()), "i"(tile_type_traits::TilesizeCode), "r"(s) ); @@ -82,19 +317,19 @@ void TMULS_TEPL(tile_shape &dst, tile_shape &src0, typename tile_shape::DType s) template void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000001, %c1\n" + "BSTART.TROWMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -102,20 +337,20 @@ void TROWMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 1, %c1\n" + "BSTART.TSUB %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -123,19 +358,19 @@ void TSUB_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TEXP_TEPL(tile_shape &dst, tile_shape &src) { asm volatile( - "BSTART.TEPL 18, %c1\n" + "BSTART.TEXP %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -143,20 +378,20 @@ void TEXP_TEPL(tile_shape &dst, tile_shape &src) { template void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 2, %c1\n" + "BSTART.TMUL %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -164,19 +399,19 @@ void TMUL_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000000, %c1\n" + "BSTART.TROWSUM %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -184,20 +419,20 @@ void TROWSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { asm volatile( - "BSTART.TEPL 0, %c1\n" + "BSTART.TADD %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -205,19 +440,19 @@ void TADD_TEPL(tile_shape &dst, tile_shape &src0, tile_shape &src1) { template void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { asm volatile( - "BSTART.TEPL 20, %c1\n" + "BSTART.TRECIP %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), "i"(tile_shape::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -225,19 +460,19 @@ void TRECIP_TEPL(tile_shape &dst, tile_shape &src) { template void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 27, %c1\n" + "BSTART.TCVT %c1\n" "B.DATR %c2, RNONE\n" "C.B.DIMI %c3, ->LB0\n" "C.B.DIMI %c4, ->LB1\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -245,15 +480,15 @@ void TCAST_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { asm volatile( - "BSTART.TEPL 0b0111011, %c1\n" + "BSTART.TEXPANDS %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [], last, ->%0<%c5>\n" + "B.IOT last, ->%q0<%c5>\n" "B.IOR [%6],[]\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape::ValidCol), "i"(tile_shape::ValidRow), @@ -266,18 +501,18 @@ void TEXPANDSCALAR_TEPL(tile_shape &dst, typename tile_shape::DType s) { template void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1000100, %c1\n" + "BSTART.TROWEXPAND %c1\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -285,19 +520,19 @@ void TROWEXPAND_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1010001, %c1\n" + "BSTART.TCOLMAX %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -305,19 +540,19 @@ void TCOLMAX_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { asm volatile( - "BSTART.TEPL 0b1010000, %c1\n" + "BSTART.TCOLSUM %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5], last, ->%0<%c6>\n" + "B.IOT %5, last, ->%q0<%c6>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_in::ValidCol), "i"(tile_shape_in::ValidRow), "i"(tile_shape_in::Cols), - "Tr"(src.data()), + "r"(src.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -325,20 +560,20 @@ void TCOLSUM_TEPL(tile_shape_out &dst, tile_shape_in &src) { template void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in &src1) { asm volatile( - "BSTART.TEPL 0b1010110, %c1\n" + "BSTART.TCOLEXPANDSUB %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_out::ValidCol), "i"(tile_shape_out::ValidRow), "i"(tile_shape_out::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } @@ -346,22 +581,24 @@ void TCOLEXPANDSUB_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in template void TCOLEXPANDMUL_TEPL(tile_shape_out &dst, tile_shape_out &src0, tile_shape_in &src1) { asm volatile( - "BSTART.TEPL 0b1010111, %c1\n" + "BSTART.TCOLEXPANDMUL %c1\n" "B.DATR Null\n" "C.B.DIMI %c2, ->LB0\n" "C.B.DIMI %c3, ->LB1\n" "C.B.DIMI %c4, ->LB2\n" - "B.IOT [%5, %6], last, ->%0<%c7>\n" + "B.IOT %5, %6, last, ->%q0<%c7>\n" "" - : "=Tr"(dst.data()) + : "=r"(dst.data()) : "i"(type_traits::TypeCode), "i"(tile_shape_out::ValidCol), "i"(tile_shape_out::ValidRow), "i"(tile_shape_out::Cols), - "Tr"(src0.data()), - "Tr"(src1.data()), + "r"(src0.data()), + "r"(src1.data()), "i"(tile_type_traits::TilesizeCode) ); } #endif + +#endif diff --git a/benchmark/two-level-arch/test/kernel/README.md b/benchmark/two-level-arch/test/kernel/README.md index 374bf5d..9660bd8 100644 --- a/benchmark/two-level-arch/test/kernel/README.md +++ b/benchmark/two-level-arch/test/kernel/README.md @@ -109,9 +109,8 @@ make clean_all # clean all ### Sort — `sort/topk`. -## Known Issues -- `fa_2d_unroll` `X=1,Y=1` / `X=2,Y=1` → `LinxV5 CallingConv Fail!` (Issue #6). -- `control` needs `-s core.singleTierMode=true` on gfsim; `.data` from `gen_data.py`. +## Runtime note +- `control` needs `-s core.singleTierMode=true` on gfsim; `.data` comes from `gen_data.py`. ## Adding a Test 1. Create `test/kernel//` with `src/`, `Makefile`, `compile.all`. diff --git a/benchmark/two-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py b/benchmark/two-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py index 139ef7e..5035af9 100644 --- a/benchmark/two-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py +++ b/benchmark/two-level-arch/test/kernel/broadcast/src/gfrun_broadcast.py @@ -10,18 +10,18 @@ import torch import torch.nn as nn import numpy as np -import signal +import signal from concurrent.futures import ThreadPoolExecutor, as_completed #1 read elf list -#2 分析elf name: matmul MNK +#2 分析elf name: matmul MNK #3 产生随机src0, src1 -> golden_dst 放到 kernel/golden/kernel_matmul_MASK_M128_N1024_K128_tM32_tK32_tN32/src0.bin, src1.bin golden.bin #4 跑gfrun 读取src0, src1 产生dst -> /remote/lms01/c00622284/janus/JanusCoreBench/compare/kernel_matmul_MASK_M128_N1024_K128_tM32_tK32_tN32/res.bin #5 跑res_compare.py #6 输出一个报告. MAX_WORKERS = 20 #parallel thread num depend on your machine -gfrun = "/remote/lms01/l00948608/project/jcore_benchmark/BlockISA/BlockISA/bin/gfrun" +gfrun = os.environ.get("GFRUN") gfrun_args = " -t 1 -f " cmp_root = os.path.abspath(os.path.dirname(__file__)+"/../../../compare") @@ -72,7 +72,9 @@ def gen_input_and_golden(elf_name, path): def run_qemu(elf): print('Start to run gfrun----------') - try: + if args.plat != "cpu" and not gfrun: + raise RuntimeError("set GFRUN to the functional-model executable") + try: if os.path.exists(elf): if args.plat == "cpu": print('plat == "cpu"') @@ -88,12 +90,12 @@ def run_qemu(elf): else: print('elf not exist') return (elf, "not exist", "") - + except subprocess.TimeoutExpired: os.killpg(os.getpgid(proc.pid), signal.SIGKILL) proc.communicate() # 确保进程资源被正确清理 return (elf, "timeout", "Timeout expired") - + except Exception as e: os.killpg(os.getpgid(proc.pid), signal.SIGKILL) proc.communicate() # 确保进程资源被正确清理 @@ -103,7 +105,7 @@ def result_compare(cmp_data, golden_data): print("cmp data: ",cmp_data) if "accelerator_fusion_fa1" in cmp_data: res = np.fromfile(cmp_data, dtype=DType_np) - res_ref = np.fromfile(golden_data, dtype=DType_np) + res_ref = np.fromfile(golden_data, dtype=DType_np) else: res = np.fromfile(cmp_data, dtype=DType_np) res_ref = np.fromfile(golden_data, dtype=DType_np) @@ -179,4 +181,3 @@ def log_result(elf, status, chk_status, loss): f.write(f"{elf}\n") - diff --git a/benchmark/two-level-arch/test/kernel/control/compile.all b/benchmark/two-level-arch/test/kernel/control/compile.all index 6ec69f0..e195dd5 100755 --- a/benchmark/two-level-arch/test/kernel/control/compile.all +++ b/benchmark/two-level-arch/test/kernel/control/compile.all @@ -1,4 +1,6 @@ -#! /bin/bash +#!/usr/bin/env bash +set -euo pipefail + for debug in on off; do if [[ "$debug" == "on" ]]; then debug_define="" diff --git a/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh b/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh index 2668212..38aa95a 100755 --- a/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh +++ b/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh @@ -1,5 +1,6 @@ -#!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms60/c00622284/janus/linxisa_compiler_v0.55/linx_blockisa_llvm_musl/bin}" +#!/usr/bin/env bash +set -euo pipefail +: "${COMPILER_DIR:?set COMPILER_DIR to compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -26,11 +27,11 @@ _binary_${name}_data_end: .equ _binary_${name}_data_size, .-_binary_${name}_data_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "inserted_slot" build_one "lookup_keys" build_one "lookup_values" -echo "Done building data object files" \ No newline at end of file +echo "Done building data object files" diff --git a/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md b/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md index de90737..91c486f 100644 --- a/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md +++ b/benchmark/two-level-arch/test/kernel/control/hashtable_lookup_simd/run_hashtable_lookup_simd.md @@ -83,7 +83,7 @@ COMPILER_DIR= bash data_obj/build_data_obj.sh data_obj ../../ ```bash # 对每个 *.data 文件执行： -$COMPILER_DIR/clang++ -target linx64v5 -c *.s -o output/.../*.o +$COMPILER_DIR/clang++ -target linx64-linx-none-elf -c *.s -o output/.../*.o ``` 生成的 `.o` 文件： @@ -101,7 +101,7 @@ output/kernel/control/hashtable_lookup_simd/data_obj/lookup_values.o # 期望 # 实际执行（make 自动推导）： cd {WORKSPACE}/JanusCoreBench/test/kernel/control $COMPILER_DIR/clang++ \ - -c -mlxbc -fenable-matrix -O2 \ + -c -target linx64-linx-none-elf -fenable-matrix -O2 \ -std=c++20 \ -I{WORKSPACE}/JanusCoreBench/include \ -I{WORKSPACE}/JanusCoreBench/test/common \ @@ -156,7 +156,7 @@ cd {WORKSPACE}/LinxBlockModel 编译器路径通过环境变量 `COMPILER_DIR` 传入，不在 Makefile 或 compile.all 中写死： ```bash -export COMPILER_DIR=/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.18/bin +export COMPILER_DIR=${WORKSPACE}/compiler/llvm/build-linxisa-clang/bin cd {WORKSPACE}/JanusCoreBench/test/kernel/control ./compile.all ``` diff --git a/benchmark/two-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh b/benchmark/two-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh index ec4819c..07076a8 100755 --- a/benchmark/two-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh +++ b/benchmark/two-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh @@ -1,5 +1,10 @@ #!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.16/bin}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-superproject-working-tree 2>/dev/null || true)" +if [[ -z "$SUPERPROJECT_ROOT" ]]; then + SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)" +fi +COMPILER_DIR="${COMPILER_DIR:-${SUPERPROJECT_ROOT}/compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -28,7 +33,7 @@ _binary_${sym_name}_end: .equ _binary_${sym_name}_size, .-_binary_${sym_name}_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "buckets.bin" diff --git a/benchmark/two-level-arch/test/kernel/reduction/README.md b/benchmark/two-level-arch/test/kernel/reduction/README.md index ca8a8f7..4bf1a96 100644 --- a/benchmark/two-level-arch/test/kernel/reduction/README.md +++ b/benchmark/two-level-arch/test/kernel/reduction/README.md @@ -7,7 +7,7 @@ This directory contains benchmark implementations for various reduction operatio The reduction operators are categorized into three main types based on their reduction direction and optimization strategy: - **Column Reduction (col)**: Reduces along the column dimension (M-axis) -- **Row Reduction (row)**: Reduces along the row dimension (N-axis) +- **Row Reduction (row)**: Reduces along the row dimension (N-axis) - **3D Column Reduction (3dcol)**: Batch column reduction for 3D tensors with layout transformation to maximize vector lane utilization ## Directory Structure @@ -162,7 +162,7 @@ using tile_shapeTmp = Tile; using tile_shapeSum = Tile; // 1×64, valid=8 ``` -Note: `tM/8 = 128/8 = 16` rows in the tile, but only `tM_VLD/8 = 120/8 = 15` are valid. The 16th row is zero-padded during DMA load (`TCOPYIN`) so that the row count remains a multiple of 8, enabling a complete 8-way tree reduction without remainder handling. +Note: `tM/8 = 128/8 = 16` rows in the tile, but only `tM_VLD/8 = 120/8 = 15` are valid. The 16th row is zero-padded during DMA load (`TLOAD`) so that the row count remains a multiple of 8, enabling a complete 8-way tree reduction without remainder handling. ##### Two-Stage Reduction @@ -222,7 +222,7 @@ All operators use 8-way unrolled tree reduction to maximize instruction-level pa ```cpp // Level 1: 8 pairwise operations sum_01 = a + b; sum_23 = c + d; sum_45 = e + f; sum_67 = g + h; -// Level 2: 4 pairwise operations +// Level 2: 4 pairwise operations sum_0123 = sum_01 + sum_23; sum_4567 = sum_45 + sum_67; // Level 3: Final reduction sum_tmp = sum_0123 + sum_4567; @@ -234,7 +234,7 @@ Uses the PTO (Parallel Tile Operations) framework: - `global_tensor`: Defines global memory layout - `Tile`: Defines tile shape and memory location (Vec/Scalar) - `global_iterator`: Iterates over tiles in global memory -- `TCOPYIN`/`TCOPYOUT`: DMA transfers between global and tile memory +- `TLOAD`/`TSTORE`: DMA transfers between global and tile memory ### 3. Corner Case Handling @@ -267,7 +267,7 @@ Uses built-in vector operations: // For reducesum_col (8192×1024 input) #define DType int32_t #define gIMs 8192 // Global M dimension -#define gINs 1024 // Global N dimension +#define gINs 1024 // Global N dimension #define tMs 32 // Tile M dimension #define tNs 128 // Tile N dimension ``` @@ -306,7 +306,7 @@ dtype output[381 * 1 * 8]; for(int i = 0; i < 381; i++) { reducesum_colsum_rand( - &input[i * 120 * 8], + &input[i * 120 * 8], &output[i * 8] ); } diff --git a/benchmark/two-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh b/benchmark/two-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh index 8128f1e..046b032 100755 --- a/benchmark/two-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh +++ b/benchmark/two-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh @@ -1,5 +1,10 @@ #!/bin/bash -COMPILER_DIR="${COMPILER_DIR:-/remote/lms01/j00827727/jcore/compilers/linx_blockisa_llvm_musl0.56.16/bin}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-superproject-working-tree 2>/dev/null || true)" +if [[ -z "$SUPERPROJECT_ROOT" ]]; then + SUPERPROJECT_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)" +fi +COMPILER_DIR="${COMPILER_DIR:-${SUPERPROJECT_ROOT}/compiler/llvm/build-linxisa-clang/bin}" DATA_OBJ_DIR="$1" OUTPUT_DIR="$2" @@ -25,10 +30,10 @@ _binary_${name}_data_end: .equ _binary_${name}_data_size, .-_binary_${name}_data_start EOF - $COMPILER_DIR/clang++ -target linx64v5 -c "$asm_file" -o "$obj_file" + $COMPILER_DIR/clang++ -target linx64-linx-none-elf -c "$asm_file" -o "$obj_file" } build_one "input_131072" build_one "top_2048_out" -echo "Done building data object files" \ No newline at end of file +echo "Done building data object files" diff --git a/benchmark/two-level-arch/test/kernel/sort/topk/topk.cpp b/benchmark/two-level-arch/test/kernel/sort/topk/topk.cpp index 1f579fd..9e36c15 100644 --- a/benchmark/two-level-arch/test/kernel/sort/topk/topk.cpp +++ b/benchmark/two-level-arch/test/kernel/sort/topk/topk.cpp @@ -53,7 +53,7 @@ int main() { using HistGT = GlobalTensor, Stride<1,1,1,16,1>>; uint32_t histResult[256]; HistGT histGlobal(histResult); - TCOPYOUT(histGlobal, high8HistTile); + TSTORE(histGlobal, high8HistTile); uint32_t global_high8_hist[256] = {0}; for (int b = 0; b < 256; b++) { @@ -92,7 +92,7 @@ int main() { uint32_t low8HistResult[256]; HistGT low8HistGlobal(low8HistResult); - TCOPYOUT(low8HistGlobal, low8HistTile); + TSTORE(low8HistGlobal, low8HistTile); uint32_t global_low8_hist_kth[256] = {0}; for (int b = 0; b < 256; b++) { diff --git a/benchmark/two-level-arch/test/tileop_api/Makefile b/benchmark/two-level-arch/test/tileop_api/Makefile index 46d4b54..67023a5 100644 --- a/benchmark/two-level-arch/test/tileop_api/Makefile +++ b/benchmark/two-level-arch/test/tileop_api/Makefile @@ -5,7 +5,7 @@ endif include ../common/Makefile.common -# The v0.56 direct-boot lane uses the scalar Linx implementations. The +# The current direct-boot lane uses the scalar Linx implementations. The # retired tile-register constraint path (`Tr`) is intentionally not enabled. DEFINES := $(filter-out -DENABLE_TENSOR_INSTR,$(DEFINES)) diff --git a/benchmark/two-level-arch/test/tileop_api/README.md b/benchmark/two-level-arch/test/tileop_api/README.md index 648f331..3e44163 100644 --- a/benchmark/two-level-arch/test/tileop_api/README.md +++ b/benchmark/two-level-arch/test/tileop_api/README.md @@ -19,4 +19,4 @@ make TESTCASE=TAdd PLAT=linx ``` `TLoad` and `TStore` are the active memory-operation names. Legacy -`TCopyIn`/`TCopyOut` aliases are not part of this manifest. +`TLoad`/`TStore` aliases are not part of this manifest. diff --git a/benchmark/two-level-arch/test/tileop_api/compile.all b/benchmark/two-level-arch/test/tileop_api/compile.all index 7682284..89673cc 100755 --- a/benchmark/two-level-arch/test/tileop_api/compile.all +++ b/benchmark/two-level-arch/test/tileop_api/compile.all @@ -12,7 +12,6 @@ make TESTCASE=TAdds make TESTCASE=TAnd make TESTCASE=TCI make TESTCASE=TCmp -make TESTCASE=TCopy make TESTCASE=TLoad make TESTCASE=TStore make TESTCASE=TCvt diff --git a/benchmark/two-level-arch/test/tileop_api/src/TCopy.cpp b/benchmark/two-level-arch/test/tileop_api/src/TCopy.cpp deleted file mode 100644 index d0fb15c..0000000 --- a/benchmark/two-level-arch/test/tileop_api/src/TCopy.cpp +++ /dev/null @@ -1,339 +0,0 @@ -#include "../data.hpp" -#include - -#ifdef LINX_PMC -#include "../linxStartEnd.hpp" -#endif - -#ifdef __linx -int main(); - -static inline __attribute__((noreturn)) void linx_supernpu_exit(uint32_t code) { - if (code == 0) { - __asm__ volatile( - "BSTART.STD\n" - "lui 65545, ->u\n" - "lui 5, ->t\n" - "addi t#1, 1365, ->t\n" - "c.swi t#1, [u#1, 0]\n" - "BSTOP\n" - ::: "memory"); - } else { - __asm__ volatile( - "BSTART.STD\n" - "lui 65545, ->u\n" - "lui 19, ->t\n" - "addi t#1, 819, ->t\n" - "c.swi t#1, [u#1, 0]\n" - "BSTOP\n" - ::: "memory"); - } - while (1) { - } -} - -extern "C" __attribute__((noreturn, section(".text._start"))) void _start(void) { - linx_supernpu_exit(static_cast(main())); -} -#endif - -template -void test_Nz(T *dst, T *src0) { - using gm_shape = global_tensor>; - using tile_shape = TileLeft; - using glb_iterator = global_iterator; - - glb_iterator gS0Iter(src0); - glb_iterator gDIter(dst); - - uint16_t block_row = gm_row / tile_row; - uint16_t block_col = gm_col / tile_col; - for (int i = 0; i < block_row; ++i) { - for (int j = 0; j < block_col; ++j) { - auto s0 = gS0Iter(i, j); - auto res = gDIter(i, j); - - tile_shape d0, d1; - TLOAD(d0, s0); - TCOPY(d1, d0); - TSTORE(res, d1); - } - } -} - -template -void test_Nz_Dynamic(T *dst, T *src0) { - using gm_shape = global_tensor>; - using tile_shape = TileLeft; - - volatile size_t tile_valid_row = tile_row - 2; - volatile size_t tile_valid_col = tile_col - 2; - - uint16_t block_row = (gm_row + tile_valid_row - 1) / tile_valid_row; - uint16_t block_col = (gm_col + tile_valid_col - 1) / tile_valid_col; - - for (int i = 0; i < block_row; ++i) { - for (int j = 0; j < block_col; ++j) { - uint16_t remainder_row = gm_row - i * tile_valid_row; - uint16_t remainder_col = gm_col - j * tile_valid_col; - - uint16_t active_row = remainder_row < tile_valid_row ? remainder_row : tile_valid_row; - uint16_t active_col = remainder_col < tile_valid_col ? remainder_col : tile_valid_col; - - int offset = i * (tile_valid_row * gm_col) + j * tile_valid_col; - gm_shape s0(src0 + offset); - gm_shape res(dst + offset); - - tile_shape d0(active_row, active_col); - tile_shape d1(active_row, active_col); - TLOAD(d0, s0); - TCOPY(d1, d0); - TSTORE(res, d1); - } - } -} - -template -void test_RowMajor(T *dst, T *src0) { - using gm_shape = global_tensor>; - using tile_shape = Tile; - - uint16_t block_row = gm_row / tile_row; - uint16_t block_col = gm_col / tile_col; - #pragma clang loop unroll(full) - for (int i = 0; i < block_row; ++i) { - #pragma clang loop unroll(full) - for (int j = 0; j < block_col; ++j) { - int offset = i * (tile_row * gm_col) + j * tile_col; - gm_shape s0(src0 + offset); - gm_shape res(dst + offset); - - tile_shape d0, d1; - TLOAD(d0, s0); - TCOPY(d1, d0); - TSTORE(res, d1); - } - } -} - -template -void test_RowMajor_Dynamic(T *dst, T *src0) { - using gm_shape = global_tensor>; - using tile_shape = Tile; - - volatile size_t tile_valid_row = tile_row - 2; - volatile size_t tile_valid_col = tile_col - 2; - - uint16_t block_row = (gm_row + tile_valid_row - 1) / tile_valid_row; - uint16_t block_col = (gm_col + tile_valid_col - 1) / tile_valid_col; - - for (int i = 0; i < block_row; ++i) { - for (int j = 0; j < block_col; ++j) { - uint16_t remainder_row = gm_row - i * tile_valid_row; - uint16_t remainder_col = gm_col - j * tile_valid_col; - - uint16_t active_row = remainder_row < tile_valid_row ? remainder_row : tile_valid_row; - uint16_t active_col = remainder_col < tile_valid_col ? remainder_col : tile_valid_col; - - int offset = i * (tile_valid_row * gm_col) + j * tile_valid_col; - gm_shape s0(src0 + offset); - gm_shape res(dst + offset); - - tile_shape d0(active_row, active_col); - tile_shape d1(active_row, active_col); - TLOAD(d0, s0); - TCOPY(d1, d0); - TSTORE(res, d1); - } - } -} - -template -void test_ColMajor(T *dst, T *src0) { - using gm_shape = global_tensor>; - using tile_shape = Tile; - - uint16_t block_row = gm_row / tile_row; - uint16_t block_col = gm_col / tile_col; - #pragma clang loop unroll(full) - for (int i = 0; i < block_col; ++i) { - #pragma clang loop unroll(full) - for (int j = 0; j < block_row; ++j) { - int offset = i * (tile_col * gm_row) + j * tile_row; - gm_shape s0(src0 + offset); - gm_shape res(dst + offset); - - tile_shape d0, d1; - TLOAD(d0, s0); - TCOPY(d1, d0); - TSTORE(res, d1); - } - } -} - -int main() { -#ifdef __linx - constexpr uint16_t gm_row = 4; - constexpr uint16_t gm_col = 4; - constexpr uint16_t tile_row = 4; - constexpr uint16_t tile_col = 4; -#else - constexpr uint16_t gm_row = 64; - constexpr uint16_t gm_col = 64; - constexpr uint16_t tile_row = 32; - constexpr uint16_t tile_col = 32; -#endif - - constexpr size_t gm_size = gm_row * gm_col; - constexpr size_t tile_size = tile_row * tile_col; - (void)tile_size; - -#ifdef __linx - static int64_t dst[gm_size]; - static int64_t src[gm_size]; - init_dst(dst, gm_size); - init_src_int(src, gm_size); - - test_RowMajor(dst, src); - - return 0; -#else - float *dst = (float *)malloc(gm_size * sizeof(float)); - check_mem_alloc(dst); - init_dst(dst, gm_size); - - float *src0 = (float *)malloc(gm_size * sizeof(float)); - check_mem_alloc(src0); - init_src_fp(src0, gm_size); - - float *dst_col = (float *)malloc(gm_size * sizeof(float)); - check_mem_alloc(dst_col); - init_dst(dst_col, gm_size); - - float *src0_col = (float *)malloc(gm_size * sizeof(float)); - check_mem_alloc(src0_col); - init_src_fp(src0_col, gm_size); - - __half *dst_f16 = (__half *)malloc(gm_size * sizeof(__half)); - check_mem_alloc(dst_f16); - init_dst(dst_f16, gm_size); - - __half *src0_f16 = (__half *)malloc(gm_size * sizeof(__half)); - check_mem_alloc(src0_f16); - init_src_fp(src0_f16, gm_size); - - int8_t *dst_i8 = (int8_t *)malloc(gm_size * sizeof(int8_t)); - check_mem_alloc(dst_i8); - init_dst(dst_i8, gm_size); - - int8_t *src0_i8 = (int8_t *)malloc(gm_size * sizeof(int8_t)); - check_mem_alloc(src0_i8); - init_src_int(src0_i8, gm_size); - - int16_t *dst_i16 = (int16_t *)malloc(gm_size * sizeof(int16_t)); - check_mem_alloc(dst_i16); - init_dst(dst_i16, gm_size); - - int16_t *src0_i16 = (int16_t *)malloc(gm_size * sizeof(int16_t)); - check_mem_alloc(src0_i16); - init_src_int(src0_i16, gm_size); - - int32_t *dst_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(dst_i32); - init_dst(dst_i32, gm_size); - - int32_t *src0_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(src0_i32); - init_src_int(src0_i32, gm_size); - - int64_t *dst_i64 = (int64_t *)malloc(gm_size * sizeof(int64_t)); - check_mem_alloc(dst_i64); - init_dst(dst_i64, gm_size); - - int64_t *src0_i64 = (int64_t *)malloc(gm_size * sizeof(int64_t)); - check_mem_alloc(src0_i64); - init_src_int(src0_i64, gm_size); - - int32_t *dst1_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(dst1_i32); - init_dst(dst1_i32, gm_size); - - int32_t *src1_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(src1_i32); - init_src_int(src1_i32, gm_size); - - int32_t *dst_nz_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(dst_nz_i32); - init_dst(dst_nz_i32, gm_size); - - int32_t *src_nz_i32 = (int32_t *)malloc(gm_size * sizeof(int32_t)); - check_mem_alloc(src_nz_i32); - init_src_int(src_nz_i32, gm_size); - -#ifdef LINX_PMC - PMC_START(); -#endif - //test for fp32 Nz - test_Nz(dst, src0); - - test_ColMajor(dst_col, src0_col); - - test_RowMajor(dst_f16, src0_f16); - - test_RowMajor(dst_i8, src0_i8); - - test_RowMajor(dst_i16, src0_i16); - - test_RowMajor(dst_i32, src0_i32); - - test_RowMajor(dst_i64, src0_i64); - - test_RowMajor_Dynamic(dst1_i32, src1_i32); - - test_Nz_Dynamic(dst_nz_i32, src_nz_i32); - -#ifdef LINX_PMC - PMC_END(); -#endif - - printf("Result:\n"); - OutArray(dst, gm_size); - OutArray(dst_f16, gm_size); - OutArray(dst_i8, gm_size); - OutArray(dst_i16, gm_size); - OutArray(dst_i32, gm_size); - OutArray(dst_i64, gm_size); - OutArray(dst1_i32, gm_size); - OutArray(dst_nz_i32, gm_size); - - free(dst); - free(src0); - - free(dst_f16); - free(src0_f16); - - free(dst_i8); - free(src0_i8); - - free(dst_i16); - free(src0_i16); - - free(dst_i32); - free(src0_i32); - - free(dst_i64); - free(src0_i64); - - free(dst1_i32); - free(src1_i32); - - free(dst_nz_i32); - free(src_nz_i32); - - return 0; -#endif -} diff --git a/compile_all.sh b/compile_all.sh index 77a0f78..865681f 100755 --- a/compile_all.sh +++ b/compile_all.sh @@ -1,4 +1,6 @@ -#!/bin/bash +#!/usr/bin/env bash +set -uo pipefail + # Top-level compilation script for all architecture backends # benchmark/two-level-arch (was benchmark-linxisa, Linx two-level block ISA) # benchmark/one-level-arch (was benchmark-ptoisa, PTO one-level tile ISA) @@ -18,7 +20,8 @@ compile_two_level() { if [ -f "$SCRIPT_DIR/benchmark/two-level-arch/compile_all.sh" ]; then bash "$SCRIPT_DIR/benchmark/two-level-arch/compile_all.sh" else - echo "Warning: benchmark/two-level-arch/compile_all.sh not found" + echo "Error: benchmark/two-level-arch/compile_all.sh not found" >&2 + return 1 fi } @@ -28,7 +31,8 @@ compile_one_level() { if [ -f "$SCRIPT_DIR/benchmark/one-level-arch/compile_all.sh" ]; then bash "$SCRIPT_DIR/benchmark/one-level-arch/compile_all.sh" else - echo "Warning: benchmark/one-level-arch/compile_all.sh not found" + echo "Error: benchmark/one-level-arch/compile_all.sh not found" >&2 + return 1 fi } @@ -40,8 +44,13 @@ case $ARCH in compile_one_level ;; all) - compile_two_level - compile_one_level + failures=() + compile_two_level || failures+=("two-level") + compile_one_level || failures+=("one-level") + if ((${#failures[@]})); then + echo "Failed architecture backends: ${failures[*]}" >&2 + exit 1 + fi ;; *) echo "Usage: $0 [two-level|one-level|all]" diff --git a/docs/assets/javascripts/supernpu.js b/docs/assets/javascripts/supernpu.js new file mode 100644 index 0000000..a7bf820 --- /dev/null +++ b/docs/assets/javascripts/supernpu.js @@ -0,0 +1,47 @@ +(function () { + function activateModel(root, target) { + const buttons = root.querySelectorAll('[role="tab"]'); + const panels = root.querySelectorAll('[role="tabpanel"]'); + + buttons.forEach((button) => { + const selected = button.dataset.target === target; + button.setAttribute("aria-selected", String(selected)); + button.tabIndex = selected ? 0 : -1; + }); + + panels.forEach((panel) => { + panel.hidden = panel.id !== target; + }); + } + + function hydrateModelSwitchers() { + document.querySelectorAll("[data-model-switcher]").forEach((root) => { + if (root.dataset.ready === "true") return; + root.dataset.ready = "true"; + + const buttons = Array.from(root.querySelectorAll('[role="tab"]')); + buttons.forEach((button, index) => { + button.addEventListener("click", () => activateModel(root, button.dataset.target)); + button.addEventListener("keydown", (event) => { + if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return; + event.preventDefault(); + let next; + if (event.key === "Home") next = buttons[0]; + else if (event.key === "End") next = buttons[buttons.length - 1]; + else { + const delta = event.key === "ArrowRight" ? 1 : -1; + next = buttons[(index + delta + buttons.length) % buttons.length]; + } + activateModel(root, next.dataset.target); + next.focus(); + }); + }); + }); + } + + if (typeof document$ !== "undefined") { + document$.subscribe(hydrateModelSwitchers); + } else { + document.addEventListener("DOMContentLoaded", hydrateModelSwitchers); + } +})(); diff --git a/docs/assets/stylesheets/supernpu.css b/docs/assets/stylesheets/supernpu.css new file mode 100644 index 0000000..3c0a0b4 --- /dev/null +++ b/docs/assets/stylesheets/supernpu.css @@ -0,0 +1,358 @@ +:root { + --snpu-ink: #172026; + --snpu-muted: #53616a; + --snpu-paper: #f7f9fa; + --snpu-panel: #ffffff; + --snpu-line: #d9e0e4; + --snpu-cyan: #006b78; + --snpu-green: #176f3e; + --snpu-amber: #b86c00; + --snpu-red: #b13a37; + --snpu-code: #111a20; +} + +[data-md-color-scheme="default"] { + --md-default-fg-color: var(--snpu-ink); + --md-default-fg-color--light: var(--snpu-muted); + --md-default-bg-color: var(--snpu-paper); + --md-primary-fg-color: #172026; + --md-primary-fg-color--light: #25323a; + --md-primary-fg-color--dark: #0e1418; + --md-accent-fg-color: var(--snpu-cyan); + --md-code-bg-color: #edf2f4; + --md-code-fg-color: #172026; + --md-typeset-a-color: #007489; +} + +[data-md-color-scheme="slate"] { + --snpu-ink: #e7edf0; + --snpu-muted: #aebbc2; + --snpu-paper: #11181c; + --snpu-panel: #172126; + --snpu-line: #344047; + --snpu-cyan: #40c4d5; + --snpu-green: #62cc8b; + --snpu-amber: #f0ae54; + --snpu-red: #f07b76; + --snpu-code: #0b1013; + --md-default-fg-color: var(--snpu-ink); + --md-default-fg-color--light: var(--snpu-muted); + --md-default-bg-color: var(--snpu-paper); + --md-primary-fg-color: #0b1114; + --md-primary-fg-color--light: #172126; + --md-primary-fg-color--dark: #080d0f; + --md-accent-fg-color: var(--snpu-cyan); + --md-code-bg-color: #0d1519; + --md-code-fg-color: #dce7eb; + --md-typeset-a-color: #52c7d6; +} + +.md-grid { + max-width: 92rem; +} + +.md-header, +.md-tabs { + box-shadow: none; + border-bottom: 1px solid color-mix(in srgb, var(--snpu-line) 65%, transparent); +} + +.md-nav__link, +.md-content__button { + color: var(--snpu-muted); +} + +.md-nav__link:hover, +.md-nav__link:focus, +.md-nav__link--active, +.md-nav__link--active:hover { + color: var(--snpu-cyan); +} + +.md-typeset { + color: var(--snpu-ink); + font-size: 0.79rem; + line-height: 1.65; +} + +.md-typeset h1, +.md-typeset h2, +.md-typeset h3 { + color: var(--snpu-ink); + letter-spacing: 0; + font-weight: 650; +} + +.md-typeset h1 { + margin-bottom: 0.65rem; +} + +.md-typeset h2 { + border-bottom: 1px solid var(--snpu-line); + padding-bottom: 0.35rem; +} + +.md-typeset code, +.md-typeset kbd, +.md-typeset pre { + font-variant-ligatures: none; +} + +.md-typeset .highlight > pre, +.md-typeset pre > code { + background: var(--snpu-code); + color: #dce7eb; +} + +.snpu-masthead { + border-top: 4px solid var(--snpu-cyan); + border-bottom: 1px solid var(--snpu-line); + padding: 2rem 0 1.7rem; + margin: 0 0 1.5rem; +} + +.snpu-masthead__label { + color: var(--snpu-cyan); + font-size: 0.68rem; + font-weight: 700; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.snpu-masthead h1 { + max-width: 21ch; + margin: 0.35rem 0 0.7rem; + font-size: clamp(1.85rem, 4vw, 3rem); + line-height: 1.05; +} + +.snpu-masthead p { + max-width: 70ch; + margin: 0; + color: var(--snpu-muted); + font-size: 0.98rem; +} + +.snpu-statusbar { + display: flex; + flex-wrap: wrap; + gap: 0.45rem 1.2rem; + margin-top: 1.1rem; + color: var(--snpu-muted); + font-family: var(--md-code-font-family); + font-size: 0.68rem; +} + +.snpu-statusbar strong { + color: var(--snpu-green); + font-weight: 700; +} + +.snpu-grid { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: 0.8rem; + margin: 1rem 0 1.8rem; +} + +.snpu-card { + position: relative; + min-width: 0; + border: 1px solid var(--snpu-line); + border-top: 3px solid var(--snpu-cyan); + border-radius: 4px; + background: var(--snpu-panel); + padding: 1rem; +} + +.snpu-card[data-accent="compute"] { border-top-color: var(--snpu-amber); } +.snpu-card[data-accent="vector"] { border-top-color: var(--snpu-green); } +.snpu-card[data-accent="warning"] { border-top-color: var(--snpu-red); } + +.snpu-card h3 { + margin: 0 0 0.3rem; + font-size: 0.88rem; +} + +.snpu-card p { + margin: 0 0 0.55rem; + color: var(--snpu-muted); +} + +.snpu-card a { + font-weight: 650; +} + +.snpu-card a::after { + position: absolute; + inset: 0; + content: ""; +} + +.snpu-card:focus-within { + outline: 2px solid var(--snpu-cyan); + outline-offset: 2px; +} + +.snpu-flow { + display: grid; + grid-template-columns: repeat(4, minmax(0, 1fr)); + border: 1px solid var(--snpu-line); + border-radius: 4px; + overflow: hidden; + margin: 1rem 0 1.5rem; + padding: 0; + list-style: none; +} + +.snpu-flow > li { + min-width: 0; + padding: 0.85rem; + border-right: 1px solid var(--snpu-line); + background: var(--snpu-panel); +} + +.snpu-flow > li:last-child { border-right: 0; } +.snpu-flow strong { display: block; margin-bottom: 0.2rem; } +.snpu-flow span { color: var(--snpu-muted); font-size: 0.7rem; } + +.snpu-model { + border: 1px solid var(--snpu-line); + border-radius: 4px; + background: var(--snpu-panel); + margin: 1rem 0 1.8rem; +} + +.snpu-model__tabs { + display: flex; + overflow-x: auto; + border-bottom: 1px solid var(--snpu-line); +} + +.snpu-model__tabs button { + appearance: none; + flex: 1 0 auto; + border: 0; + border-right: 1px solid var(--snpu-line); + border-bottom: 3px solid transparent; + background: transparent; + color: var(--snpu-muted); + cursor: pointer; + font: inherit; + font-size: 0.7rem; + font-weight: 650; + padding: 0.7rem 0.8rem; +} + +.snpu-model__tabs button:last-child { border-right: 0; } +.snpu-model__tabs button:hover { color: var(--snpu-ink); } +.snpu-model__tabs button:focus-visible { + outline: 2px solid var(--snpu-cyan); + outline-offset: -3px; +} +.snpu-model__tabs button[aria-selected="true"] { + border-bottom-color: var(--snpu-cyan); + color: var(--snpu-ink); +} + +.snpu-model__panel { + min-height: 9.5rem; + padding: 1rem 1.1rem; +} + +.snpu-model__panel[hidden] { display: none; } +.snpu-model__panel h3 { margin-top: 0; } +.snpu-model__panel p:last-child { margin-bottom: 0; } + +.snpu-source { + border-left: 3px solid var(--snpu-green); + background: color-mix(in srgb, var(--snpu-green) 7%, var(--snpu-panel)); + padding: 0.75rem 0.9rem; + margin: 1rem 0; +} + +.snpu-source strong { color: var(--snpu-green); } + +.snpu-contract { + display: grid; + grid-template-columns: 10rem 1fr; + gap: 0; + border: 1px solid var(--snpu-line); + border-bottom: 0; + margin: 1rem 0; +} + +.snpu-contract > * { + border-bottom: 1px solid var(--snpu-line); + padding: 0.55rem 0.7rem; +} + +.snpu-contract dt { + background: color-mix(in srgb, var(--snpu-cyan) 7%, var(--snpu-panel)); + font-family: var(--md-code-font-family); + font-size: 0.7rem; + font-weight: 700; +} + +.snpu-contract dd { margin: 0; } + +.md-typeset table:not([class]) { + border: 1px solid var(--snpu-line); + border-radius: 0; + font-size: 0.7rem; +} + +.md-typeset table:not([class]) th { + background: color-mix(in srgb, var(--snpu-ink) 7%, var(--snpu-panel)); + color: var(--snpu-ink); +} + +.md-typeset .admonition, +.md-typeset details { + border-radius: 4px; + box-shadow: none; +} + +.md-typeset .tabbed-set { + border: 1px solid var(--snpu-line); + border-radius: 4px; + background: var(--snpu-panel); +} + +.md-typeset .tabbed-content { + padding: 0 0.9rem 0.3rem; +} + +.snpu-kicker { + color: var(--snpu-muted); + font-family: var(--md-code-font-family); + font-size: 0.68rem; + text-transform: uppercase; +} + +@media screen and (max-width: 76.25em) { + .snpu-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } +} + +@media screen and (max-width: 44.9375em) { + .md-typeset { font-size: 0.76rem; } + .snpu-masthead { padding-top: 1.2rem; } + .snpu-masthead h1 { font-size: 1.9rem; } + .snpu-grid { grid-template-columns: 1fr; } + .snpu-flow { grid-template-columns: 1fr 1fr; } + .snpu-flow > li:nth-child(2) { border-right: 0; } + .snpu-flow > li:nth-child(-n + 2) { border-bottom: 1px solid var(--snpu-line); } + .snpu-contract { grid-template-columns: 1fr; } + .snpu-contract dt { border-bottom: 0; } + .snpu-model__panel { min-height: 12rem; } + .snpu-model__tabs button { + flex: 1 0 5.2rem; + min-width: 5.2rem; + padding: 0.65rem 0.35rem; + font-size: 0.68rem; + white-space: nowrap; + scroll-snap-align: start; + } + + .snpu-model__tabs { scroll-snap-type: x proximity; } +} diff --git a/docs/benchmarks/catalog.json b/docs/benchmarks/catalog.json new file mode 100644 index 0000000..8f06b24 --- /dev/null +++ b/docs/benchmarks/catalog.json @@ -0,0 +1,3298 @@ +{ + "schema": 3, + "active_build_variants": 53, + "source_implementations": 20, + "families": [ + { + "backend": "One-level", + "family": "broadcast", + "page": "benchmarks/catalog/one-level/broadcast/index.md", + "implementations": 4, + "variants": 6, + "intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TDIVS", + "TEXPANDS", + "TINSERT", + "TLOAD", + "TMULS", + "TREMS", + "TROWEXPAND", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "concat", + "page": "benchmarks/catalog/one-level/concat/index.md", + "implementations": 2, + "variants": 4, + "intrinsics": [ + "MGATHER", + "MSCATTER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB" + ] + }, + { + "backend": "One-level", + "family": "control", + "page": "benchmarks/catalog/one-level/control/index.md", + "implementations": 1, + "variants": 6, + "intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ] + }, + { + "backend": "One-level", + "family": "element_wise/gelu", + "page": "benchmarks/catalog/one-level/element-wise-gelu/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "TADDS", + "TCVT", + "TEXP", + "TLOAD", + "TMAXS", + "TMINS", + "TMUL", + "TMULS", + "TRECIP", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "fa", + "page": "benchmarks/catalog/one-level/fa/index.md", + "implementations": 2, + "variants": 9, + "intrinsics": [ + "TADD", + "TCOLEXPANDMUL", + "TCOLEXPANDSUB", + "TCOLMAX", + "TCOLSUM", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TMAX", + "TMUL", + "TMULS", + "TQUANT", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ] + }, + { + "backend": "One-level", + "family": "gather", + "page": "benchmarks/catalog/one-level/gather/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "MGATHER", + "TLOAD", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "page": "benchmarks/catalog/one-level/matmul/index.md", + "implementations": 3, + "variants": 16, + "intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TMUL", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "reduction/reducemax_col", + "page": "benchmarks/catalog/one-level/reduction-reducemax-col/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "TCOLMAX", + "TEXPANDS", + "TLOAD", + "TMAX", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "reduction/reducemax_row", + "page": "benchmarks/catalog/one-level/reduction-reducemax-row/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "TEXPANDS", + "TLOAD", + "TMAX", + "TROWMAX", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "reduction/reducesum_col", + "page": "benchmarks/catalog/one-level/reduction-reducesum-col/index.md", + "implementations": 1, + "variants": 2, + "intrinsics": [ + "TADD", + "TCOLSUM", + "TEXPANDS", + "TLOAD", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "reduction/reducesum_row", + "page": "benchmarks/catalog/one-level/reduction-reducesum-row/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "TADD", + "TEXPANDS", + "TLOAD", + "TROWSUM", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "sort", + "page": "benchmarks/catalog/one-level/sort/index.md", + "implementations": 1, + "variants": 1, + "intrinsics": [ + "TEXPANDS", + "TSTORE" + ] + }, + { + "backend": "One-level", + "family": "transpose", + "page": "benchmarks/catalog/one-level/transpose/index.md", + "implementations": 1, + "variants": 4, + "intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB", + "TTRANS" + ] + } + ], + "builds": [ + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 47, + "command": "make TESTCASE=broadcast_vec_07 DType=__half tMs=16 MAX_DIMs=2 IN_SHAPEs=1443,1 OUT_SHAPEs=1443,129 IN_DIMs=2 OUT_DIMs=2 gIMs=1443*1 gOMs=1443*129 IN_SHAPE_NAME=1443_1 OUT_SHAPE_NAME=1443_129 diss", + "testcase": "broadcast_vec_07", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_07.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-vec-07-ba7002f4.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TLOAD", + "TROWEXPAND", + "TSTORE" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TROWEXPAND": [ + "TROWEXPAND" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 52, + "command": "make TESTCASE=broadcast_vec_019 DType=__half tMs=8 kInners=49 MAX_DIMs=3 IN_SHAPEs=1280,1,49 OUT_SHAPEs=1280,8,49 IN_DIMs=3 OUT_DIMs=3 gIMs=1280*1*49 gOMs=1280*8*49 IN_SHAPE_NAME=1280_1_49 OUT_SHAPE_NAME=1280_8_49 diss", + "testcase": "broadcast_vec_019", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_019.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-vec-019-8713b1a8.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TINSERT", + "TLOAD", + "TSTORE" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TINSERT": [ + "TINSERT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 57, + "command": "make TESTCASE=broadcast_vec_039 DType=__half tMs=8 kInners=16 MAX_DIMs=3 IN_SHAPEs=8192,1,16 OUT_SHAPEs=8192,8,16 IN_DIMs=3 OUT_DIMs=3 gIMs=8192*1*16 gOMs=8192*8*16 IN_SHAPE_NAME=8192_1_16 OUT_SHAPE_NAME=8192_8_16 diss", + "testcase": "broadcast_vec_039", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_039.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-vec-039-ac6775c3.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TINSERT", + "TLOAD", + "TSTORE" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TINSERT": [ + "TINSERT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 70, + "command": "make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=2 IN_SHAPEs=1042,1 OUT_SHAPEs=1042,129 IN_DIMs=2 OUT_DIMs=2 gIMs=1042*1 gOMs=1042*129 TESTNAME=broadcast_to_07 IN_SHAPE_NAME=1042_1 OUT_SHAPE_NAME=1042_129 res_check=off diss", + "testcase": "broadcast", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TDIVS", + "TEXPANDS", + "TMULS", + "TREMS", + "TSTORE" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TREMS": [ + "TREMS" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 77, + "command": "make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=3 IN_SHAPEs=1280,1,49 OUT_SHAPEs=1280,8,49 IN_DIMs=3 OUT_DIMs=3 gIMs=1280*1*49 gOMs=1280*8*49 TESTNAME=broadcast_to_ABA_019_new IN_SHAPE_NAME=1280_1_49 OUT_SHAPE_NAME=1280_8_49 res_check=off diss", + "testcase": "broadcast", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TDIVS", + "TEXPANDS", + "TMULS", + "TREMS", + "TSTORE" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TREMS": [ + "TREMS" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "broadcast", + "manifest": "benchmark/one-level-arch/test/kernel/broadcast/compile.all", + "manifest_line": 84, + "command": "make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=5 IN_SHAPEs=1,1,1,65,128 OUT_SHAPEs=1,1,7,65,128 IN_DIMs=5 OUT_DIMs=5 gIMs=1*1*1*65*128 gOMs=1*1*7*65*128 TESTNAME=BroadcastTo_ND_NCDHW_float16_int64_HunyuanImage21_MLLM_1015_000004 IN_SHAPE_NAME=1_1_1_65_128 OUT_SHAPE_NAME=1_1_7_65_128 res_check=off diss", + "testcase": "broadcast", + "source": "benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp", + "page": "benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TDIVS", + "TEXPANDS", + "TMULS", + "TREMS", + "TSTORE" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TREMS": [ + "TREMS" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "concat", + "manifest": "benchmark/one-level-arch/test/kernel/concat/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=concat_gather COMPILER_DIR=$COMPILER_DIR DType=int32_t MAX_DIM=8 tM=512 IN_SHAPE=64,2 OUT_SHAPE=64,2000 IN_SHAPE_NAME=64_2 OUT_SHAPE_NAME=64_2000 DATA_DIM=2 CONCAT_DIM=1 gIM='1000*64*2' gOM='64*2000'", + "testcase": "concat_gather", + "source": "benchmark/one-level-arch/test/kernel/concat/src/concat_gather.cpp", + "page": "benchmarks/catalog/one-level/concat/concat-gather-1c6990b2.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TMULS", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "concat", + "manifest": "benchmark/one-level-arch/test/kernel/concat/compile.all", + "manifest_line": 18, + "command": "make TESTCASE=concat_gather COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=256,8 OUT_SHAPE=256,8000 IN_SHAPE_NAME=256_8 OUT_SHAPE_NAME=256_8000 DATA_DIM=2 CONCAT_DIM=1 gIM='4000*256*8' gOM='256*8000'", + "testcase": "concat_gather", + "source": "benchmark/one-level-arch/test/kernel/concat/src/concat_gather.cpp", + "page": "benchmarks/catalog/one-level/concat/concat-gather-1c6990b2.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TMULS", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "concat", + "manifest": "benchmark/one-level-arch/test/kernel/concat/compile.all", + "manifest_line": 28, + "command": "make TESTCASE=concat_scatter COMPILER_DIR=$COMPILER_DIR DType=int32_t MAX_DIM=8 tM=512 IN_SHAPE=64,2 OUT_SHAPE=64,2000 IN_SHAPE_NAME=64_2 OUT_SHAPE_NAME=64_2000 DATA_DIM=2 CONCAT_DIM=1 gIM='1000*64*2' gOM='64*2000'", + "testcase": "concat_scatter", + "source": "benchmark/one-level-arch/test/kernel/concat/src/concat_scatter.cpp", + "page": "benchmarks/catalog/one-level/concat/concat-scatter-29559c6c.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MSCATTER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSUB" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MSCATTER": [ + "MSCATTER" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "concat", + "manifest": "benchmark/one-level-arch/test/kernel/concat/compile.all", + "manifest_line": 35, + "command": "make TESTCASE=concat_scatter COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=256,8 OUT_SHAPE=256,8000 IN_SHAPE_NAME=256_8 OUT_SHAPE_NAME=256_8000 DATA_DIM=2 CONCAT_DIM=1 gIM='4000*256*8' gOM='256*8000'", + "testcase": "concat_scatter", + "source": "benchmark/one-level-arch/test/kernel/concat/src/concat_scatter.cpp", + "page": "benchmarks/catalog/one-level/concat/concat-scatter-29559c6c.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MSCATTER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSUB" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MSCATTER": [ + "MSCATTER" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col1024_debug_off EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=1024 -DFOR_GFSIM\" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col1024_debug_on EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=1024 \" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col256_debug_off EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=256 -DFOR_GFSIM\" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col256_debug_on EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=256 \" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col512_debug_off EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=512 -DFOR_GFSIM\" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "control", + "manifest": "benchmark/one-level-arch/test/kernel/control/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col512_debug_on EXTRA_DEFINES=\"-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=512 \" diss", + "testcase": "hashtable_lookup_simd", + "source": "benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp", + "page": "benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TADDS", + "TAND", + "TCMP", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMULS", + "TOR", + "TREM", + "TSEL", + "TSHLS", + "TSHRS", + "TSTORE", + "TXOR" + ], + "intrinsic_forms": { + "TSHLS": [ + "TSHLS" + ], + "TSHRS": [ + "TSHRS" + ], + "TOR": [ + "TOR" + ], + "TCVT": [ + "TCAST", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TXOR": [ + "TXOR" + ], + "TADDS": [ + "TADDS" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TAND": [ + "TAND" + ], + "TREM": [ + "TREM" + ], + "TADD": [ + "TADD" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TCMP": [ + "TCMP" + ], + "TSEL": [ + "TSELECT" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "element_wise/gelu", + "manifest": "benchmark/one-level-arch/test/kernel/element_wise/gelu/compile.all", + "manifest_line": 10, + "command": "make TESTCASE=gelu DTYPE=__bf16 tMs=2048 gMs=24*8*1024 SHAPE_NAME=24_8_1024 Approximate=false diss", + "testcase": "gelu", + "source": "benchmark/one-level-arch/test/kernel/element_wise/gelu/src/gelu.cpp", + "page": "benchmarks/catalog/one-level/element-wise-gelu/gelu-fdf67353.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADDS", + "TCVT", + "TEXP", + "TLOAD", + "TMAXS", + "TMINS", + "TMUL", + "TMULS", + "TRECIP", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "TCVT" + ], + "TMAXS": [ + "TMAXS" + ], + "TMINS": [ + "TMINS" + ], + "TMUL": [ + "TMUL" + ], + "TMULS": [ + "TMULS" + ], + "TADDS": [ + "TADDS" + ], + "TEXP": [ + "TEXP" + ], + "TRECIP": [ + "TRECIP" + ], + "TLOAD": [ + "TLOAD" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 8, + "command": "make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=2", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 9, + "command": "make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=1 Y=2", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 12, + "command": "make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=4", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 13, + "command": "make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=1 Y=4", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 16, + "command": "make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=2 Y=2", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 17, + "command": "make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=2 Y=2", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 20, + "command": "make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=2 Y=4", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 21, + "command": "make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=2 Y=4", + "testcase": "fa_2d_unroll", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMAX", + "TMUL", + "TMULS", + "TRECIP", + "TROWEXPANDMUL", + "TROWEXPANDSUB", + "TROWMAX", + "TROWSUM", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TMULS": [ + "TMULS" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TROWEXPANDSUB": [ + "TROWEXPANDSUB" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TROWEXPANDMUL": [ + "TROWEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "fa", + "manifest": "benchmark/one-level-arch/test/kernel/fa/compile.all", + "manifest_line": 26, + "command": "make TESTCASE=fa_HIF4_HIF4 MODE=BF16_NOGATHER Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=1", + "testcase": "fa_HIF4_HIF4", + "source": "benchmark/one-level-arch/test/kernel/fa/src/fa_hif4.cpp", + "page": "benchmarks/catalog/one-level/fa/fa-hif4-86b1f297.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCOLEXPANDMUL", + "TCOLEXPANDSUB", + "TCOLMAX", + "TCOLSUM", + "TCVT", + "TEXP", + "TEXPANDS", + "TLOAD", + "TMATMUL_MX", + "TMAX", + "TMUL", + "TMULS", + "TQUANT", + "TRECIP", + "TSTORE", + "TSUB" + ], + "intrinsic_forms": { + "TMULS": [ + "TMULS" + ], + "TCOLMAX": [ + "TCOLMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSUB": [ + "TSUB" + ], + "TEXP": [ + "TEXP" + ], + "TMUL": [ + "TMUL" + ], + "TCOLEXPANDSUB": [ + "TCOLEXPANDSUB" + ], + "TCOLSUM": [ + "TCOLSUM" + ], + "TADD": [ + "TADD" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TCOLEXPANDMUL": [ + "TCOLEXPANDMUL" + ], + "TRECIP": [ + "TRECIP" + ], + "TQUANT": [ + "TQUANT" + ], + "TLOAD": [ + "TLOAD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "gather", + "manifest": "benchmark/one-level-arch/test/kernel/gather/compile.all", + "manifest_line": 26, + "command": "make TESTCASE=gather COMPILER_DIR=$COMPILER_DIR DType=__fp32 OType=uint32_t gKs=131072 gMs=32 gNs=256 tMs=32 tNs=64", + "testcase": "gather", + "source": "benchmark/one-level-arch/test/kernel/gather/src/gather.cpp", + "page": "benchmarks/catalog/one-level/gather/gather-4948597c.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TLOAD", + "TSTORE" + ], + "intrinsic_forms": { + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 26, + "command": "make TESTCASE=matmul TYPE=HIF4_HIF4 VER=MX_NOGATHER M=256 N=2048 K=2048 tM=64 tN=64 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp", + "page": "benchmarks/catalog/one-level/matmul/hif4-hif4-f0fd84bb.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 27, + "command": "make TESTCASE=matmul TYPE=HIF4_HIF4 VER=MX_NOGATHER M=512 N=1280 K=4096 tM=64 tN=64 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp", + "page": "benchmarks/catalog/one-level/matmul/hif4-hif4-f0fd84bb.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 32, + "command": "make TESTCASE=matmul TYPE=HIF4_HIF4 VER=MX_NOGATHER_REUSEA M=256 N=2048 K=2048 tM=64 tN=64 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp", + "page": "benchmarks/catalog/one-level/matmul/hif4-hif4-f0fd84bb.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 33, + "command": "make TESTCASE=matmul TYPE=HIF4_HIF4 VER=MX_NOGATHER_REUSEA M=512 N=1280 K=4096 tM=64 tN=64 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp", + "page": "benchmarks/catalog/one-level/matmul/hif4-hif4-f0fd84bb.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 38, + "command": "make TESTCASE=matmul TYPE=A16W4 M=256 N=2048 K=2048 tM=16 tN=16 tK=128", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/A16W4.cpp", + "page": "benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 39, + "command": "make TESTCASE=matmul TYPE=A16W4 M=512 N=1280 K=2048 tM=16 tN=16 tK=128", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/A16W4.cpp", + "page": "benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 40, + "command": "make TESTCASE=matmul TYPE=A16W4 M=512 N=512 K=4096 tM=16 tN=16 tK=128", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/A16W4.cpp", + "page": "benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCOLEXPAND", + "TCVT", + "TEXPANDS", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMUL", + "TSTORE" + ], + "intrinsic_forms": { + "TSTORE": [ + "TSTORE", + "TSTORE_ACC" + ], + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TLOAD": [ + "TLOAD" + ], + "MGATHER": [ + "MGATHER" + ], + "TMATMUL": [ + "MATMUL", + "TMATMUL" + ], + "TMATMUL_ACC": [ + "MATMACC" + ], + "TCOLEXPAND": [ + "TCOLEXPAND" + ], + "TMUL": [ + "TMUL" + ], + "TADD": [ + "TADD" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ] + }, + "helpers": [ + "MATMACCMX", + "MATMULMX" + ] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 46, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32 M=256 N=256 K=256 tM=32 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 47, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32_REUSEA M=256 N=256 K=256 tM=32 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 48, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32_REUSEB M=256 N=256 K=256 tM=32 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 53, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16 M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 54, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16_REUSEA M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 55, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16_REUSEB M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 58, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8 M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 59, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8_REUSEA M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "matmul", + "manifest": "benchmark/one-level-arch/test/kernel/matmul/compile.all", + "manifest_line": 60, + "command": "make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8_REUSEB M=256 N=256 K=256 tM=64 tN=32 tK=64", + "testcase": "matmul", + "source": "benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp", + "page": "benchmarks/catalog/one-level/matmul/matmul-a008ac76.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCVT", + "TLOAD", + "TMATMUL", + "TMATMUL_ACC", + "TMATMUL_MX", + "TSTORE" + ], + "intrinsic_forms": { + "TCVT": [ + "ACCCVT", + "TCVT" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TMATMUL": [ + "TMATMUL" + ], + "TMATMUL_ACC": [ + "TMATMUL_ACC" + ], + "TADD": [ + "TADD" + ], + "TMATMUL_MX": [ + "TMATMUL_MX" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "reduction/reducemax_col", + "manifest": "benchmark/one-level-arch/test/kernel/reduction/reducemax_col/compile.all", + "manifest_line": 7, + "command": "make TESTCASE=reducemax_col COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=32 tN_s=64 gM_s=2048 gN_s=64", + "testcase": "reducemax_col", + "source": "benchmark/one-level-arch/test/kernel/reduction/reducemax_col/src/reducemax_col.cpp", + "page": "benchmarks/catalog/one-level/reduction-reducemax-col/reducemax-col-1c7de9cb.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TCOLMAX", + "TEXPANDS", + "TLOAD", + "TMAX", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TLOAD": [ + "TLOAD" + ], + "TCOLMAX": [ + "TCOLMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "reduction/reducemax_row", + "manifest": "benchmark/one-level-arch/test/kernel/reduction/reducemax_row/compile.all", + "manifest_line": 7, + "command": "make TESTCASE=reducemax_row COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=16 tN_s=128 gM_s=16 gN_s=8192", + "testcase": "reducemax_row", + "source": "benchmark/one-level-arch/test/kernel/reduction/reducemax_row/src/reducemax_row.cpp", + "page": "benchmarks/catalog/one-level/reduction-reducemax-row/reducemax-row-6e95758c.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TEXPANDS", + "TLOAD", + "TMAX", + "TROWMAX", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TLOAD": [ + "TLOAD" + ], + "TROWMAX": [ + "TROWMAX" + ], + "TMAX": [ + "TMAX" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "reduction/reducesum_col", + "manifest": "benchmark/one-level-arch/test/kernel/reduction/reducesum_col/compile.all", + "manifest_line": 7, + "command": "make TESTCASE=reducesum_col COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=32 tN_s=64 gM_s=2048 gN_s=64", + "testcase": "reducesum_col", + "source": "benchmark/one-level-arch/test/kernel/reduction/reducesum_col/src/reducesum_col.cpp", + "page": "benchmarks/catalog/one-level/reduction-reducesum-col/reducesum-col-d8b6df3d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCOLSUM", + "TEXPANDS", + "TLOAD", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TLOAD": [ + "TLOAD" + ], + "TCOLSUM": [ + "TCOLSUM" + ], + "TADD": [ + "TADD" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "reduction/reducesum_col", + "manifest": "benchmark/one-level-arch/test/kernel/reduction/reducesum_col/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=reducesum_col COMPILER_DIR=$COMPILER_DIR DType=__half tM_s=32 tN_s=64 gM_s=2048 gN_s=64", + "testcase": "reducesum_col", + "source": "benchmark/one-level-arch/test/kernel/reduction/reducesum_col/src/reducesum_col.cpp", + "page": "benchmarks/catalog/one-level/reduction-reducesum-col/reducesum-col-d8b6df3d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TCOLSUM", + "TEXPANDS", + "TLOAD", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TLOAD": [ + "TLOAD" + ], + "TCOLSUM": [ + "TCOLSUM" + ], + "TADD": [ + "TADD" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "reduction/reducesum_row", + "manifest": "benchmark/one-level-arch/test/kernel/reduction/reducesum_row/compile.all", + "manifest_line": 7, + "command": "make TESTCASE=reducesum_row COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=16 tN_s=128 gM_s=16 gN_s=8192", + "testcase": "reducesum_row", + "source": "benchmark/one-level-arch/test/kernel/reduction/reducesum_row/src/reducesum_row.cpp", + "page": "benchmarks/catalog/one-level/reduction-reducesum-row/reducesum-row-f79bc562.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TADD", + "TEXPANDS", + "TLOAD", + "TROWSUM", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TLOAD": [ + "TLOAD" + ], + "TROWSUM": [ + "TROWSUM" + ], + "TADD": [ + "TADD" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "sort", + "manifest": "benchmark/one-level-arch/test/kernel/sort/compile.all", + "manifest_line": 2, + "command": "make TESTCASE=topk", + "testcase": "topk", + "source": "benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp", + "page": "benchmarks/catalog/one-level/sort/topk-0a98c8a8.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "TEXPANDS", + "TSTORE" + ], + "intrinsic_forms": { + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TSTORE": [ + "TSTORE" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "transpose", + "manifest": "benchmark/one-level-arch/test/kernel/transpose/compile.all", + "manifest_line": 11, + "command": "make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1476,32 OUT_SHAPE=32,1476 IN_SHAPE_NAME=1476_32 OUT_SHAPE_NAME=32_1476 IN_DIM=2 OUT_DIM=2 TRANSPOSE_DIM1=0 TRANSPOSE_DIM0=1 gIM='1476*32' gOM='32*1476'", + "testcase": "transpose", + "source": "benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp", + "page": "benchmarks/catalog/one-level/transpose/transpose-dece711d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB", + "TTRANS" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TTRANS": [ + "TTRANS" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "transpose", + "manifest": "benchmark/one-level-arch/test/kernel/transpose/compile.all", + "manifest_line": 24, + "command": "make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1,8,4096,3 OUT_SHAPE=1,8,3,4096 IN_SHAPE_NAME=1_8_4096_3 OUT_SHAPE_NAME=1_8_3_4096 IN_DIM=4 OUT_DIM=4 TRANSPOSE_DIM1=2 TRANSPOSE_DIM0=3 gIM='1*8*4096*3' gOM='1*8*3*4096'", + "testcase": "transpose", + "source": "benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp", + "page": "benchmarks/catalog/one-level/transpose/transpose-dece711d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB", + "TTRANS" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TTRANS": [ + "TTRANS" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "transpose", + "manifest": "benchmark/one-level-arch/test/kernel/transpose/compile.all", + "manifest_line": 33, + "command": "make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1,32,8,32 OUT_SHAPE=1,8,32,32 IN_SHAPE_NAME=1_32_8_32 OUT_SHAPE_NAME=1_8_32_32 IN_DIM=4 OUT_DIM=4 TRANSPOSE_DIM1=1 TRANSPOSE_DIM0=2 gIM='1*32*8*32' gOM='1*8*32*32'", + "testcase": "transpose", + "source": "benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp", + "page": "benchmarks/catalog/one-level/transpose/transpose-dece711d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB", + "TTRANS" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TTRANS": [ + "TTRANS" + ] + }, + "helpers": [] + }, + { + "backend": "One-level", + "family": "transpose", + "manifest": "benchmark/one-level-arch/test/kernel/transpose/compile.all", + "manifest_line": 46, + "command": "make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=8,64,4,16,7 OUT_SHAPE=8,16,4,64,7 IN_SHAPE_NAME=1_8_64_4_16_7 OUT_SHAPE_NAME=1_8_16_4_64_7 IN_DIM=5 OUT_DIM=5 TRANSPOSE_DIM1=1 TRANSPOSE_DIM0=3 gIM='1*8*64*4*16*7' gOM='1*8*16*4*64*7'", + "testcase": "transpose", + "source": "benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp", + "page": "benchmarks/catalog/one-level/transpose/transpose-dece711d.md", + "intrinsic_surface_scope": "source-union", + "source_intrinsics": [ + "MGATHER", + "TADD", + "TCI", + "TCVT", + "TDIVS", + "TEXPANDS", + "TLOAD", + "TMULS", + "TSTORE", + "TSUB", + "TTRANS" + ], + "intrinsic_forms": { + "TCI": [ + "TCI" + ], + "TEXPANDS": [ + "TEXPANDSCALAR" + ], + "TCVT": [ + "TCVT" + ], + "TDIVS": [ + "TDIVS" + ], + "TMULS": [ + "TMULS" + ], + "TSUB": [ + "TSUB" + ], + "TADD": [ + "TADD" + ], + "MGATHER": [ + "MGATHER" + ], + "TSTORE": [ + "TSTORE" + ], + "TLOAD": [ + "TLOAD" + ], + "TTRANS": [ + "TTRANS" + ] + }, + "helpers": [] + } + ] +} diff --git a/docs/benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md b/docs/benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md new file mode 100644 index 0000000..8e60b53 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/broadcast/broadcast-9f406253.md @@ -0,0 +1,360 @@ +# broadcast: broadcast.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `broadcast` | +| Implementation source | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp` | +| Active build variants | 3 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="broadcast.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "broadcast/broadcast_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 512 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 3 +#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 64,1,49 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 64,8,49 +#endif + +#ifndef IN_DIMs +#define IN_DIMs 3 +#endif + +#ifndef OUT_DIMs +#define OUT_DIMs 3 +#endif + +#ifndef gIMs +#define gIMs 64*1*49 //4 014 080 +#endif + +#ifndef gOMs +#define gOMs 64*8*49 //8,028,160 +#endif + + +// 【静态随机偏移】手动写死，模拟内存地址不是数组起始位置 +#ifndef OFFSET_INs +#define OFFSET_INs 11 // 输入静态偏移 +#endif +#ifndef OFFSET_OUTs +#define OFFSET_OUTs 17 // 输出静态偏移 +#endif +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; + + // ========================== + // 申明空间，留出静态偏移空间 + // ========================== + dtype input_buf[gIMs + OFFSET_INs]; // 前面留空，模拟随机地址 + dtype output_buf[gOMs + OFFSET_OUTs]; + + // ========================== + // 【静态随机地址】 + // ========================== + dtype* input = input_buf + OFFSET_INs; + dtype* output = output_buf + OFFSET_OUTs; + + // ========================== + // 从 bin 文件读取输入数据 + // ========================== + // printf("---cpp: start to read input data---"); + // printf("---cpp: start to read input data---"); + // printf("in_shape[0]=%d\n",in_shape[0]); + // printf("in_shape[1]=%d\n",in_shape[1]); + // printf("in_shape[2]=%d\n",in_shape[2]); + // printf("in_shape[3]=%d\n",in_shape[3]); + + #ifdef RES_CHECK + #define INPUT_PATH CHK_DIR "/input.bin" + #define OUTPUT_PATH CHK_DIR "/output.bin" + // #define INPUT_PATH "./input.bin" + // #define OUTPUT_PATH "./output.bin" + readBinaryFile(INPUT_PATH, (uint8_t*)input, gIMs * sizeof(dtype)); + printf("input[0]=%d\n",input[0]); + printf("input[1]=%d\n",input[1]); + printf("input[2]=%d\n",input[2]); + printf("input[3]=%d\n",input[3]); + #endif + + broadcast(input, output, in_shape, out_shape); + + #ifdef RES_CHECK + writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gOMs * sizeof(dtype)); + #endif + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp" + + ```cpp title="broadcast_pto.hpp" linenums="1" + // ============================================================================ + // Broadcast 算子 — PTO 一层编程模型 + // + // 以 Tile_ISA_Reference 文档为准，使用 PTO ISA 定义的 C++ Intrinsic 接口。 + // 当前编译器 (Linx-TileOP-API) 尚未完全实现 PTO ISA，本文件按 ISA 文档书写， + // 暂不保证可编译通过。 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TCI │ API 有，二层实现 │ pto_tileop.hpp 有声明； │ + // │ │ │ jcore/TCI.hpp 用 __vec__ + blkv_get_* │ + // │ │ │ 实现，非真正一层 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TEXPANDS │ API 有(名不同)， │ Pto ISA 名 TEXPANDS； │ + // │ │ 二层实现 │ 当前编译器名 TEXPANDSCALAR； │ + // │ │ │ jcore 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TREMS │ 完全缺失 │ pto_tileop.hpp 中无此 API； │ + // │ │ │ 当前编译器无任何标量取余实现 │ + // │ │ │ (仅有 tile-tile 的 TREM，且仅 int32) │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TDIVS │ API 有，二层实现 │ jcore/TDivs.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TMULS │ API 有，二层实现 │ jcore/TMuls.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TADD │ API 有，二层实现 │ jcore/TAdd.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ MGATHER │ 已支持(一层) │ template_asm.h 用 asm volatile │ + // │ │ │ (BSTART.MGATHER) 实现； │ + // │ │ │ 但缺少 Coalesce::Elem 模板参数支持 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ Pto ISA 名 TSTORE； │ + // │ │ 二层实现 │ 当前编译器名 TSTORE； │ + // │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include // Tile, GlobalTensor 等类型 (当前编译器已有) + #include // global_iterator 工具类型 (当前编译器已有) + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ============================================================================ + // 维度规则：从后面对齐，前面自动补 1，维度=1 可广播 + // ============================================================================ + + // ---------------------------------------------------------------------------- + // gen_offset_pto: 用 PTO ISA Tile 指令计算广播偏移 (一层编程，无 __vec__ 块) + // + // 算法 (stride 累加法，从最内维到最外维): + // 1. TCI 生成索引序列 base, base+1, ..., base+N-1 + // 2. TEXPANDS 将输出偏移 tile 初始化为 0 + // 3. 对每个输出维 d (从 OUT_DIM-1 到 0): + // TREMS coord = idx % out_shape[d] (标量取余) + // TDIVS idx = idx / out_shape[d] (标量整除) + // 若 d 对应输入维 i = d-(OUT_DIM-IN_DIM) >= 0: + // 非广播维 (in_shape[i]!=1): + // TMULS tmp = coord * stride + // TADD out += tmp + // stride *= in_shape[i] + // + // 注意: PTO ISA MGATHER 使用元素索引 (非字节偏移), + // 所以这里不再乘 sizeof(dtype)。 + // 原始 broadcast.hpp 的 __vec__ 版本需要乘 sizeof(dtype) 是因为 + // template_asm.h 里的旧 MGATHER 按字节偏移取数。 + // ---------------------------------------------------------------------------- + template + void gen_offset_pto( + tile_shapeOffset &out, + const size_t *in_shape, + const size_t *out_shape, + const size_t base, + const size_t total_elements) + { + static_assert(tile_shapeOffset::ValidRow != -1 && tile_shapeOffset::ValidCol != -1, + "Only static shape supported"); + + using off_t = typename tile_shapeOffset::DType; // uint32_t + + tile_shapeOffset idxTile; // 当前正在分解的线性索引 + tile_shapeOffset coordTile; // 当前输出维坐标 + tile_shapeOffset tmpTile; // TREMS 需要 (A2A3 要求 tmp >= 1 行, 列数 >= dst) + + // ---- Step 1: TCI 生成索引序列 [base, base+1, ..., base+N-1] ---- + // [当前编译器] TCI 在 pto_tileop.hpp 有声明，但 jcore 实现为 __vec__ (二层) + TCI(idxTile, (off_t)base); + + // ---- Step 2: TEXPANDS 初始化 out = 0 ---- + // [当前编译器] PTO ISA 名 TEXPANDS，当前编译器名 TEXPANDSCALAR；jcore 为 __vec__ + TEXPANDSCALAR(out, (off_t)0); + + size_t stride = 1; + + // ---- Step 3: 从最内维到最外维逐维 divmod + stride 累加 ---- + #pragma clang loop unroll(full) + for (int d = (int)OUT_DIM - 1; d >= 0; d--) { + off_t out_d = (off_t)out_shape[d]; + + // TREMS: coord = idx % out_d + // [当前编译器] 完全缺失！pto_tileop.hpp 无 TREMS API。 + // 退路: 用 TDIVS+TMULS+TSUB 三条拼出取余。 + TREMS(coordTile, idxTile, out_d); + + // TDIVS: idx = idx / out_d (推进到下一维) + // [当前编译器] API 有，jcore 为 __vec__ + TDIVS(idxTile, idxTile, out_d); + + // 该输出维是否对应输入维 + int i = d - (int)(OUT_DIM - IN_DIM); + if (i >= 0) { + if (in_shape[i] != 1) { // 非广播维才累加 + // TMULS: tmp = coord * stride + // [当前编译器] API 有，jcore 为 __vec__ + TMULS(tmpTile, coordTile, (off_t)stride); + // TADD: out += tmp + // [当前编译器] API 有，jcore 为 __vec__ + TADD(out, out, tmpTile); + } + stride *= in_shape[i]; // stride 更新 (广播维 ==1 不变) + } + } + + // 不再乘 sizeof(dtype): + // PTO ISA MGATHER 按"元素索引"取数 (dst[i,j] = src[idx[i,j]])， + // 而非旧 MGATHER 的字节偏移。 + } + + + // ---------------------------------------------------------------------------- + // broadcast: 接口与原 broadcast.hpp 一致 + // ---------------------------------------------------------------------------- + template + void broadcast( + dtype *in_ptr, + dtype *out_ptr, + const size_t *in_shape, + const size_t *out_shape + ) { + const size_t Mb = gOM / tM; + const size_t rmd_M = gOM % tM; + + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeOffset = Tile; + using tile_shapeData_rmd = Tile; + using tile_shapeOffset_rmd= Tile; + + gm_shapeIn inGm(in_ptr); + tile_shapeData outTile; + tile_shapeOffset offsetTile; + tile_shapeData_rmd outTile_rmd; + tile_shapeOffset_rmd offsetTile_rmd; + size_t base = 0; + + using itOut = global_iterator; + itOut gOIter(out_ptr); + + size_t total_elements = tM; + for (int i = 0; i < Mb; ++i) { + auto gO = gOIter(0, i); + + // 计算偏移 tile (元素索引) + gen_offset_pto( + offsetTile, in_shape, out_shape, base, total_elements); + base += total_elements; + + // MGATHER: 按 offsetTile 中的元素索引从 inGm 取数 + // [当前编译器] template_asm.h 的 MGATHER 已用 asm volatile (一层)， + // 但不支持 Coalesce::Elem 模板参数； + // 且旧实现按字节偏移取数，与 PTO ISA 的元素索引语义不同。 + MGATHER(outTile, inGm, offsetTile); + + // TSTORE: 将 outTile 写回 global memory + // [当前编译器] PTO ISA 名 TSTORE，当前编译器名 TSTORE；jcore 为 __vec__ + TSTORE(gO, outTile); + } + if constexpr (rmd_M) { + auto gO = gOIter(0, Mb); + total_elements = rmd_M; + gen_offset_pto( + offsetTile_rmd, in_shape, out_shape, base, total_elements); + base += total_elements; + MGATHER(outTile_rmd, inGm, offsetTile_rmd); + TSTORE(gO, outTile_rmd); + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MGATHER`](../../../../intrinsics/mgather.md) | `MGATHER` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TCI`](../../../../intrinsics/tci.md) | `TCI` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TDIVS`](../../../../intrinsics/tdivs.md) | `TDIVS` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TREMS`](../../../../intrinsics/trems.md) | `TREMS` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/broadcast` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 70 | `make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=2 IN_SHAPEs=1042,1 OUT_SHAPEs=1042,129 IN_DIMs=2 OUT_DIMs=2 gIMs=1042*1 gOMs=1042*129 TESTNAME=broadcast_to_07 IN_SHAPE_NAME=1042_1 OUT_SHAPE_NAME=1042_129 res_check=off diss` | +| 77 | `make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=3 IN_SHAPEs=1280,1,49 OUT_SHAPEs=1280,8,49 IN_DIMs=3 OUT_DIMs=3 gIMs=1280*1*49 gOMs=1280*8*49 TESTNAME=broadcast_to_ABA_019_new IN_SHAPE_NAME=1280_1_49 OUT_SHAPE_NAME=1280_8_49 res_check=off diss` | +| 84 | `make TESTCASE=broadcast DType=__half tMs=2048 MAX_DIMs=5 IN_SHAPEs=1,1,1,65,128 OUT_SHAPEs=1,1,7,65,128 IN_DIMs=5 OUT_DIMs=5 gIMs=1*1*1*65*128 gOMs=1*1*7*65*128 TESTNAME=BroadcastTo_ND_NCDHW_float16_int64_HunyuanImage21_MLLM_1015_000004 IN_SHAPE_NAME=1_1_1_65_128 OUT_SHAPE_NAME=1_1_7_65_128 res_check=off diss` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/broadcast/broadcast_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp` diff --git a/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-019-8713b1a8.md b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-019-8713b1a8.md new file mode 100644 index 0000000..78d2861 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-019-8713b1a8.md @@ -0,0 +1,267 @@ +# broadcast_vec_019: broadcast_vec_019.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `broadcast` | +| Implementation source | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_019.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then transforms tile layout or selects tile elements, then commits the result to global memory. + +```cpp title="broadcast_vec_019.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "broadcast/broadcast_vec_019_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 16 +#endif + +#ifndef kInners +#define kInners 49 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 3 +#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 1280,1,49 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 1280,8,49 +#endif + +#ifndef IN_DIMs +#define IN_DIMs 3 +#endif + +#ifndef OUT_DIMs +#define OUT_DIMs 3 +#endif + +#ifndef gIMs +#define gIMs 1280*1*49 +#endif + +#ifndef gOMs +#define gOMs 1280*8*49 +#endif + +#ifndef OFFSET_INs +#define OFFSET_INs 11 +#endif +#ifndef OFFSET_OUTs +#define OFFSET_OUTs 17 +#endif + +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; + + dtype input_buf[gIMs + OFFSET_INs]; + dtype output_buf[gOMs + OFFSET_OUTs]; + + dtype* input = input_buf + OFFSET_INs; + dtype* output = output_buf + OFFSET_OUTs; + + broadcast( + input, output, in_shape, out_shape); + + #ifdef RES_CHECK + #define INPUT_PATH CHK_DIR "/input.bin" + #define OUTPUT_PATH CHK_DIR "/output.bin" + readBinaryFile(INPUT_PATH, (uint8_t*)input, gIMs * sizeof(dtype)); + writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gOMs * sizeof(dtype)); + #endif +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp" + + ```cpp title="broadcast_vec_019_pto.hpp" linenums="1" + // ============================================================================ + // Broadcast (B,1,K) -> (B,N,K) — PTO 一层编程模型 + // + // 原始 broadcast_vec_019.hpp 策略: + // TLOAD (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TSTORE (kTileBatch,N*K) + // __vec__ 块: dst[batch*RowStride + copy*K + x] = src[batch*RowStride + x] + // + // PTO 一层策略: + // TLOAD (kTileBatch,K) -> TINSERT × N (将输入 tile 插入输出 tile 的 N 个列偏移) -> TSTORE (kTileBatch,N*K) + // TINSERT 语义: dst[indexRow+i, indexCol+j] = src[i,j] + // 对 copy c = 0..N-1: TINSERT(outTile, inTile, 0, c*K) + // N 次插入互不重叠, 合起来恰好填满输出 tile 的 valid 区域 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ + // │ │ 二层实现 │ 当前编译器名 TLOAD； │ + // │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TINSERT │ 完全缺失 │ pto_tileop.hpp 中无此 API； │ + // │ │ │ 当前编译器无 TINSERT 实现 │ + // │ │ │ (仅有反向操作 TEXTRACT) │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ + // │ │ 二层实现 │ 当前编译器名 TSTORE； │ + // │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include + #include + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ===================================================================== + // Broadcast (B,1,K) -> (B,N,K) via TLOAD + TINSERT×N + TSTORE + // + // Data layout (row-major): + // Input: [B][1][K] -> flat = B*K, batch b data at offset b*K + // Output: [B][N][K] -> flat = B*N*K, batch b data at offset b*N*K + // Broadcast along dim1: 1 -> N, dim0 & dim2 preserved. + // + // Template params (与原 broadcast_vec_019 一致): + // dtype - data type (__half, float, etc.) + // MAX_DIM - max dimensions (kept for compat) + // IN_DIM - input dim count (kept for compat) + // OUT_DIM - output dim count (kept for compat) + // gIM - total input elements = B * K + // gOM - total output elements = B * N * K + // kTileBatch - batches per tile, power-of-2 + // kInner - inner dimension K (e.g. 49, need not be power-of-2) + // ===================================================================== + + template + void broadcast(dtype *in_ptr, dtype *out_ptr, + const size_t * /*in_shape*/, const size_t * /*out_shape*/) { + constexpr size_t kBCast = gOM / gIM; + constexpr size_t kBatch = gIM / kInner; + constexpr size_t tileCols = 512; + + static_assert(gOM % gIM == 0, + "gOM must be divisible by gIM for (B,1,K)->(B,N,K) broadcast"); + static_assert(gIM % kInner == 0, + "gIM must be divisible by kInner (B = gIM/kInner must be integer)"); + static_assert((kTileBatch & (kTileBatch - 1)) == 0, + "kTileBatch must be power of 2 for 512B tile alignment"); + static_assert((kBCast & (kBCast - 1)) == 0, + "kBCast (N) must be power of 2 for bitwise division in SIMT"); + static_assert(tileCols >= kBCast * kInner, + "padded tileCols (512) must >= broadcast target width (N*K)"); + + using tile_in = Tile; + using tile_out = Tile; + using gm_in = global_tensor>; + using gm_out = global_tensor>; + + constexpr size_t Nb = kBatch / kTileBatch; + constexpr size_t rmd = kBatch % kTileBatch; + + tile_in inTile; + tile_out outTile; + + for (size_t i = 0; i < Nb; i++) { + gm_in gsrc(in_ptr + i * kTileBatch * kInner); + gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); + + // TLOAD: GM -> UB, 加载 (kTileBatch, kInner) 输入 tile + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ + TLOAD(inTile, gsrc); + + // TINSERT × kBCast: 将输入 tile 插入输出 tile 的 N 个列偏移 + // 每次 TINSERT 写入 kInner 列, N 次互不重叠, 合起来填满 N*kInner 列 + // [当前编译器] 完全缺失! pto_tileop.hpp 无 TINSERT API + #pragma clang loop unroll(full) + for (size_t c = 0; c < kBCast; c++) { + TINSERT(outTile, inTile, /*indexRow=*/0, /*indexCol=*/(uint16_t)(c * kInner)); + } + + // TSTORE: UB -> GM, 写回 (kTileBatch, kBCast*kInner) 输出 tile + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ + TSTORE(gdst, outTile); + } + + using tile_in_r = Tile; + using tile_out_r = Tile; + tile_in_r inTile_rmd; + tile_out_r outTile_rmd; + if constexpr (rmd > 0) { + gm_in gsrc(in_ptr + Nb * kTileBatch * kInner); + gm_out gdst(out_ptr + Nb * kTileBatch * kBCast * kInner); + + TLOAD(inTile_rmd, gsrc); + + #pragma clang loop unroll(full) + for (size_t c = 0; c < kBCast; c++) { + TINSERT(outTile_rmd, inTile_rmd, 0, (uint16_t)(c * kInner)); + } + + TSTORE(gdst, outTile_rmd); + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TINSERT`](../../../../intrinsics/tinsert.md) | `TINSERT` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/broadcast` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 52 | `make TESTCASE=broadcast_vec_019 DType=__half tMs=8 kInners=49 MAX_DIMs=3 IN_SHAPEs=1280,1,49 OUT_SHAPEs=1280,8,49 IN_DIMs=3 OUT_DIMs=3 gIMs=1280*1*49 gOMs=1280*8*49 IN_SHAPE_NAME=1280_1_49 OUT_SHAPE_NAME=1280_8_49 diss` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_019_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_019.cpp` diff --git a/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-039-ac6775c3.md b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-039-ac6775c3.md new file mode 100644 index 0000000..558b944 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-039-ac6775c3.md @@ -0,0 +1,309 @@ +# broadcast_vec_039: broadcast_vec_039.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `broadcast` | +| Implementation source | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_039.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then transforms tile layout or selects tile elements, then commits the result to global memory. + +```cpp title="broadcast_vec_039.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "broadcast/broadcast_vec_039_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 16 +#endif + +#ifndef kInners +#define kInners 16 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 3 +#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 8192,1,16 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 8192,8,16 +#endif + +#ifndef IN_DIMs +#define IN_DIMs 3 +#endif + +#ifndef OUT_DIMs +#define OUT_DIMs 3 +#endif + +#ifndef gIMs +#define gIMs 8192*1*16 +#endif + +#ifndef gOMs +#define gOMs 8192*8*16 +#endif + +#ifndef OFFSET_INs +#define OFFSET_INs 11 +#endif +#ifndef OFFSET_OUTs +#define OFFSET_OUTs 17 +#endif + +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; + + dtype input_buf[gIMs + OFFSET_INs]; + dtype output_buf[gOMs + OFFSET_OUTs]; + + dtype* input = input_buf + OFFSET_INs; + dtype* output = output_buf + OFFSET_OUTs; + + // for (size_t i = 0; i < (size_t)gIMs; i++) { + // if constexpr (sizeof(dtype) == 2) { + // ((uint16_t*)input)[i] = (uint16_t)(i % 1000); + // } else { + // input[i] = (dtype)(i % 1000); + // } + // } + + broadcast( + input, output, in_shape, out_shape); + + // size_t pass_count = 0; + // for (size_t b = 0; b < in_shape[0]; b++) { + // for (size_t n = 0; n < out_shape[1]; n++) { + // for (size_t k = 0; k < in_shape[2]; k++) { + // size_t in_idx = b * in_shape[2] + k; + // size_t out_idx = b * out_shape[1] * out_shape[2] + n * out_shape[2] + k; + + // dtype expected; + // if constexpr (sizeof(dtype) == 2) { + // expected = (dtype)(uint16_t)(in_idx % 1000); + // } else { + // expected = (dtype)(in_idx % 1000); + // } + + // dtype actual = output[out_idx]; + // if (actual != expected) { + // if (pass_count < 10) { + // printf("FAIL: [%zu][%zu][%zu] expected=%d, got=%d\n", + // b, n, k, (int)expected, (int)actual); + // } + // pass_count++; + // } + // } + // } + // } + // if (pass_count == 0) { + // printf("PASS: all %zu elements correct\n", (size_t)gOMs); + // } else { + // printf("FAIL: %zu mismatches\n", pass_count); + // } + + #ifdef RES_CHECK + #define INPUT_PATH CHK_DIR "/input.bin" + #define OUTPUT_PATH CHK_DIR "/output.bin" + readBinaryFile(INPUT_PATH, (uint8_t*)input, gIMs * sizeof(dtype)); + writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gOMs * sizeof(dtype)); + #endif + + // return (pass_count == 0) ? 0 : 1; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp" + + ```cpp title="broadcast_vec_039_pto.hpp" linenums="1" + // ============================================================================ + // Broadcast (B,1,K) -> (B,N,K) — PTO 一层编程模型 + // + // 原始 broadcast_vec_039.hpp 策略: + // TLOAD (kTileBatch,K) -> __vec__ 行复制 (K 个元素复制 N 次) -> TSTORE (kTileBatch,N*K) + // __vec__ 块: dst[y*RowStride + x] = src[y*RowStride + (x & (K-1))] + // (K 为 2 的幂, 用位与代替取余) + // + // PTO 一层策略: + // TLOAD (kTileBatch,K) -> TINSERT × N (将输入 tile 插入输出 tile 的 N 个列偏移) -> TSTORE (kTileBatch,N*K) + // TINSERT 语义: dst[indexRow+i, indexCol+j] = src[i,j] + // 对 copy c = 0..N-1: TINSERT(outTile, inTile, 0, c*K) + // N 次插入互不重叠, 合起来恰好填满输出 tile 的 valid 区域 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ + // │ │ 二层实现 │ 当前编译器名 TLOAD； │ + // │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TINSERT │ 完全缺失 │ pto_tileop.hpp 中无此 API； │ + // │ │ │ 当前编译器无 TINSERT 实现 │ + // │ │ │ (仅有反向操作 TEXTRACT) │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ + // │ │ 二层实现 │ 当前编译器名 TSTORE； │ + // │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include + #include + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ===================================================================== + // Broadcast (B,1,K) -> (B,N,K) via TLOAD + TINSERT×N + TSTORE + // + // Data layout (row-major): + // Input: [B][1][K] -> flat = B*K, batch b data at offset b*K + // Output: [B][N][K] -> flat = B*N*K, batch b data at offset b*N*K + // Broadcast along dim1: 1 -> N, dim0 & dim2 preserved. + // + // Template params (与原 broadcast_vec_039 一致): + // dtype - data type (__half, float, etc.) + // MAX_DIM - max dimensions (kept for compat) + // IN_DIM - input dim count (kept for compat) + // OUT_DIM - output dim count (kept for compat) + // gIM - total input elements = B * K (e.g. 8192*16 = 131072) + // gOM - total output elements = B * N * K (e.g. 8192*8*16 = 1048576) + // kTileBatch - batches per tile, power-of-2 (e.g. 1,2,4,8,16,32,64) + // kInner - inner dimension K, power-of-2 (e.g. 16) + // ===================================================================== + + template + void broadcast(dtype *in_ptr, dtype *out_ptr, + const size_t * /*in_shape*/, const size_t * /*out_shape*/) { + constexpr size_t kBCast = gOM / gIM; + constexpr size_t kBatch = gIM / kInner; + constexpr size_t tileCols = 256; + + static_assert(gOM % gIM == 0, + "gOM must be divisible by gIM for (B,1,K)->(B,N,K) broadcast"); + static_assert(gIM % kInner == 0, + "gIM must be divisible by kInner (B = gIM/kInner must be integer)"); + static_assert((kInner & (kInner - 1)) == 0, + "kInner must be power of 2 for bit-op index decomposition"); + static_assert((kTileBatch & (kTileBatch - 1)) == 0, + "kTileBatch must be power of 2 for 512B tile alignment"); + static_assert(tileCols >= kBCast * kInner, + "padded tileCols (256) must >= broadcast target width (N*K)"); + + using tile_in = Tile; + using tile_out = Tile; + using gm_in = global_tensor>; + using gm_out = global_tensor>; + + constexpr size_t Nb = kBatch / kTileBatch; + constexpr size_t rmd = kBatch % kTileBatch; + + tile_in inTile; + tile_out outTile; + + for (size_t i = 0; i < Nb; i++) { + gm_in gsrc(in_ptr + i * kTileBatch * kInner); + gm_out gdst(out_ptr + i * kTileBatch * kBCast * kInner); + + // TLOAD: GM -> UB, 加载 (kTileBatch, kInner) 输入 tile + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ + TLOAD(inTile, gsrc); + + // TINSERT × kBCast: 将输入 tile 插入输出 tile 的 N 个列偏移 + // 每次 TINSERT 写入 kInner 列, N 次互不重叠, 合起来填满 N*kInner 列 + // [当前编译器] 完全缺失! pto_tileop.hpp 无 TINSERT API + #pragma clang loop unroll(full) + for (size_t c = 0; c < kBCast; c++) { + TINSERT(outTile, inTile, /*indexRow=*/0, /*indexCol=*/(uint16_t)(c * kInner)); + } + + // TSTORE: UB -> GM, 写回 (kTileBatch, kBCast*kInner) 输出 tile + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ + TSTORE(gdst, outTile); + } + + using tile_in_r = Tile; + using tile_out_r = Tile; + tile_in_r inTile_rmd; + tile_out_r outTile_rmd; + if constexpr (rmd > 0) { + gm_in gsrc(in_ptr + Nb * kTileBatch * kInner); + gm_out gdst(out_ptr + Nb * kTileBatch * kBCast * kInner); + + TLOAD(inTile_rmd, gsrc); + + #pragma clang loop unroll(full) + for (size_t c = 0; c < kBCast; c++) { + TINSERT(outTile_rmd, inTile_rmd, 0, (uint16_t)(c * kInner)); + } + + TSTORE(gdst, outTile_rmd); + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TINSERT`](../../../../intrinsics/tinsert.md) | `TINSERT` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/broadcast` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 57 | `make TESTCASE=broadcast_vec_039 DType=__half tMs=8 kInners=16 MAX_DIMs=3 IN_SHAPEs=8192,1,16 OUT_SHAPEs=8192,8,16 IN_DIMs=3 OUT_DIMs=3 gIMs=8192*1*16 gOMs=8192*8*16 IN_SHAPE_NAME=8192_1_16 OUT_SHAPE_NAME=8192_8_16 diss` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_039_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_039.cpp` diff --git a/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-07-ba7002f4.md b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-07-ba7002f4.md new file mode 100644 index 0000000..f24e497 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/broadcast/broadcast-vec-07-ba7002f4.md @@ -0,0 +1,277 @@ +# broadcast_vec_07: broadcast_vec_07.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `broadcast` | +| Implementation source | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_07.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then reduces or broadcasts along tile axes, then commits the result to global memory. + +```cpp title="broadcast_vec_07.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "broadcast/broadcast_vec_07_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 32 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 2 +#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 1443,1 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 1443,129 +#endif + +#ifndef IN_DIMs +#define IN_DIMs 2 +#endif + +#ifndef OUT_DIMs +#define OUT_DIMs 2 +#endif + +#ifndef gIMs +#define gIMs 1443*1 +#endif + +#ifndef gOMs +#define gOMs 1443*129 +#endif + +#ifndef OFFSET_INs +#define OFFSET_INs 11 +#endif +#ifndef OFFSET_OUTs +#define OFFSET_OUTs 17 +#endif + +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; + + dtype input_buf[gIMs + OFFSET_INs]; + dtype output_buf[gOMs + OFFSET_OUTs]; + + dtype* input = input_buf + OFFSET_INs; + dtype* output = output_buf + OFFSET_OUTs; + + // for (size_t i = 0; i < gIMs; i++) { + // if constexpr (sizeof(dtype) == 2) { + // ((uint16_t*)input)[i] = (uint16_t)(i % 1000); + // } else { + // input[i] = (dtype)(i % 1000); + // } + // } + + broadcast( + input, output, in_shape, out_shape); + + // size_t pass_count = 0; + // for (size_t r = 0; r < in_shape[0]; r++) { + // dtype expected; + // if constexpr (sizeof(dtype) == 2) { + // expected = (dtype)(uint16_t)(r % 1000); + // } else { + // expected = (dtype)(r % 1000); + // } + // for (size_t c = 0; c < out_shape[1]; c++) { + // dtype actual = output[r * out_shape[1] + c]; + // if (actual != expected) { + // if (pass_count < 10) { + // printf("FAIL: [%zu][%zu] expected=%d, got=%d\n", + // r, c, (int)expected, (int)actual); + // } + // pass_count++; + // } + // } + // } + // if (pass_count == 0) { + // printf("PASS: all %zu elements correct\n", (size_t)gOMs); + // } else { + // printf("FAIL: %zu mismatches\n", pass_count); + // } + + #ifdef RES_CHECK + #define INPUT_PATH CHK_DIR "/input.bin" + #define OUTPUT_PATH CHK_DIR "/output.bin" + readBinaryFile(INPUT_PATH, (uint8_t*)input, gIMs * sizeof(dtype)); + writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gOMs * sizeof(dtype)); + #endif + + // return (pass_count == 0) ? 0 : 1; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp" + + ```cpp title="broadcast_vec_07_pto.hpp" linenums="1" + // ============================================================================ + // Broadcast (N,1) -> (N,C) — PTO 一层编程模型 + // + // 原始 broadcast_vec_07.hpp 策略: + // TLOAD (kTileRows,1) -> __vec__ 行广播 -> TSTORE (kTileRows,C) + // __vec__ 块: dst[x + y*RowStride] = src[y*RowStride] (col 0 -> all cols) + // + // PTO 一层策略: + // TLOAD (kTileRows,1) -> TROWEXPAND (col 0 广播到全部 C 列) -> TSTORE (kTileRows,C) + // TROWEXPAND 语义: dst[i,j] = src[i,0], 恰好是行广播 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD； │ + // │ │ 二层实现 │ 当前编译器名 TLOAD； │ + // │ │ │ jcore/TLoad.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │TROWEXPAND│ API 有(名不同)， │ PTO ISA 名 TROWEXPAND； │ + // │ │ 二层实现 │ 当前编译器名 TEXPANDROW； │ + // │ │ │ jcore/TExpandRow.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE； │ + // │ │ 二层实现 │ 当前编译器名 TSTORE； │ + // │ │ │ jcore/TStore.hpp 用 __vec__ 实现 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include + #include + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ===================================================================== + // Broadcast (N,1) -> (N,C) via TLOAD + TROWEXPAND + TSTORE + // + // Template params (与原 broadcast_vec_07 一致): + // dtype - data type (__half, float, etc.) + // MAX_DIM - max dimensions (unused, kept for compat) + // IN_DIM - input dim count (unused) + // OUT_DIM - output dim count (unused) + // gIM - total input elements = N*1 = N (e.g. 1443) + // gOM - total output elements = N*C (e.g. 1443*129) + // kTileRows - rows per tile, must be power-of-2 (e.g. 1,2,4,..,64) + // ===================================================================== + + template + void broadcast(dtype *in_ptr, dtype *out_ptr, + const size_t * /*in_shape*/, const size_t * /*out_shape*/) { + constexpr size_t kN = gIM; + constexpr size_t kC = gOM / gIM; + constexpr size_t tileCols = 256; + + static_assert(gOM % gIM == 0, + "gOM must be divisible by gIM for (N,1)->(N,C) broadcast"); + static_assert(tileCols >= kC, + "padded tileCols (256) must >= broadcast target columns"); + static_assert((kTileRows & (kTileRows - 1)) == 0, + "kTileRows must be power of 2 for 512B tile alignment"); + + using tile_in = Tile; + using tile_out = Tile; + using gm_in = global_tensor>; + using gm_out = global_tensor>; + + constexpr size_t Nb = kN / kTileRows; + constexpr size_t rmd = kN % kTileRows; + + tile_in inTile; + tile_out outTile; + + for (size_t i = 0; i < Nb; i++) { + gm_in gsrc(in_ptr + i * kTileRows); + gm_out gdst(out_ptr + i * kTileRows * kC); + + // TLOAD: GM -> UB, 加载 (kTileRows, 1) 输入 tile + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ + TLOAD(inTile, gsrc); + + // TROWEXPAND: 将每行 col 0 广播到全部 kC 列 -> (kTileRows, kC) + // [当前编译器] 名为 TEXPANDROW, jcore 为 __vec__ + TROWEXPAND(outTile, inTile); + + // TSTORE: UB -> GM, 写回 (kTileRows, kC) 输出 tile + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ + TSTORE(gdst, outTile); + } + + using tile_in_r = Tile; + using tile_out_r = Tile; + tile_in_r inTile_rmd; + tile_out_r outTile_rmd; + if constexpr (rmd > 0) { + gm_in gsrc(in_ptr + Nb * kTileRows); + gm_out gdst(out_ptr + Nb * kTileRows * kC); + + TLOAD(inTile_rmd, gsrc); + TROWEXPAND(outTile_rmd, inTile_rmd); + TSTORE(gdst, outTile_rmd); + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp` | +| [`TROWEXPAND`](../../../../intrinsics/trowexpand.md) | `TROWEXPAND` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/broadcast` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 47 | `make TESTCASE=broadcast_vec_07 DType=__half tMs=16 MAX_DIMs=2 IN_SHAPEs=1443,1 OUT_SHAPEs=1443,129 IN_DIMs=2 OUT_DIMs=2 gIMs=1443*1 gOMs=1443*129 IN_SHAPE_NAME=1443_1 OUT_SHAPE_NAME=1443_129 diss` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/broadcast/broadcast_vec_07_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_07.cpp` diff --git a/docs/benchmarks/catalog/one-level/broadcast/index.md b/docs/benchmarks/catalog/one-level/broadcast/index.md new file mode 100644 index 0000000..0e376d7 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/broadcast/index.md @@ -0,0 +1,13 @@ +# One-level: broadcast + + + +This family contains **4** source implementations and +**6** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [broadcast](broadcast-9f406253.md) | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast.cpp` | 3 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCI`](../../../../intrinsics/tci.md), [`TDIVS`](../../../../intrinsics/tdivs.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TREMS`](../../../../intrinsics/trems.md), [`TSTORE`](../../../../intrinsics/tstore.md) | +| [broadcast_vec_019](broadcast-vec-019-8713b1a8.md) | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_019.cpp` | 1 | [`TINSERT`](../../../../intrinsics/tinsert.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TSTORE`](../../../../intrinsics/tstore.md) | +| [broadcast_vec_039](broadcast-vec-039-ac6775c3.md) | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_039.cpp` | 1 | [`TINSERT`](../../../../intrinsics/tinsert.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TSTORE`](../../../../intrinsics/tstore.md) | +| [broadcast_vec_07](broadcast-vec-07-ba7002f4.md) | `benchmark/one-level-arch/test/kernel/broadcast/src/broadcast_vec_07.cpp` | 1 | [`TLOAD`](../../../../intrinsics/tload.md), [`TROWEXPAND`](../../../../intrinsics/trowexpand.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/concat/concat-gather-1c6990b2.md b/docs/benchmarks/catalog/one-level/concat/concat-gather-1c6990b2.md new file mode 100644 index 0000000..da022ab --- /dev/null +++ b/docs/benchmarks/catalog/one-level/concat/concat-gather-1c6990b2.md @@ -0,0 +1,350 @@ +# concat_gather: concat_gather.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `concat` | +| Implementation source | `benchmark/one-level-arch/test/kernel/concat/src/concat_gather.cpp` | +| Active build variants | 2 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="concat_gather.cpp" linenums="1" +#include +//#include "template_asm.h" + +#include +#include + +#include "fileop.h" +#include "concat/concat_gather_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 1024 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 8 +#endif + +//#ifndef NUMs +//#define NUMs 1000 +//#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 64,2 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 64,2000 +#endif +//#define IN_SHAPE {64, 8, 64, 4, 128} +//#define OUT_SHAPE {64, 8, 128, 4, 64} + +//#define IN_DIM 3 +//#define OUT_DIM 3 +//#define IN_DIM 4 +//#define OUT_DIM 4 + +#ifndef DATA_DIMs +#define DATA_DIMs 2 +#endif + +#ifndef CONCAT_DIMs +#define CONCAT_DIMs 1 +#endif +//#define IN_DIM 5 +//#define OUT_DIM 5 + + +#ifndef gIMs +#define gIMs (1000*64*2) +#endif + +#ifndef gOMs +#define gOMs (64*2000) +#endif +//#define gIM (64 * 8 * 64 * 4 * 128) +//#define gOM (64 * 8 * 128 * 4 * 64) + + + +// ============================================================================ +// main +// ============================================================================ + + + + +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; +// size_t in_dim = IN_DIMs; +// size_t out_dim = OUT_DIMs; +// size_t transpose_dim1 = TRANSPOSE_DIM1; +// size_t transpose_dim0 = TRANSPOSE_DIM0; + + dtype input_buf[gIMs]; + dtype output_buf[gOMs]; + + + dtype* input = input_buf; + dtype* output = output_buf; + + + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/concat_gather/src/data_1000_64_2.bin", (uint8_t*)input, gIMs * sizeof(dtype)); + concat_gather(input, output, in_shape, out_shape); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/concat_gather/src/result.bin", (uint8_t*)output, gOMs * sizeof(dtype)); + + + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp" + + ```cpp title="concat_gather_pto.hpp" linenums="1" + #ifndef CONCAT_GATHER_KERNEL_HPP + #define CONCAT_GATHER_KERNEL_HPP + + #include + #include + #include + + using namespace pto; + + /** + * @brief 通用 N 维 concat gather（Tile ISA 实现） + * + * 算法：对输出张量的每个元素，计算其多维坐标，确定来自哪个输入张量（n） + * 以及在输入张量内的偏移，用 MGATHER 从输入中 gather 数据。 + * + * @tparam DType 数据类型 + * @tparam MAX_DIM 最大维度数 + * @tparam gIM 输入总元素数 + * @tparam gOM 输出总元素数 + * @tparam tM 每个 tile 处理的元素数 + * @tparam DATA_DIM 数据维度数（编译期常量） + * @tparam CONCAT_DIM 拼接维度索引（编译期常量） + */ + template + void concat_gather( + DType *in_ptr, + DType *out_ptr, + size_t *in_shape, + size_t *out_shape + ) + { + constexpr int kFullTiles = gOM / tM; + constexpr int kTailElements = gOM % tM; + + using InputGlobal = pto::global_tensor>; + using OutputGlobal = pto::global_tensor>; + using DataTile = pto::Tile; + using OffsetTile = pto::Tile; + using OutputIterator = global_iterator; + + InputGlobal input_global(in_ptr); + OutputIterator output_iter(out_ptr); + + uint32_t output_base = 0; + for (int tile_index = 0; tile_index < kFullTiles; ++tile_index) { + auto output_global = output_iter(0, tile_index); + DataTile output_tile; + OffsetTile offset_tile; + + OffsetTile linear_index; + OffsetTile quotient; + OffsetTile cycle; + OffsetTile cycle_base; + OffsetTile coordinate; + OffsetTile contribution; + + TCI(linear_index, output_base); + TEXPANDSCALAR(offset_tile, static_cast(0)); + + // 计算输入 stride（标量核心，运行时计算） + uint32_t input_stride[DATA_DIM]; + input_stride[DATA_DIM - 1] = 1; + for (int i = static_cast(DATA_DIM) - 2; i >= 0; --i) { + input_stride[i] = input_stride[i + 1] * static_cast(in_shape[i + 1]); + } + uint32_t input_size = input_stride[0] * static_cast(in_shape[0]); + + // 对每个维度提取坐标并计算 byte offset + for (int d = static_cast(DATA_DIM) - 1; d >= 0; --d) { + uint32_t output_stride = 1; + for (int dd = d + 1; dd < static_cast(DATA_DIM); ++dd) { + output_stride *= static_cast(out_shape[dd]); + } + + if (output_stride == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, output_stride); + } + + uint32_t out_dim_size = static_cast(out_shape[d]); + TDIVS(cycle, quotient, out_dim_size); + TMULS(cycle_base, cycle, out_dim_size); + TSUB(coordinate, quotient, cycle_base); + + if (static_cast(d) == CONCAT_DIM) { + uint32_t in_dim_size = static_cast(in_shape[d]); + OffsetTile n_tile; + OffsetTile offset_in; + OffsetTile n_times_shape; + + TDIVS(n_tile, coordinate, in_dim_size); + TMULS(n_times_shape, n_tile, in_dim_size); + TSUB(offset_in, coordinate, n_times_shape); + + TMULS(contribution, offset_in, + input_stride[d] * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + + TMULS(contribution, n_tile, + input_size * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } else { + TMULS(contribution, coordinate, + input_stride[d] * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } + } + + MGATHER(output_tile, input_global, offset_tile); + TSTORE(output_global, output_tile); + + output_base += tM; + } + + if constexpr (kTailElements != 0) { + using TailDataTile = pto::Tile; + using TailOffsetTile = pto::Tile; + + auto output_global = output_iter(0, kFullTiles); + TailDataTile output_tile; + TailOffsetTile offset_tile; + + TailOffsetTile linear_index; + TailOffsetTile quotient; + TailOffsetTile cycle; + TailOffsetTile cycle_base; + TailOffsetTile coordinate; + TailOffsetTile contribution; + + TCI(linear_index, output_base); + TEXPANDSCALAR(offset_tile, static_cast(0)); + + uint32_t input_stride[DATA_DIM]; + input_stride[DATA_DIM - 1] = 1; + for (int i = static_cast(DATA_DIM) - 2; i >= 0; --i) { + input_stride[i] = input_stride[i + 1] * static_cast(in_shape[i + 1]); + } + uint32_t input_size = input_stride[0] * static_cast(in_shape[0]); + + for (int d = static_cast(DATA_DIM) - 1; d >= 0; --d) { + uint32_t output_stride = 1; + for (int dd = d + 1; dd < static_cast(DATA_DIM); ++dd) { + output_stride *= static_cast(out_shape[dd]); + } + + if (output_stride == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, output_stride); + } + + uint32_t out_dim_size = static_cast(out_shape[d]); + TDIVS(cycle, quotient, out_dim_size); + TMULS(cycle_base, cycle, out_dim_size); + TSUB(coordinate, quotient, cycle_base); + + if (static_cast(d) == CONCAT_DIM) { + uint32_t in_dim_size = static_cast(in_shape[d]); + TailOffsetTile n_tile; + TailOffsetTile offset_in; + TailOffsetTile n_times_shape; + + TDIVS(n_tile, coordinate, in_dim_size); + TMULS(n_times_shape, n_tile, in_dim_size); + TSUB(offset_in, coordinate, n_times_shape); + + TMULS(contribution, offset_in, + input_stride[d] * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + + TMULS(contribution, n_tile, + input_size * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } else { + TMULS(contribution, coordinate, + input_stride[d] * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } + } + + MGATHER(output_tile, input_global, offset_tile); + TSTORE(output_global, output_tile); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MGATHER`](../../../../intrinsics/mgather.md) | `MGATHER` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TCI`](../../../../intrinsics/tci.md) | `TCI` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `TCVT` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TDIVS`](../../../../intrinsics/tdivs.md) | `TDIVS` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | +| [`TSUB`](../../../../intrinsics/tsub.md) | `TSUB` | `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/concat` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 11 | `make TESTCASE=concat_gather COMPILER_DIR=$COMPILER_DIR DType=int32_t MAX_DIM=8 tM=512 IN_SHAPE=64,2 OUT_SHAPE=64,2000 IN_SHAPE_NAME=64_2 OUT_SHAPE_NAME=64_2000 DATA_DIM=2 CONCAT_DIM=1 gIM='1000*64*2' gOM='64*2000'` | +| 18 | `make TESTCASE=concat_gather COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=256,8 OUT_SHAPE=256,8000 IN_SHAPE_NAME=256_8 OUT_SHAPE_NAME=256_8000 DATA_DIM=2 CONCAT_DIM=1 gIM='4000*256*8' gOM='256*8000'` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/concat/concat_gather_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/concat/src/concat_gather.cpp` diff --git a/docs/benchmarks/catalog/one-level/concat/concat-scatter-29559c6c.md b/docs/benchmarks/catalog/one-level/concat/concat-scatter-29559c6c.md new file mode 100644 index 0000000..f9dc1d5 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/concat/concat-scatter-29559c6c.md @@ -0,0 +1,314 @@ +# concat_scatter: concat_scatter.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `concat` | +| Implementation source | `benchmark/one-level-arch/test/kernel/concat/src/concat_scatter.cpp` | +| Active build variants | 2 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="concat_scatter.cpp" linenums="1" +#include +//#include "template_asm.h" + +#include +#include + +#include "fileop.h" +#include "concat/concat_scatter_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 1024 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 8 +#endif + +#ifndef NUMs +#define NUMs 1000 +#endif + +#ifndef IN_SHAPEs +#define IN_SHAPEs 64,2 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 64,2000 +#endif +//#define IN_SHAPE {64, 8, 64, 4, 128} +//#define OUT_SHAPE {64, 8, 128, 4, 64} + +//#define IN_DIM 3 +//#define OUT_DIM 3 +//#define IN_DIM 4 +//#define OUT_DIM 4 + +#ifndef DATA_DIMs +#define DATA_DIMs 2 +#endif + +#ifndef CONCAT_DIMs +#define CONCAT_DIMs 1 +#endif +//#define IN_DIM 5 +//#define OUT_DIM 5 + + +#ifndef gIMs +#define gIMs (1000*64*2) +#endif + +#ifndef gOMs +#define gOMs (64*2000) +#endif +//#define gIM (64 * 8 * 64 * 4 * 128) +//#define gOM (64 * 8 * 128 * 4 * 64) + + + +// ============================================================================ +// main +// ============================================================================ + + + + +int main() { + using dtype = DType; + size_t in_shape[MAX_DIMs] = {IN_SHAPEs}; + size_t out_shape[MAX_DIMs] = {OUT_SHAPEs}; +// size_t in_dim = IN_DIMs; +// size_t out_dim = OUT_DIMs; +// size_t transpose_dim1 = TRANSPOSE_DIM1; +// size_t transpose_dim0 = TRANSPOSE_DIM0; + + dtype input_buf[gIMs]; + dtype output_buf[gOMs]; + + + dtype* input = input_buf; + dtype* output = output_buf; + + + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/concat_scatter/src/data_1000_64_2.bin", (uint8_t*)input, gIMs * sizeof(dtype)); + concat_scatter(input, output, in_shape, out_shape); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/concat_scatter/src/result.bin", (uint8_t*)output, gOMs * sizeof(dtype)); + + + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp" + + ```cpp title="concat_scatter_pto.hpp" linenums="1" + #ifndef CONCAT_SCATTER_KERNEL_HPP + #define CONCAT_SCATTER_KERNEL_HPP + + #include + #include + #include + + using namespace pto; + + /** + * @brief 通用 N 维 concat scatter（Tile ISA 实现） + * + * 算法：对输入张量的每个元素，计算其多维坐标，用输出 stride 计算输出 byte offset， + * 用 MSCATTER 将数据写入输出。 + * + * 注意：当前实现中 n（来自哪个输入张量）始终为 0，即输入被放置在输出的起始位置。 + * 这与原始 __vec__ 版本行为一致。 + * + * @tparam DType 数据类型 + * @tparam MAX_DIM 最大维度数 + * @tparam gIM 输入总元素数 + * @tparam gOM 输出总元素数 + * @tparam tM 每个 tile 处理的元素数 + * @tparam DATA_DIM 数据维度数（编译期常量） + * @tparam CONCAT_DIM 拼接维度索引（编译期常量） + */ + template + void concat_scatter( + DType *in_ptr, + DType *out_ptr, + size_t *in_shape, + size_t *out_shape + ) + { + constexpr int kFullTiles = gIM / tM; + constexpr int kTailElements = gIM % tM; + + using InputGlobal = pto::global_tensor>; + using OutputGlobal = pto::global_tensor>; + using DataTile = pto::Tile; + using OffsetTile = pto::Tile; + using InputIterator = global_iterator; + + InputGlobal input_global(in_ptr); + OutputGlobal output_global(out_ptr); + InputIterator input_iter(in_ptr); + + uint32_t input_base = 0; + for (int tile_index = 0; tile_index < kFullTiles; ++tile_index) { + auto input_tile_global = input_iter(0, tile_index); + DataTile data_tile; + OffsetTile offset_tile; + + OffsetTile linear_index; + OffsetTile quotient; + OffsetTile cycle; + OffsetTile cycle_base; + OffsetTile coordinate; + OffsetTile contribution; + + TCI(linear_index, static_cast(input_base)); + TEXPANDSCALAR(offset_tile, static_cast(0)); + + // 计算输出 stride（标量核心，运行时计算） + uint16_t output_stride[DATA_DIM]; + output_stride[DATA_DIM - 1] = 1; + for (int i = static_cast(DATA_DIM) - 2; i >= 0; --i) { + output_stride[i] = output_stride[i + 1] * static_cast(out_shape[i + 1]); + } + + // 对每个维度提取输入坐标并计算输出 byte offset + for (int d = static_cast(DATA_DIM) - 1; d >= 0; --d) { + uint32_t input_stride_for_dim = 1; + for (int dd = d + 1; dd < static_cast(DATA_DIM); ++dd) { + input_stride_for_dim *= static_cast(in_shape[dd]); + } + + if (input_stride_for_dim == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, static_cast(input_stride_for_dim)); + } + + uint16_t in_dim_size = static_cast(in_shape[d]); + TDIVS(cycle, quotient, in_dim_size); + TMULS(cycle_base, cycle, in_dim_size); + TSUB(coordinate, quotient, cycle_base); + + TMULS(contribution, coordinate, + static_cast(output_stride[d] * sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } + + TLOAD(data_tile, input_tile_global); + MSCATTER(output_global, data_tile, offset_tile); + + input_base += tM; + } + + if constexpr (kTailElements != 0) { + using TailDataTile = pto::Tile; + using TailOffsetTile = pto::Tile; + + auto input_tile_global = input_iter(0, kFullTiles); + TailDataTile data_tile; + TailOffsetTile offset_tile; + + TailOffsetTile linear_index; + TailOffsetTile quotient; + TailOffsetTile cycle; + TailOffsetTile cycle_base; + TailOffsetTile coordinate; + TailOffsetTile contribution; + + TCI(linear_index, static_cast(input_base)); + TEXPANDSCALAR(offset_tile, static_cast(0)); + + uint16_t output_stride[DATA_DIM]; + output_stride[DATA_DIM - 1] = 1; + for (int i = static_cast(DATA_DIM) - 2; i >= 0; --i) { + output_stride[i] = output_stride[i + 1] * static_cast(out_shape[i + 1]); + } + + for (int d = static_cast(DATA_DIM) - 1; d >= 0; --d) { + uint32_t input_stride_for_dim = 1; + for (int dd = d + 1; dd < static_cast(DATA_DIM); ++dd) { + input_stride_for_dim *= static_cast(in_shape[dd]); + } + + if (input_stride_for_dim == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, static_cast(input_stride_for_dim)); + } + + uint16_t in_dim_size = static_cast(in_shape[d]); + TDIVS(cycle, quotient, in_dim_size); + TMULS(cycle_base, cycle, in_dim_size); + TSUB(coordinate, quotient, cycle_base); + + TMULS(contribution, coordinate, + static_cast(output_stride[d] * sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + } + + TLOAD(data_tile, input_tile_global); + MSCATTER(output_global, data_tile, offset_tile); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MSCATTER`](../../../../intrinsics/mscatter.md) | `MSCATTER` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TCI`](../../../../intrinsics/tci.md) | `TCI` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `TCVT` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TDIVS`](../../../../intrinsics/tdivs.md) | `TDIVS` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | +| [`TSUB`](../../../../intrinsics/tsub.md) | `TSUB` | `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/concat` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 28 | `make TESTCASE=concat_scatter COMPILER_DIR=$COMPILER_DIR DType=int32_t MAX_DIM=8 tM=512 IN_SHAPE=64,2 OUT_SHAPE=64,2000 IN_SHAPE_NAME=64_2 OUT_SHAPE_NAME=64_2000 DATA_DIM=2 CONCAT_DIM=1 gIM='1000*64*2' gOM='64*2000'` | +| 35 | `make TESTCASE=concat_scatter COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=256,8 OUT_SHAPE=256,8000 IN_SHAPE_NAME=256_8 OUT_SHAPE_NAME=256_8000 DATA_DIM=2 CONCAT_DIM=1 gIM='4000*256*8' gOM='256*8000'` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/concat/concat_scatter_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/concat/src/concat_scatter.cpp` diff --git a/docs/benchmarks/catalog/one-level/concat/index.md b/docs/benchmarks/catalog/one-level/concat/index.md new file mode 100644 index 0000000..0a8c132 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/concat/index.md @@ -0,0 +1,11 @@ +# One-level: concat + + + +This family contains **2** source implementations and +**4** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [concat_gather](concat-gather-1c6990b2.md) | `benchmark/one-level-arch/test/kernel/concat/src/concat_gather.cpp` | 2 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCI`](../../../../intrinsics/tci.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TDIVS`](../../../../intrinsics/tdivs.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TSTORE`](../../../../intrinsics/tstore.md), [`TSUB`](../../../../intrinsics/tsub.md) | +| [concat_scatter](concat-scatter-29559c6c.md) | `benchmark/one-level-arch/test/kernel/concat/src/concat_scatter.cpp` | 2 | [`MSCATTER`](../../../../intrinsics/mscatter.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCI`](../../../../intrinsics/tci.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TDIVS`](../../../../intrinsics/tdivs.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TSUB`](../../../../intrinsics/tsub.md) | diff --git a/docs/benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md b/docs/benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md new file mode 100644 index 0000000..e7d287c --- /dev/null +++ b/docs/benchmarks/catalog/one-level/control/hashtable-lookup-simd-5bc02e85.md @@ -0,0 +1,418 @@ +# hashtable_lookup_simd: hashtable_lookup_simd.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `control` | +| Implementation source | `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp` | +| Active build variants | 6 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="hashtable_lookup_simd.cpp" linenums="1" +#include +#include "benchmark.h" +#include "fileop.h" +#include "template_asm.h" +#include "linx_print.h" + +#include "control/hashtable_lookup_simd.hpp" + +// ============================================================================ +// Constants +// ============================================================================ + +constexpr uint32_t kCapacity = 2550000; // 40800000 / 16 +#ifndef kNum +constexpr int32_t kNum = 256; // 3276800 / 8 +#endif + +#ifndef MAX_PROBE +#define MAX_PROBE 512 +#endif + +#ifndef NUM_COL +#define NUM_COL 256 +#endif + +constexpr int32_t kMaxProbe = MAX_PROBE; +constexpr int32_t kNumRow = 1; +constexpr int32_t kNumCol = NUM_COL; + +constexpr int32_t kBatchSize = kNumCol; // 1D tile of size NUM_COL + +// ============================================================================ +// ELF Data layout +// ============================================================================ + +extern "C" { + extern const uint8_t _binary_inserted_slot_data_start[]; + extern const uint8_t _binary_inserted_slot_data_end[]; + extern const uint8_t _binary_lookup_keys_data_start[]; + extern const uint8_t _binary_lookup_keys_data_end[]; + extern const uint8_t _binary_lookup_values_data_start[]; + extern const uint8_t _binary_lookup_values_data_end[]; +} + +static TableEntry* g_hashtable_ro = reinterpret_cast( + const_cast(_binary_inserted_slot_data_start)); +static int64_t* g_query_keys = reinterpret_cast( + const_cast(_binary_lookup_keys_data_start)); +static int32_t* g_lookup_values = reinterpret_cast( + const_cast(_binary_lookup_values_data_start)); + +static int32_t g_output[kNum]; + +// ============================================================================ +// main +// ============================================================================ + +int main() { + // Initialize all output to kNotFound + for (int i = 0; i < kNum; i++) { + g_output[i] = kNotFound; + } + + // Process all lookup keys in batches of kBatchSize (256) + constexpr int32_t num_batches = (kNum + kBatchSize - 1) / kBatchSize; + + BENCHSTART; + for (int32_t batch = 0; batch < num_batches; batch++) { + int32_t batch_start = batch * kBatchSize; + int32_t batch_count = (batch_start + kBatchSize <= kNum) ? kBatchSize : (kNum - batch_start); + + // For the last batch that may be smaller, we still launch with full tile + // but only the valid elements matter + LaunchHashFind( + g_output + batch_start, g_hashtable_ro, g_query_keys + batch_start); + } + BENCHEND; + + // Verify results. Output goes through linxi_put (raw UART MMIO), which is + // visible on both qemu virt and gfsim ("linx_uart:" lines) without any + // libc/syscall dependency -- so this runs identically under both back-ends. + int match = 0; + int mismatch_count = 0; + for (int i = 0; i < kNum; i++) { + if (g_output[i] == g_lookup_values[i]) { + match++; + } else { + mismatch_count++; + } + } + + // NOTE: counts are printed in hex via linxi_put_hex (shift/mask based). The + // decimal helper linxi_put_i64 uses scalar unsigned divide (divu), which does + // not terminate on the current model, so it is avoided here. + linxi_puts("=== hashtable_lookup_simd ==="); + linxi_put("Match(hex): "); + linxi_put_hex((unsigned long long)(unsigned)match); + linxi_put(" / "); + linxi_put_hex((unsigned long long)(unsigned)kNum); + linxi_putc('\n'); + + int ret = (match == kNum) ? 0 : 1; + linxi_puts(ret == 0 ? "PASS" : "FAIL"); + return ret; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp" + + ```cpp title="hashtable_lookup_simd.hpp" linenums="1" + #ifndef HASHTABLE_LOOKUP_SIMD_HPP + #define HASHTABLE_LOOKUP_SIMD_HPP + + #include + #include "template_asm.h" + #include + + // ============================================================================ + // Pure-SIMD (tile-level) hashtable lookup. + // + // Everything below is expressed with whole-tile operations from the tile-op API + // (TMULS/TADDS/TXOR/TSHLS/TSHRS/TOR/TAND/TREM for the hash, MGATHER/TCMP/TSELECT for + // the probe). There are NO __vec__ SIMT kernels, no <<<>>> launches, and no per-lane + // control flow (blkv_get_index / blkv_rdadd / data-dependent branches). The probe loop + // is a host loop bounded by kMaxProbe with NO data-dependent early break: a tile-level + // early exit would need to reduce the "still-searching" mask to a host scalar, and the + // tile-op API has no tile->scalar-register reduction (only TROWSUM->tile and + // TSTORE->memory). Since TSELECT keeps already-found lanes immune to later probes, + // running all kMaxProbe iterations is correct; the loop is pure tile ops + a host counter. + // ============================================================================ + + // MurmurHash3_x86_32 constants + #define DEFAULT_HASH_SEED 0u + #define c1 0xcc9e2d51u + #define c2 0x1b873593u + #define c3 0xe6546b64u + #define rot_c1 15u + #define rot_c2 13u + + struct TableEntry { + int64_t key; + int32_t value; + int32_t padding; + }; + + constexpr int32_t kNotFound = -1; + + template + struct HashFindTypes { + using TileU32 = Tile; + using TileI64 = Tile; + using TileI32 = Tile; + }; + + // dst = rotl32(x, r) = (x << r) | (x >>(logical) (32 - r)) (in-place safe) + template + inline void tileRotl32(TileU32& dst, TileU32& x, unsigned r, TileU32& tA, TileU32& tB) { + TSHLS(tA, x, r); + TSHRS(tB, x, 32u - r); + TOR(dst, tA, tB); + } + + // One MurmurHash3 mixing round for a 32-bit block: folds `block` into `h`. + template + inline void murmurRound(TileU32& h, TileU32& block, TileU32& k, TileU32& tA, TileU32& tB) { + TCVT(k, block); + TMULS(k, k, c1); + tileRotl32(k, k, rot_c1, tA, tB); + TMULS(k, k, c2); + TXOR(h, h, k); + tileRotl32(h, h, rot_c2, tA, tB); + TMULS(h, h, 5u); + TADDS(h, h, c3); + } + + // Unsigned modulo of a full-range uint32 tile by a (compile-time-known) modulus, + // using only signed-int32 TREM (the ISA has no unsigned rem). Identity: + // h % m == (2 * ((h >> 1) % m) + (h & 1)) % m + // (h>>1) < 2^31 so it is a non-negative int32; 2*(m-1)+1 < 2^31 so the second + // operand stays non-negative -> signed TREM yields the correct unsigned result. + template + __attribute__((always_inline)) inline void + uModU32(typename HashFindTypes::TileI32& dst, + typename HashFindTypes::TileU32& h) + { + using Types = HashFindTypes; + using TileU32 = typename Types::TileU32; + using TileI32 = typename Types::TileI32; + + TileU32 halfU, bitU, oneU; + TileI32 halfI, bitI, capI, tmp; + + TSHRS(halfU, h, 1u); // h >> 1 (0 .. 2^31-1) + TEXPANDSCALAR(oneU, 1u); + TAND(bitU, h, oneU); // h & 1 + + TCAST(halfI, halfU); // -> non-negative int32 + TCAST(bitI, bitU); + TEXPANDSCALAR(capI, static_cast(kMod)); + + TREM(tmp, halfI, capI); // (h>>1) % m + TMULS(tmp, tmp, 2); // 2 * r1 + TADD(tmp, tmp, bitI); // 2*r1 + (h&1) + TREM(dst, tmp, capI); // == h % m + } + + // Compute per-key probe byte-offset = (MurmurHash3(key) % kCap) * sizeof(TableEntry) + template + __attribute__((always_inline)) inline void + computeProbeOffsets( + typename HashFindTypes::TileI32& probeOff, + typename HashFindTypes::TileI64& queryKeys) + { + using Types = HashFindTypes; + using TileU32 = typename Types::TileU32; + using TileI64 = typename Types::TileI64; + + TileU32 h, k, blk, tA, tB, tmp; + TileI64 keyHi; + + // block0 = low 32 bits, block1 = high 32 bits of each 8-byte key + TileU32 block0, block1; + TCAST(block0, queryKeys); // static_cast truncates to low 32 + TSHRS(keyHi, queryKeys, 32u); // logical >> 32 (unsigned), high 32 -> low + TCAST(block1, keyHi); + + TEXPANDSCALAR(h, DEFAULT_HASH_SEED); + TCVT(blk, block0); + murmurRound(h, blk, k, tA, tB); + TCVT(blk, block1); + murmurRound(h, blk, k, tA, tB); + + // finalize: h ^= len(=8); h ^= h>>16; h *= ..; h ^= h>>13; h *= ..; h ^= h>>16 + TileU32 lenTile; + TEXPANDSCALAR(lenTile, 8u); + TXOR(h, h, lenTile); + TSHRS(tmp, h, 16u); TXOR(h, h, tmp); + TMULS(h, h, 0x85ebca6bu); + TSHRS(tmp, h, 13u); TXOR(h, h, tmp); + TMULS(h, h, 0xc2b2ae35u); + TSHRS(tmp, h, 16u); TXOR(h, h, tmp); + + // slot = h % kCap (unsigned) ; byte offset = slot * sizeof(TableEntry) + uModU32(kCap)>(probeOff, h); + TMULS(probeOff, probeOff, static_cast(sizeof(TableEntry))); + } + + // ============================================================================ + // runHashFind - one batch of kTileRows x kTileCols keys, pure tile ops + // ============================================================================ + template + void runHashFind(int32_t __out__ *out, + TableEntry __in__ *table, + int64_t __in__ *queries) + { + using Types = HashFindTypes; + using TileI64 = typename Types::TileI64; + using TileI32 = typename Types::TileI32; + + using BatchShape = Shape<1, 1, 1, kTileRows, kTileCols>; + using BatchStride = Stride<1, 1, 1, kTileCols, 1>; + using KeyGT = GlobalTensor; + using OutGT = GlobalTensor; + + // Whole table viewed as flat arrays; MGATHER uses byte offsets from the base. + using TableKeyGT = GlobalTensor, Stride<1,1,1,1,1>>; + using TableValGT = GlobalTensor, Stride<1,1,1,1,1>>; + TableKeyGT tableKeyGlobal(reinterpret_cast(table)); + TableValGT tableValGlobal(reinterpret_cast(reinterpret_cast(table) + 8)); + + TileI64 queryKeyTile; + TileI32 probeOffTile; + TileI32 outTile; + + KeyGT key_gt(queries); + TLOAD(queryKeyTile, key_gt); + + computeProbeOffsets(probeOffTile, queryKeyTile); + + TEXPANDSCALAR(outTile, kNotFound); + + // TCMP EQ supports int32 only, so compare the 64-bit keys as two int32 halves. + TileI32 queryLo, queryHi; + { + TileI64 qHi; + TCAST(queryLo, queryKeyTile); // low 32 bits + TSHRS(qHi, queryKeyTile, 32u); + TCAST(queryHi, qHi); // high 32 bits + } + + TileI64 tableKeyTile; + TileI64 tableKeyHi; + TileI32 tableLo, tableHi; + TileI32 tableValTile; + TileI32 matchLo, matchHi, matchMask; + TileI32 capBytesTile; + TEXPANDSCALAR(capBytesTile, static_cast(kCap) * static_cast(sizeof(TableEntry))); + + for (int probe = 0; probe < kMaxProbe; probe++) { + MGATHER(tableKeyTile, tableKeyGlobal, probeOffTile); + MGATHER(tableValTile, tableValGlobal, probeOffTile); + // 64-bit key equality = (low32 == low32) && (high32 == high32) + TCAST(tableLo, tableKeyTile); + TSHRS(tableKeyHi, tableKeyTile, 32u); + TCAST(tableHi, tableKeyHi); + TCMP(matchLo, queryLo, tableLo, CmpMode::EQ); + TCMP(matchHi, queryHi, tableHi, CmpMode::EQ); + TAND(matchMask, matchLo, matchHi); + TSELECT(outTile, matchMask, tableValTile, outTile); // match ? value : keep + + // NOTE: no early break. A data-dependent early exit would require reducing the + // "still-searching" mask to a host scalar (TROWSUMEXPAND -> TSTORE -> scalar + // load), i.e. a tile->memory->scalar round-trip every iteration. The tile-op API + // has no tile->scalar-register reduction, so that round-trip is the only option + // and it is exactly the fragile path we avoid. TSELECT already makes already-found + // lanes immune to further probes, so running all kMaxProbe iterations is correct; + // we simply trade the early-exit speedup for a pure-tile loop body. + + // advance: probeOff = (probeOff + sizeof(TableEntry)) % (kCap * sizeof(TableEntry)) + TADDS(probeOffTile, probeOffTile, static_cast(sizeof(TableEntry))); + TREM(probeOffTile, probeOffTile, capBytesTile); + } + + OutGT outGlobal(out); + TSTORE(outGlobal, outTile); + } + + template + void LaunchHashFind(int32_t *out, TableEntry* table, int64_t* queries) + { + runHashFind(out, table, queries); + } + + #endif // HASHTABLE_LOOKUP_SIMD_HPP + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MGATHER`](../../../../intrinsics/mgather.md) | `MGATHER` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TADDS`](../../../../intrinsics/tadds.md) | `TADDS` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TAND`](../../../../intrinsics/tand.md) | `TAND` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TCMP`](../../../../intrinsics/tcmp.md) | `TCMP` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `TCAST`, `TCVT` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TOR`](../../../../intrinsics/tor.md) | `TOR` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TREM`](../../../../intrinsics/trem.md) | `TREM` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TSEL`](../../../../intrinsics/tsel.md) | `TSELECT` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TSHLS`](../../../../intrinsics/tshls.md) | `TSHLS` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TSHRS`](../../../../intrinsics/tshrs.md) | `TSHRS` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | +| [`TXOR`](../../../../intrinsics/txor.md) | `TXOR` | `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/control` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col256_debug_on EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=256 " diss` | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col512_debug_on EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=512 " diss` | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col1024_debug_on EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=1024 " diss` | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col256_debug_off EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=256 -DFOR_GFSIM" diss` | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col512_debug_off EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=512 -DFOR_GFSIM" diss` | +| 11 | `make TESTCASE=hashtable_lookup_simd SUFFIX=_kNum6144_kMaxProbe512_knum_col1024_debug_off EXTRA_DEFINES="-DkNum=6144 -DMAX_PROBE=512 -DNUM_COL=1024 -DFOR_GFSIM" diss` | + +## Required Data Objects + +The benchmark build or run path uses these checked-in generators/data inputs: + +- `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/build_data_obj.sh` +- `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/inserted_slot.data` +- `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/lookup_keys.data` +- `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/lookup_values.data` +- `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/data_obj/probe_statistics.md` +- `benchmark/one-level-arch/test/kernel/control/hkv/data_obj/build_data_obj.sh` + +## Resolved One-Level Source Closure + +??? info "8 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/control/hashtable_lookup_simd.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/linx_print.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/src/benchmark.h` + - `benchmark/one-level-arch/test/common/template_asm.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp` diff --git a/docs/benchmarks/catalog/one-level/control/index.md b/docs/benchmarks/catalog/one-level/control/index.md new file mode 100644 index 0000000..599e239 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/control/index.md @@ -0,0 +1,10 @@ +# One-level: control + + + +This family contains **1** source implementations and +**6** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [hashtable_lookup_simd](hashtable-lookup-simd-5bc02e85.md) | `benchmark/one-level-arch/test/kernel/control/hashtable_lookup_simd/hashtable_lookup_simd.cpp` | 6 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TADDS`](../../../../intrinsics/tadds.md), [`TAND`](../../../../intrinsics/tand.md), [`TCMP`](../../../../intrinsics/tcmp.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TOR`](../../../../intrinsics/tor.md), [`TREM`](../../../../intrinsics/trem.md), [`TSEL`](../../../../intrinsics/tsel.md), [`TSHLS`](../../../../intrinsics/tshls.md), [`TSHRS`](../../../../intrinsics/tshrs.md), [`TSTORE`](../../../../intrinsics/tstore.md), [`TXOR`](../../../../intrinsics/txor.md) | diff --git a/docs/benchmarks/catalog/one-level/element-wise-gelu/gelu-fdf67353.md b/docs/benchmarks/catalog/one-level/element-wise-gelu/gelu-fdf67353.md new file mode 100644 index 0000000..67e8367 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/element-wise-gelu/gelu-fdf67353.md @@ -0,0 +1,359 @@ +# gelu: gelu.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `element_wise/gelu` | +| Implementation source | `benchmark/one-level-arch/test/kernel/element_wise/gelu/src/gelu.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="gelu.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "element_wise/gelu_pto.hpp" + + +#ifndef DTYPE +#define DTYPE int32_t +#endif + +#ifndef tMs +#define tMs 512 +#endif + +#ifndef gMs +#define gMs (24 * 512 * 1024) +#endif + +#ifndef Approximate +#define Approximate false +#endif + +#ifndef DTYPE +#define DTYPE true +#endif +// 【静态随机偏移】手动写死，模拟内存地址不是数组起始位置 +#define OFFSET_IN 11 // 输入静态偏移 +#define OFFSET_OUT 17 // 输出静态偏移 +// ============================================================================ +// main +// ============================================================================ +int main() { + + // printf("start !!!!!!!!!!!!!!!!!"); + using dtype = DTYPE; + // ========================== + // 申明空间，留出静态偏移空间 + // ========================== + dtype input_buf[gMs + OFFSET_IN]; // 前面留空，模拟随机地址 + dtype output_buf[gMs + OFFSET_OUT]; + + // ========================== + // 【静态随机地址】 + // ========================== + dtype* input = input_buf + OFFSET_IN; + dtype* output = output_buf + OFFSET_OUT; + + // ========================== + // 从 bin 文件读取输入数据 + // ========================== + + // #ifdef RES_CHECK + // // #ifndef CHK_DIR + // // #define CHK_DIR "/remote/lms01/l00948608/project/jcore_benchmark/JanusCoreBench/test/gelu/src/data" + // // #endif + // #define INPUT_PATH CHK_DIR "/input.bin" + // #define OUTPUT_PATH CHK_DIR "/output.bin" + // printf("input path \n"); + // printf(INPUT_PATH); + // printf("\n"); + // // printf(OUTPUT_PATH); + // readBinaryFile(INPUT_PATH, (uint8_t*)input, gMs * sizeof(dtype)); + // // printf("input %.4f \n", input[0]); + // // printf("input %.4f \n", input[1]); + // // printf("input %.4f \n", input[2]); + // #endif + + gelu(input, output, Approximate); + // printf("output %.4f \n", output[0]); + // printf("output %.4f \n", output[1]); + // printf("output %.4f \n", output[2]); + + // #ifdef RES_CHECK + // writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gMs * sizeof(dtype)); + // #endif +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp" + + ```cpp title="gelu_pto.hpp" linenums="1" + // ============================================================================ + // GELU 算子 — PTO 一层编程模型 + // + // 原始 gelu.hpp 策略: + // TLOAD (half) -> __vec__ gelu_simd (多项式拟合) -> TSTORE (half) + // __vec__ 块逐元素: fp16→fp32, clamp, Horner 多项式, exp, 除法, fp32→fp16 + // + // PTO 一层策略: + // TLOAD (half) -> TCVT(fp16→fp32) -> tile 指令链计算 GELU -> TCVT(fp32→fp16) -> TSTORE (half) + // 全部用 Tile 级内联函数, 无 __vec__ 块 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD；当前编译器名 TLOAD； │ + // │ │ 二层实现 │ jcore/TLoad.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TCVT │ API 有(签名不同)，│ PTO ISA: TCVT(dst,src,tmp,mode,satMode) │ + // │ │ 二层实现 │ 当前编译器: TCVT(dst,src) 无 tmp/mode； │ + // │ │ │ jcore/TCvt.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TMAXS │ API 有，二层实现 │ jcore/TMaxs.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TMINS │ API 有，二层实现 │ jcore/TMins.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TMUL │ API 有，二层实现 │ jcore/TMul.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TMULS │ API 有，二层实现 │ jcore/TMuls.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TADDS │ API 有，二层实现 │ jcore/TAdds.hpp 用 __vec__ 实现 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TEXP │ API 有，二层实现 │ jcore/TExp.hpp 用 __vec__ 实现 │ + // │ │ │ (template_asm.h 有 TEXP_TEPL 内联汇编, │ + // │ │ │ 但不在 pto_tileop.hpp API 中) │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TRECIP │ API 有，二层实现 │ jcore/TRecip.hpp 用 __vec__ 实现 │ + // │ │ │ (template_asm.h 有 TRECIP_TEPL 内联汇编, │ + // │ │ │ 但不在 pto_tileop.hpp API 中) │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE；当前编译器名 TSTORE；│ + // │ │ 二层实现 │ jcore/TStore.hpp 用 __vec__ 实现 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include + #include + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ============================================================================ + // GELU 多项式拟合系数 (与原始 gelu.hpp 一致) + // GELU(x) = x / (1 + exp(t * P(t²))) + // P(t²) = A5*t²⁵ + A4*t²⁴ + A3*t²³ + A2*t²² + A1*t² + A0 + AM1/t² + // (Horner: p = ((((A5*t2 + A4)*t2 + A3)*t2 + A2)*t2 + A1)*t2 + A0)*t2 + AM1) + // ============================================================================ + namespace gelu_coeffs { + constexpr float A5 = -3.5123395303315874e-09f; + constexpr float A4 = 2.6452661927578447e-07f; + constexpr float A3 = -7.9294877650681883e-06f; + constexpr float A2 = 1.1061238183174282e-04f; + constexpr float A1 = 6.5189960878342390e-05f; + constexpr float A0 = -7.2666168212890625e-02f; + constexpr float AM1 = -1.5957698822021484e+00f; + constexpr float CLAMP_MAX = 5.75f; + } + + // ---------------------------------------------------------------------------- + // gelu_impl: 用 PTO ISA Tile 指令计算 GELU (一层编程, 无 __vec__ 块) + // + // 输入: inTile — fp16 tile, shape (1, tM) + // 输出: outTile — fp16 tile, shape (1, tM) + // 中间: 全部在 fp32 tile 上计算 + // + // 算法: + // x = (float)in + // t = clamp(x, -5.75, 5.75) + // t2 = t * t + // p = Horner(t2, [A5,A4,A3,A2,A1,A0,AM1]) + // e = exp(t * p) + // y = x * (1 / (1 + e)) // 用 TRECIP + TMUL 代替除法 + // out = (half)y + // ---------------------------------------------------------------------------- + template + void gelu_impl( + tile_shapeData &inTile, + tile_shapeData &outTile, + tile_shapeFP32 &tmpCvt // TCVT 需要的临时 tile + ) { + using fp_t = typename tile_shapeFP32::DType; // float + + tile_shapeFP32 xTile; // x = (float)input + tile_shapeFP32 tTile; // t = clamp(x) + tile_shapeFP32 t2Tile; // t² + tile_shapeFP32 pTile; // 多项式值 + tile_shapeFP32 scratchTile; // 复用: tp -> exp -> denom -> recip -> y + + // ---- Step 1: fp16 -> fp32 ---- + // [当前编译器] TCVT(dst, src) 无 tmp/mode/satMode 参数; jcore 为 __vec__ + TCVT(xTile, inTile); + + // ---- Step 2: clamp x to [-5.75, 5.75] ---- + // [当前编译器] TMAXS/TMINS API 有, jcore 为 __vec__ + TMAXS(tTile, xTile, (fp_t)(-gelu_coeffs::CLAMP_MAX)); // t = max(x, -5.75) + TMINS(tTile, tTile, (fp_t)gelu_coeffs::CLAMP_MAX); // t = min(t, 5.75) + + // ---- Step 3: t² = t * t ---- + // [当前编译器] TMUL API 有, jcore 为 __vec__ + TMUL(t2Tile, tTile, tTile); + + // ---- Step 4: 多项式 Horner 法 ---- + // p = A5*t2 + A4 + // [当前编译器] TMULS/TADDS API 有, jcore 为 __vec__ + TMULS(pTile, t2Tile, gelu_coeffs::A5); + TADDS(pTile, pTile, gelu_coeffs::A4); + + // p = p*t2 + A3 + TMUL(pTile, pTile, t2Tile); + TADDS(pTile, pTile, gelu_coeffs::A3); + + // p = p*t2 + A2 + TMUL(pTile, pTile, t2Tile); + TADDS(pTile, pTile, gelu_coeffs::A2); + + // p = p*t2 + A1 + TMUL(pTile, pTile, t2Tile); + TADDS(pTile, pTile, gelu_coeffs::A1); + + // p = p*t2 + A0 + TMUL(pTile, pTile, t2Tile); + TADDS(pTile, pTile, gelu_coeffs::A0); + + // p = p*t2 + AM1 + TMUL(pTile, pTile, t2Tile); + TADDS(pTile, pTile, gelu_coeffs::AM1); + + // ---- Step 5: exp_val = exp(t * p) ---- + // scratch = t * p + TMUL(scratchTile, tTile, pTile); + // exp_val = exp(scratch) + // [当前编译器] TEXP API 有, jcore 为 __vec__ + // (template_asm.h 有 TEXP_TEPL 内联汇编, 但不在 pto_tileop.hpp 中) + TEXP(scratchTile, scratchTile); // scratch = exp(t*p) + + // ---- Step 6: y = x / (1 + exp_val) ---- + // denom = 1 + exp_val + TADDS(scratchTile, scratchTile, (fp_t)1.0f); // scratch = 1 + exp + // recip = 1 / denom + // [当前编译器] TRECIP API 有, jcore 为 __vec__ + // (template_asm.h 有 TRECIP_TEPL 内联汇编, 但不在 pto_tileop.hpp 中) + TRECIP(scratchTile, scratchTile); // scratch = 1 / (1+exp) + // y = x * recip + TMUL(scratchTile, xTile, scratchTile); // scratch = x * recip = y + + // ---- Step 7: fp32 -> fp16 ---- + TCVT(outTile, scratchTile); + } + + + // ---------------------------------------------------------------------------- + // gelu: 主入口, 接口与原 gelu.hpp 一致 + // ---------------------------------------------------------------------------- + template + void gelu( + dtype *in_ptr, + dtype *out_ptr, + bool approximate = false + ) { + using gm_shape = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeFP32 = Tile; + using tile_shapeData_rmd = Tile; + using tile_shapeFP32_rmd = Tile; + + const int Mb = gM / tM; + const int rmd_M = gM % tM; + + using itIn = global_iterator; + using itOut = global_iterator; + + itIn gIIter(in_ptr); + itOut gOIter(out_ptr); + + tile_shapeData inTile, outTile; + tile_shapeFP32 tmpCvt; // TCVT 临时 tile + tile_shapeData_rmd inTile_rmd, outTile_rmd; + tile_shapeFP32_rmd tmpCvt_rmd; + + for (int i = 0; i < Mb; ++i) { + auto gI = gIIter(0, i); + auto gO = gOIter(0, i); + + // TLOAD: GM -> UB + // [当前编译器] 名为 TLOAD, jcore 为 __vec__ + TLOAD(inTile, gI); + + gelu_impl(inTile, outTile, tmpCvt); + + // TSTORE: UB -> GM + // [当前编译器] 名为 TSTORE, jcore 为 __vec__ + TSTORE(gO, outTile); + } + if constexpr (rmd_M) { + auto gI = gIIter(0, Mb); + auto gO = gOIter(0, Mb); + + TLOAD(inTile_rmd, gI); + gelu_impl(inTile_rmd, outTile_rmd, tmpCvt_rmd); + TSTORE(gO, outTile_rmd); + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADDS`](../../../../intrinsics/tadds.md) | `TADDS` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `TCVT` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TEXP`](../../../../intrinsics/texp.md) | `TEXP` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TMAXS`](../../../../intrinsics/tmaxs.md) | `TMAXS` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TMINS`](../../../../intrinsics/tmins.md) | `TMINS` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TMUL`](../../../../intrinsics/tmul.md) | `TMUL` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TRECIP`](../../../../intrinsics/trecip.md) | `TRECIP` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/element_wise/gelu` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 10 | `make TESTCASE=gelu DTYPE=__bf16 tMs=2048 gMs=24*8*1024 SHAPE_NAME=24_8_1024 Approximate=false diss` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/element_wise/gelu_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/element_wise/gelu/src/gelu.cpp` diff --git a/docs/benchmarks/catalog/one-level/element-wise-gelu/index.md b/docs/benchmarks/catalog/one-level/element-wise-gelu/index.md new file mode 100644 index 0000000..2728af8 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/element-wise-gelu/index.md @@ -0,0 +1,10 @@ +# One-level: element_wise/gelu + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [gelu](gelu-fdf67353.md) | `benchmark/one-level-arch/test/kernel/element_wise/gelu/src/gelu.cpp` | 1 | [`TADDS`](../../../../intrinsics/tadds.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXP`](../../../../intrinsics/texp.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMAXS`](../../../../intrinsics/tmaxs.md), [`TMINS`](../../../../intrinsics/tmins.md), [`TMUL`](../../../../intrinsics/tmul.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TRECIP`](../../../../intrinsics/trecip.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md b/docs/benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md new file mode 100644 index 0000000..a09488a --- /dev/null +++ b/docs/benchmarks/catalog/one-level/fa/fa-2d-unroll-b46797ec.md @@ -0,0 +1,504 @@ +# fa_2d_unroll: fa_2d_unroll.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `fa` | +| Implementation source | `benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp` | +| Active build variants | 8 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then executes matrix or matrix-vector work on cube/accumulator tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="fa_2d_unroll.cpp" linenums="1" +#include +#include "benchmark.h" +#include "fileop.h" +#include "fa/fa_2d_unroll_pto.hpp" + +#define B 1 +#define H 1 + +#ifndef Tsq +#define Sq 512 +#else +#define Sq Tsq +#endif + +#ifndef Tskv +#define Skv 512 +#else +#define Skv Tskv +#endif + +#define qD 128 +#define vD 128 + +#ifndef Tm +#define kTm 128 +#else +#define kTm Tm +#endif + +#ifndef Tk +#define kTk 128 +#else +#define kTk Tk +#endif + +#define ALIGN_MASK 0xfffffffffffff000ull +#define ALIGN 4*1024 + +int main(){ + using dtype = __half; + + dtype qp[B*H*Sq*qD + 2*ALIGN]; + dtype kp[B*H*Skv*qD + 2*ALIGN]; + dtype vp[B*H*Skv*vD + 2*ALIGN]; + dtype outp[B*H*Sq*vD + 2*ALIGN]; + + dtype* q = (dtype *)(((uint64_t)qp & ALIGN_MASK) + ALIGN); + dtype* k = (dtype *)(((uint64_t)kp & ALIGN_MASK) + ALIGN); + dtype* v = (dtype *)(((uint64_t)vp & ALIGN_MASK) + ALIGN); + dtype* out = (dtype *)(((uint64_t)outp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRCQ_PATH CHK_DIR "/srcq.bin" + #define SRCK_PATH CHK_DIR "/srck.bin" + #define SRCV_PATH CHK_DIR "/srcv.bin" + readBinaryFile(SRCQ_PATH, (uint8_t*)q, B*H*Sq*qD*sizeof(dtype)); + readBinaryFile(SRCK_PATH, (uint8_t*)k, B*H*Skv*qD*sizeof(dtype)); + readBinaryFile(SRCV_PATH, (uint8_t*)v, B*H*Skv*vD*sizeof(dtype)); + #endif + + BENCHSTART; + for(int i=0;i( + out + i*H*Sq*vD + j*Sq*vD, + q + i*H*Sq*qD + j*Sq*qD, + k + i*H*Skv*qD + j*Skv*qD, + v + i*H*Skv*vD + j*Skv*vD + ); + } + } + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)out, B*H*Sq*vD*sizeof(dtype)); + #endif + + return 0; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp" + + ```cpp title="fa_2d_unroll_pto.hpp" linenums="1" + template + struct tileW_type { + using DType = dtype; + }; + + template + void flash_attention_2d_unroll_pto(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, dtype* v_ptr) { + // 计算语义: + // O = softmax((Q * K^T) / sqrt(scaleD)) * V + // Q: [Sq, qD], K: [Skv, qD], V: [Skv, vD], O: [Sq, vD] + // + // 该实现按 Q 维度切成 Qb 个 kTm 行块，按 K/V 的 sequence 维度切成 Kb 个 kTk 列块。 + // Xdim 表示一次 unroll 处理多少个 Q block，Ydim 表示一次 unroll 处理多少个 K/V block。 + // 对每个 Q block 使用 online softmax: 逐个 K/V block 更新 row max、row sum 和 partial output。 + + // 全局张量形状和内存布局。 + // gmK/gmO 都使用 RowMajor，和测试侧线性 buffer 的 [Skv,qD]/[Sq,vD] 布局一致。 + // K load 成 [kTk,qD] 的 Right tile，作为 TMATMUL 右操作数时等价参与 Q * K^T。 + using gmQ = global_tensor>; // Q global: [Sq, qD] + using gmK = global_tensor>; // K global: [Skv, qD] + using gmV = global_tensor>; // V global: [Skv, vD] + using gmO = global_tensor>; // O global: [Sq, vD] + + // tile 寄存器形状。 + // + // Q/K: + // tileQ: L0A/Left tile, logical shape [kTm, qD]。 + // tileK: L0B/Right tile, logical loaded shape [kTk, qD] from row-major K. + // It is consumed as the transposed right operand for Q * K^T. + // qD==192 时物理 qD 维 pad 到 256，逻辑有效列仍然是 qD。 + // + // QK score: + // tileW_out: Acc tile, TMATMUL 输出，logical shape [kTm, kTk]，dtype=float。 + // tileW: Vec tile, ACCCVT 后的 score tile，ColMajor，logical shape [kTm, kTk]。 + // tileW_cast: softmax 后 score cast 成 TMATMUL 左操作数需要的 dtype。 + // tileW_left: L0A/Left tile，用于后续 softmax(score) * V。 + // + // Output: + // tileO_out: Acc tile, PV 矩阵乘输出，logical shape [kTm, vD]。 + // tileO: Vec tile，online softmax 的 float partial output，logical shape [kTm, vD]。 + // tileO_cast: 写回前从 float cast 到输出 dtype 的 tile，logical shape [kTm, vD]。 + // + // V: + // tileV: L0B/Right tile，logical shape [kTk, vD]。 + // + // Softmax row state: + // tileMax/tileSum/tileScale 的逻辑 shape 都是 [kTm, 1]。 + // 这里第二维物理写成 8，valid col 为 1，用于满足 Vec tile 对齐/active size 要求。 + using tileQ = TileLeft; + using tileK = TileRight; + using tileW_out = TileAcc; + using tileW = Tile; + using tileW_cast = Tile::DType, kTm, kTk, BLayout::ColMajor>; + using tileW_left = TileLeft; + + using tileO_out = TileAcc; + using tileO = Tile; + using tileO_cast = Tile; + + using tileV = TileRight; + using tileMax = Tile; + using tileSum = Tile; + using tileScale = Tile; + + // 全局迭代器。iterator 的 block 坐标与对应 tile 的 logical shape 匹配: + // gIterQ(q_block, 0) -> Q[q_block*kTm : (q_block+1)*kTm, 0:qD] + // gIterK(k_block, 0) -> K[k_block*kTk : (k_block+1)*kTk, 0:qD] + // gIterV(k_block, 0) -> V[k_block*kTk : (k_block+1)*kTk, 0:vD] + // gIterO(q_block, 0) -> O[q_block*kTm : (q_block+1)*kTm, 0:vD] + using itQ = global_iterator; + using itK = global_iterator; + using itV = global_iterator; + using itO = global_iterator; + + itQ gIterQ(q_ptr); + itK gIterK(k_ptr); + itV gIterV(v_ptr); + itO gIterO(out_ptr); + + // FlashAttention score scale。默认 scaleD=qD，与标准 attention 的 1/sqrt(qD) 一致。 + const float scale = 1.0f / sqrt((float)scaleD); + const int Qb = (Sq + kTm - 1) / kTm; + const int Kb = (Skv + kTk - 1) / kTk; + + #ifdef _2D_UNROLL_PTO + // 当前实现没有为尾块做 mask/pad 分支，因此要求 block 数能被 unroll 因子整除。 + static_assert(Qb%Xdim==0, "Qb needs to be a multiple of Xdim"); + static_assert(Kb%Ydim==0, "Kb needs to be a multiple of Ydim"); + static_assert(type_traits::bits == 4 || type_traits::DType>::bits == type_traits::bits, "when dtype=fp8 or fp16 or fp32, tileW_cast dtype must the same"); + #endif + + // 外层遍历 Q block。每轮并行/展开处理 Xdim 个 Q tile。 + for (int i = 0; i < Qb; i+=Xdim) { + + tileQ tQ[Xdim]; + + // TLOAD: + // global Q block -> tileQ + // before: gQ points to Q block [kTm, qD] + // after : tQ[x] is Left tile, logical shape [kTm, qD] + #pragma clang loop unroll(full) + for(int x=0;x tileK + // before: gK points to row-major K block [kTk, qD] + // after : tK[y] is Right tile, logical loaded shape [kTk, qD] + #pragma clang loop unroll(full) + for(int y=0;y tileW_out [kTm, kTk] Acc(float) + // ACCCVT : Acc/NZ-like layout -> Vec ColMajor tileW [kTm, kTk] + // TMULS : tileW *= 1/sqrt(scaleD) + // + // After this block: + // tW[x][y] holds scaled attention logits for Q block (i+x) and K block (j+y). + #pragma clang loop unroll(full) + for(int x=0;x Vec ColMajor + ACCCVT(tW[x][y], tW_out); + TMULS(tW[x][y], tW[x][y], scale); + } + } + + tileMax tNewMax[Xdim]; + tileSum tNewSum[Xdim]; + + tileW_cast tExpW[Xdim][Ydim]; + + tileMax tLocalMax[Xdim][Ydim]; + tileSum tLocalSum[Xdim][Ydim]; + tileSum tScaledOldSum[Xdim]; + + // Online softmax for current Ydim K blocks. + // 数学形式按行计算: + // m_new = max(m_old, rowmax(score_block_0), ..., rowmax(score_block_Y-1)) + // scale_old = exp(m_old - m_new) + // l_new = l_old * scale_old + sum_y rowsum(exp(score_y - m_new)) + // p_y = exp(score_y - m_new) + #pragma clang loop unroll(full) + for(int x=0;x::DType. + // tExpW[x][y] still has logical shape [kTm, kTk]. + #pragma clang loop unroll(full) + for(int y=0;y tileV + // before: gV points to V block [kTk, vD] + // after : tV[y] is Right tile, logical shape [kTk, vD] + #pragma clang loop unroll(full) + for(int y=0;y tileW_left Left [kTm,kTk] + // TMATMUL : first y, tPV_out = p_y * V_y + // TMATMUL_ACC: next y, tPV_out += p_y * V_y + // ACCCVT : Acc [kTm,vD] -> Vec float tPV[x] [kTm,vD] + // + // After this block: + // tPV[x] = sum_y exp(score_y - m_new) * V_y, for current Ydim K/V blocks. + tileW_left tW_left[Xdim][Ydim]; + #pragma clang loop unroll(full) + for(int x=0;x dtype output tile + // TSTORE = store O block [kTm, vD] back to global output. + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + TRECIP(tInvSum[x], tSum[x]); + TROWEXPANDMUL(tO[x], tO[x], tInvSum[x]); + TCVT(tO_cast[x], tO[x]); + auto dstO = gIterO(i+x, 0); + TSTORE(dstO, tO_cast[x]); + } + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `ACCCVT`, `TCVT` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TEXP`](../../../../intrinsics/texp.md) | `TEXP` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TMATMUL`](../../../../intrinsics/tmatmul.md) | `TMATMUL` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md) | `TMATMUL_ACC` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TMAX`](../../../../intrinsics/tmax.md) | `TMAX` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TMUL`](../../../../intrinsics/tmul.md) | `TMUL` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TRECIP`](../../../../intrinsics/trecip.md) | `TRECIP` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TROWEXPANDMUL`](../../../../intrinsics/trowexpandmul.md) | `TROWEXPANDMUL` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TROWEXPANDSUB`](../../../../intrinsics/trowexpandsub.md) | `TROWEXPANDSUB` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TROWMAX`](../../../../intrinsics/trowmax.md) | `TROWMAX` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TROWSUM`](../../../../intrinsics/trowsum.md) | `TROWSUM` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | +| [`TSUB`](../../../../intrinsics/tsub.md) | `TSUB` | `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/fa` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 8 | `make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=2` | +| 9 | `make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=1 Y=2` | +| 12 | `make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=4` | +| 13 | `make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=1 Y=4` | +| 16 | `make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=2 Y=2` | +| 17 | `make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=2 Y=2` | +| 20 | `make TESTCASE=fa_2d_unroll Sq=256 Skv=512 Tm=16 Tk=32 X=2 Y=4` | +| 21 | `make TESTCASE=fa_2d_unroll Sq=512 Skv=512 Tm=16 Tk=32 X=2 Y=4` | + +## Resolved One-Level Source Closure + +??? info "6 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/fa/fa_2d_unroll_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/src/benchmark.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp` diff --git a/docs/benchmarks/catalog/one-level/fa/fa-hif4-86b1f297.md b/docs/benchmarks/catalog/one-level/fa/fa-hif4-86b1f297.md new file mode 100644 index 0000000..a49c40b --- /dev/null +++ b/docs/benchmarks/catalog/one-level/fa/fa-hif4-86b1f297.md @@ -0,0 +1,530 @@ +# fa_HIF4_HIF4: fa_hif4.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `fa` | +| Implementation source | `benchmark/one-level-arch/test/kernel/fa/src/fa_hif4.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then executes matrix or matrix-vector work on cube/accumulator tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="fa_hif4.cpp" linenums="1" +#include +#include "benchmark.h" +#include "fileop.h" +#include "fa/fa_hif4.hpp" + +#define B 1 +#define H 1 //2 + +#ifndef Tsq +#define Sq 512 //1024 +#else +#define Sq Tsq +#endif + +#ifndef Tskv +#define Skv 512 //1024 +#else +#define Skv Tskv +#endif + +#define qD 128 +#define vD 128 + +#ifndef Tm +#define kTm 128 +#else +#define kTm Tm +#endif + +#ifndef Tk +#define kTk 128 +#else +#define kTk Tk +#endif + +#define ALIGN_MASK 0xfffffffffffff000ull +#define ALIGN 4*1024 + +#ifndef MBUF +#define MBUF 2 + +// #define HIF4_BF16x2 +// #define HIF4 +// #define HIF4_NOGATHER +#define STR(x) #x +#endif + +int main(){ + // using typep = __half; + using typep = __fp4_e1m2x2; + typep qp[B*H*Sq*qD + 2*ALIGN]; + typep kp[B*H*Skv*qD + 2*ALIGN]; + typep vp[B*H*Skv*vD + 2*ALIGN]; + typep outp[B*H*Sq*vD + 2*ALIGN]; + uint8_t qmx[B*H*Sq*qD + 2*ALIGN]; + uint8_t kmx[B*H*Sq*qD + 2*ALIGN]; + uint8_t vmx[B*H*Sq*vD + 2*ALIGN]; + + typep* q = (typep *)(((uint64_t)qp & ALIGN_MASK) + ALIGN); + typep* k = (typep *)(((uint64_t)kp & ALIGN_MASK) + ALIGN); + typep* v = (typep *)(((uint64_t)vp & ALIGN_MASK) + ALIGN); + typep* out = (typep *)(((uint64_t)outp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRCQ_PATH CHK_DIR "/srcq.bin" + #define SRCK_PATH CHK_DIR "/srck.bin" + #define SRCV_PATH CHK_DIR "/srcv.bin" + readBinaryFile(SRCQ_PATH, (uint8_t*)q, B*H*S*qD*sizeof(__half)); + readBinaryFile(SRCK_PATH, (uint8_t*)k, B*H*S*qD*sizeof(__half)); + readBinaryFile(SRCV_PATH, (uint8_t*)v, B*H*S*vD*sizeof(__half)); + #endif + + // uint32_t a; + // uint32_t *b = &a; + // float *c = (float*)b; + // c = c + 1; + // printf("%f\n", *c); + + BENCHSTART; + for(int i=0;i(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(BF16x2) + flash_attention_2d_unroll_hif4<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(BF16x2_NOGATHER) + flash_attention_2d_unroll_hif4_nogather<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(BF16_NOGATHER) + flash_attention_2d_unroll_hif4_nogather<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(OPT) + // support X1 Y4 + flash_attention_2d_unroll_hif4_optsoftmax<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(OPT_LOAD) + // opt lhi -> lwi + flash_attention_2d_unroll_hif4_optsoftmax_loadx2<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(OPT_OFFLOAD) + // support X1 Y4 + flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + #elif defined(OPT_OFFLOAD2) + // support X1 Y4 + flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2<__fp4_e1m2x2, Sq, Skv, qD, vD, kTm, kTk, 16, __bf16x2>(out, q+i*H*Sq*qD+j*Sq*qD, k+i*H*Skv*qD+j*Skv*qD, v+i*H*Skv*vD+j*Skv*vD, qmx, kmx, vmx); + + #endif + } + } + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)out, B*H*S*vD*sizeof(__half)); + #endif +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/fa/fa_hif4.hpp" + + ```cpp title="fa_hif4.hpp" linenums="1" + #ifndef FA_HIF4_HPP + #define FA_HIF4_HPP + + #include + #include + + using namespace pto; + + template + void pto_flash_softmax_block(CastTile &src_exp, MaxTile &new_max, SumTile &new_sum, + ScaleTile &rescale, SrcTile &src, MaxTile &old_max, + SumTile &old_sum) { + SrcTile scaled_src; + TMULS(scaled_src, src, 1.0f / sqrt((float)scaleD)); + + MaxTile local_max; + TCOLMAX(local_max, scaled_src); + TMAX(new_max, old_max, local_max); + + TSUB(rescale, old_max, new_max); + TEXP(rescale, rescale); + + SumTile scaled_old_sum; + TMUL(scaled_old_sum, old_sum, rescale); + + TCOLEXPANDSUB(scaled_src, scaled_src, new_max); + TEXP(scaled_src, scaled_src); + + SumTile local_sum; + TCOLSUM(local_sum, scaled_src); + TADD(new_sum, scaled_old_sum, local_sum); + + TCVT(src_exp, scaled_src); + } + + template + void pto_online_update(OutTile &out, OldTile &old_out, PvTile &pv, ScaleTile &scale) { + TCOLEXPANDMUL(out, old_out, scale); + TADD(out, out, pv); + } + + template + void pto_normalize_by_sum(OutTile &out, SumTile &sum) { + SumTile inv_sum; + TRECIP(inv_sum, sum); + TCOLEXPANDMUL(out, out, inv_sum); + } + + template + void pto_quantize_softmax_to_hif4(QuantTile &dst, SrcTile &src) { + TQUANT(dst, src); + } + + template + void flash_attention_2d_unroll_hif4(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, + dtype* v_ptr, uint8_t* scale_q, + uint8_t* scale_k, uint8_t* scale_v) { + static_assert(qD == vD); + + using gmQ = global_tensor>; + using gmK = global_tensor>; + using gmV = global_tensor>; + using gmO = global_tensor>; + + using gmQScale = global_tensor>; + using gmKScale = global_tensor>; + using gmVScale = global_tensor>; + + using tileQ = TileLeft; + using tileK = TileRight; + using tileV = TileRight; + + using tileQScale = Tile; + using tileKScale = Tile; + using tileVScale = Tile; + + using tileWAcc = TileAcc; + using tileW = Tile; + using tileWCast = Tile; + using tilePHif4 = Tile; + using tilePLeft = TileLeft; + + using tileOAcc = TileAcc; + using tileO = Tile; + using tileOCast = Tile; + + using tileMax = Tile; + using tileSum = Tile; + using tileScale = Tile; + + using itQ = global_iterator; + using itK = global_iterator; + using itV = global_iterator; + using itO = global_iterator; + using itQScale = global_iterator; + using itKScale = global_iterator; + using itVScale = global_iterator; + + itQ gIterQ(q_ptr); + itK gIterK(k_ptr); + itV gIterV(v_ptr); + itO gIterO(out_ptr); + itQScale gIterQScale(scale_q); + itKScale gIterKScale(scale_k); + itVScale gIterVScale(scale_v); + + const int Qb = (Sq + kTm - 1) / kTm; + const int Kb = (Skv + kTk - 1) / kTk; + + #ifdef _2D_UNROLL_PTO + static_assert(Qb % Xdim == 0, "Qb needs to be a multiple of Xdim"); + static_assert(Kb % Ydim == 0, "Kb needs to be a multiple of Ydim"); + #endif + + for (int i = 0; i < Qb; i += Xdim) { + tileQ tQ[Xdim]; + tileQScale tQScale[Xdim]; + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + auto gQ = gIterQ(i + x, 0); + auto gQScale = gIterQScale(i + x, 0); + TLOAD(tQ[x], gQ); + TLOAD(tQScale[x], gQScale); + } + + tileMax tMax[Xdim]; + tileSum tSum[Xdim]; + tileO tO[Xdim]; + tileO tPV[Xdim]; + tileScale tRescale[Xdim]; + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + TEXPANDSCALAR(tMax[x], -1e30f); + TEXPANDSCALAR(tSum[x], 0.0f); + TEXPANDSCALAR(tO[x], 0.0f); + } + + for (int j = 0; j < Kb; j += Ydim) { + tileK tK[Ydim]; + tileKScale tKScale[Ydim]; + + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + auto gK = gIterK(0, j + y); + auto gKScale = gIterKScale(0, j + y); + TLOAD(tK[y], gK); + TLOAD(tKScale[y], gKScale); + } + + tileW tW[Xdim][Ydim]; + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + tileWAcc tWAcc; + TMATMUL_MX(tWAcc, tQ[x], tQScale[x], tK[y], tKScale[y]); + ACCCVT(tW[x][y], tWAcc); + } + } + + tileMax tNewMax[Xdim]; + tileSum tNewSum[Xdim]; + tileWCast tExpW[Xdim][Ydim]; + tilePHif4 tP[Xdim][Ydim]; + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + tileMax tLocalMax[Ydim]; + + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + TCOLMAX(tLocalMax[y], tW[x][y]); + } + + #if Ydim == 1 + TMAX(tNewMax[x], tMax[x], tLocalMax[0]); + #elif Ydim == 2 + tileMax tMax01; + TMAX(tMax01, tLocalMax[0], tLocalMax[1]); + TMAX(tNewMax[x], tMax[x], tMax01); + #elif Ydim == 4 + tileMax tMax01; + tileMax tMax23; + tileMax tMax0123; + TMAX(tMax01, tLocalMax[0], tLocalMax[1]); + TMAX(tMax23, tLocalMax[2], tLocalMax[3]); + TMAX(tMax0123, tMax01, tMax23); + TMAX(tNewMax[x], tMax[x], tMax0123); + #else + static_assert(Ydim == 1 || Ydim == 2 || Ydim == 4, + "PTO HIF4 FA currently supports Ydim 1/2/4"); + #endif + + TSUB(tRescale[x], tMax[x], tNewMax[x]); + TEXP(tRescale[x], tRescale[x]); + + tileSum tScaledOldSum; + TMUL(tScaledOldSum, tSum[x], tRescale[x]); + + tileSum tLocalSum[Ydim]; + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + TCOLEXPANDSUB(tW[x][y], tW[x][y], tNewMax[x]); + TEXP(tW[x][y], tW[x][y]); + TCOLSUM(tLocalSum[y], tW[x][y]); + TCVT(tExpW[x][y], tW[x][y]); + pto_quantize_softmax_to_hif4(tP[x][y], tExpW[x][y]); + } + + #if Ydim == 1 + TADD(tNewSum[x], tScaledOldSum, tLocalSum[0]); + #elif Ydim == 2 + tileSum tSum01; + TADD(tSum01, tLocalSum[0], tLocalSum[1]); + TADD(tNewSum[x], tScaledOldSum, tSum01); + #elif Ydim == 4 + tileSum tSum01; + tileSum tSum23; + tileSum tSum0123; + TADD(tSum01, tLocalSum[0], tLocalSum[1]); + TADD(tSum23, tLocalSum[2], tLocalSum[3]); + TADD(tSum0123, tSum01, tSum23); + TADD(tNewSum[x], tScaledOldSum, tSum0123); + #endif + } + + tileV tV[Ydim]; + tileVScale tVScale[Ydim]; + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + auto gV = gIterV(j + y, 0); + auto gVScale = gIterVScale(j + y, 0); + TLOAD(tV[y], gV); + TLOAD(tVScale[y], gVScale); + } + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + #if Ydim == 1 + tilePLeft tPLeft; + TCVT(tPLeft, tP[x][0]); + tileOAcc tPVAcc; + TMATMUL_MX(tPVAcc, tPLeft, tQScale[x], tV[0], tVScale[0]); + ACCCVT(tPV[x], tPVAcc); + #else + tileO tPVSum; + #pragma clang loop unroll(full) + for (int y = 0; y < Ydim; ++y) { + tilePLeft tPLeft; + TCVT(tPLeft, tP[x][y]); + tileOAcc tPVAcc; + tileO tPVPart; + TMATMUL_MX(tPVAcc, tPLeft, tQScale[x], tV[y], tVScale[y]); + ACCCVT(tPVPart, tPVAcc); + if (y == 0) { + tPVSum = tPVPart; + } else { + TADD(tPVSum, tPVSum, tPVPart); + } + } + tPV[x] = tPVSum; + #endif + + pto_online_update(tO[x], tO[x], tPV[x], tRescale[x]); + } + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + tMax[x] = tNewMax[x]; + tSum[x] = tNewSum[x]; + } + } + + #pragma clang loop unroll(full) + for (int x = 0; x < Xdim; ++x) { + tileOCast tOCast; + pto_normalize_by_sum(tO[x], tSum[x]); + TCVT(tOCast, tO[x]); + auto gO = gIterO(i + x, 0); + TSTORE(gO, tOCast); + } + } + } + + template + void flash_attention_2d_unroll_hif4_nogather(dtype* out_ptr, dtype* q_ptr, dtype* k_ptr, + dtype* v_ptr, uint8_t* scale_q, + uint8_t* scale_k, uint8_t* scale_v) { + flash_attention_2d_unroll_hif4( + out_ptr, q_ptr, k_ptr, v_ptr, scale_q, scale_k, scale_v); + } + + template + void flash_attention_2d_unroll_hif4_optsoftmax(dtype* out_ptr, dtype* q_ptr, + dtype* k_ptr, dtype* v_ptr, + uint8_t* scale_q, uint8_t* scale_k, + uint8_t* scale_v) { + flash_attention_2d_unroll_hif4( + out_ptr, q_ptr, k_ptr, v_ptr, scale_q, scale_k, scale_v); + } + + template + void flash_attention_2d_unroll_hif4_optsoftmax_loadx2(dtype* out_ptr, dtype* q_ptr, + dtype* k_ptr, dtype* v_ptr, + uint8_t* scale_q, uint8_t* scale_k, + uint8_t* scale_v) { + flash_attention_2d_unroll_hif4( + out_ptr, q_ptr, k_ptr, v_ptr, scale_q, scale_k, scale_v); + } + + template + void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload(dtype* out_ptr, dtype* q_ptr, + dtype* k_ptr, dtype* v_ptr, + uint8_t* scale_q, + uint8_t* scale_k, + uint8_t* scale_v) { + flash_attention_2d_unroll_hif4( + out_ptr, q_ptr, k_ptr, v_ptr, scale_q, scale_k, scale_v); + } + + template + void flash_attention_2d_unroll_hif4_optsoftmax_cubeoffload2(dtype* out_ptr, dtype* q_ptr, + dtype* k_ptr, dtype* v_ptr, + uint8_t* scale_q, + uint8_t* scale_k, + uint8_t* scale_v) { + flash_attention_2d_unroll_hif4( + out_ptr, q_ptr, k_ptr, v_ptr, scale_q, scale_k, scale_v); + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TCOLEXPANDMUL`](../../../../intrinsics/tcolexpandmul.md) | `TCOLEXPANDMUL` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TCOLEXPANDSUB`](../../../../intrinsics/tcolexpandsub.md) | `TCOLEXPANDSUB` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TCOLMAX`](../../../../intrinsics/tcolmax.md) | `TCOLMAX` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TCOLSUM`](../../../../intrinsics/tcolsum.md) | `TCOLSUM` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `ACCCVT`, `TCVT` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TEXP`](../../../../intrinsics/texp.md) | `TEXP` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TMATMUL_MX`](../../../../intrinsics/tmatmul_mx.md) | `TMATMUL_MX` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TMAX`](../../../../intrinsics/tmax.md) | `TMAX` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TMUL`](../../../../intrinsics/tmul.md) | `TMUL` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TQUANT`](../../../../intrinsics/tquant.md) | `TQUANT` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TRECIP`](../../../../intrinsics/trecip.md) | `TRECIP` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | +| [`TSUB`](../../../../intrinsics/tsub.md) | `TSUB` | `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/fa` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 26 | `make TESTCASE=fa_HIF4_HIF4 MODE=BF16_NOGATHER Sq=256 Skv=512 Tm=16 Tk=32 X=1 Y=1` | + +## Resolved One-Level Source Closure + +??? info "6 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/fa/fa_hif4.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/src/benchmark.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/fa/src/fa_hif4.cpp` diff --git a/docs/benchmarks/catalog/one-level/fa/index.md b/docs/benchmarks/catalog/one-level/fa/index.md new file mode 100644 index 0000000..74b1b6e --- /dev/null +++ b/docs/benchmarks/catalog/one-level/fa/index.md @@ -0,0 +1,11 @@ +# One-level: fa + + + +This family contains **2** source implementations and +**9** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [fa_2d_unroll](fa-2d-unroll-b46797ec.md) | `benchmark/one-level-arch/test/kernel/fa/src/fa_2d_unroll.cpp` | 8 | [`TADD`](../../../../intrinsics/tadd.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXP`](../../../../intrinsics/texp.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMATMUL`](../../../../intrinsics/tmatmul.md), [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md), [`TMAX`](../../../../intrinsics/tmax.md), [`TMUL`](../../../../intrinsics/tmul.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TRECIP`](../../../../intrinsics/trecip.md), [`TROWEXPANDMUL`](../../../../intrinsics/trowexpandmul.md), [`TROWEXPANDSUB`](../../../../intrinsics/trowexpandsub.md), [`TROWMAX`](../../../../intrinsics/trowmax.md), [`TROWSUM`](../../../../intrinsics/trowsum.md), [`TSTORE`](../../../../intrinsics/tstore.md), [`TSUB`](../../../../intrinsics/tsub.md) | +| [fa_HIF4_HIF4](fa-hif4-86b1f297.md) | `benchmark/one-level-arch/test/kernel/fa/src/fa_hif4.cpp` | 1 | [`TADD`](../../../../intrinsics/tadd.md), [`TCOLEXPANDMUL`](../../../../intrinsics/tcolexpandmul.md), [`TCOLEXPANDSUB`](../../../../intrinsics/tcolexpandsub.md), [`TCOLMAX`](../../../../intrinsics/tcolmax.md), [`TCOLSUM`](../../../../intrinsics/tcolsum.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXP`](../../../../intrinsics/texp.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMATMUL_MX`](../../../../intrinsics/tmatmul_mx.md), [`TMAX`](../../../../intrinsics/tmax.md), [`TMUL`](../../../../intrinsics/tmul.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TQUANT`](../../../../intrinsics/tquant.md), [`TRECIP`](../../../../intrinsics/trecip.md), [`TSTORE`](../../../../intrinsics/tstore.md), [`TSUB`](../../../../intrinsics/tsub.md) | diff --git a/docs/benchmarks/catalog/one-level/gather/gather-4948597c.md b/docs/benchmarks/catalog/one-level/gather/gather-4948597c.md new file mode 100644 index 0000000..a411458 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/gather/gather-4948597c.md @@ -0,0 +1,333 @@ +# gather: gather.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `gather` | +| Implementation source | `benchmark/one-level-arch/test/kernel/gather/src/gather.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then commits the result to global memory. + +```cpp title="gather.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "gather/gather_pto.hpp" + + +#ifndef DType +#define DType float +#endif + +#ifndef OType +#define OType int32_t +#endif + + + +#ifndef gKs +#define gKs 754 +#endif + +#ifndef gMs +#define gMs 17 +#endif + +#ifndef gNs +#define gNs 4096 +#endif + +#ifndef tMs +#define tMs 32 +#endif + +#ifndef tNs +#define tNs 32 +#endif + +// 【静态随机偏移】手动写死，模拟内存地址不是数组起始位置 +#ifndef OFFSET_INs +#define OFFSET_INs 11 // 输入静态偏移 +#endif +#ifndef OFFSET_OUTs +#define OFFSET_OUTs 17 // 输出静态偏移 +#endif +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + using otype = OType; + // ========================== + // 申明空间，留出静态偏移空间 + // ========================== + + // printf("gKs = %d\n",gKs); + // printf("gMs = %d\n",gMs); + // printf("gNs = %d\n",gNs); + // printf("gMs * sizeof(otype) = %d\n",gMs * sizeof(otype)); + // printf("gKs * gNs * sizeof(dtype) = %d\n",gKs * gNs * sizeof(dtype)); + + dtype input_buf[gKs * gNs + OFFSET_INs]; // 前面留空，模拟随机地址 + otype input_offset_buf[gMs + OFFSET_INs]; + dtype output_buf[gNs * gMs + OFFSET_OUTs]; + + // ========================== + // 【静态随机地址】 + // ========================== + dtype* input = input_buf + OFFSET_INs; + otype* input_offset = input_offset_buf + OFFSET_INs; + dtype* output = output_buf + OFFSET_OUTs; + + // ========================== + // 从 bin 文件读取输入数据 + // ========================== + // printf("---cpp: start to read input data---"); + // printf("---cpp: start to read input data---"); + + #ifdef RES_CHECK + #define INPUT_PATH CHK_DIR "/input.bin" + #define INPUT_OFFSET_PATH CHK_DIR "/input_offset.bin" + #define OUTPUT_PATH CHK_DIR "/output.bin" + readBinaryFile(INPUT_PATH, (uint8_t*)input, gKs * gNs * sizeof(dtype)); + readBinaryFile(INPUT_OFFSET_PATH, (uint8_t*)input_offset, gMs * sizeof(otype)); + + // printf("input_offset[200] = %d\n",input_offset[200]); + // printf("input_offset[1] = %d\n",input_offset[1]); + // printf("input_offset[2] = %d\n",input_offset[2]); + // printf("input_offset[3] = %d\n",input_offset[3]); + + // printf("input[200] = %f\n",input[200]); + // printf("input[1] = %f\n",input[1]); + // printf("input[2] = %f\n",input[2]); + // printf("input[3] = %f\n",input[3]); + printf("input[351*17] = %f\n",input[351*17]); + #endif + + gather(input, input_offset, output); + + #ifdef RES_CHECK + writeBinaryFile(OUTPUT_PATH, (uint8_t*)output, gMs * gNs * sizeof(dtype)); + #endif + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/gather/gather_pto.hpp" + + ```cpp title="gather_pto.hpp" linenums="1" + // ============================================================================ + // Gather 算子 — PTO 一层编程模型 + // + // 原始 gather.hpp 策略: + // 对每个输出 tile (tM, tN): + // 1. TLOAD 加载 offset tile (行索引, 1×tM) from GM + // 2. __vec__ gen_offset 计算字节偏移: + // offset[row,col] = (in_offset[row] * gN + n_base + col) * sizeof(dtype) + // 3. MGATHER 按字节偏移从数据表取数 (旧 MGATHER, 字节偏移语义) + // 4. TSTORE 写回输出 + // + // PTO 一层策略: + // 对每个输出 tile (tM, tN): + // 1. TLOAD 加载 offset tile (行索引, 1×tM) from GM + // 2. MGATHER 按行索引直接取数: + // dst[r,:] = table[idx[r], :] + // 通过调整 table 指针 (+n_base) 处理列偏移 + // (PTO ISA MGATHER 使用行索引, 非 字节偏移, 无需 gen_offset) + // 3. TSTORE 写回输出 + // + // gen_offset __vec__ 块完全消除: MGATHER 内部完成 + // 行索引 → 行地址 的转换 (tablePtr + idx * tableRowStride)。 + // + // ┌─────────────────────────────────────────────────────────────────────────┐ + // │ 当前编译器不支持 / 不完整的指令汇总 │ + // ├──────────┬──────────────────┬──────────────────────────────────────────┤ + // │ Pto ISA │ 当前编译器状态 │ 说明 │ + // │ 指令 │ │ │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TLOAD │ API 有(名不同)， │ PTO ISA 名 TLOAD；当前编译器名 TLOAD； │ + // │ │ │ 核心参数 (dst, src) 和行为一致 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ MGATHER │ 部分支持 │ template_asm.h 有 MGATHER (asm volatile);│ + // │ │ │ 但缺 Coalesce 模板参数 (无法选 Row/Elem);│ + // │ │ │ 且旧实现按字节偏移取数, │ + // │ │ │ PTO ISA Coalesce::Row 按行索引取数 │ + // ├──────────┼──────────────────┼──────────────────────────────────────────┤ + // │ TSTORE │ API 有(名不同)， │ PTO ISA 名 TSTORE；当前编译器名 TSTORE；│ + // │ │ │ 核心参数 (dst, src) 和行为一致 │ + // └──────────┴──────────────────┴──────────────────────────────────────────┘ + // + // ============================================================================ + + #include + #include + // #include // [!] PTO ISA C++ Intrinsic — 当前编译器未提供 + + #include + #include + + // ---------------------------------------------------------------------------- + // gather: 按行索引从数据表中 gather 行 + // + // 输入: + // in_data_ptr — 数据表, shape (gK, gN), row-major + // in_offset_ptr — 行索引数组, shape (gM,), 每个元素是数据表的行号 + // out_ptr — 输出, shape (gM, gN), out[j,:] = in_data[in_offset[j],:] + // + // 模板参数 (与原 gather.hpp 一致): + // dtype — 数据类型 (float, half, ...) + // otype — 索引类型 (uint32_t, int32_t) + // gK — 数据表行数 + // gM — 输出行数 (= 索引数组长度) + // gN — 数据表列数 (= 每行元素数) + // tM — tile 行数 + // tN — tile 列数 + // ---------------------------------------------------------------------------- + template + void gather( + dtype *in_data_ptr, + otype *in_offset_ptr, + dtype *out_ptr + ) { + const size_t Mb = gM / tM; + const size_t Nb = gN / tN; + const size_t rmd_M = gM % tM; + const size_t rmd_N = gN % tN; + + using gm_shapeInOffset = global_tensor>; + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + + using tile_shapeInOffset = Tile; + using tile_shapeData = Tile; + using tile_shapeInOffset_rmd_n = Tile; + using tile_shapeData_rmd_n = Tile; + using tile_shapeInOffset_rmd_mn = Tile; + using tile_shapeData_rmd_mn = Tile; + using tile_shapeInOffset_rmd_m = Tile; + using tile_shapeData_rmd_m = Tile; + + tile_shapeInOffset inOffsetTile; + tile_shapeData outTile; + tile_shapeInOffset_rmd_n inOffsetTile_rmd_n; + tile_shapeData_rmd_n outTile_rmd_n; + tile_shapeInOffset_rmd_mn inOffsetTile_rmd_mn; + tile_shapeData_rmd_mn outTile_rmd_mn; + tile_shapeInOffset_rmd_m inOffsetTile_rmd_m; + tile_shapeData_rmd_m outTile_rmd_m; + + using itInOffset = global_iterator; + using itOut = global_iterator; + + itInOffset gInOffsetIter(in_offset_ptr); + itOut gOIter(out_ptr); + + // ---- 主循环: Mb × Nb 个完整 tile ---- + for (int j = 0; j < Mb; ++j) { + for (int i = 0; i < Nb; ++i) { + auto gInOffset = gInOffsetIter(0, j); + auto gO = gOIter(j, i); + size_t n_base = i * tN; + + // TLOAD: 加载行索引 tile (1, tM) from GM + // [当前编译器] 名为 TLOAD + TLOAD(inOffsetTile, gInOffset); + + // MGATHER: 按行索引从数据表取数 + // dst[r,:] = table[idx[r], :] + // table 指针偏移 n_base 个元素, 使取数起始列为 n_base + // (tablePtr + idx * gN + n_base 定位到 row idx, col n_base) + // [当前编译器] template_asm.h 的 MGATHER 无 Coalesce 模板参数; + // 且按字节偏移取数, 非行索引 + gm_shapeIn adjustedGm(in_data_ptr + n_base); + MGATHER(outTile, adjustedGm, inOffsetTile); + + // TSTORE: 写回输出 tile (tM, tN) to GM + // [当前编译器] 名为 TSTORE + TSTORE(gO, outTile); + } + + // ---- rmd_N: 最后一个列块不完整 ---- + if constexpr (rmd_N) { + auto gInOffset = gInOffsetIter(0, j); + auto gO = gOIter(j, Nb); + size_t n_base = Nb * tN; + + TLOAD(inOffsetTile_rmd_n, gInOffset); + gm_shapeIn adjustedGm(in_data_ptr + n_base); + MGATHER(outTile_rmd_n, adjustedGm, inOffsetTile_rmd_n); + TSTORE(gO, outTile_rmd_n); + } + } + + // ---- rmd_M: 最后一个行块不完整 ---- + if constexpr (rmd_M) { + for (int i = 0; i < Nb; ++i) { + auto gInOffset = gInOffsetIter(0, Mb); + auto gO = gOIter(Mb, i); + size_t n_base = i * tN; + + TLOAD(inOffsetTile_rmd_m, gInOffset); + gm_shapeIn adjustedGm(in_data_ptr + n_base); + MGATHER(outTile_rmd_m, adjustedGm, inOffsetTile_rmd_m); + TSTORE(gO, outTile_rmd_m); + } + + // ---- rmd_M + rmd_N: 右下角不完整 ---- + if constexpr (rmd_N) { + auto gInOffset = gInOffsetIter(0, Mb); + auto gO = gOIter(Mb, Nb); + size_t n_base = Nb * tN; + + TLOAD(inOffsetTile_rmd_mn, gInOffset); + gm_shapeIn adjustedGm(in_data_ptr + n_base); + MGATHER(outTile_rmd_mn, adjustedGm, inOffsetTile_rmd_mn); + TSTORE(gO, outTile_rmd_mn); + } + } + } + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MGATHER`](../../../../intrinsics/mgather.md) | `MGATHER` | `benchmark/one-level-arch/kernels/gather/gather_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/gather/gather_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/gather/gather_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/gather` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 26 | `make TESTCASE=gather COMPILER_DIR=$COMPILER_DIR DType=__fp32 OType=uint32_t gKs=131072 gMs=32 gNs=256 tMs=32 tNs=64` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/gather/gather_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/gather/src/gather.cpp` diff --git a/docs/benchmarks/catalog/one-level/gather/index.md b/docs/benchmarks/catalog/one-level/gather/index.md new file mode 100644 index 0000000..4afc9b2 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/gather/index.md @@ -0,0 +1,10 @@ +# One-level: gather + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [gather](gather-4948597c.md) | `benchmark/one-level-arch/test/kernel/gather/src/gather.cpp` | 1 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md b/docs/benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md new file mode 100644 index 0000000..98fa2c7 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/matmul/a16w4-f1f175ff.md @@ -0,0 +1,1859 @@ +# matmul: A16W4.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `matmul` | +| Implementation source | `benchmark/one-level-arch/test/kernel/matmul/src/A16W4.cpp` | +| Active build variants | 3 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then executes matrix or matrix-vector work on cube/accumulator tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="A16W4.cpp" linenums="1" +#include +#include +#include "fileop.h" +#include "common.h" +#include "benchmark.h" + +#ifndef globM +#define globM 120 +#endif + +#ifndef globN +#define globN 120 +#endif + +#ifndef globK +#define globK 120 +#endif + +#ifndef tilM +#define tilM 16 +#endif + +#ifndef tilN +#define tilN 16 +#endif + +#ifndef tilK +#define tilK 16 +#endif + +#ifndef Batch +#define Batch 1 +#endif + +#define ALIGN_MASK 0xfffffffffffff000ull +#define ALIGN 4*1024 + +#include "matmul/matmul_mx.hpp" + + +int main() { + // bf16*fp4 + using fp4_t = __fp4_e2m1x2; + using bf16_t = __bf16; + static_assert(tilM % 2 == 0); // 暂时假定tile是偶数的，方便取地址，奇数tile实现需要末尾padding 0对齐地址 + static_assert(tilN % 2 == 0); + static_assert(tilK == (128)); + bf16_t src0[(globM+1)/2*(globK+1)/2*4]; // 保证是偶数M,K,N,奇数MKN末尾pad0 + fp4_t src1[(globK+1)/2*(globN+1)/2*4]; + float src2[(globK+1)/2*(globN+1)/2*4]; + float dst[(globM+1)/2*(globN+1)/2*4]; + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(bf16_t)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(fp4_t)); + readBinaryFile(SRC1_PATH, (uint8_t*)src2, globK*globN*sizeof(float)); + #endif + + BENCHSTART; + matmul_mp(dst, src0, src1, src2); + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)dst, globM*globN*sizeof(float)); + #endif + + return 0; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp" + + ```cpp title="matmul_mx.hpp" linenums="1" + #ifndef MATMUL_MX_HPP + #define MATMUL_MX_HPP + + #include + #ifdef LINXISA_MATMUL_MP_TILEOP + #include + + namespace pto::blkv { + template + void blkv_for_2d(uint32_t, uint32_t, Fn &&); + } // namespace pto::blkv + + uint16_t blkv_get_index_x(); + uint16_t blkv_get_index_y(); + + template + void gen_ND2ZZ_offset_Impl(Gm &, Tile &, OffsetTile &, uint32_t, uint32_t); + + template + void gen_ND2NN_offset_Impl(Gm &, Tile &, OffsetTile &, uint32_t, uint32_t); + #else + #include + #include "template_asm.h" + #endif + #include + #include + + + // #define DUMP_TILE(label, TileVar, DumpBuf, Rows, Cols) \ + // do { \ + // GlobalTensor, \ + // Stride<1,1,1,Cols,1>> _g(DumpBuf); \ + // TSTORE(_g, TileVar); \ + // printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ + // for (int ri = 0; ri < Rows; ri++) { \ + // printf(" row%2d: ", ri); \ + // for (int ci = 0; ci < Cols; ci++) \ + // printf("%.4f ", DumpBuf[ri * Cols + ci]); \ + // printf("\n"); \ + // } \ + // fflush(stdout); \ + // } while (0) + + #ifndef Batch + #define Batch 1 + #endif + + using namespace pto; + + // TODO, move to utils.cpp + template + void TSTORE_ACC(GmOut &Gout, TileAcc &tAcc){ + using TileAccOut = Tile; + TileAccOut tAccOut; + if constexpr (TileAcc::Loc == Location::Acc) { + ACCCVT(tAccOut, tAcc); + } else { + TCVT(tAccOut, tAcc); + } + TSTORE(Gout, tAccOut); + } + + // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， + // smatrix_wfactor : scaling matrix 与计算matrix位宽比 + template + void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + // static_assert(gM % tM == 0); + // static_assert(gN % tN == 0); + // static_assert(gK % tK == 0); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + gm_shapeAMX gAMX(src0_mx); + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + using tile_ND2ZZOffset = Tile; + tile_ND2ZZOffset nd2zz_offset; + + using gm_shapeBMX = global_tensor>; + gm_shapeBMX gBMX(src1_mx); + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + using tile_ND2NNOffset = Tile; + tile_ND2NNOffset nd2nn_offset; + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i(gAMX, tAMX, nd2zz_offset, i, k); + // DUMP_TILE("111", nd2zz_offset, g_dump_intTile, 1, tM*tK/smatrix_wfactor); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(i,Kb); + auto gB = gBIter(Kb,j); + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_trows tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); + // DUMP_TILE("111", nd2zz_offset, g_dump_intTile, 1, tM*tK/smatrix_wfactor); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcols tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + tile_shapeC_trows tACC; + for(int k=0;k(gAMX, tAMX, nd2zz_offset, i, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_trows tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_trows tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_trows tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcorner tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcols tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcorner tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcols tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcols tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_trows tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcorner tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcorner tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + } + + template + void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + static_assert(typeb_wfactor == 1 ); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + tile_shapeC_trows tACC; + for(int k=0;k0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, k); + auto gBMX = gBMXIter(k, j); + tile_shapeAMX_tcols tAMX; + tile_shapeBMX tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, Kb); + auto gBMX = gBMXIter(Kb, j); + tile_shapeAMX_tcorner tAMX; + tile_shapeBMX_tcols tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, k); + auto gBMX = gBMXIter(k, Nb); + tile_shapeAMX_tcols tAMX; + tile_shapeBMX_trows tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, Kb); + auto gBMX = gBMXIter(Kb, Nb); + tile_shapeAMX_tcorner tAMX; + tile_shapeBMX_tcorner tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + } + + struct ResA { + int m; + int k; + int val; + }; + + struct ResB { + int n; + int k; + int val; + }; + + constexpr ResA find_reuseA(int Mb, int Kb, int MAX_TILE_NUM) { + int best_m = 0, best_k = 0, best_val = -1; + + #pragma clang loop unroll(full) + for (int m = 1; m <= Mb; ++m) { + #pragma clang loop unroll(full) + for (int k = 1; k <= Kb; ++k) { + int v = m * k; + if (v <= MAX_TILE_NUM && v > best_val) { + best_val = v; + best_m = m; + best_k = k; + } + } + } + return {best_m, best_k, best_val}; + } + + template + void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + static_assert(typeb_wfactor == 1 ); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + static const uint32_t MAX_TILE_NUM = 24; // TODO, check this value + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; + using itAMX = global_iterator; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // 计算复用因子 + constexpr ResA R = find_reuseA(Mb < 1 ? 1 : Mb, Kb, MAX_TILE_NUM); + const int dM = R.m == 0 ? 0 : Mb / R.m; + const int rM = R.m == 0 ? 0 : Mb % R.m; + + static_assert(R.val <= MAX_TILE_NUM, "R.val is bigger than MAX_TILE_NUM"); + + #pragma clang loop unroll(full) + for(int i=0; i0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + + auto gC = gCIter(i*R.m+ii, j); + TSTORE_ACC(gC, tACC); + } + + // [m, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + + auto gC = gCIter(i*R.m+ii, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + if constexpr(rM>0){ + tile_shapeA tA[rM][R.k]; + tile_shapeAMX tAMX[rM][R.k]; + + #pragma clang loop unroll(full) + for(int i=0; i0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(i+dM*R.m, j); + TSTORE_ACC(gC, tACC); + } + + // [rM, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(i+dM*R.m, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + // [rmd_M, n, k] + if constexpr (rmd_M) { + tile_shapeA_tcols tA[R.k]; + tile_shapeAMX_tcols tAMX[R.k]; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(Mb, j); + TSTORE_ACC(gC, tACC); + } + + // [rmd_M, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(Mb, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， + // smatrix_wfactor : scaling matrix 与计算matrix位宽比 + template + void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + static_assert(gM % tM == 0); + static_assert(gN % tN == 0); + static_assert(gK % tK == 0); + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + // gm_shapeAMX gAMX(src0_mx); + // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + // using tile_ND2ZZOffset = Tile; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + // gm_shapeBMX gBMX(src1_mx); + // using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using itBMX = global_iterator; + // using tile_ND2NNOffset = Tile; + // tile_ND2NNOffset nd2nn_offset; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i + void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + static_assert(gM % tM == 0); + static_assert(gN % tN == 0); + static_assert(gK % tK == 0); + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + // gm_shapeAMX gAMX(src0_mx); + // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + // using tile_ND2ZZOffset = Tile; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + // gm_shapeBMX gBMX(src1_mx); + // using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using itBMX = global_iterator; + // using tile_ND2NNOffset = Tile; + // tile_ND2NNOffset nd2nn_offset; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i + void __vec__ dequant_acc( + typename psum_tensor::TileDType __in__ data, + typename scale_tensor::TileDType __in__ scale, + typename out_tensor::TileDType __in__ adder, + typename out_tensor::TileDType __out__ out + ) { + + uint16_t x = blkv_get_index_x(); + uint16_t y = blkv_get_index_y(); + uint16_t src_idx = x * psum_tensor::RowStride + y * psum_tensor::ColStride; + // uint16_t scale_idx = (x/128) * scale_tensor::RowStride + y * scale_tensor::ColStride; + uint16_t scale_idx = y * scale_tensor::ColStride; + blkv_get_tile_ptr(out)[src_idx] = blkv_get_tile_ptr(data)[src_idx] * blkv_get_tile_ptr(scale)[scale_idx] + blkv_get_tile_ptr(adder)[src_idx]; + } + + template + void __vec__ dequant_acc_new( + typename psum_tensor::TileDType __in__ data, + typename scale_tensor::TileDType __in__ scale, + typename out_tensor::TileDType __in__ adder, + typename out_tensor::TileDType __out__ out + ) { + + uint16_t x = blkv_get_index_x(); + uint16_t y = blkv_get_index_y(); + uint16_t src_idx = x * psum_tensor::RowStride + y * psum_tensor::ColStride; + // uint16_t scale_idx = (x/128) * scale_tensor::RowStride + y * scale_tensor::ColStride; + uint16_t scale_idx = y * scale_tensor::ColStride; + blkv_get_tile_ptr(out)[src_idx] = blkv_get_tile_ptr(data)[src_idx] * blkv_get_tile_ptr(scale)[scale_idx] + blkv_get_tile_ptr(adder)[src_idx]; + } + + template + void dequant_acc_tileop(OutTile &out, PsumTile &data, ScaleTile &scale, OutTile &adder) { + OutTile expanded_scale; + OutTile scaled; + TCOLEXPAND(expanded_scale, scale); + TMUL(scaled, data, expanded_scale); + TADD(out, scaled, adder); + } + + // mixed precision matmul dequant(A*B)， fp4x2 width_factor: 2 + template + void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { + static_assert(gK % (128) == 0); + static_assert(tK == 128); + static const uint32_t trow = (tM >= 16) ? tM : 16; + static const uint32_t tcol = (tN >= 16) ? tN : 16; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + // 伪量化固定float, group 大小128， 128个fp4共享一个scaling factor, 128的partial sum* scale + using gm_shape_scale = global_tensor>; + using gm_shapeACC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shape_scale = Tile; + using tile_shape_dequant = Tile; + using tile_shapeACC = TileAcc; + // copy of acc, input as vector + using tile_ACCin = Tile; + + using itA = global_iterator; + using itB = global_iterator; + using itScale = global_iterator; + using itACC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itScale gScaleIter(c_ptr); + itACC gACCIter(acc_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + using tile_shape_scale_trows = Tile; + using tile_shape_scale_tcols = Tile; + using tile_shape_scale_tcorner = Tile; + + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; + + using tile_shape_dequant_trows = Tile; + using tile_shape_dequant_tcols = Tile; + using tile_shape_dequant_tcorner = Tile; + for(int i=0;i1 , 256 -> 2 scaling factor + // static_assert(tile_shapeB::ValidCol % (width_factor*128) == 0); // TODO, 暂不考虑padding，假设形状是规整的, 方便处理, taccin*ts_adder=tc_dequant + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + k++; + if constexpr (rmd_K) { + // to do check acc shape + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, j); + auto gS = gScaleIter(Kb,j); + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + tile_shape_scale_tcols ts; + tile_shape_dequant tC_dequant; + + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + TSTORE(gACC, tAdder[(k+1)%2]); + } + // if constexpr (rmd_N) // TODO + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gACC = gACCIter(Mb, j); + tile_shapeC_tcols tACC; + tile_ACCin_tcols tACCin; + tile_shape_dequant_tcols tAdder[2]; + TEXPANDSCALAR(tAdder[0], 0.); + int k = 0; + #pragma clang loop unroll(full) + for (; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + auto gS = gScaleIter(k,j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + tile_shape_scale ts; + tile_shape_dequant_tcols tC_dequant; + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + k++; + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + auto gS = gScaleIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + tile_shape_scale_tcols ts; + tile_shape_dequant tC_dequant; + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + TSTORE(gACC, tAdder[(k+1)%2]); + } + // todo + // if constexpr (rmd_N) { + // auto gC = gCIter(Mb, Nb); + + // tile_shapeC_tcorner tACC; + // if constexpr(Kb>0){ + // auto gA = gAIter(Mb, 0); + // auto gB = gBIter(0, Nb); + + // tile_shapeA_tcols tA; + // tile_shapeB_trows tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMUL(tACC, tA, tB); + // } + // #pragma clang loop unroll(full) + // for (int k = 1; k < Kb; ++k) { + // auto gA = gAIter(Mb, k); + // auto gB = gBIter(k, Nb); + + // tile_shapeA_tcols tA; + // tile_shapeB_trows tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMACC(tACC, tA, tB); + // } + // if constexpr (rmd_K) { + // auto gA = gAIter(Mb, Kb); + // auto gB = gBIter(Kb, Nb); + + // tile_shapeA_tcorner tA; + // tile_shapeB_tcorner tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // if constexpr(Kb>0){ + // MATMACC(tACC, tA, tB); + // } else { + // MATMUL(tACC, tA, tB); + // } + // } + // TSTORE_ACC(gC, tACC); + // } + } + } + + // mixed precision matmul dequant(A*B), fp4x2 width_factor: 2, using LinxISA tileops for dequant accumulation. + template + void matmul_mp_tileop(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { + static_assert(gK % (128) == 0); + static_assert(tK == 128); + static const uint32_t trow = (tM >= 16) ? tM : 16; + static const uint32_t tcol = (tN >= 16) ? tN : 16; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shape_scale = global_tensor>; + using gm_shapeACC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shape_scale = Tile; + using tile_shape_dequant = Tile; + using tile_shapeACC = TileAcc; + using tile_ACCin = Tile; + + using itA = global_iterator; + using itB = global_iterator; + using itScale = global_iterator; + using itACC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itScale gScaleIter(c_ptr); + itACC gACCIter(acc_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + using tile_shape_scale_trows = Tile; + using tile_shape_scale_tcols = Tile; + using tile_shape_scale_tcorner = Tile; + + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; + + using tile_shape_dequant_trows = Tile; + using tile_shape_dequant_tcols = Tile; + using tile_shape_dequant_tcorner = Tile; + for(int i=0;i + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `matmul` | +| Implementation source | `benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp` | +| Active build variants | 4 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then executes matrix or matrix-vector work on cube/accumulator tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="HiF4_HiF4.cpp" linenums="1" +#include +#include +#include "fileop.h" +#include "common.h" +#include "benchmark.h" + +#ifndef globM +#define globM 120 +#endif + +#ifndef globN +#define globN 120 +#endif + +#ifndef globK +#define globK 120 +#endif + +#ifndef tilM +#define tilM 16 +#endif + +#ifndef tilN +#define tilN 16 +#endif + +#ifndef tilK +#define tilK 16 +#endif + +#ifndef Batch +#define Batch 1 +#endif + +#define ALIGN_MASK 0xfffffffffffff000ull +#define ALIGN 4*1024 + +#include "matmul/matmul_mx.hpp" + + +int main() { + using fp4_t = __fp4_hif4x2; + constexpr int globKv = globK / 2; + constexpr int tilKv = tilK / 2; + static_assert(tilM % 2 == 0); // 暂时假定tile是偶数的，方便取地址，奇数tile实现需要末尾padding 0对齐地址 + static_assert(tilN % 2 == 0); + static_assert(tilKv % 2 == 0); + fp4_t src0p[(globM+1)/2*(globK+1)/2*4 + 2*ALIGN]; // 保证是偶数M,K,N,奇数MKN末尾pad0, ALGIN保证地址256B对齐 + fp4_t src1p[(globK+1)/2*(globN+1)/2*4 + 2*ALIGN]; + uint8_t src0_mxp[(globM+1)/2*(globK+1)/2*4/16 + 2*ALIGN]; //scaling matrix 64元素共享 4 Bytes + uint8_t src1_mxp[(globM+1)/2*(globK+1)/2*4/16 + 2*ALIGN]; + float dstp[(globM+1)/2*(globN+1)/2*4]; + fp4_t *src0 = (fp4_t *)(((uint64_t)src0p & ALIGN_MASK) + ALIGN); + fp4_t *src1 = (fp4_t *)(((uint64_t)src1p & ALIGN_MASK) + ALIGN); + uint8_t *src0_mx = (uint8_t *)(((uint64_t)src0_mxp & ALIGN_MASK) + ALIGN); + uint8_t *src1_mx = (uint8_t *)(((uint64_t)src1_mxp & ALIGN_MASK) + ALIGN); + float *dst = (float*)(((uint64_t)dstp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(fp4_t)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(fp4_t)); + #endif + + BENCHSTART; + #ifdef NOMX_NOGATHER + matmul_fp_notcvt(dst, src0, src1, src0_mx, src1_mx); + #elif MX_NOGATHER + matmul_mxfp_notcvt(dst, src0, src1, src0_mx, src1_mx); + #elif MX_NOGATHER_REUSEA + matmul_mxfp_notcvt_reuseA(dst, src0, src1, src0_mx, src1_mx); + #else + matmul_mxfp(dst, src0, src1, src0_mx, src1_mx); + #endif + // matmul_mxfp_notcvt(dst, src0, src1, src0_mx, src1_mx); + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)dst, globM*globN*sizeof(float)); + #endif + + return 0; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/matmul/matmul_mx.hpp" + + ```cpp title="matmul_mx.hpp" linenums="1" + #ifndef MATMUL_MX_HPP + #define MATMUL_MX_HPP + + #include + #ifdef LINXISA_MATMUL_MP_TILEOP + #include + + namespace pto::blkv { + template + void blkv_for_2d(uint32_t, uint32_t, Fn &&); + } // namespace pto::blkv + + uint16_t blkv_get_index_x(); + uint16_t blkv_get_index_y(); + + template + void gen_ND2ZZ_offset_Impl(Gm &, Tile &, OffsetTile &, uint32_t, uint32_t); + + template + void gen_ND2NN_offset_Impl(Gm &, Tile &, OffsetTile &, uint32_t, uint32_t); + #else + #include + #include "template_asm.h" + #endif + #include + #include + + + // #define DUMP_TILE(label, TileVar, DumpBuf, Rows, Cols) \ + // do { \ + // GlobalTensor, \ + // Stride<1,1,1,Cols,1>> _g(DumpBuf); \ + // TSTORE(_g, TileVar); \ + // printf("[DUMP] %s (shape=%dx%d):\n", label, Rows, Cols); \ + // for (int ri = 0; ri < Rows; ri++) { \ + // printf(" row%2d: ", ri); \ + // for (int ci = 0; ci < Cols; ci++) \ + // printf("%.4f ", DumpBuf[ri * Cols + ci]); \ + // printf("\n"); \ + // } \ + // fflush(stdout); \ + // } while (0) + + #ifndef Batch + #define Batch 1 + #endif + + using namespace pto; + + // TODO, move to utils.cpp + template + void TSTORE_ACC(GmOut &Gout, TileAcc &tAcc){ + using TileAccOut = Tile; + TileAccOut tAccOut; + if constexpr (TileAcc::Loc == Location::Acc) { + ACCCVT(tAccOut, tAcc); + } else { + TCVT(tAccOut, tAcc); + } + TSTORE(Gout, tAccOut); + } + + // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， + // smatrix_wfactor : scaling matrix 与计算matrix位宽比 + template + void matmul_mxfp(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + // static_assert(gM % tM == 0); + // static_assert(gN % tN == 0); + // static_assert(gK % tK == 0); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + gm_shapeAMX gAMX(src0_mx); + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + using tile_ND2ZZOffset = Tile; + tile_ND2ZZOffset nd2zz_offset; + + using gm_shapeBMX = global_tensor>; + gm_shapeBMX gBMX(src1_mx); + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + using tile_ND2NNOffset = Tile; + tile_ND2NNOffset nd2nn_offset; + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i(gAMX, tAMX, nd2zz_offset, i, k); + // DUMP_TILE("111", nd2zz_offset, g_dump_intTile, 1, tM*tK/smatrix_wfactor); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(i,Kb); + auto gB = gBIter(Kb,j); + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_trows tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); + // DUMP_TILE("111", nd2zz_offset, g_dump_intTile, 1, tM*tK/smatrix_wfactor); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcols tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + tile_shapeC_trows tACC; + for(int k=0;k(gAMX, tAMX, nd2zz_offset, i, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_trows tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_trows tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_trows tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, i, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcorner tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcols tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcorner tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcols tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, j); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcols tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, k); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_trows tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, k, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + tile_shapeAMX_tcorner tAMX; + gen_ND2ZZ_offset_Impl(gAMX, tAMX, nd2zz_offset, Mb, Kb); + MGATHER(tAMX, gAMX, nd2zz_offset); // ND2ZZ, tile masked + tile_shapeBMX_tcorner tBMX; + gen_ND2NN_offset_Impl(gBMX, tBMX, nd2nn_offset, Kb, Nb); + MGATHER(tBMX, gBMX, nd2nn_offset); // ND2NN, tile masked + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + } + + template + void matmul_mxfp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + static_assert(typeb_wfactor == 1 ); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + tile_shapeC_trows tACC; + for(int k=0;k0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, k); + auto gBMX = gBMXIter(k, j); + tile_shapeAMX_tcols tAMX; + tile_shapeBMX tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, Kb); + auto gBMX = gBMXIter(Kb, j); + tile_shapeAMX_tcorner tAMX; + tile_shapeBMX_tcols tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, k); + auto gBMX = gBMXIter(k, Nb); + tile_shapeAMX_tcols tAMX; + tile_shapeBMX_trows tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + + if(k==0){ + MATMULMX(tACC, tA, tAMX, tB, tBMX); + }else{ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + + auto gAMX = gAMXIter(Mb, Kb); + auto gBMX = gBMXIter(Kb, Nb); + tile_shapeAMX_tcorner tAMX; + tile_shapeBMX_tcorner tBMX; + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + if constexpr(Kb>0){ + MATMACCMX(tACC, tA, tAMX, tB, tBMX); + } else { + MATMULMX(tACC, tA, tAMX, tB, tBMX); + } + } + TSTORE_ACC(gC, tACC); + } + } + } + + struct ResA { + int m; + int k; + int val; + }; + + struct ResB { + int n; + int k; + int val; + }; + + constexpr ResA find_reuseA(int Mb, int Kb, int MAX_TILE_NUM) { + int best_m = 0, best_k = 0, best_val = -1; + + #pragma clang loop unroll(full) + for (int m = 1; m <= Mb; ++m) { + #pragma clang loop unroll(full) + for (int k = 1; k <= Kb; ++k) { + int v = m * k; + if (v <= MAX_TILE_NUM && v > best_val) { + best_val = v; + best_m = m; + best_k = k; + } + } + } + return {best_m, best_k, best_val}; + } + + template + void matmul_mxfp_notcvt_reuseA(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + static_assert(typeb_wfactor == 1 ); + static const uint32_t valid_row = (tM > gM) ? gM : tM; + static const uint32_t valid_col = (tN > gN) ? gN : tN; + static const uint32_t MAX_TILE_NUM = 24; // TODO, check this value + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + using tile_shapeAMX = Tile; + using itAMX = global_iterator; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + using tile_shapeBMX = Tile; + using itBMX = global_iterator; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeAMX_trows = Tile; + using tile_shapeAMX_tcols = Tile; + using tile_shapeAMX_tcorner = Tile; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeBMX_trows = Tile; + using tile_shapeBMX_tcols = Tile; + using tile_shapeBMX_tcorner = Tile; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // 计算复用因子 + constexpr ResA R = find_reuseA(Mb < 1 ? 1 : Mb, Kb, MAX_TILE_NUM); + const int dM = R.m == 0 ? 0 : Mb / R.m; + const int rM = R.m == 0 ? 0 : Mb % R.m; + + static_assert(R.val <= MAX_TILE_NUM, "R.val is bigger than MAX_TILE_NUM"); + + #pragma clang loop unroll(full) + for(int i=0; i0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + + auto gC = gCIter(i*R.m+ii, j); + TSTORE_ACC(gC, tACC); + } + + // [m, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + + auto gC = gCIter(i*R.m+ii, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + if constexpr(rM>0){ + tile_shapeA tA[rM][R.k]; + tile_shapeAMX tAMX[rM][R.k]; + + #pragma clang loop unroll(full) + for(int i=0; i0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(i+dM*R.m, j); + TSTORE_ACC(gC, tACC); + } + + // [rM, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(i+dM*R.m, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + // [rmd_M, n, k] + if constexpr (rmd_M) { + tile_shapeA_tcols tA[R.k]; + tile_shapeAMX_tcols tAMX[R.k]; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(Mb, j); + TSTORE_ACC(gC, tACC); + } + + // [rmd_M, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + + #pragma clang loop unroll(full) + for(int k=0; k0){ + MATMACCMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } else { + MATMULMX(tACC, tA_tmp, tAMX_tmp, tB, tBMX); + } + } + auto gC = gCIter(Mb, Nb); + TSTORE_ACC(gC, tACC); + } + } + } + + // typeb_wfactor 表明typeA和typeB的位宽比例，比如fp8是fp4x2的两倍， + // smatrix_wfactor : scaling matrix 与计算matrix位宽比 + template + void matmul_mxfp_notcvt_old(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + static_assert(gM % tM == 0); + static_assert(gN % tN == 0); + static_assert(gK % tK == 0); + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + // gm_shapeAMX gAMX(src0_mx); + // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + // using tile_ND2ZZOffset = Tile; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + // gm_shapeBMX gBMX(src1_mx); + // using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using itBMX = global_iterator; + // using tile_ND2NNOffset = Tile; + // tile_ND2NNOffset nd2nn_offset; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i + void matmul_fp_notcvt(float *dst, dtypeA *src0, dtypeB *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + // only support regular shape now for this operator! + static_assert(gM % tM == 0); + static_assert(gN % tN == 0); + static_assert(gK % tK == 0); + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + using gm_shapeAMX = global_tensor>; + // gm_shapeAMX gAMX(src0_mx); + // using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using tile_shapeAMX = Tile; // 实际tile尺寸, 需初始化为0 + using itAMX = global_iterator; + // using tile_ND2ZZOffset = Tile; + itAMX gAMXIter(src0_mx); + + using gm_shapeBMX = global_tensor>; + // gm_shapeBMX gBMX(src1_mx); + // using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using tile_shapeBMX = Tile; // 8*4B, tload 256B, + using itBMX = global_iterator; + // using tile_ND2NNOffset = Tile; + // tile_ND2NNOffset nd2nn_offset; + itBMX gBMXIter(src1_mx); + + const int Mb = (gM) / tM; + const int Nb = (gN) / tN; + const int Kb = (gK) / tK; + + alignas(256) static uint16_t g_dump_intTile[tM*tK/smatrix_wfactor]; + for(int i=0;i + void __vec__ dequant_acc( + typename psum_tensor::TileDType __in__ data, + typename scale_tensor::TileDType __in__ scale, + typename out_tensor::TileDType __in__ adder, + typename out_tensor::TileDType __out__ out + ) { + + uint16_t x = blkv_get_index_x(); + uint16_t y = blkv_get_index_y(); + uint16_t src_idx = x * psum_tensor::RowStride + y * psum_tensor::ColStride; + // uint16_t scale_idx = (x/128) * scale_tensor::RowStride + y * scale_tensor::ColStride; + uint16_t scale_idx = y * scale_tensor::ColStride; + blkv_get_tile_ptr(out)[src_idx] = blkv_get_tile_ptr(data)[src_idx] * blkv_get_tile_ptr(scale)[scale_idx] + blkv_get_tile_ptr(adder)[src_idx]; + } + + template + void __vec__ dequant_acc_new( + typename psum_tensor::TileDType __in__ data, + typename scale_tensor::TileDType __in__ scale, + typename out_tensor::TileDType __in__ adder, + typename out_tensor::TileDType __out__ out + ) { + + uint16_t x = blkv_get_index_x(); + uint16_t y = blkv_get_index_y(); + uint16_t src_idx = x * psum_tensor::RowStride + y * psum_tensor::ColStride; + // uint16_t scale_idx = (x/128) * scale_tensor::RowStride + y * scale_tensor::ColStride; + uint16_t scale_idx = y * scale_tensor::ColStride; + blkv_get_tile_ptr(out)[src_idx] = blkv_get_tile_ptr(data)[src_idx] * blkv_get_tile_ptr(scale)[scale_idx] + blkv_get_tile_ptr(adder)[src_idx]; + } + + template + void dequant_acc_tileop(OutTile &out, PsumTile &data, ScaleTile &scale, OutTile &adder) { + OutTile expanded_scale; + OutTile scaled; + TCOLEXPAND(expanded_scale, scale); + TMUL(scaled, data, expanded_scale); + TADD(out, scaled, adder); + } + + // mixed precision matmul dequant(A*B)， fp4x2 width_factor: 2 + template + void matmul_mp(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { + static_assert(gK % (128) == 0); + static_assert(tK == 128); + static const uint32_t trow = (tM >= 16) ? tM : 16; + static const uint32_t tcol = (tN >= 16) ? tN : 16; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + // 伪量化固定float, group 大小128， 128个fp4共享一个scaling factor, 128的partial sum* scale + using gm_shape_scale = global_tensor>; + using gm_shapeACC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shape_scale = Tile; + using tile_shape_dequant = Tile; + using tile_shapeACC = TileAcc; + // copy of acc, input as vector + using tile_ACCin = Tile; + + using itA = global_iterator; + using itB = global_iterator; + using itScale = global_iterator; + using itACC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itScale gScaleIter(c_ptr); + itACC gACCIter(acc_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + using tile_shape_scale_trows = Tile; + using tile_shape_scale_tcols = Tile; + using tile_shape_scale_tcorner = Tile; + + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; + + using tile_shape_dequant_trows = Tile; + using tile_shape_dequant_tcols = Tile; + using tile_shape_dequant_tcorner = Tile; + for(int i=0;i1 , 256 -> 2 scaling factor + // static_assert(tile_shapeB::ValidCol % (width_factor*128) == 0); // TODO, 暂不考虑padding，假设形状是规整的, 方便处理, taccin*ts_adder=tc_dequant + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + k++; + if constexpr (rmd_K) { + // to do check acc shape + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, j); + auto gS = gScaleIter(Kb,j); + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + tile_shape_scale_tcols ts; + tile_shape_dequant tC_dequant; + + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + TSTORE(gACC, tAdder[(k+1)%2]); + } + // if constexpr (rmd_N) // TODO + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gACC = gACCIter(Mb, j); + tile_shapeC_tcols tACC; + tile_ACCin_tcols tACCin; + tile_shape_dequant_tcols tAdder[2]; + TEXPANDSCALAR(tAdder[0], 0.); + int k = 0; + #pragma clang loop unroll(full) + for (; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + auto gS = gScaleIter(k,j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + tile_shape_scale ts; + tile_shape_dequant_tcols tC_dequant; + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + k++; + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + auto gS = gScaleIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + tile_shape_scale_tcols ts; + tile_shape_dequant tC_dequant; + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(ts, gS); + MATMUL(tACC, tA, tB); + ACCCVT(tACCin, tACC); + dequant_acc<<>>(tACCin.data(), ts.data(), tAdder[k%2].data(), tC_dequant.data()); + tAdder[(k+1)%2] = tC_dequant; + } + TSTORE(gACC, tAdder[(k+1)%2]); + } + // todo + // if constexpr (rmd_N) { + // auto gC = gCIter(Mb, Nb); + + // tile_shapeC_tcorner tACC; + // if constexpr(Kb>0){ + // auto gA = gAIter(Mb, 0); + // auto gB = gBIter(0, Nb); + + // tile_shapeA_tcols tA; + // tile_shapeB_trows tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMUL(tACC, tA, tB); + // } + // #pragma clang loop unroll(full) + // for (int k = 1; k < Kb; ++k) { + // auto gA = gAIter(Mb, k); + // auto gB = gBIter(k, Nb); + + // tile_shapeA_tcols tA; + // tile_shapeB_trows tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // MATMACC(tACC, tA, tB); + // } + // if constexpr (rmd_K) { + // auto gA = gAIter(Mb, Kb); + // auto gB = gBIter(Kb, Nb); + + // tile_shapeA_tcorner tA; + // tile_shapeB_tcorner tB; + // TLOAD(tA, gA); + // TLOAD(tB, gB); + // if constexpr(Kb>0){ + // MATMACC(tACC, tA, tB); + // } else { + // MATMUL(tACC, tA, tB); + // } + // } + // TSTORE_ACC(gC, tACC); + // } + } + } + + // mixed precision matmul dequant(A*B), fp4x2 width_factor: 2, using LinxISA tileops for dequant accumulation. + template + void matmul_mp_tileop(float *acc_ptr, dtypeA *a_ptr, dtypeB *b_ptr, float *c_ptr) { + static_assert(gK % (128) == 0); + static_assert(tK == 128); + static const uint32_t trow = (tM >= 16) ? tM : 16; + static const uint32_t tcol = (tN >= 16) ? tN : 16; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shape_scale = global_tensor>; + using gm_shapeACC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shape_scale = Tile; + using tile_shape_dequant = Tile; + using tile_shapeACC = TileAcc; + using tile_ACCin = Tile; + + using itA = global_iterator; + using itB = global_iterator; + using itScale = global_iterator; + using itACC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itScale gScaleIter(c_ptr); + itACC gACCIter(acc_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + using tile_shape_scale_trows = Tile; + using tile_shape_scale_tcols = Tile; + using tile_shape_scale_tcorner = Tile; + + using tile_ACCin_trows = Tile; + using tile_ACCin_tcols = Tile; + using tile_ACCin_tconer = Tile; + + using tile_shape_dequant_trows = Tile; + using tile_shape_dequant_tcols = Tile; + using tile_shape_dequant_tcorner = Tile; + for(int i=0;i + +This family contains **3** source implementations and +**16** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [matmul](a16w4-f1f175ff.md) | `benchmark/one-level-arch/test/kernel/matmul/src/A16W4.cpp` | 3 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCOLEXPAND`](../../../../intrinsics/tcolexpand.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMATMUL`](../../../../intrinsics/tmatmul.md), [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md), [`TMUL`](../../../../intrinsics/tmul.md), [`TSTORE`](../../../../intrinsics/tstore.md) | +| [matmul](hif4-hif4-f0fd84bb.md) | `benchmark/one-level-arch/test/kernel/matmul/src/HiF4_HiF4.cpp` | 4 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCOLEXPAND`](../../../../intrinsics/tcolexpand.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMATMUL`](../../../../intrinsics/tmatmul.md), [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md), [`TMUL`](../../../../intrinsics/tmul.md), [`TSTORE`](../../../../intrinsics/tstore.md) | +| [matmul](matmul-a008ac76.md) | `benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp` | 9 | [`TADD`](../../../../intrinsics/tadd.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMATMUL`](../../../../intrinsics/tmatmul.md), [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md), [`TMATMUL_MX`](../../../../intrinsics/tmatmul_mx.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/matmul/matmul-a008ac76.md b/docs/benchmarks/catalog/one-level/matmul/matmul-a008ac76.md new file mode 100644 index 0000000..90d4855 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/matmul/matmul-a008ac76.md @@ -0,0 +1,4082 @@ +# matmul: matmul.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `matmul` | +| Implementation source | `benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp` | +| Active build variants | 9 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then executes matrix or matrix-vector work on cube/accumulator tiles, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="matmul.cpp" linenums="1" +#include +#include +#include "fileop.h" +#include "common.h" +#include "benchmark.h" + +#ifndef globM +#define globM 120 +#endif + +#ifndef globN +#define globN 120 +#endif + +#ifndef globK +#define globK 120 +#endif + +#ifndef tilM +#define tilM 16 +#endif + +#ifndef tilN +#define tilN 16 +#endif + +#ifndef tilK +#define tilK 16 +#endif + +#ifndef Batch +#define Batch 1 +#endif + +#define ALIGN_MASK 0xfffffffffffff000ull +#define ALIGN 4*1024 + +#include "matmul/matmul.hpp" + +int main() { + + #if defined(VEC) + float src0[globM*globK]; + float src1[globK*globN]; + float dst[globM*globN]; + + BENCHSTART; + matmul_vec(dst,src0, src1); + BENCHEND; + + #elif defined(FRAC) + float src0[globM*globK]; + float src1[globK*globN]; + float dst[globM*globN]; + + BENCHSTART; + matmul_frac(dst, src0, src1); + BENCHEND; + + #elif defined(MASK_FP32) || defined(MASK_FP32_REUSEA) || defined(MASK_FP32_REUSEB) || defined(MASK_FP32_REUSEAB) || defined(MASK_FP32_DYNAMIC) || \ + defined(MASK_FP32_DYNAMIC_REUSE) + // char* p; + // p = (char *)malloc(globM*globK*sizeof(__fp8_e4m3)+2*ALIGN); + // src0 = (__fp8_e4m3 *)((uint64_t)p & ALIGN_MASK); + // p = (char *)malloc(globK*globN*sizeof(__fp8_e4m3)+2*ALIGN); + // src1 = (__fp8_e4m3 *)((uint64_t)p & ALIGN_MASK); + // p = (char *)malloc(globM*globN*sizeof(float)+2*ALIGN); + // dst = (float*)((uint64_t)p & ALIGN_MASK); + // printf("src0 is %p src1 is %p dst is %p\n", src0, src1, dst); + + float src0[globM*globK]; + float src1[globK*globN]; + float dst [globM*globN]; + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(float)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(float)); + #endif + + BENCHSTART; + #if defined(MASK_FP32) + matmul_mask_tileop(dst, src0, src1); + #elif defined(MASK_FP32_REUSEA) + matmul_mask_reuseA_tileop(dst, src0, src1); + #elif defined(MASK_FP32_REUSEB) + matmul_mask_reuseB_tileop(dst, src0, src1); + #elif defined(MASK_FP32_REUSEAB) + matmul_mask_reuseAB_tileop(dst, src0, src1); + #elif defined(MASK_FP32_DYNAMIC) + matmul_dynamic_new(dst, src0, src1, globM, globN, globK); + #elif defined(MASK_FP32_DYNAMIC_REUSE) + matmul_dynamic_reuseA(dst, src0, src1, globM, globN, globK); + #endif + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)dst, globM*globN*sizeof(float)); + #endif + + #elif defined(MASK_FP16_REUSEA_OPT) || (MASK_FP16_REUSEA_OPT2)|| (MASK_FP16_REUSEB_OPT2)|| defined(MASK_FP16) || defined(MASK_FP16_REUSEA) || defined(MASK_FP16_REUSEB) || defined(MASK_FP16_REUSEAB) || defined(MASK_FP16_DYNAMIC) || \ + defined(MASK_FP16_DYNAMIC_REUSE) + using datatype = __half; + // using datatype = __bf16; + using output_type = float; + datatype *src0; + datatype *src1; + output_type *dst; + + datatype src0p[globM*globK + 2*ALIGN]; + datatype src1p[globK*globN + 2*ALIGN]; + output_type dstp[globM*globN + 2*ALIGN]; + + src0 = (datatype *)(((uint64_t)src0p & ALIGN_MASK) + ALIGN); + src1 = (datatype *)(((uint64_t)src1p & ALIGN_MASK) + ALIGN); + dst = (output_type*)(((uint64_t)dstp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(datatype)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(datatype)); + #endif + + BENCHSTART; + #if defined(MASK_FP16) + matmul_mask_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEA) + matmul_mask_reuseA_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEA_OPT) + matmul_mask_reuseA_OPT_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEA_OPT2) + matmul_mask_reuseA_OPT2_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEB_OPT2) + matmul_mask_reuseB_OPT2_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEB) + matmul_mask_reuseB_tileop(dst, src0, src1); + #elif defined(MASK_FP16_REUSEAB) + matmul_mask_reuseAB_tileop(dst, src0, src1); + #elif defined(MASK_FP16_DYNAMIC) + matmul_dynamic_new(dst, src0, src1, globM, globN, globK); + #elif defined(MASK_FP16_DYNAMIC_REUSE) + matmul_dynamic_reuseA(dst, src0, src1, globM, globN, globK); + #endif + BENCHEND; + + #elif defined(MASK_FP8) || defined(MASK_FP8_2LVL) || defined(MASK_FP8_MULTI4_B) || defined(MASK_FP8_MULTI4_AB) || defined(MASK_FP8_REUSEA) || \ + defined(MASK_FP8_REUSEB) || defined(MASK_FP8_REUSEAB) || defined(MASK_FP8_DYNAMIC) || defined(MASK_FP8_DYNAMIC_REUSE) + __fp8_e4m3 *src0; + __fp8_e4m3 *src1; + float *dst; + + __fp8_e4m3 src0p[globM*globK + 2*ALIGN]; + __fp8_e4m3 src1p[globK*globN + 2*ALIGN]; + float dstp[globM*globN + 2*ALIGN]; + + src0 = (__fp8_e4m3 *)(((uint64_t)src0p & ALIGN_MASK) + ALIGN); + src1 = (__fp8_e4m3 *)(((uint64_t)src1p & ALIGN_MASK) + ALIGN); + dst = (float*)(((uint64_t)dstp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(__fp8_e4m3)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(__fp8_e4m3)); + #endif + + BENCHSTART; + #if defined(MASK_FP8) + matmul_mask_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_2LVL) + matmul_mask_2lvl_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_MULTI4_B) + matmul_mask_multi4_B_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_MULTI4_AB) + matmul_mask_multi4_AB_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_REUSEA) + matmul_mask_reuseA_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_REUSEB) + matmul_mask_reuseB_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_REUSEAB) + matmul_mask_reuseAB_tileop<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1); + #elif defined(MASK_FP8_DYNAMIC) + matmul_dynamic_new<__fp8_e4m3, tilM, tilN, tilK>(dst, src0, src1, globM, globN, globK); + #elif defined(MASK_FP8_DYNAMIC_REUSE) + matmul_dynamic_reuseA<__fp8_e4m3, tilM, tilN, tilK>(dst, src0, src1, globM, globN, globK); + #endif + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)dst, globM*globN*sizeof(float)); + #endif + + #elif defined(MX_FP8) + __fp8_e4m3 *src0; + __fp8_e4m3 *src1; + uint8_t *src0_mx; + uint8_t *src1_mx; + float *dst; + + __fp8_e4m3 src0p[globM*globK + 2*ALIGN]; + __fp8_e4m3 src1p[globK*globN + 2*ALIGN]; + uint8_t src0_mxp[(globM*globK + 2*ALIGN)/32]; + uint8_t src1_mxp[(globK*globN + 2*ALIGN)/32]; + float dstp[globM*globN + 2*ALIGN]; + + src0 = (__fp8_e4m3 *)(((uint64_t)src0p & ALIGN_MASK) + ALIGN); + src1 = (__fp8_e4m3 *)(((uint64_t)src1p & ALIGN_MASK) + ALIGN); + src0_mx = (uint8_t *)(((uint64_t)src0_mxp & ALIGN_MASK) + ALIGN); + src1_mx = (uint8_t *)(((uint64_t)src1_mxp & ALIGN_MASK) + ALIGN); + dst = (float*)(((uint64_t)dstp & ALIGN_MASK) + ALIGN); + + #ifdef RES_CHECK + #define SRC0_PATH CHK_DIR "/src0.bin" + #define SRC1_PATH CHK_DIR "/src1.bin" + readBinaryFile(SRC0_PATH, (uint8_t*)src0, globM*globK*sizeof(__fp8_e4m3)); + readBinaryFile(SRC1_PATH, (uint8_t*)src1, globK*globN*sizeof(__fp8_e4m3)); + #endif + + BENCHSTART; + matmul_mx<__fp8_e4m3, globM, globN, globK, tilM, tilN, tilK>(dst, src0, src1, src0_mx, src1_mx); + BENCHEND; + + #ifdef RES_CHECK + #define RES_PATH CHK_DIR "/res.bin" + writeBinaryFile(RES_PATH, (uint8_t*)dst, globM*globN*sizeof(float)); + #endif + + #else + #error "No Available Macro!"; + #endif + + return 0; +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/matmul/matmul.hpp" + + ```cpp title="matmul.hpp" linenums="1" + #ifndef MATMUL_TILEOP_KERNEL_HPP + #define MATMUL_TILEOP_KERNEL_HPP + + #include + + #ifndef Batch + #define Batch 1 + #endif + + using namespace pto; + + template + void store_acc_tile_tileop(GmOut &Gout, TileAcc &tAcc){ + using TileAccOut = Tile; + TileAccOut tAccOut; + if constexpr (TileAcc::Loc == Location::Acc) { + ACCCVT(tAccOut, tAcc); + } else { + TCVT(tAccOut, tAcc); + } + TSTORE(Gout, tAccOut); + } + + template + void store_acc_tile_dynamic_tileop(GmOut &Gout, TileAcc &tAcc, size_t valid_row, size_t valid_col){ + using TileAccOut = Tile; + TileAccOut tAccOut(valid_row, valid_col); + if constexpr (TileAcc::Loc == Location::Acc) { + ACCCVT(tAccOut, tAcc); + } else { + TCVT(tAccOut, tAcc); + } + TSTORE(Gout, tAccOut); + } + + // A * B -> C with any shape + // activation * weight( int8_t->FP16/FP8-> FP32 -> int8_t) + template + void matmul_mask_tileop(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + // using tile_shapecast = Tile; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itC gCIter(c_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + for (int b=0;b0){ + auto gA = gAIter(i, 0); + auto gB = gBIter(0, j); + + tile_shapeA tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(i, k); + auto gB = gBIter(k, j); + + tile_shapeA tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + // TCVT(tCast, tACC); + // TSTORE(gC, tCast); + store_acc_tile_tileop(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + + tile_shapeC_trows tACC; + if constexpr(Kb>0){ + auto gA = gAIter(i, 0); + auto gB = gBIter(0, Nb); + + tile_shapeA tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(i, k); + auto gB = gBIter(k, Nb); + + tile_shapeA tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_trows tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + if constexpr(Kb>0){ + auto gA = gAIter(Mb, 0); + auto gB = gBIter(0, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + if constexpr(Kb>0){ + auto gA = gAIter(Mb, 0); + auto gB = gBIter(0, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + } + } + } + + template + void matmul_frac_tileop(float* dst, dtype* src0, dtype* src1){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itC gCIter(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + for(int i=0;i0){ + auto gA = gAIter(i, 0); + auto gB = gBIter(0, j); + + tile_shapeA tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for(int k=1;k best_val) { + best_val = v; + best_m = m; + best_k = k; + } + } + } + return {best_m, best_k, best_val}; + } + + constexpr ResB find_reuseB(int Nb, int Kb, int MAX_TILE_NUM) { + int best_n = 0, best_k = 0, best_val = -1; + + #pragma clang loop unroll(full) + for (int n = 1; n <= Nb; ++n) { + #pragma clang loop unroll(full) + for (int k = 1; k <= Kb; ++k) { + int v = n * k; + if (v <= MAX_TILE_NUM && v > best_val) { + best_val = v; + best_n = n; + best_k = k; + } + } + } + return {best_n, best_k, best_val}; + } + + template + void matmul_mask_reuseA_tileop(float *dst, dtype *src0, dtype *src1){ + // const int MAX_TILE_NUM = gK/tK; + const int MAX_TILE_NUM = 24; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + // static_assert(Kb == 56); + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // constexpr ResA R = find_reuseA(Mb, Kb, MAX_TILE_NUM); + constexpr ResA R = {1, (Kb > MAX_TILE_NUM) ? MAX_TILE_NUM : Kb, R.m * R.k}; + // R.m = 1; + // R.k = (Kb > MAX_TILE_NUM) ? MAX_TILE_NUM : Kb; + // R.val = R.m * R.k; + + const int dM = R.m == 0? 0 : Mb / R.m; + const int rM = R.m == 0? 0 : Mb % R.m; + + static_assert(R.val <= MAX_TILE_NUM, "R.val is biggger than MAX_TILE_NUM"); + //printf("R.m is %d, R.k is %d\n", R.m, R.k); + static_assert(Batch == 1); + for (int b=0;b0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + + auto gC = gIterC(i*R.m+ii,j); + store_acc_tile_tileop(gC, tACC); + } + + // [m, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0;k Kb); + if constexpr(R.k < Kb){ + + for(int k=R.k;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + + auto gC = gIterC(i*R.m+ii,Nb); + store_acc_tile_tileop(gC, tACC); + } + + } + } + + if constexpr(rM>0){ + tile_shapeA tA[rM][R.k]; + + #pragma clang loop unroll(full) + for(int i=0;i0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(i+dM*R.m,j); + store_acc_tile_tileop(gC, tACC); + } + + // [rM, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(i+dM*R.m,Nb); + store_acc_tile_tileop(gC, tACC); + } + } + } + + // [rmd_M, n, k] + if constexpr (rmd_M) { + tile_shapeA_tcols tA[R.k]; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(Mb,j); + store_acc_tile_tileop(gC, tACC); + } + + // [rmd_M, rmd_N, k] + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(Mb,Nb); + store_acc_tile_tileop(gC, tACC); + } + } + }// Batch + } + + template + void matmul_mask_reuseA_OPT_tileop(float *dst, dtype *src0, dtype *src1){ + const int MAX_TILE_NUM = 20; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + constexpr int Mb = gM / tM; + constexpr int Nb = gN / tN; + constexpr int Kb = gK / tK; + + constexpr int rmd_M = gM % tM; + constexpr int rmd_N = gN % tN; + constexpr int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + constexpr ResA R = find_reuseA(Mb, Kb, MAX_TILE_NUM); + static_assert(R.val <= MAX_TILE_NUM, "R.val is bigger than MAX_TILE_NUM"); + + const int dM = (R.m == 0) ? 0 : Mb / R.m; + const int rM = (R.m == 0) ? 0 : Mb % R.m; + + // 剩余 K 轴（超出 R.k 的部分）按 MAX_TILE_NUM 再做切分 + constexpr int remain_K = (Kb > R.k) ? (Kb - R.k) : 0; + constexpr int K2_chunks = (remain_K > 0) ? (remain_K / MAX_TILE_NUM) : 0; + constexpr int K2_rem = (remain_K > 0) ? (remain_K % MAX_TILE_NUM) : 0; + + for (int b = 0; b < Batch; b++) { + + // ======================================================== + // SECTION 1: 主 M 块 (dM * R.m 行) + // ======================================================== + #pragma clang loop unroll(full) + for (int i = 0; i < dM; i++) { + + #pragma clang loop unroll(full) + for (int ii = 0; ii < R.m; ii++) { + const int row = i * R.m + ii; + + // Phase A: 加载 A[row][0..R.k)，复用于所有 N 列 + tile_shapeA tA_phase0[R.k]; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + auto gA = gIterA(row, k); + TLOAD(tA_phase0[k], gA); + } + + // --- N 主列 --- + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB tB; + auto gB = gIterB(k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + // --- N 余列 (rmd_N) --- + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + + // Phase B-1: 剩余 K 轴 Full chunks (每块 MAX_TILE_NUM 个 k tile) + // K 在外加载 A，N 在内复用 A，每块输出一个部分和 + if constexpr (K2_chunks > 0) { + #pragma clang loop unroll(full) + for (int kc = 0; kc < K2_chunks; kc++) { + const int k_base = R.k + kc * MAX_TILE_NUM; + + tile_shapeA tA_chunk[MAX_TILE_NUM]; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA_chunk[k], gA); + } + + // --- N 主列 --- + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + // --- N 余列 --- + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + } + + // Phase B-2: 剩余 K 轴 Tail chunk (K2_rem 个 k tile) + if constexpr (K2_rem > 0) { + const int k_base = R.k + K2_chunks * MAX_TILE_NUM; + + tile_shapeA tA_tail[K2_rem]; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA_tail[k], gA); + } + + // --- N 主列 --- + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + // --- N 余列 --- + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + // Phase C: rmd_K 尾列（A 只有一个 tile，直接复用于所有 N） + if constexpr (rmd_K) { + tile_shapeA_trows tA_rmdK; + auto gA = gIterA(row, Kb); + TLOAD(tA_rmdK, gA); + + // --- N 主列 --- + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + tile_shapeB_tcols tB; + auto gB = gIterB(Kb, j); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + // --- N 余列 --- + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + tile_shapeB_tcorner tB; + auto gB = gIterB(Kb, Nb); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + } // ii + } // i (dM) + + // ======================================================== + // SECTION 2: 余 M 块 (rM 行) + // ======================================================== + if constexpr (rM > 0) { + #pragma clang loop unroll(full) + for (int i = 0; i < rM; i++) { + const int row = dM * R.m + i; + + // Phase A + tile_shapeA tA_phase0[R.k]; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + auto gA = gIterA(row, k); + TLOAD(tA_phase0[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB tB; + auto gB = gIterB(k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + + // Phase B-1: Full chunks + if constexpr (K2_chunks > 0) { + #pragma clang loop unroll(full) + for (int kc = 0; kc < K2_chunks; kc++) { + const int k_base = R.k + kc * MAX_TILE_NUM; + + tile_shapeA tA_chunk[MAX_TILE_NUM]; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA_chunk[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + } + + // Phase B-2: Tail chunk + if constexpr (K2_rem > 0) { + const int k_base = R.k + K2_chunks * MAX_TILE_NUM; + + tile_shapeA tA_tail[K2_rem]; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA_tail[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + // Phase C: rmd_K + if constexpr (rmd_K) { + tile_shapeA_trows tA_rmdK; + auto gA = gIterA(row, Kb); + TLOAD(tA_rmdK, gA); + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + tile_shapeB_tcols tB; + auto gB = gIterB(Kb, j); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(row, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_trows tACC; + tile_shapeB_tcorner tB; + auto gB = gIterB(Kb, Nb); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(row, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + } // i (rM) + } + + // ======================================================== + // SECTION 3: rmd_M 尾行 (行索引 = Mb，A 类型为 tcols) + // ======================================================== + if constexpr (rmd_M) { + + // Phase A + tile_shapeA_tcols tA_phase0[R.k]; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + auto gA = gIterA(Mb, k); + TLOAD(tA_phase0[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB tB; + auto gB = gIterB(k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(Mb, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < R.k; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_phase0[k], tB); + else TMATMUL_ACC(tACC, tA_phase0[k], tB); + } + auto gC = gIterC(Mb, Nb); + store_acc_tile_tileop(gC, tACC); + } + + // Phase B-1: Full chunks (rmd_M 行，A 类型为 tcols) + if constexpr (K2_chunks > 0) { + #pragma clang loop unroll(full) + for (int kc = 0; kc < K2_chunks; kc++) { + const int k_base = R.k + kc * MAX_TILE_NUM; + + tile_shapeA_tcols tA_chunk[MAX_TILE_NUM]; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA_chunk[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(Mb, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < MAX_TILE_NUM; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_chunk[k], tB); + else TMATMUL_ACC(tACC, tA_chunk[k], tB); + } + auto gC = gIterC(Mb, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + } + + // Phase B-2: Tail chunk (rmd_M 行) + if constexpr (K2_rem > 0) { + const int k_base = R.k + K2_chunks * MAX_TILE_NUM; + + tile_shapeA_tcols tA_tail[K2_rem]; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA_tail[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(Mb, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < K2_rem; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA_tail[k], tB); + else TMATMUL_ACC(tACC, tA_tail[k], tB); + } + auto gC = gIterC(Mb, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + // Phase C: rmd_K (rmd_M 行，A 类型为 tcorner) + if constexpr (rmd_K) { + tile_shapeA_tcorner tA_rmdK; + auto gA = gIterA(Mb, Kb); + TLOAD(tA_rmdK, gA); + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeC_tcols tACC; + tile_shapeB_tcols tB; + auto gB = gIterB(Kb, j); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(Mb, j); + store_acc_tile_tileop(gC, tACC); + } + + if constexpr (rmd_N) { + tile_shapeC_tcorner tACC; + tile_shapeB_tcorner tB; + auto gB = gIterB(Kb, Nb); + TLOAD(tB, gB); + if constexpr (Kb > 0) TMATMUL_ACC(tACC, tA_rmdK, tB); + else TMATMUL(tACC, tA_rmdK, tB); + auto gC = gIterC(Mb, Nb); + store_acc_tile_tileop(gC, tACC); + } + } + } // rmd_M + } // Batch + } + + template + void matmul_mask_reuseA_OPT2_tileop(float *dst, dtype *src0, dtype *src1){ + constexpr int MAX_TILE_NUM = 14; + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft ; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc ; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + constexpr int Mb = gM / tM; + constexpr int Nb = gN / tN; + constexpr int Kb = gK / tK; + + constexpr int rmd_M = gM % tM; + constexpr int rmd_N = gN % tN; + constexpr int rmd_K = gK % tK; + + // —— A / B 余块 —— + using tile_shapeA_trows = TileLeft ; + using tile_shapeA_tcols = TileLeft ; + using tile_shapeA_tcorner = TileLeft ; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + // —— ACC 余块 —— + using tile_shapeACC_trows = TileAcc; + using tile_shapeACC_tcols = TileAcc; + using tile_shapeACC_tcorner = TileAcc; + + // —— Vec 常驻 C（fp32, ColMajor）—— + using tile_C_out = Tile; + using tile_C_out_trows = Tile; + using tile_C_out_tcols = Tile; + using tile_C_out_tcorner = Tile; + + // —— 写回时的 bf16 临时 tile（RowMajor）—— + using tile_C_bf16 = Tile; + using tile_C_bf16_trows = Tile; + using tile_C_bf16_tcols = Tile; + using tile_C_bf16_tcorner = Tile; + + // —— K 切分 —— + constexpr int K_chunks = Kb / MAX_TILE_NUM; + constexpr int K_rem = Kb % MAX_TILE_NUM; + + static_assert(K_chunks > 0 || K_rem > 0 || rmd_K > 0, "empty K"); + + for (int b = 0; b < Batch; b++) { + + // Vec 常驻 C + tile_C_out tC_main[Mb][Nb]; + tile_C_out_trows tC_rcol[Mb]; + tile_C_out_tcols tC_rrow[Nb]; + tile_C_out_tcorner tC_corner; + + // ============================================================ + // 段 1：等长 chunks (LEN = MAX_TILE_NUM) + // chunk[0] : is_first = true + // chunk[1..*] : is_first = false + // ============================================================ + if constexpr (K_chunks > 0) { + + // ---------- chunk[0]: is_first = true ---------- + { + constexpr int LEN = MAX_TILE_NUM; + constexpr int k_base = 0; + + // ----- 主 M 块 ----- + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeA tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + TCVT(tC_main[row][j], tACC); // is_first: 直接覆盖 + } + + if constexpr (rmd_N) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + TCVT(tC_rcol[row], tACC); + } + } // row + + // ----- rmd_M 尾行 ----- + if constexpr (rmd_M) { + tile_shapeA_tcols tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + TCVT(tC_rrow[j], tACC); + } + + if constexpr (rmd_N) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + TCVT(tC_corner, tACC); + } + } + } // chunk[0] + + // ---------- chunk[1 .. K_chunks-1]: is_first = false ---------- + if constexpr (K_chunks > 1) { + #pragma clang loop unroll(full) + for (int kc = 1; kc < K_chunks; kc++) { + constexpr int LEN = MAX_TILE_NUM; + const int k_base = kc * MAX_TILE_NUM; + + // 主 M 块 + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeA tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][j], tC_main[row][j], tACC_vec); + } + + if constexpr (rmd_N) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + } // row + + if constexpr (rmd_M) { + tile_shapeA_tcols tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[j], tC_rrow[j], tACC_vec); + } + + if constexpr (rmd_N) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } // kc + } // K_chunks > 1 + } // K_chunks > 0 + + // ============================================================ + // 段 2：K_rem (LEN = K_rem) + // is_first = (K_chunks == 0) + // ============================================================ + if constexpr (K_rem > 0) { + constexpr int LEN = K_rem; + constexpr int k_base = K_chunks * MAX_TILE_NUM; + constexpr bool is_first = (K_chunks == 0); + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeA tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(row, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + if constexpr (is_first) { + TCVT(tC_main[row][j], tACC); + } else { + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][j], tC_main[row][j], tACC_vec); + } + } + + if constexpr (rmd_N) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + if constexpr (is_first) { + TCVT(tC_rcol[row], tACC); + } else { + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + } + } + + if constexpr (rmd_M) { + tile_shapeA_tcols tA[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA[k], gA); + } + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB tB; + auto gB = gIterB(k_base + k, j); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + if constexpr (is_first) { + TCVT(tC_rrow[j], tACC); + } else { + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[j], tC_rrow[j], tACC_vec); + } + } + + if constexpr (rmd_N) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeB_trows tB; + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB, gB); + if (k == 0) TMATMUL(tACC, tA[k], tB); + else TMATMUL_ACC(tACC, tA[k], tB); + } + if constexpr (is_first) { + TCVT(tC_corner, tACC); + } else { + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } + } // K_rem + + // ============================================================ + // 段 3：rmd_K + // is_first = (K_chunks == 0) && (K_rem == 0) + // ============================================================ + if constexpr (rmd_K) { + constexpr bool is_first = (K_chunks == 0) && (K_rem == 0); + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeA_trows tA_rmdK; + auto gA = gIterA(row, Kb); + TLOAD(tA_rmdK, gA); + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC tACC; + tile_shapeB_tcols tB; + auto gB = gIterB(Kb, j); + TLOAD(tB, gB); + TMATMUL(tACC, tA_rmdK, tB); + + if constexpr (is_first) { + TCVT(tC_main[row][j], tACC); + } else { + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][j], tC_main[row][j], tACC_vec); + } + } + + if constexpr (rmd_N) { + tile_shapeACC_trows tACC; + tile_shapeB_tcorner tB; + auto gB = gIterB(Kb, Nb); + TLOAD(tB, gB); + TMATMUL(tACC, tA_rmdK, tB); + + if constexpr (is_first) { + TCVT(tC_rcol[row], tACC); + } else { + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + } + } + + if constexpr (rmd_M) { + tile_shapeA_tcorner tA_rmdK; + auto gA = gIterA(Mb, Kb); + TLOAD(tA_rmdK, gA); + + #pragma clang loop unroll(full) + for (int j = 0; j < Nb; j++) { + tile_shapeACC_tcols tACC; + tile_shapeB_tcols tB; + auto gB = gIterB(Kb, j); + TLOAD(tB, gB); + TMATMUL(tACC, tA_rmdK, tB); + + if constexpr (is_first) { + TCVT(tC_rrow[j], tACC); + } else { + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[j], tC_rrow[j], tACC_vec); + } + } + + if constexpr (rmd_N) { + tile_shapeACC_tcorner tACC; + tile_shapeB_tcorner tB; + auto gB = gIterB(Kb, Nb); + TLOAD(tB, gB); + TMATMUL(tACC, tA_rmdK, tB); + + if constexpr (is_first) { + TCVT(tC_corner, tACC); + } else { + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } + } // rmd_K + + // ============================================================ + // 全部 K 累加完成后，统一写回 global + // ============================================================ + #pragma clang loop unroll(full) + for (int m = 0; m < Mb; m++) { + #pragma clang loop unroll(full) + for (int n = 0; n < Nb; n++) { + auto gC = gIterC(m, n); + TSTORE(gC, tC_main[m][n]); + } + if constexpr (rmd_N) { + auto gC = gIterC(m, Nb); + TSTORE(gC, tC_rcol[m]); + } + } + if constexpr (rmd_M) { + #pragma clang loop unroll(full) + for (int n = 0; n < Nb; n++) { + auto gC = gIterC(Mb, n); + TSTORE(gC, tC_rrow[n]); + } + if constexpr (rmd_N) { + auto gC = gIterC(Mb, Nb); + TSTORE(gC, tC_corner); + } + } + + } // Batch + } + + template + void matmul_mask_reuseB_OPT2_tileop(float *dst, dtype *src0, dtype *src1){ + constexpr int MAX_TILE_NUM = 14; + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft ; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc ; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + constexpr int Mb = gM / tM; + constexpr int Nb = gN / tN; + constexpr int Kb = gK / tK; + + constexpr int rmd_M = gM % tM; + constexpr int rmd_N = gN % tN; + constexpr int rmd_K = gK % tK; + + // —— A / B 余块 —— + using tile_shapeA_trows = TileLeft ; + using tile_shapeA_tcols = TileLeft ; + using tile_shapeA_tcorner = TileLeft ; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + // —— ACC 余块 —— + using tile_shapeACC_trows = TileAcc; + using tile_shapeACC_tcols = TileAcc; + using tile_shapeACC_tcorner = TileAcc; + + // —— Vec 常驻 C（fp32, ColMajor）—— + using tile_C_out = Tile; + using tile_C_out_trows = Tile; + using tile_C_out_tcols = Tile; + using tile_C_out_tcorner = Tile; + + // —— 写回时的 bf16 临时 tile（RowMajor）—— + using tile_C_bf16 = Tile; + using tile_C_bf16_trows = Tile; + using tile_C_bf16_tcols = Tile; + using tile_C_bf16_tcorner = Tile; + + // —— K 切分 —— + constexpr int K_chunks = Kb / MAX_TILE_NUM; + constexpr int K_rem = Kb % MAX_TILE_NUM; + + static_assert(K_chunks > 0 || K_rem > 0 || rmd_K > 0, "empty K"); + + for (int b = 0; b < Batch; b++) { + + // Vec 常驻 C + tile_C_out tC_main[Mb][Nb]; + tile_C_out_trows tC_rcol[Mb]; + tile_C_out_tcols tC_rrow[Nb]; + tile_C_out_tcorner tC_corner; + + // ============================================================ + // 段 1：等长 chunks (LEN = MAX_TILE_NUM) + // chunk[0] : is_first = true + // chunk[1..*] : is_first = false + // ============================================================ + if constexpr (K_chunks > 0) { + + // ---------- chunk[0]: is_first = true ---------- + { + constexpr int LEN = MAX_TILE_NUM; + constexpr int k_base = 0; + + // ----- 主 N 块 ----- + #pragma clang loop unroll(full) + for (int col = 0; col < Nb; col++) { + tile_shapeB tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, col); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + TCVT(tC_main[row][col], tACC); // is_first: 直接覆盖 + } + + if constexpr (rmd_M) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + TCVT(tC_rrow[col], tACC); + } + } // col + + // ----- rmd_N 尾列 ----- + if constexpr (rmd_N) { + tile_shapeB_trows tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + TCVT(tC_rcol[row], tACC); + } + + if constexpr (rmd_M) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + TCVT(tC_corner, tACC); + } + } + } // chunk[0] + + // ---------- chunk[1 .. K_chunks-1]: is_first = false ---------- + if constexpr (K_chunks > 1) { + #pragma clang loop unroll(full) + for (int kc = 1; kc < K_chunks; kc++) { + constexpr int LEN = MAX_TILE_NUM; + const int k_base = kc * MAX_TILE_NUM; + + // 主 N 块 + #pragma clang loop unroll(full) + for (int col = 0; col < Nb; col++) { + tile_shapeB tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, col); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][col], tC_main[row][col], tACC_vec); + } + + if constexpr (rmd_M) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[col], tC_rrow[col], tACC_vec); + } + } // col + + if constexpr (rmd_N) { + tile_shapeB_trows tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + + if constexpr (rmd_M) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } // kc + } // K_chunks > 1 + } // K_chunks > 0 + + // ============================================================ + // 段 2：K_rem (LEN = K_rem) + // is_first = (K_chunks == 0) + // ============================================================ + if constexpr (K_rem > 0) { + constexpr int LEN = K_rem; + constexpr int k_base = K_chunks * MAX_TILE_NUM; + constexpr bool is_first = (K_chunks == 0); + + #pragma clang loop unroll(full) + for (int col = 0; col < Nb; col++) { + tile_shapeB tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, col); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + if constexpr (is_first) { + TCVT(tC_main[row][col], tACC); + } else { + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][col], tC_main[row][col], tACC_vec); + } + } + + if constexpr (rmd_M) { + tile_shapeACC_tcols tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + if constexpr (is_first) { + TCVT(tC_rrow[col], tACC); + } else { + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[col], tC_rrow[col], tACC_vec); + } + } + } + + if constexpr (rmd_N) { + tile_shapeB_trows tB[LEN]; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + auto gB = gIterB(k_base + k, Nb); + TLOAD(tB[k], gB); + } + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC_trows tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA tA; + auto gA = gIterA(row, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + if constexpr (is_first) { + TCVT(tC_rcol[row], tACC); + } else { + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + } + + if constexpr (rmd_M) { + tile_shapeACC_tcorner tACC; + #pragma clang loop unroll(full) + for (int k = 0; k < LEN; k++) { + tile_shapeA_tcols tA; + auto gA = gIterA(Mb, k_base + k); + TLOAD(tA, gA); + if (k == 0) TMATMUL(tACC, tA, tB[k]); + else TMATMUL_ACC(tACC, tA, tB[k]); + } + if constexpr (is_first) { + TCVT(tC_corner, tACC); + } else { + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } + } // K_rem + + // ============================================================ + // 段 3：rmd_K + // is_first = (K_chunks == 0) && (K_rem == 0) + // ============================================================ + if constexpr (rmd_K) { + constexpr bool is_first = (K_chunks == 0) && (K_rem == 0); + + #pragma clang loop unroll(full) + for (int col = 0; col < Nb; col++) { + tile_shapeB_tcols tB_rmdK; + auto gB = gIterB(Kb, col); + TLOAD(tB_rmdK, gB); + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC tACC; + tile_shapeA_trows tA; + auto gA = gIterA(row, Kb); + TLOAD(tA, gA); + TMATMUL(tACC, tA, tB_rmdK); + + if constexpr (is_first) { + TCVT(tC_main[row][col], tACC); + } else { + tile_C_out tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_main[row][col], tC_main[row][col], tACC_vec); + } + } + + if constexpr (rmd_M) { + tile_shapeACC_tcols tACC; + tile_shapeA_tcorner tA; + auto gA = gIterA(Mb, Kb); + TLOAD(tA, gA); + TMATMUL(tACC, tA, tB_rmdK); + + if constexpr (is_first) { + TCVT(tC_rrow[col], tACC); + } else { + tile_C_out_tcols tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rrow[col], tC_rrow[col], tACC_vec); + } + } + } + + if constexpr (rmd_N) { + tile_shapeB_tcorner tB_rmdK; + auto gB = gIterB(Kb, Nb); + TLOAD(tB_rmdK, gB); + + #pragma clang loop unroll(full) + for (int row = 0; row < Mb; row++) { + tile_shapeACC_trows tACC; + tile_shapeA_trows tA; + auto gA = gIterA(row, Kb); + TLOAD(tA, gA); + TMATMUL(tACC, tA, tB_rmdK); + + if constexpr (is_first) { + TCVT(tC_rcol[row], tACC); + } else { + tile_C_out_trows tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_rcol[row], tC_rcol[row], tACC_vec); + } + } + + if constexpr (rmd_M) { + tile_shapeACC_tcorner tACC; + tile_shapeA_tcorner tA; + auto gA = gIterA(Mb, Kb); + TLOAD(tA, gA); + TMATMUL(tACC, tA, tB_rmdK); + + if constexpr (is_first) { + TCVT(tC_corner, tACC); + } else { + tile_C_out_tcorner tACC_vec; + TCVT(tACC_vec, tACC); + TADD(tC_corner, tC_corner, tACC_vec); + } + } + } + } // rmd_K + + // ============================================================ + // 全部 K 累加完成后，统一写回 global + // ============================================================ + #pragma clang loop unroll(full) + for (int m = 0; m < Mb; m++) { + #pragma clang loop unroll(full) + for (int n = 0; n < Nb; n++) { + auto gC = gIterC(m, n); + TSTORE(gC, tC_main[m][n]); + } + if constexpr (rmd_N) { + auto gC = gIterC(m, Nb); + TSTORE(gC, tC_rcol[m]); + } + } + if constexpr (rmd_M) { + #pragma clang loop unroll(full) + for (int n = 0; n < Nb; n++) { + auto gC = gIterC(Mb, n); + TSTORE(gC, tC_rrow[n]); + } + if constexpr (rmd_N) { + auto gC = gIterC(Mb, Nb); + TSTORE(gC, tC_corner); + } + } + + } // Batch + } + + + template + void matmul_mask_reuseB_tileop(float *dst, dtype *src0, dtype *src1){ + const int MAX_TILE_NUM = 24; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + // constexpr ResB R = find_reuseB(Nb, Kb, MAX_TILE_NUM); + constexpr ResB R = {1, (Kb > MAX_TILE_NUM) ? MAX_TILE_NUM : Kb, R.n * R.k}; + // static_assert(R.n == 1); + const int dN = R.n == 0? 0 : Nb / R.n; + const int rN = R.n == 0? 0 : Nb % R.n; + + static_assert(R.val <= MAX_TILE_NUM, "R.val is biggger than MAX_TILE_NUM"); + //printf("R.m is %d, R.k is %d\n", R.m, R.k); + + for (int b=0;b0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + + auto gC = gIterC(j, i*R.n+ii); + store_acc_tile_tileop(gC, tACC); + } + + // [n, rmd_M, k] + if constexpr (rmd_M) { + tile_shapeC_tcols tACC; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + + auto gC = gIterC(Mb, i*R.n+ii); + store_acc_tile_tileop(gC, tACC); + } + + } + } + + // [rN, m, k] + if constexpr(rN>0){ + tile_shapeB tB[R.k][rN]; + + #pragma clang loop unroll(full) + for(int i=0;i0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(j, i+dN*R.n); + store_acc_tile_tileop(gC, tACC); + } + + // [rN, rmd_M, k] + if constexpr (rmd_M) { + tile_shapeC_tcols tACC; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(Mb, i+dN*R.n); + store_acc_tile_tileop(gC, tACC); + } + } + } + + // [rmd_N, m, k] + if constexpr (rmd_N) { + tile_shapeB_trows tB[R.k]; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(j, Nb); + store_acc_tile_tileop(gC, tACC); + } + + // [rmd_N, rmd_M, k] + if constexpr (rmd_M) { + tile_shapeC_tcorner tACC; + + #pragma clang loop unroll(full) + for(int k=0;k0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + auto gC = gIterC(Mb,Nb); + store_acc_tile_tileop(gC, tACC); + } + } + + }// Batch + } + + struct ResAB { + int m; + int n; + int k; + int val; + }; + + constexpr ResAB find_reuseAB(int Mb, int Nb, int Kb, int MAX_TILE_NUM) { + int best_m = 0, best_k = 0, best_n = 0, best_val = -1; + + #pragma clang loop unroll(full) + for (int n = 1; n <= Nb; ++n) { + #pragma clang loop unroll(full) + for (int m = 1; m <= Mb; ++m) { + #pragma clang loop unroll(full) + for (int k = 1; k <= Kb; ++k) { + int v = m * k + n * k; + if (v <= MAX_TILE_NUM && v > best_val) { + best_val = v; + best_k = k; + best_m = m; + best_n = n; + } + } + } + } + return {best_m, best_n, best_k, best_val}; + } + + + template + void matmul_mask_reuseAB_tileop(float *dst, dtype *src0, dtype *src1){ + const int MAX_TILE_NUM = 24; + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + const int Mb = (gM + tM - 1) / tM; + const int Nb = (gN + tN - 1) / tN; + const int Kb = (gK + tK - 1) / tK; + + constexpr ResAB R = find_reuseAB(Mb, Nb, Kb, MAX_TILE_NUM); + + const int dM = Mb / R.m; + const int rM = Mb % R.m; + + const int dN = Nb / R.n; + const int rN = Nb % R.n; + + const int dK = Kb / R.k; + const int rK = Kb % R.k; + + static_assert(R.val <= MAX_TILE_NUM, "R.val is biggger than MAX_TILE_NUM"); + // printf("R.m is %d, R.n is %d, R.k is %d\n", R.m, R.n, R.k); + + for (int b=0;b + void matmul_mask_multi4_B_tileop(float *dst, dtype *src0, dtype *src1){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + for (int b=0;b + void matmul_mask_multi4_AB_tileop(float *dst, dtype *src0, dtype *src1){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gIterA(src0); + itB gIterB(src1); + itC gIterC(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + #pragma clang loop unroll(full) + for(int i=0;i + __attribute__((noinline)) void matmul_dynamic_new_tileop(float* dst, dtype* src0, dtype* src1, int gM, int gN, int gK){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + if (gM <= 0 || gN <= 0 || gK <= 0) + return; + + for (int b=0;b tK ? tK : gK - k; + tile_shapeA tA(dyn_m, dyn_k); + tile_shapeB tB(dyn_k, dyn_n); + TLOAD(tA, gA); + TLOAD(tB, gB); + if(k==0){ + TMATMUL(tACC, tA, tB); + }else{ + TMATMUL_ACC(tACC, tA, tB); + } + } + store_acc_tile_dynamic_tileop(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + } + } + } + } + + template + __attribute__((noinline)) void matmul_dynamic_tileop(float* dst, dtype* src0, dtype* src1, int gM, int gN, int gK){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + int Mb = (gM + tM - 1) / tM; + int Nb = (gN + tN - 1) / tN; + int Kb = (gK + tK - 1) / tK; + + int rem_m = gM % tM; + int rem_n = gN % tN; + int rem_k = gK % tK; + + for (int b=0;b gM ? rem_m:tM; + int dyn_n = (j+1) * tN > gN ? rem_n:tN; + + tile_shapeACC tACC(dyn_m, dyn_n); + for(int k=0;k gK ? rem_k:tK; + tile_shapeA tA(dyn_m, dyn_k); + tile_shapeB tB(dyn_k, dyn_n); + TLOAD(tA, gA); + TLOAD(tB, gB); + if(k==0){ + TMATMUL(tACC, tA, tB); + }else{ + TMATMUL_ACC(tACC, tA, tB); + } + } + store_acc_tile_dynamic_tileop(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + } + } + } + } + + /* + ResA find_reuseA_dynamic(int Mb, int Kb, int MAX_TILE_NUM) { + int best_m = 0, best_k = 0, best_val = -1; + + for (int m = 1; m <= Mb; ++m) { + for (int k = 1; k <= Kb; ++k) { + int v = m * k; + if (v <= MAX_TILE_NUM && v > best_val) { + best_val = v; + best_m = m; + best_k = k; + } + } + } + return {best_m, best_k, best_val}; + } + + template + __attribute__((noinline)) void matmul_dynamic_reuseA_tileop(float* dst, dtype* src0, dtype* src1, int gM, int gN, int gK){ + const int MAX_TILE_NUM = 24; + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + int Mb = (gM + tM - 1) / tM; + int Nb = (gN + tN - 1) / tN; + int Kb = (gK + tK - 1) / tK; + + int rem_m = gM % tM; + int rem_n = gN % tN; + int rem_k = gK % tK; + + ResA R = find_reuseA_dynamic(Mb, Kb, MAX_TILE_NUM); + + int dM = R.m == 0? 0 : Mb / R.m; + int rM = R.m == 0? 0 : Mb % R.m; + + // printf("Mb:%d, Nb:%d Kb:%d, R.m:%d, R.k:%d\n", Mb, Nb, Kb, R.m, R.k); + for (int b=0;b gM ){ + tA[mm][kk]= tile_shapeA(rem_m, tK); + }else{ + tA[mm][kk]= tile_shapeA(tM, tK); + } + } + } + + for(int ii=0;ii gM? rem_m:tM; + + for(int j=0;j gN ? rem_n:tN; + tile_shapeACC tACC(dyn_m, dyn_n); + + for(int k=0;k gK ? rem_k:tK; + tile_shapeA tA(dyn_m, dyn_k); + tile_shapeB tB(dyn_k, dyn_n); + + TLOAD(tA, gA); + TLOAD(tB, gB); + if(k==0){ + TMATMUL(tACC, tA, tB); + }else{ + TMATMUL_ACC(tACC, tA, tB); + } + } + } + + size_t offset_C = (i+ii) * gN * tile_shapeACC::Rows + j * tile_shapeACC::Cols; + gm_shapeC gC(dst + offset_C, gM, gN); + store_acc_tile_dynamic_tileop(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + } + } + + i+= m_step; + } + } + } + + template + __attribute__((noinline)) void matmul_dynamic_reuseB_tileop(float* dst, dtype* src0, dtype* src1, int gM, int gN, int gK){ + const int MAX_TILE_NUM = 24; + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + + int Mb = (gM + tM - 1) / tM; + int Nb = (gN + tN - 1) / tN; + int Kb = (gK + tK - 1) / tK; + + int rem_m = gM % tM; + int rem_n = gN % tN; + int rem_k = gK % tK; + + ResA Ra = find_reuseA_dynamic(Nb, Kb, MAX_TILE_NUM); + ResB R; + R.n = Ra.m; + R.k = Ra.k; + R.val = Ra.val; + + int dN = R.n == 0? 0 : Nb / R.n; + int rN = R.n == 0? 0 : Nb % R.n; + + for (int b=0;b gN ){ + tB[kk][nn]= tile_shapeB(tK, rem_n); + }else{ + tB[kk][nn]= tile_shapeB(tK, tN); + } + } + } + + for(int ii=0;ii gN? rem_n:tN; + + for(int j=0;j gM ? rem_m:tM; + tile_shapeACC tACC(dyn_m, dyn_n); + + for(int k=0;k gK ? rem_k:tK; + tile_shapeA tA(dyn_m, dyn_k); + tile_shapeB tB(dyn_k, dyn_n); + + TLOAD(tA, gA); + TLOAD(tB, gB); + if(k==0){ + TMATMUL(tACC, tA, tB); + }else{ + TMATMUL_ACC(tACC, tA, tB); + } + } + } + + size_t offset_C = j * gN * tile_shapeACC::Rows + (i+ii) * tile_shapeACC::Cols; + gm_shapeC gC(dst + offset_C, gM, gN); + store_acc_tile_dynamic_tileop(gC, tACC, tACC.GetValidRow(), tACC.GetValidCol()); + } + } + + i+= n_step; + } + } + } + */ + + + template + void matmul_mx_tileop(float *dst, dtype *src0, dtype *src1, uint8_t *src0_mx, uint8_t *src1_mx) { + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeAMX = global_tensor>; + using gm_shapeBMX = global_tensor>; + using gm_shapeC = global_tensor>; + + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeAMX = Tile; + using tile_shapeBMX = Tile; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itAMX = global_iterator; + using itBMX = global_iterator; + using itC = global_iterator; + + itA gAIter(src0); + itB gBIter(src1); + itAMX gAMXIter(src0_mx); + itBMX gBMXIter(src1_mx); + itC gCIter(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + for(int i=0;i 0) { + auto gA = gAIter(i,0); + auto gB = gBIter(0,j); + auto gAMX = gAMXIter(i,0); + auto gBMX = gBMXIter(0,j); + tile_shapeA tA; + tile_shapeB tB; + tile_shapeAMX tAMX; + tile_shapeBMX tBMX; + TLOAD(tA, gA); + TLOAD(tB, gB); + TLOAD(tAMX, gAMX); + TLOAD(tBMX, gBMX); + TMATMUL_MX(tACC, tA, tAMX, tB, tBMX); + } + if constexpr (Kb > 1) { + #pragma clang loop unroll(full) + for(int k=1;k + void matmul_mask_2lvl_tileop(float *c_ptr, dtype *a_ptr, dtype *b_ptr) { + + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = TileLeft; + using tile_shapeB = TileRight; + using tile_shapeACC = TileAcc; + using itA = global_iterator; + using itB = global_iterator; + using itC = global_iterator; + + itA gAIter(a_ptr); + itB gBIter(b_ptr); + itC gCIter(c_ptr); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + const int rmd_M = gM % tM; + const int rmd_N = gN % tN; + const int rmd_K = gK % tK; + + using tile_shapeA_trows = TileLeft; + using tile_shapeA_tcols = TileLeft; + using tile_shapeA_tcorner = TileLeft; + + using tile_shapeB_trows = TileRight; + using tile_shapeB_tcols = TileRight; + using tile_shapeB_tcorner = TileRight; + + using tile_shapeC_trows = TileAcc; + using tile_shapeC_tcols = TileAcc; + using tile_shapeC_tcorner = TileAcc; + + for (int b=0;b0){ + auto gA = gAIter(i, 0); + auto gB = gBIter(0, j); + + tile_shapeA tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(i, k); + auto gB = gBIter(k, j); + + tile_shapeA tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_trows tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(i, Nb); + + tile_shapeC_trows tACC; + if constexpr(Kb>0){ + auto gA = gAIter(i, 0); + auto gB = gBIter(0, Nb); + + tile_shapeA tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(i, k); + auto gB = gBIter(k, Nb); + + tile_shapeA tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(i, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_trows tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + } + if constexpr (rmd_M) { + for (int j = 0; j < Nb; ++j) { + auto gC = gCIter(Mb, j); + + tile_shapeC_tcols tACC; + if constexpr(Kb>0){ + auto gA = gAIter(Mb, 0); + auto gB = gBIter(0, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, j); + + tile_shapeA_tcols tA; + tile_shapeB tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, j); + + tile_shapeA_tcorner tA; + tile_shapeB_tcols tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + if constexpr (rmd_N) { + auto gC = gCIter(Mb, Nb); + + tile_shapeC_tcorner tACC; + if constexpr(Kb>0){ + auto gA = gAIter(Mb, 0); + auto gB = gBIter(0, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL(tACC, tA, tB); + } + #pragma clang loop unroll(full) + for (int k = 1; k < Kb; ++k) { + auto gA = gAIter(Mb, k); + auto gB = gBIter(k, Nb); + + tile_shapeA_tcols tA; + tile_shapeB_trows tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + TMATMUL_ACC(tACC, tA, tB); + } + if constexpr (rmd_K) { + auto gA = gAIter(Mb, Kb); + auto gB = gBIter(Kb, Nb); + + tile_shapeA_tcorner tA; + tile_shapeB_tcorner tB; + TLOAD(tA, gA); + TLOAD(tB, gB); + if constexpr(Kb>0){ + TMATMUL_ACC(tACC, tA, tB); + } else { + TMATMUL(tACC, tA, tB); + } + } + store_acc_tile_tileop(gC, tACC); + } + } + } + } + + template + void matmul_vec_tileop(float* dst, float* src0, float* src1){ + using gm_shapeA = global_tensor>; + using gm_shapeB = global_tensor>; + using gm_shapeC = global_tensor>; + using tile_shapeA = Tile; + using tile_shapeB = Tile; + using tile_shapeACC = TileAcc; + using gm_iteratorA = global_iterator; + using gm_iteratorB = global_iterator; + using gm_iteratorC = global_iterator; + + gm_iteratorA gAIter(src0); + gm_iteratorB gBIter(src1); + gm_iteratorC gCIter(dst); + + const int Mb = gM / tM; + const int Nb = gN / tN; + const int Kb = gK / tK; + + for(int i=0;i + void matmul_tile_vec_tileop(float* dst, float* src0, float* src1) { + using gm_shape_A = global_tensor>; + using gm_shape_B = global_tensor>; + using gm_shape_C = global_tensor>; + + using tile_shape_A = TileLeft; + using tile_shape_B = TileRight; + using tile_shape_C = TileAcc; + + gm_shape_A s0(src0); + gm_shape_B s1(src1); + gm_shape_C res(dst); + + tile_shape_A d0; + tile_shape_B d1; + tile_shape_C d2; + + TLOAD(d0, s0); + TLOAD(d1, s1); + TMATMUL(d2, d0, d1); + store_acc_tile_tileop(res, d2); + } + + template + void matmul_tile_frac_tileop(float* dst, float* src0, float* src1) { + using gm_shape_A = global_tensor>; + using gm_shape_B = global_tensor>; + using gm_shape_C = global_tensor>; + + using tile_shape_A = TileLeft; + using tile_shape_B = TileRight; + using tile_shape_C = TileAcc; + + gm_shape_A s0(src0); + gm_shape_B s1(src1); + gm_shape_C res(dst); + + tile_shape_A d0; + tile_shape_B d1; + tile_shape_C d2; + + TLOAD(d0, s0); + TLOAD(d1, s1); + TMATMUL(d2, d0, d1); + store_acc_tile_tileop(res, d2); + } + + + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `ACCCVT`, `TCVT` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TMATMUL`](../../../../intrinsics/tmatmul.md) | `TMATMUL` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TMATMUL_ACC`](../../../../intrinsics/tmatmul_acc.md) | `TMATMUL_ACC` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TMATMUL_MX`](../../../../intrinsics/tmatmul_mx.md) | `TMATMUL_MX` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/matmul/matmul.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/matmul` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 46 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32 M=256 N=256 K=256 tM=32 tN=32 tK=64` | +| 47 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32_REUSEA M=256 N=256 K=256 tM=32 tN=32 tK=64` | +| 48 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP32_REUSEB M=256 N=256 K=256 tM=32 tN=32 tK=64` | +| 53 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16 M=256 N=256 K=256 tM=64 tN=32 tK=64` | +| 54 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16_REUSEA M=256 N=256 K=256 tM=64 tN=32 tK=64` | +| 55 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP16_REUSEB M=256 N=256 K=256 tM=64 tN=32 tK=64` | +| 58 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8 M=256 N=256 K=256 tM=64 tN=32 tK=64` | +| 59 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8_REUSEA M=256 N=256 K=256 tM=64 tN=32 tK=64` | +| 60 | `make TESTCASE=matmul TYPE=MASK MODE=MASK_FP8_REUSEB M=256 N=256 K=256 tM=64 tN=32 tK=64` | + +## Resolved One-Level Source Closure + +??? info "7 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/matmul/matmul.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/src/benchmark.h` + - `benchmark/one-level-arch/test/common/src/common.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/matmul/src/matmul.cpp` diff --git a/docs/benchmarks/catalog/one-level/reduction-reducemax-col/index.md b/docs/benchmarks/catalog/one-level/reduction-reducemax-col/index.md new file mode 100644 index 0000000..5bbf03c --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducemax-col/index.md @@ -0,0 +1,10 @@ +# One-level: reduction/reducemax_col + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [reducemax_col](reducemax-col-1c7de9cb.md) | `benchmark/one-level-arch/test/kernel/reduction/reducemax_col/src/reducemax_col.cpp` | 1 | [`TCOLMAX`](../../../../intrinsics/tcolmax.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMAX`](../../../../intrinsics/tmax.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/reduction-reducemax-col/reducemax-col-1c7de9cb.md b/docs/benchmarks/catalog/one-level/reduction-reducemax-col/reducemax-col-1c7de9cb.md new file mode 100644 index 0000000..f8fd510 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducemax-col/reducemax-col-1c7de9cb.md @@ -0,0 +1,193 @@ +# reducemax_col: reducemax_col.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `reduction/reducemax_col` | +| Implementation source | `benchmark/one-level-arch/test/kernel/reduction/reducemax_col/src/reducemax_col.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="reducemax_col.cpp" linenums="1" +#include + +#include + +#include "fileop.h" +#include "reduction/reducemax_colvec_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + + +#ifndef tM +#define tM 32 +#endif + +#ifndef tN +#define tN 64 +#endif + + + + +#define gIM 256 +#define gIN 256 +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + + dtype input_buf[gIM*gIN]; +// dtype zero_buf[1*gIN]; + dtype output_buf[1*gIN]; + + dtype* input=input_buf; +// dtype* zero=zero_buf; + dtype* output=output_buf; + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducemax_col/src/data_256x256.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducemax_col/src/data_8192x1024.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducemax_col/src/data1x256_zero.bin", (uint8_t*)zero, gIN * sizeof(dtype)); +// reducesum_colsum_rand(input, output); + reducemax_col_rand(input, output); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducemax_col/src/result_max.bin", (uint8_t*)output, gIN * sizeof(dtype)); + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp" + + ```cpp title="reducemax_colvec_pto.hpp" linenums="1" + #ifndef REDUCEMAXCOLVEC_KERNEL_HPP + #define REDUCEMAXCOLVEC_KERNEL_HPP + + #include + #include + #include + + using namespace pto; + + template + void reducemax_col_rand( + dtype *in_ptr, + dtype *out_ptr + ) + { + const int Mb = gIM / tM; + const int Nb = gIN / tN; + const int rmd_M = gIM % tM; + const int rmd_N = gIN % tN; + + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeData_col = Tile; + using tile_shapeMax = Tile; + using tile_shapeData_row = Tile; + using tile_shapeData_cor = Tile; + using tile_shapeMax_row = Tile; + + gm_shapeIn inGm(in_ptr); + gm_shapeOut outGm(out_ptr); + + tile_shapeData dataTile; + tile_shapeData_col dataTile_col; + tile_shapeMax MaxTile; + tile_shapeMax oldMaxTile; + tile_shapeData_row dataTile_row; + tile_shapeData_cor dataTile_cor; + tile_shapeMax_row MaxTile_row; + tile_shapeMax_row oldMaxTile_row; + + using itIn = global_iterator; + using itOut = global_iterator; + + itIn gIIter(in_ptr); + itOut gOIter(out_ptr); + + for (int j = 0; j < Nb; ++j) { + auto gO = gOIter(0, j); + TEXPANDSCALAR(oldMaxTile, static_cast(0)); + + for (int i = 0; i < Mb; ++i) { + auto gI = gIIter(i, j); + TLOAD(dataTile, gI); + TCOLMAX(MaxTile, dataTile); + TMAX(oldMaxTile, oldMaxTile, MaxTile); + } + if constexpr (rmd_M > 0) { + auto gI = gIIter(Mb, j); + TLOAD(dataTile_col, gI); + TCOLMAX(MaxTile, dataTile_col); + TMAX(oldMaxTile, oldMaxTile, MaxTile); + } + TSTORE(gO, oldMaxTile); + } + if constexpr (rmd_N > 0) { + auto gO = gOIter(0, Nb); + TEXPANDSCALAR(oldMaxTile_row, static_cast(0)); + + for (int i = 0; i < Mb; ++i) { + auto gI = gIIter(i, Nb); + TLOAD(dataTile_row, gI); + TCOLMAX(MaxTile_row, dataTile_row); + TMAX(oldMaxTile_row, oldMaxTile_row, MaxTile_row); + } + if constexpr (rmd_M > 0) { + auto gI = gIIter(Mb, Nb); + TLOAD(dataTile_cor, gI); + TCOLMAX(MaxTile_row, dataTile_cor); + TMAX(oldMaxTile_row, oldMaxTile_row, MaxTile_row); + } + TSTORE(gO, oldMaxTile_row); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TCOLMAX`](../../../../intrinsics/tcolmax.md) | `TCOLMAX` | `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` | +| [`TMAX`](../../../../intrinsics/tmax.md) | `TMAX` | `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/reduction/reducemax_col` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 7 | `make TESTCASE=reducemax_col COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=32 tN_s=64 gM_s=2048 gN_s=64` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/reduction/reducemax_colvec_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/reduction/reducemax_col/src/reducemax_col.cpp` diff --git a/docs/benchmarks/catalog/one-level/reduction-reducemax-row/index.md b/docs/benchmarks/catalog/one-level/reduction-reducemax-row/index.md new file mode 100644 index 0000000..6c72b18 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducemax-row/index.md @@ -0,0 +1,10 @@ +# One-level: reduction/reducemax_row + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [reducemax_row](reducemax-row-6e95758c.md) | `benchmark/one-level-arch/test/kernel/reduction/reducemax_row/src/reducemax_row.cpp` | 1 | [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMAX`](../../../../intrinsics/tmax.md), [`TROWMAX`](../../../../intrinsics/trowmax.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/reduction-reducemax-row/reducemax-row-6e95758c.md b/docs/benchmarks/catalog/one-level/reduction-reducemax-row/reducemax-row-6e95758c.md new file mode 100644 index 0000000..8163851 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducemax-row/reducemax-row-6e95758c.md @@ -0,0 +1,198 @@ +# reducemax_row: reducemax_row.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `reduction/reducemax_row` | +| Implementation source | `benchmark/one-level-arch/test/kernel/reduction/reducemax_row/src/reducemax_row.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="reducemax_row.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +#include "reduction/reducemax_rowvec_pto.hpp" + + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 128 +#endif + +#ifndef tNs +#define tNs 64 +#endif + +#ifndef gIMs +#define gIMs 1024 +#endif + +#ifndef gINs +#define gINs 8192 +#endif +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + + dtype input[gIMs*gINs]; + dtype output[gIMs*1]; + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/data_256x256.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/data_1024x8192.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); + reducemax_row_rand(input, output); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/result_rowsum.bin", (uint8_t*)output, gIM * sizeof(dtype)); +//每个tile只有前两个位置有数。 +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp" + + ```cpp title="reducemax_rowvec_pto.hpp" linenums="1" + #ifndef REDUCEMAXROWVEC_KERNEL_HPP + #define REDUCEMAXROWVEC_KERNEL_HPP + + #pragma once + #include + #include + #include + + using namespace pto; + + template + void reducemax_row_rand( + dtype *in_ptr, + dtype *out_ptr + ) + { + const int Mb = gIM / tM; + const int Nb = gIN / tN; + const int rmd_M = gIM % tM; + const int rmd_N = gIN % tN; + + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeData_row = Tile; + using tile_shapeMax = Tile; + using tile_shapeData_col = Tile; + using tile_shapeData_cor = Tile; + using tile_shapeMax_col = Tile; + using tile_shapeTmp = Tile; + using tile_shapeTmp_row = Tile; + using tile_shapeTmp_col = Tile; + using tile_shapeTmp_cor = Tile; + + gm_shapeIn inGm(in_ptr); + gm_shapeOut outGm(out_ptr); + + tile_shapeData dataTile; + tile_shapeData_row dataTile_row; + tile_shapeData_col dataTile_col; + tile_shapeData_cor dataTile_cor; + tile_shapeMax MaxTile; + tile_shapeMax oldMaxTile; + tile_shapeMax_col MaxTile_col; + tile_shapeMax_col oldMaxTile_col; + tile_shapeTmp tmpTile; + tile_shapeTmp_row tmpTile_row; + tile_shapeTmp_col tmpTile_col; + tile_shapeTmp_cor tmpTile_cor; + + using itIn = global_iterator; + using itOut = global_iterator; + + itIn gIIter(in_ptr); + itOut gOIter(out_ptr); + + for (int j = 0; j < Mb; ++j) { + auto gO = gOIter(j, 0); + TEXPANDSCALAR(oldMaxTile, static_cast(0)); + + for (int i = 0; i < Nb; ++i) { + auto gI = gIIter(j, i); + TLOAD(dataTile, gI); + TROWMAX(MaxTile, dataTile); + TMAX(oldMaxTile, oldMaxTile, MaxTile); + } + if constexpr (rmd_N > 0) { + auto gI = gIIter(j, Nb); + TLOAD(dataTile_row, gI); + TROWMAX(MaxTile, dataTile_row); + TMAX(oldMaxTile, oldMaxTile, MaxTile); + } + TSTORE(gO, oldMaxTile); + } + if constexpr (rmd_M > 0) { + auto gO = gOIter(Mb, 0); + TEXPANDSCALAR(oldMaxTile_col, static_cast(0)); + + for (int i = 0; i < Nb; ++i) { + auto gI = gIIter(Mb, i); + TLOAD(dataTile_col, gI); + TROWMAX(MaxTile_col, dataTile_col); + TMAX(oldMaxTile_col, oldMaxTile_col, MaxTile_col); + } + if constexpr (rmd_N > 0) { + auto gI = gIIter(Mb, Nb); + TLOAD(dataTile_cor, gI); + TROWMAX(MaxTile_col, dataTile_cor); + TMAX(oldMaxTile_col, oldMaxTile_col, MaxTile_col); + } + TSTORE(gO, oldMaxTile_col); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` | +| [`TMAX`](../../../../intrinsics/tmax.md) | `TMAX` | `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` | +| [`TROWMAX`](../../../../intrinsics/trowmax.md) | `TROWMAX` | `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/reduction/reducemax_row` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 7 | `make TESTCASE=reducemax_row COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=16 tN_s=128 gM_s=16 gN_s=8192` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/reduction/reducemax_rowvec_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/reduction/reducemax_row/src/reducemax_row.cpp` diff --git a/docs/benchmarks/catalog/one-level/reduction-reducesum-col/index.md b/docs/benchmarks/catalog/one-level/reduction-reducesum-col/index.md new file mode 100644 index 0000000..b234b9a --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducesum-col/index.md @@ -0,0 +1,10 @@ +# One-level: reduction/reducesum_col + + + +This family contains **1** source implementations and +**2** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [reducesum_col](reducesum-col-d8b6df3d.md) | `benchmark/one-level-arch/test/kernel/reduction/reducesum_col/src/reducesum_col.cpp` | 2 | [`TADD`](../../../../intrinsics/tadd.md), [`TCOLSUM`](../../../../intrinsics/tcolsum.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/reduction-reducesum-col/reducesum-col-d8b6df3d.md b/docs/benchmarks/catalog/one-level/reduction-reducesum-col/reducesum-col-d8b6df3d.md new file mode 100644 index 0000000..02747d6 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducesum-col/reducesum-col-d8b6df3d.md @@ -0,0 +1,210 @@ +# reducesum_col: reducesum_col.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `reduction/reducesum_col` | +| Implementation source | `benchmark/one-level-arch/test/kernel/reduction/reducesum_col/src/reducesum_col.cpp` | +| Active build variants | 2 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="reducesum_col.cpp" linenums="1" +#include + +#include + +#include "fileop.h" +#include "reduction/reducesum_colvec_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef gIMs +#define gIMs 8192 +#endif + +#ifndef gINs +#define gINs 1024 +#endif + + +#ifndef tMs +#define tMs 32 +#endif + +#ifndef tNs +#define tNs 128 +#endif + + + + +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + + dtype input_buf[gIMs*gINs]; +// dtype zero_buf[1*gIN]; + dtype output_buf[1*gINs]; + + dtype* input=input_buf; +// dtype* zero=zero_buf; + dtype* output=output_buf; + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_colsum/src/data_256x256.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_colsum/src/data_8192x1024.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_colsum/src/data1x256_zero.bin", (uint8_t*)zero, gIN * sizeof(dtype)); +// reducesum_colsum_rand(input, output); + reducesum_colsum_rand(input, output); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_colsum/src/result.bin", (uint8_t*)output, gIN * sizeof(dtype)); + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp" + + ```cpp title="reducesum_colvec_pto.hpp" linenums="1" + #ifndef REDUCESUMCOLVEC_KERNEL_HPP + #define REDUCESUMCOLVEC_KERNEL_HPP + + #include + #include + #include + + using namespace pto; + + template + void reducesum_colsum_rand( + dtype *in_ptr, + dtype *out_ptr + ) + { + const int Mb = gIM / tM; + const int Nb = gIN / tN; + const int rmd_M = gIM % tM; + const int rmd_N = gIN % tN; + + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeData_col = Tile; + using tile_shapeSum = Tile; + using tile_shapeData_row = Tile; + using tile_shapeData_cor = Tile; + using tile_shapeSum_row = Tile; + + using tile_shapeTmp = Tile; + using tile_shapeTmp_col = Tile; + using tile_shapeTmp_row = Tile; + using tile_shapeTmp_cor = Tile; + + gm_shapeIn inGm(in_ptr); + gm_shapeOut outGm(out_ptr); + + tile_shapeData dataTile; + tile_shapeData_col dataTile_col; + tile_shapeSum SumTile; + tile_shapeSum oldSumTile; + tile_shapeData_row dataTile_row; + tile_shapeData_cor dataTile_cor; + tile_shapeSum_row SumTile_row; + tile_shapeSum_row oldSumTile_row; + + tile_shapeTmp tmpTile; + tile_shapeTmp_col tmpTile_col; + tile_shapeTmp_row tmpTile_row; + tile_shapeTmp_cor tmpTile_cor; + + using itIn = global_iterator; + using itOut = global_iterator; + + itIn gIIter(in_ptr); + itOut gOIter(out_ptr); + + for (int j = 0; j < Nb; ++j) { + auto gO = gOIter(0, j); + TEXPANDSCALAR(oldSumTile, static_cast(0)); + + for (int i = 0; i < Mb; ++i) { + auto gI = gIIter(i, j); + TLOAD(dataTile, gI); + TCOLSUM(SumTile, dataTile); + TADD(oldSumTile, oldSumTile, SumTile); + } + if constexpr (rmd_M > 0) { + auto gI = gIIter(Mb, j); + TLOAD(dataTile_col, gI); + TCOLSUM(SumTile, dataTile_col); + TADD(oldSumTile, oldSumTile, SumTile); + } + TSTORE(gO, oldSumTile); + } + if constexpr (rmd_N > 0) { + auto gO = gOIter(0, Nb); + TEXPANDSCALAR(oldSumTile_row, static_cast(0)); + + for (int i = 0; i < Mb; ++i) { + auto gI = gIIter(i, Nb); + TLOAD(dataTile_row, gI); + TCOLSUM(SumTile_row, dataTile_row); + TADD(oldSumTile_row, oldSumTile_row, SumTile_row); + } + if constexpr (rmd_M > 0) { + auto gI = gIIter(Mb, Nb); + TLOAD(dataTile_cor, gI); + TCOLSUM(SumTile_row, dataTile_cor); + TADD(oldSumTile_row, oldSumTile_row, SumTile_row); + } + TSTORE(gO, oldSumTile_row); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` | +| [`TCOLSUM`](../../../../intrinsics/tcolsum.md) | `TCOLSUM` | `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/reduction/reducesum_col` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 7 | `make TESTCASE=reducesum_col COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=32 tN_s=64 gM_s=2048 gN_s=64` | +| 11 | `make TESTCASE=reducesum_col COMPILER_DIR=$COMPILER_DIR DType=__half tM_s=32 tN_s=64 gM_s=2048 gN_s=64` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/reduction/reducesum_colvec_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/reduction/reducesum_col/src/reducesum_col.cpp` diff --git a/docs/benchmarks/catalog/one-level/reduction-reducesum-row/index.md b/docs/benchmarks/catalog/one-level/reduction-reducesum-row/index.md new file mode 100644 index 0000000..a1a3d65 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducesum-row/index.md @@ -0,0 +1,10 @@ +# One-level: reduction/reducesum_row + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [reducesum_row](reducesum-row-f79bc562.md) | `benchmark/one-level-arch/test/kernel/reduction/reducesum_row/src/reducesum_row.cpp` | 1 | [`TADD`](../../../../intrinsics/tadd.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TROWSUM`](../../../../intrinsics/trowsum.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/reduction-reducesum-row/reducesum-row-f79bc562.md b/docs/benchmarks/catalog/one-level/reduction-reducesum-row/reducesum-row-f79bc562.md new file mode 100644 index 0000000..f7f9851 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/reduction-reducesum-row/reducesum-row-f79bc562.md @@ -0,0 +1,199 @@ +# reducesum_row: reducesum_row.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `reduction/reducesum_row` | +| Implementation source | `benchmark/one-level-arch/test/kernel/reduction/reducesum_row/src/reducesum_row.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then reduces or broadcasts along tile axes, then applies vector elementwise arithmetic, then commits the result to global memory. + +```cpp title="reducesum_row.cpp" linenums="1" +#include + +#include +#include + +#include "fileop.h" +//#include "reduction/reducesum_rowvec.hpp" +#include "reduction/reducesum_rowvec_pto.hpp" + + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 128 +#endif + +#ifndef tNs +#define tNs 64 +#endif + +#ifndef gIMs +#define gIMs 1024 +#endif + +#ifndef gINs +#define gINs 8192 +#endif +// ============================================================================ +// main +// ============================================================================ +int main() { + using dtype = DType; + + dtype input[gIMs*gINs]; + dtype output[gIMs*1]; + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/data_256x256.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/data_1024x8192.bin", (uint8_t*)input, gIM * gIN * sizeof(dtype)); + reducesum_trowsum_rand(input, output); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/reducesum_trowsum/src/result_rowsum.bin", (uint8_t*)output, gIM * sizeof(dtype)); +//每个tile只有前两个位置有数。 +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp" + + ```cpp title="reducesum_rowvec_pto.hpp" linenums="1" + #ifndef REDUCESUMTROWSUM_KERNEL_HPP + #define REDUCESUMTROWSUM_KERNEL_HPP + + #pragma once + #include + #include + #include + + using namespace pto; + + template + void reducesum_trowsum_rand( + dtype *in_ptr, + dtype *out_ptr + ) + { + const int Mb = gIM / tM; + const int Nb = gIN / tN; + const int rmd_M = gIM % tM; + const int rmd_N = gIN % tN; + + using gm_shapeIn = global_tensor>; + using gm_shapeOut = global_tensor>; + using tile_shapeData = Tile; + using tile_shapeData_row = Tile; + using tile_shapeSum = Tile; + using tile_shapeData_col = Tile; + using tile_shapeData_cor = Tile; + using tile_shapeSum_col = Tile; + using tile_shapeTmp = Tile; + using tile_shapeTmp_row = Tile; + using tile_shapeTmp_col = Tile; + using tile_shapeTmp_cor = Tile; + + gm_shapeIn inGm(in_ptr); + gm_shapeOut outGm(out_ptr); + + tile_shapeData dataTile; + tile_shapeData_row dataTile_row; + tile_shapeData_col dataTile_col; + tile_shapeData_cor dataTile_cor; + tile_shapeSum SumTile; + tile_shapeSum oldSumTile; + tile_shapeSum_col SumTile_col; + tile_shapeSum_col oldSumTile_col; + tile_shapeTmp tmpTile; + tile_shapeTmp_row tmpTile_row; + tile_shapeTmp_col tmpTile_col; + tile_shapeTmp_cor tmpTile_cor; + + using itIn = global_iterator; + using itOut = global_iterator; + + itIn gIIter(in_ptr); + itOut gOIter(out_ptr); + + for (int j = 0; j < Mb; ++j) { + auto gO = gOIter(j, 0); + TEXPANDSCALAR(oldSumTile, static_cast(0)); + + for (int i = 0; i < Nb; ++i) { + auto gI = gIIter(j, i); + TLOAD(dataTile, gI); + TROWSUM(SumTile, dataTile); + TADD(oldSumTile, oldSumTile, SumTile); + } + if constexpr (rmd_N > 0) { + auto gI = gIIter(j, Nb); + TLOAD(dataTile_row, gI); + TROWSUM(SumTile, dataTile_row); + TADD(oldSumTile, oldSumTile, SumTile); + } + TSTORE(gO, oldSumTile); + } + if constexpr (rmd_M > 0) { + auto gO = gOIter(Mb, 0); + TEXPANDSCALAR(oldSumTile_col, static_cast(0)); + + for (int i = 0; i < Nb; ++i) { + auto gI = gIIter(Mb, i); + TLOAD(dataTile_col, gI); + TROWSUM(SumTile_col, dataTile_col); + TADD(oldSumTile_col, oldSumTile_col, SumTile_col); + } + if constexpr (rmd_N > 0) { + auto gI = gIIter(Mb, Nb); + TLOAD(dataTile_cor, gI); + TROWSUM(SumTile_col, dataTile_cor); + TADD(oldSumTile_col, oldSumTile_col, SumTile_col); + } + TSTORE(gO, oldSumTile_col); + } + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` | +| [`TROWSUM`](../../../../intrinsics/trowsum.md) | `TROWSUM` | `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/reduction/reducesum_row` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 7 | `make TESTCASE=reducesum_row COMPILER_DIR=$COMPILER_DIR DType=int32_t tM_s=16 tN_s=128 gM_s=16 gN_s=8192` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/reduction/reducesum_rowvec_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/reduction/reducesum_row/src/reducesum_row.cpp` diff --git a/docs/benchmarks/catalog/one-level/sort/index.md b/docs/benchmarks/catalog/one-level/sort/index.md new file mode 100644 index 0000000..96fd1c2 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/sort/index.md @@ -0,0 +1,10 @@ +# One-level: sort + + + +This family contains **1** source implementations and +**1** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [topk](topk-0a98c8a8.md) | `benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp` | 1 | [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TSTORE`](../../../../intrinsics/tstore.md) | diff --git a/docs/benchmarks/catalog/one-level/sort/topk-0a98c8a8.md b/docs/benchmarks/catalog/one-level/sort/topk-0a98c8a8.md new file mode 100644 index 0000000..a8e06f4 --- /dev/null +++ b/docs/benchmarks/catalog/one-level/sort/topk-0a98c8a8.md @@ -0,0 +1,246 @@ +# topk: topk.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `sort` | +| Implementation source | `benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp` | +| Active build variants | 1 | + +## How the Code Is Written + +The implementation commits the result to global memory. + +```cpp title="topk.cpp" linenums="1" +#include +#include "benchmark.h" +#include "fileop.h" +#include "template_asm.h" +#include +#include + +#include "sort/topk.hpp" + +// #define FOR_GFSIM +// ============================================================================ +// ELF Data layout +// ============================================================================ + +extern "C" { + extern const uint8_t _binary_input_131072_data_start[]; + extern const uint8_t _binary_input_131072_data_end[]; + extern const uint8_t _binary_top_2048_out_data_start[]; + extern const uint8_t _binary_top_2048_out_data_end[]; +} + +static uint16_t* g_input = reinterpret_cast( + const_cast(_binary_input_131072_data_start)); +static uint16_t* g_expected = reinterpret_cast( + const_cast(_binary_top_2048_out_data_start)); + +// ============================================================================ +// Global-scope buffers +// ============================================================================ + +static uint16_t g_output[kInputCount]; + +// ============================================================================ +// main +// ============================================================================ + +int main() { +#ifndef FOR_GFSIM + printf("=== TopK Test (SIMT per-bucket) ===\n"); + printf("Input: %d TopK: %d Tiles: %d TileSize: %d\n", + kInputCount, kTopK, kNumTiles, kTileSize); + fflush(stdout); +#endif + + // ------------------------------------------------------------------------- + // Phase 1: SIMT high8 histogram (1 block × 256 lanes, each lane = 1 bucket) + // ------------------------------------------------------------------------- + TileU32 high8HistTile; + TEXPANDSCALAR(high8HistTile, static_cast(0)); + ExtractHigh8Hist_Impl< TileU32 >(high8HistTile, g_input); + + // Copy histogram results out and reduce to global 256-bin histogram + using HistGT = GlobalTensor, Stride<1,1,1,16,1>>; + uint32_t histResult[256]; + HistGT histGlobal(histResult); + TSTORE(histGlobal, high8HistTile); + + uint32_t global_high8_hist[256] = {0}; + for (int b = 0; b < 256; b++) { + global_high8_hist[b] = histResult[b]; + } + +#ifndef FOR_GFSIM + printf("\nPhase 1: high8 histograms built (1 SIMT launch, 256 lanes).\n"); + fflush(stdout); +#endif + + // ------------------------------------------------------------------------- + // Phase 2: Scalar prefix scan → kth_bin and need_from_kth_bin + // ------------------------------------------------------------------------- + int need_from_kth_bin = 0; + int kth_bin = find_kth_bin(global_high8_hist, kTopK, need_from_kth_bin); + +#ifndef FOR_GFSIM + printf("\nPhase 2: kth_bin=%d need_from_kth_bin=%d\n", + kth_bin, need_from_kth_bin); + uint64_t total_above = 0; + for (int b = kth_bin + 1; b < 256; b++) total_above += global_high8_hist[b]; + printf(" Elements in bins > kth_bin: %lu (expected ~%d)\n", + total_above, kTopK - need_from_kth_bin); + printf(" Elements in bin == kth_bin: %u\n", global_high8_hist[kth_bin]); + fflush(stdout); +#endif + + // ------------------------------------------------------------------------- + // Phase 3: SIMT low8 histogram for kth_bin elements + // ------------------------------------------------------------------------- + TileU32 low8HistTile; + TEXPANDSCALAR(low8HistTile, static_cast(0)); + ExtractLow8HistForKthBin_Impl< TileU32 >(low8HistTile, g_input, + static_cast(kth_bin)); + + uint32_t low8HistResult[256]; + HistGT low8HistGlobal(low8HistResult); + TSTORE(low8HistGlobal, low8HistTile); + + uint32_t global_low8_hist_kth[256] = {0}; + for (int b = 0; b < 256; b++) { + global_low8_hist_kth[b] = low8HistResult[b]; + } + + // ------------------------------------------------------------------------- + // Phase 4: Scalar prefix scan → low8_boundary + // ------------------------------------------------------------------------- + int low8_boundary = 0; + uint64_t cumsum_low = 0; + for (int b = 255; b >= 0; b--) { + cumsum_low += global_low8_hist_kth[b]; + if (cumsum_low >= static_cast(need_from_kth_bin)) { + low8_boundary = b; + break; + } + } + +#ifndef FOR_GFSIM + printf("\nPhase 4: low8_boundary=%d\n", low8_boundary); + printf(" Global low8 hist (kth bin) total: %lu\n", cumsum_low); + fflush(stdout); +#endif + + // ------------------------------------------------------------------------- + // Phase 5: Scalar masked scatter (directly on g_input / g_output) + // ------------------------------------------------------------------------- + memset(g_output, 0, sizeof(g_output)); + for (int i = 0; i < kInputCount; i++) { + uint16_t val = g_input[i]; + uint8_t high8 = static_cast(val >> 8); + int low8 = static_cast(val & 0xFF); + int include = (high8 > kth_bin) || + ((high8 == kth_bin) & (low8 >= low8_boundary)); + if (include) { + g_output[i] = val; + } + } + + // ------------------------------------------------------------------------- + // Host: collect non-zero entries + // ------------------------------------------------------------------------- + uint16_t result[kTopK]; + int out_count = 0; + for (int i = 0; i < kInputCount && out_count < kTopK; i++) { + if (g_output[i] != 0) { + result[out_count++] = g_output[i]; + } + } + +#ifndef FOR_GFSIM + printf("\nPhase 5: Collected %d output elements (expected %d)\n", + out_count, kTopK); + fflush(stdout); +#endif + + // ------------------------------------------------------------------------- + // Verification + // ------------------------------------------------------------------------- + int cmp_count = (out_count < kTopK) ? out_count : kTopK; + + uint16_t result_sorted[2048]; + memcpy(result_sorted, result, sizeof(result_sorted)); + for (int i = 0; i < cmp_count; i++) { + for (int j = i + 1; j < cmp_count; j++) { + if (result_sorted[i] < result_sorted[j]) { + uint16_t tmp = result_sorted[i]; + result_sorted[i] = result_sorted[j]; + result_sorted[j] = tmp; + } + } + } + + int match = 0; + for (int i = 0; i < cmp_count; i++) { + if (result_sorted[i] == g_expected[i]) match++; + } + +#ifndef FOR_GFSIM + printf("\n=== Verification (vs. embedded standard answer) ===\n"); + printf("Match: %d/%d (%.1f%%)\n", match, cmp_count, 100.0 * match / cmp_count); + printf("Output[0..9]: "); + for (int i = 0; i < 10 && i < out_count; i++) printf("0x%04x ", result_sorted[i]); + printf("\nExpected[0..9]: "); + for (int i = 0; i < 10; i++) printf("0x%04x ", g_expected[i]); + printf("\n"); +#endif + + int ret = (match == cmp_count) ? 0 : 1; +#ifndef FOR_GFSIM + printf("%s\n", ret ? "FAIL" : "PASS"); + fflush(stdout); +#endif + return ret; +} +``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/sort` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 2 | `make TESTCASE=topk` | + +## Required Data Objects + +The benchmark build or run path uses these checked-in generators/data inputs: + +- `benchmark/one-level-arch/test/kernel/sort/topk/data_obj/build_data_obj.sh` + +## Resolved One-Level Source Closure + +??? info "7 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/sort/topk.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/src/benchmark.h` + - `benchmark/one-level-arch/test/common/template_asm.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/sort/topk/topk.cpp` diff --git a/docs/benchmarks/catalog/one-level/transpose/index.md b/docs/benchmarks/catalog/one-level/transpose/index.md new file mode 100644 index 0000000..c46010d --- /dev/null +++ b/docs/benchmarks/catalog/one-level/transpose/index.md @@ -0,0 +1,10 @@ +# One-level: transpose + + + +This family contains **1** source implementations and +**4** active build variants. + +| Implementation | Source | Variants | PTO source-union surface | +| --- | --- | ---: | --- | +| [transpose](transpose-dece711d.md) | `benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp` | 4 | [`MGATHER`](../../../../intrinsics/mgather.md), [`TADD`](../../../../intrinsics/tadd.md), [`TCI`](../../../../intrinsics/tci.md), [`TCVT`](../../../../intrinsics/tcvt.md), [`TDIVS`](../../../../intrinsics/tdivs.md), [`TEXPANDS`](../../../../intrinsics/texpands.md), [`TLOAD`](../../../../intrinsics/tload.md), [`TMULS`](../../../../intrinsics/tmuls.md), [`TSTORE`](../../../../intrinsics/tstore.md), [`TSUB`](../../../../intrinsics/tsub.md), [`TTRANS`](../../../../intrinsics/ttrans.md) | diff --git a/docs/benchmarks/catalog/one-level/transpose/transpose-dece711d.md b/docs/benchmarks/catalog/one-level/transpose/transpose-dece711d.md new file mode 100644 index 0000000..97b2d3b --- /dev/null +++ b/docs/benchmarks/catalog/one-level/transpose/transpose-dece711d.md @@ -0,0 +1,621 @@ +# transpose: transpose.cpp + + + +| Field | Value | +| --- | --- | +| Benchmark surface | One-level | +| Family | `transpose` | +| Implementation source | `benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp` | +| Active build variants | 4 | + +## How the Code Is Written + +The implementation loads or gathers global data into typed tiles, then applies vector elementwise arithmetic, then transforms tile layout or selects tile elements, then commits the result to global memory. + +```cpp title="transpose.cpp" linenums="1" +#include +//#include "template_asm.h" + +#include +#include + +#include "fileop.h" +#include "transpose/transpose_pto.hpp" + + +#ifndef DType +#define DType int32_t +#endif + +#ifndef tMs +#define tMs 1024 +#endif + +#ifndef MAX_DIMs +#define MAX_DIMs 8 +#endif + +//#define IN_SHAPE {2, 128, 64} +//#define OUT_SHAPE {2, 64, 128} +//#define IN_SHAPE {8, 1024, 2048} +//#define OUT_SHAPE {8, 2048, 1024} +//#define IN_SHAPE {64, 8, 4096, 3} +//#define OUT_SHAPE {64, 8, 3, 4096} +#ifndef IN_SHAPEs +#define IN_SHAPEs 64,8,64,4,16,7 +#endif + +#ifndef OUT_SHAPEs +#define OUT_SHAPEs 64,8,16,4,64,7 +#endif +//#define IN_SHAPE {64, 8, 64, 4, 128} +//#define OUT_SHAPE {64, 8, 128, 4, 64} + +//#define IN_DIM 3 +//#define OUT_DIM 3 +//#define IN_DIM 4 +//#define OUT_DIM 4 + +#ifndef IN_DIMs +#define IN_DIMs 6 +#endif + +#ifndef OUT_DIMs +#define OUT_DIMs 6 +#endif +//#define IN_DIM 5 +//#define OUT_DIM 5 + + +//#define TRANSPOSE_DIM1 1 +//#define TRANSPOSE_DIM0 2 +//#define TRANSPOSE_DIM1 2 +//#define TRANSPOSE_DIM0 3 +#ifndef TRANSPOSE_DIM1s +#define TRANSPOSE_DIM1s 2 +#endif + +#ifndef TRANSPOSE_DIM0s +#define TRANSPOSE_DIM0s 4 +#endif +//#define TRANSPOSE_DIM1 2 +//#define TRANSPOSE_DIM0 4 + +//#define gIM (2 * 128 * 64) +//#define gOM (2 * 64 * 128) +//#define gIM (8 * 1024 * 2048) +//#define gOM (8 * 2048 * 1024) +//#define gIM (64 * 8 * 4096 * 3) +//#define gOM (64 * 8 * 3 * 4096) + +#ifndef gIMs +#define gIMs (64 * 8 * 64 * 4 * 16 * 7) +#endif + +#ifndef gOMs +#define gOMs (64 * 8 * 16 * 4 * 64 * 7) +#endif +//#define gIM (64 * 8 * 64 * 4 * 128) +//#define gOM (64 * 8 * 128 * 4 * 64) + + + +// ============================================================================ +// main +// ============================================================================ + + + + +int main() { + using dtype = DType; + uint32_t in_shape[128] = {IN_SHAPEs}; + uint32_t out_shape[128] = {OUT_SHAPEs}; + size_t in_dim = IN_DIMs; + size_t out_dim = OUT_DIMs; +// size_t transpose_dim1 = TRANSPOSE_DIM1; +// size_t transpose_dim0 = TRANSPOSE_DIM0; + + dtype input_buf[gIMs]; + dtype output_buf[gOMs]; + + + dtype* input = input_buf; + dtype* output = output_buf; + + +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_2x128x64.bin", (uint8_t*)input, gIM * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_8x1024x2048.bin", (uint8_t*)input, gIM * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_64x8x4096x3.bin", (uint8_t*)input, gIM * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_64x8x64x4x16x7.bin", (uint8_t*)input, gIMs * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_2x2x4x4x8x4.bin", (uint8_t*)input, gIMs * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_2x2x4x4x8.bin", (uint8_t*)input, gIMs * sizeof(dtype)); +// readBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/data_64x8x64x4x128.bin", (uint8_t*)input, gIM * sizeof(dtype)); + transpose(input, output, in_shape, out_shape); +// transpose(input, output); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/result_64x8x4096x3.bin", (uint8_t*)output, gOM * sizeof(dtype)); + // writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/result_2x2x4x4x8x4.bin", (uint8_t*)output, gOMs * sizeof(dtype)); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/result_2x2x4x4x8.bin", (uint8_t*)output, gOMs * sizeof(dtype)); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/result_64x8x64x4x16x7.bin", (uint8_t*)output, gOMs * sizeof(dtype)); +// writeBinaryFile("/remote/lms01/q50057645/jcore_project/JanusCoreBench/test/ascpp/transpose/src/result_64x8x64x4x128.bin", (uint8_t*)output, gOM * sizeof(dtype)); +// printf("output"); + + +} +``` + +## PTO-Bearing Local Implementations + +These local include files contain the PTO calls reached by this entrypoint. + +??? code "benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp" + + ```cpp title="transpose_pto.hpp" linenums="1" + /** + * @file transpose_tile_isa.hpp + * @brief 基于纯 Tile ISA 的转置算子实现 + * + * 本文件提供两个转置算子： + * 1. tile_transpose_nd: 通用 N 维任意轴交换，使用 offset 计算 + MGATHER + * 2. tile_transpose_2d: 2D 矩阵转置，使用硬件原生 TTRANS 指令 + * + * 核心设计思想： + * - 将"标量计算"与"数据并行计算"分离到不同执行路径 + * - 维度循环在标量核心上运行（编译期模板展开） + * - 每个维度内的坐标提取用 tile 操作并行处理所有元素 + * - 避免使用 __vec__ 的 per-element 标量循环模式 + */ + + #ifndef SUPERNPU_TRANSPOSE_TILE_ISA_HPP + #define SUPERNPU_TRANSPOSE_TILE_ISA_HPP + + #include + #include + #include + + // TTRANS 指令的 API 兼容性配置 + // 旧 API: TTRANS(dst, src) + // 新 API: TTRANS(dst, src, tmp) - 需要显式传入临时 tile + // 根据实际 PTO 版本设置此宏 + #ifndef SUPERNPU_PTO_TTRANS_NEEDS_TMP + #define SUPERNPU_PTO_TTRANS_NEEDS_TMP 0 + #endif + + namespace supernpu::tile_isa { + + /** + * @brief 通用 N 维转置（任意轴交换） + * + * 算法核心： + * 1. 对输出张量的每个元素，计算其多维坐标 + * 2. 通过轴映射得到对应的输入坐标 + * 3. 用输入 stride 计算 byte offset + * 4. 用 MGATHER 按 offset 从输入中 gather 数据 + * + * 关键优化： + * - Rank、Axis0、Axis1 是模板参数 → 维度循环编译期完全展开 + * - 每个维度的坐标提取用 tile 操作并行处理 tile 内所有元素 + * - 不是"per-element 串行迭代"，而是"per-dimension 并行处理" + * + * @tparam DType 数据类型 + * @tparam Rank 张量维度数（编译期常量，> 1） + * @tparam Axis0 转置轴 0（编译期常量，[0, Rank)） + * @tparam Axis1 转置轴 1（编译期常量，[0, Rank) 且 != Axis0） + * @tparam Elements 总元素数（编译期常量，> 0） + * @tparam TileElements 每个 tile 处理的元素数（默认 512） + * + * @param input 输入数据指针（展平为一维） + * @param output 输出数据指针（展平为一维） + * @param input_shape 输入张量各维度尺寸（运行时数组） + * @param output_shape 输出张量各维度尺寸（运行时数组） + */ + template + void tile_transpose_nd(DType *input, DType *output, std::uint32_t *input_shape, + std::uint32_t *output_shape) { + // ========== 编译期安全检查 ========== + static_assert(Rank > 1, "转置至少需要 2 维"); + static_assert(Axis0 >= 0 && Axis0 < Rank, "Axis0 超出维度范围"); + static_assert(Axis1 >= 0 && Axis1 < Rank, "Axis1 超出维度范围"); + static_assert(Axis0 != Axis1, "转置的两个轴必须不同"); + static_assert(Elements > 0, "元素数必须为正"); + static_assert(TileElements > 0, "TileElements 必须为正"); + // Tile 行必须 32 字节对齐（硬件要求） + static_assert(TileElements * static_cast(sizeof(DType)) % 32 == 0, + "RowMajor Tile 的行必须 32 字节对齐"); + // MGATHER 的 offset tile 使用 uint32 byte offset，不能溢出 + static_assert(static_cast(Elements) * sizeof(DType) <= + 0x100000000ULL, + "MGATHER 的 byte offset 使用 uint32，不能溢出"); + + // ========== 分块参数 ========== + constexpr int kFullTiles = Elements / TileElements; // 完整 tile 数量 + constexpr int kTailElements = Elements % TileElements; // 尾部剩余元素数 + + // ========== 类型定义 ========== + // 输入/输出全局张量：展平为一维（转置不改变存储，只改变逻辑索引） + using InputGlobal = pto::global_tensor>; + using OutputGlobal = pto::global_tensor>; + + // 数据 tile：存储实际数据 + using DataTile = pto::Tile; + // Offset tile：存储每个元素的 byte offset（用于 MGATHER） + using OffsetTile = pto::Tile; + // 输出迭代器：用于遍历输出张量的 tile + using OutputIterator = global_iterator; + + // ========== 初始化全局张量和迭代器 ========== + InputGlobal input_global(input); + OutputIterator output_iter(output); + + // ========== 处理完整 tile ========== + std::uint32_t output_base = 0; // 当前 tile 在输出中的起始线性索引 + for (int tile_index = 0; tile_index < kFullTiles; ++tile_index) { + auto output_global = output_iter(0, tile_index); + DataTile output_tile; // 存储 gather 回来的数据 + OffsetTile offset_tile; // 存储每个元素的 byte offset + + // 临时 tile（用于坐标提取和 offset 计算） + OffsetTile linear_index; // 线性索引 [output_base, output_base+1, ...] + OffsetTile quotient; // 除法商 + OffsetTile cycle; // 周期数（用于取模） + OffsetTile cycle_base; // 周期基址 + OffsetTile coordinate; // 当前维度的坐标 + OffsetTile contribution; // 当前维度对 offset 的贡献 + + // Step 1: 生成线性索引序列 [output_base, output_base+1, ..., output_base+TileElements-1] + TCI(linear_index, output_base); + + // Step 2: 初始化 offset 累加器为 0 + TEXPANDSCALAR(offset_tile, static_cast(0)); + + // Step 3: 对每个维度计算坐标并累加 offset + // 关键：这个循环在标量核心上运行，由于 Rank 是模板参数，编译期完全展开 + // 每次迭代用 tile 操作并行处理 tile 内所有元素的当前维度 + std::uint32_t input_stride = 1; // 输入张量当前维度的 stride（从内层开始） + for (int input_dim = Rank - 1; input_dim >= 0; --input_dim) { + // 轴映射：交换 Axis0 和 Axis1 + // 例如：如果 input_dim == Axis0，则对应的 output_dim == Axis1 + int output_dim = input_dim; + if (input_dim == Axis0) { + output_dim = Axis1; + } else if (input_dim == Axis1) { + output_dim = Axis0; + } + + // 计算 output_dim 维度的 stride（标量计算，在标量核心上完成） + // output_stride = output_shape[output_dim+1] * output_shape[output_dim+2] * ... * output_shape[Rank-1] + std::uint32_t output_stride = 1; + for (int dim = output_dim + 1; dim < Rank; ++dim) { + output_stride *= output_shape[dim]; + } + + // 提取当前维度的坐标：coordinate = (linear_index / output_stride) % output_shape[output_dim] + // 用 tile 操作并行处理所有元素 + + // quotient = linear_index / output_stride + // 优化：如果 output_stride == 1，直接拷贝（避免除法） + if (output_stride == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, output_stride); + } + + // 取模实现：a % b = a - (a / b) * b + // cycle = quotient / output_shape[output_dim] + TDIVS(cycle, quotient, output_shape[output_dim]); + // cycle_base = cycle * output_shape[output_dim] + TMULS(cycle_base, cycle, output_shape[output_dim]); + // coordinate = quotient - cycle_base （即 quotient % output_shape[output_dim]） + TSUB(coordinate, quotient, cycle_base); + + // 计算当前维度对 byte offset 的贡献 + // contribution = coordinate * input_stride * sizeof(DType) + // 注意：coordinate 是 output_coord[output_dim]，由于轴映射，它等于 input_coord[input_dim] + TMULS(contribution, coordinate, + input_stride * static_cast(sizeof(DType))); + + // 累加到总 offset + TADD(offset_tile, offset_tile, contribution); + + // 更新 input_stride（从内层向外层累积） + input_stride *= input_shape[input_dim]; + } + + // Step 4: 按 byte offset 从输入中 gather 数据 + MGATHER(output_tile, input_global, offset_tile); + + // Step 5: 写回到输出全局内存 + TSTORE(output_global, output_tile); + + // 更新下一个 tile 的起始索引 + output_base += TileElements; + } + + // ========== 处理尾部元素 ========== + // 如果总元素数不是 TileElements 的整数倍，处理剩余元素 + if constexpr (kTailElements != 0) { + // 尾部 tile 的类型定义（物理尺寸相同，但 valid region 缩小） + using TailDataTile = pto::Tile; + using TailOffsetTile = + pto::Tile; + + auto output_global = output_iter(0, kFullTiles); + TailDataTile output_tile; + TailOffsetTile offset_tile; + + TailOffsetTile linear_index; + TailOffsetTile quotient; + TailOffsetTile cycle; + TailOffsetTile cycle_base; + TailOffsetTile coordinate; + TailOffsetTile contribution; + + // 尾部 tile 的处理逻辑与完整 tile 完全一致 + // 区别仅在于 valid region 更小（kTailElements 而非 TileElements） + // tile 操作以 valid region 为迭代域，因此自动只处理有效元素 + TCI(linear_index, output_base); + TEXPANDSCALAR(offset_tile, static_cast(0)); + + std::uint32_t input_stride = 1; + for (int input_dim = Rank - 1; input_dim >= 0; --input_dim) { + int output_dim = input_dim; + if (input_dim == Axis0) { + output_dim = Axis1; + } else if (input_dim == Axis1) { + output_dim = Axis0; + } + + std::uint32_t output_stride = 1; + for (int dim = output_dim + 1; dim < Rank; ++dim) { + output_stride *= output_shape[dim]; + } + + if (output_stride == 1) { + TCVT(quotient, linear_index); + } else { + TDIVS(quotient, linear_index, output_stride); + } + + TDIVS(cycle, quotient, output_shape[output_dim]); + TMULS(cycle_base, cycle, output_shape[output_dim]); + TSUB(coordinate, quotient, cycle_base); + TMULS(contribution, coordinate, + input_stride * static_cast(sizeof(DType))); + TADD(offset_tile, offset_tile, contribution); + + input_stride *= input_shape[input_dim]; + } + + MGATHER(output_tile, input_global, offset_tile); + TSTORE(output_global, output_tile); + } + } + + /** + * @brief 2D 矩阵转置（高性能路径） + * + * 使用硬件原生 TTRANS 指令，性能最优。 + * 算法：标准分块转置 + * 1. 将 Rows × Cols 矩阵划分为 TileRows × TileCols 的块 + * 2. 对每个块：TLOAD → TTRANS → TSTORE + * 3. 输入块 TileRows × TileCols，转置后输出块 TileCols × TileRows + * 4. 输出位置为 (col_tile, row_tile)（行列互换） + * + * @tparam DType 数据类型 + * @tparam Rows 输入矩阵行数 + * @tparam Cols 输入矩阵列数 + * @tparam TileRows tile 块行数（默认 16） + * @tparam TileCols tile 块列数（默认 16） + * + * @param input 输入矩阵指针（Rows × Cols） + * @param output 输出矩阵指针（Cols × Rows） + */ + template + void tile_transpose_2d(DType *input, DType *output) { + // ========== 编译期安全检查 ========== + static_assert(Rows > 0 && Cols > 0, "矩阵维度必须为正"); + static_assert(TileRows > 0 && TileCols > 0, "tile 维度必须为正"); + // TTRANS 对行宽有 32 字节对齐要求 + static_assert(TileRows * static_cast(sizeof(DType)) % 32 == 0, + "TTRANS 输出行宽必须 32 字节对齐"); + static_assert(TileCols * static_cast(sizeof(DType)) % 32 == 0, + "TTRANS 输入行宽必须 32 字节对齐"); + + // ========== 分块参数 ========== + constexpr int kRowTiles = Rows / TileRows; // 行方向完整 tile 数 + constexpr int kColTiles = Cols / TileCols; // 列方向完整 tile 数 + constexpr int kTailRows = Rows % TileRows; // 行方向尾部元素数 + constexpr int kTailCols = Cols % TileCols; // 列方向尾部元素数 + + // ========== 类型定义 ========== + using InputGlobal = pto::global_tensor>; + using OutputGlobal = pto::global_tensor>; + + // 输入 tile：TileRows × TileCols + using SrcTile = pto::Tile; + // 输出 tile：TileCols × TileRows（转置后） + using DstTile = pto::Tile; + + using InputIterator = global_iterator; + using OutputIterator = global_iterator; + + InputIterator input_iter(input); + OutputIterator output_iter(output); + + // ========== 处理完整 tile 块 ========== + for (int row_tile = 0; row_tile < kRowTiles; ++row_tile) { + for (int col_tile = 0; col_tile < kColTiles; ++col_tile) { + // 输入位置：(row_tile, col_tile) + auto input_global = input_iter(row_tile, col_tile); + // 输出位置：(col_tile, row_tile) - 行列互换 + auto output_global = output_iter(col_tile, row_tile); + + SrcTile src_tile; + DstTile dst_tile; + + // 从全局内存加载输入 tile + TLOAD(src_tile, input_global); + + // 硬件转置：TileRows × TileCols → TileCols × TileRows + #if SUPERNPU_PTO_TTRANS_NEEDS_TMP + SrcTile tmp_tile; + TTRANS(dst_tile, src_tile, tmp_tile); + #else + TTRANS(dst_tile, src_tile); + #endif + + // 写回到全局内存 + TSTORE(output_global, dst_tile); + } + + // ========== 处理右侧尾部 tile（列方向） ========== + if constexpr (kTailCols != 0) { + // 输入 tile：TileRows × kTailCols（valid region） + using SrcRightTile = + pto::Tile; + // 输出 tile：kTailCols × TileRows（转置后） + using DstBottomTile = + pto::Tile; + + auto input_global = input_iter(row_tile, kColTiles); + auto output_global = output_iter(kColTiles, row_tile); + + SrcRightTile src_tile; + DstBottomTile dst_tile; + + TLOAD(src_tile, input_global); + #if SUPERNPU_PTO_TTRANS_NEEDS_TMP + SrcRightTile tmp_tile; + TTRANS(dst_tile, src_tile, tmp_tile); + #else + TTRANS(dst_tile, src_tile); + #endif + TSTORE(output_global, dst_tile); + } + } + + // ========== 处理底部尾部 tile（行方向） ========== + if constexpr (kTailRows != 0) { + for (int col_tile = 0; col_tile < kColTiles; ++col_tile) { + // 输入 tile：kTailRows × TileCols（valid region） + using SrcBottomTile = + pto::Tile; + // 输出 tile：TileCols × kTailRows（转置后） + using DstRightTile = + pto::Tile; + + auto input_global = input_iter(kRowTiles, col_tile); + auto output_global = output_iter(col_tile, kRowTiles); + + SrcBottomTile src_tile; + DstRightTile dst_tile; + + TLOAD(src_tile, input_global); + #if SUPERNPU_PTO_TTRANS_NEEDS_TMP + SrcBottomTile tmp_tile; + TTRANS(dst_tile, src_tile, tmp_tile); + #else + TTRANS(dst_tile, src_tile); + #endif + TSTORE(output_global, dst_tile); + } + + // ========== 处理右下角尾部 tile ========== + if constexpr (kTailCols != 0) { + // 输入 tile：kTailRows × kTailCols（valid region） + using SrcCornerTile = + pto::Tile; + // 输出 tile：kTailCols × kTailRows（转置后） + using DstCornerTile = + pto::Tile; + + auto input_global = input_iter(kRowTiles, kColTiles); + auto output_global = output_iter(kColTiles, kRowTiles); + + SrcCornerTile src_tile; + DstCornerTile dst_tile; + + TLOAD(src_tile, input_global); + #if SUPERNPU_PTO_TTRANS_NEEDS_TMP + SrcCornerTile tmp_tile; + TTRANS(dst_tile, src_tile, tmp_tile); + #else + TTRANS(dst_tile, src_tile); + #endif + TSTORE(output_global, dst_tile); + } + } + } + + } // namespace supernpu::tile_isa + + // Global adapter matching the test harness signature + // transpose + template + void transpose(DType *input, DType *output, std::uint32_t *in_shape, + std::uint32_t *out_shape) { + ::supernpu::tile_isa::tile_transpose_nd( + input, output, in_shape, out_shape); + } + + #endif + ``` + +## Supported PTO Intrinsics + +This is the source-level union across the compile-time paths in this +implementation. A build command may select a subset through its macros; +the exact source spellings below preserve aliases and masked forms. + +| Intrinsic contract | Source spelling | Called from | +| --- | --- | --- | +| [`MGATHER`](../../../../intrinsics/mgather.md) | `MGATHER` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TADD`](../../../../intrinsics/tadd.md) | `TADD` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TCI`](../../../../intrinsics/tci.md) | `TCI` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TCVT`](../../../../intrinsics/tcvt.md) | `TCVT` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TDIVS`](../../../../intrinsics/tdivs.md) | `TDIVS` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TEXPANDS`](../../../../intrinsics/texpands.md) | `TEXPANDSCALAR` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TLOAD`](../../../../intrinsics/tload.md) | `TLOAD` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TMULS`](../../../../intrinsics/tmuls.md) | `TMULS` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TSTORE`](../../../../intrinsics/tstore.md) | `TSTORE` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TSUB`](../../../../intrinsics/tsub.md) | `TSUB` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | +| [`TTRANS`](../../../../intrinsics/ttrans.md) | `TTRANS` | `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` | + +## Active Build Commands + +Run these from `benchmark/one-level-arch/test/kernel/transpose` after setting +`COMPILER_DIR` and `LINX_SYSROOT` as described in the build guide. + +| Manifest line | Command | +| ---: | --- | +| 11 | `make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1476,32 OUT_SHAPE=32,1476 IN_SHAPE_NAME=1476_32 OUT_SHAPE_NAME=32_1476 IN_DIM=2 OUT_DIM=2 TRANSPOSE_DIM1=0 TRANSPOSE_DIM0=1 gIM='1476*32' gOM='32*1476'` | +| 24 | `make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1,8,4096,3 OUT_SHAPE=1,8,3,4096 IN_SHAPE_NAME=1_8_4096_3 OUT_SHAPE_NAME=1_8_3_4096 IN_DIM=4 OUT_DIM=4 TRANSPOSE_DIM1=2 TRANSPOSE_DIM0=3 gIM='1*8*4096*3' gOM='1*8*3*4096'` | +| 33 | `make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=1,32,8,32 OUT_SHAPE=1,8,32,32 IN_SHAPE_NAME=1_32_8_32 OUT_SHAPE_NAME=1_8_32_32 IN_DIM=4 OUT_DIM=4 TRANSPOSE_DIM1=1 TRANSPOSE_DIM0=2 gIM='1*32*8*32' gOM='1*8*32*32'` | +| 46 | `make TESTCASE=transpose COMPILER_DIR=$COMPILER_DIR DType=__half MAX_DIM=8 tM=512 IN_SHAPE=8,64,4,16,7 OUT_SHAPE=8,16,4,64,7 IN_SHAPE_NAME=1_8_64_4_16_7 OUT_SHAPE_NAME=1_8_16_4_64_7 IN_DIM=5 OUT_DIM=5 TRANSPOSE_DIM1=1 TRANSPOSE_DIM0=3 gIM='1*8*64*4*16*7' gOM='1*8*16*4*64*7'` | + +## Resolved One-Level Source Closure + +??? info "5 one-level source files reached through local includes" + + - `benchmark/one-level-arch/kernels/transpose/transpose_pto.hpp` + - `benchmark/one-level-arch/test/common/fileop.h` + - `benchmark/one-level-arch/test/common/readBinary.h` + - `benchmark/one-level-arch/test/common/writeBinary.h` + - `benchmark/one-level-arch/test/kernel/transpose/src/transpose.cpp` diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md new file mode 100644 index 0000000..c3f009a --- /dev/null +++ b/docs/benchmarks/index.md @@ -0,0 +1,37 @@ +# Benchmark Programming Catalog + + + +This catalog is generated from every active one-level `compile.all` command. Each +implementation page presents the complete entry source, PTO-bearing local +include implementations, the canonical PTO intrinsic surface, exact build +commands, and required data objects. + +**Coverage:** 53 active build variants, 20 source-backed +implementations, and 13 benchmark families. + +| Surface | Family | Implementations | Build variants | PTO source-union surface | +| --- | --- | ---: | ---: | --- | +| One-level | [broadcast](catalog/one-level/broadcast/index.md) | 4 | 6 | 11 intrinsics | +| One-level | [concat](catalog/one-level/concat/index.md) | 2 | 4 | 11 intrinsics | +| One-level | [control](catalog/one-level/control/index.md) | 1 | 6 | 16 intrinsics | +| One-level | [element_wise/gelu](catalog/one-level/element-wise-gelu/index.md) | 1 | 1 | 10 intrinsics | +| One-level | [fa](catalog/one-level/fa/index.md) | 2 | 9 | 23 intrinsics | +| One-level | [gather](catalog/one-level/gather/index.md) | 1 | 1 | 3 intrinsics | +| One-level | [matmul](catalog/one-level/matmul/index.md) | 3 | 16 | 11 intrinsics | +| One-level | [reduction/reducemax_col](catalog/one-level/reduction-reducemax-col/index.md) | 1 | 1 | 5 intrinsics | +| One-level | [reduction/reducemax_row](catalog/one-level/reduction-reducemax-row/index.md) | 1 | 1 | 5 intrinsics | +| One-level | [reduction/reducesum_col](catalog/one-level/reduction-reducesum-col/index.md) | 1 | 2 | 5 intrinsics | +| One-level | [reduction/reducesum_row](catalog/one-level/reduction-reducesum-row/index.md) | 1 | 1 | 5 intrinsics | +| One-level | [sort](catalog/one-level/sort/index.md) | 1 | 1 | 2 intrinsics | +| One-level | [transpose](catalog/one-level/transpose/index.md) | 1 | 4 | 11 intrinsics | + +## Reading a Benchmark Page + +1. Start with **How the Code Is Written** for the complete entry source. +2. Expand **PTO-Bearing Local Implementations** to inspect the reached kernel code. +3. Use **Supported PTO Intrinsics** to open the normative intrinsic contract. +4. Reproduce a listed command from the directory shown on the page. + +Only `benchmark/one-level-arch` entrypoints are published here. Other repository +surfaces are outside this programming-model profile. diff --git a/docs/examples/README.md b/docs/examples/README.md new file mode 100644 index 0000000..3d5ad5c --- /dev/null +++ b/docs/examples/README.md @@ -0,0 +1,20 @@ +# Compile-checked documentation examples + +These files are included verbatim in the MkDocs tutorials. Run them from the +repository root with the active Linx LLVM toolchain: + +```bash +export COMPILER_DIR=/path/to/linx-isa/compiler/llvm/build-linxisa-clang/bin +export LINX_SYSROOT=/path/to/linx-isa/out/libc/musl/install/phase-b +bash docs/examples/check.sh +``` + +`cpp_language.cpp` covers the general C++20 features described in the guide. +`tile_add.cpp` and `gemm_tile.cpp` use the canonical builtin-backed one-level +PTO surface from the sibling `pto_kernels` workload, which is also used by Linx +QEMU tile validation. +`flash_attention.cpp` compiles a larger QK/PV dataflow probe from that same +canonical surface. It intentionally omits online softmax; the full benchmark +sources provide the numerically complete algorithm. +They are compile-only teaching kernels; linked, runnable programs live in the +benchmark test harness. diff --git a/docs/examples/check.sh b/docs/examples/check.sh new file mode 100755 index 0000000..1840e90 --- /dev/null +++ b/docs/examples/check.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +CASE="${1:-all}" + +if [[ -z "${COMPILER_DIR:-}" ]]; then + echo "error: COMPILER_DIR must point to the Linx LLVM bin directory" >&2 + exit 2 +fi + +if [[ ! -x "$COMPILER_DIR/clang++" ]]; then + echo "error: Linx clang++ not found at $COMPILER_DIR/clang++" >&2 + exit 2 +fi + +if [[ "${KEEP_DOC_OBJECTS:-0}" == "1" ]]; then + OUT_DIR="$SCRIPT_DIR/output" + mkdir -p "$OUT_DIR" +else + OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/supernpu-doc-examples.XXXXXX")" + trap 'rm -rf "$OUT_DIR"' EXIT +fi + +COMMON_FLAGS=( + -target linx64-linx-none-elf + -std=c++20 + -O2 + -c +) + +CANONICAL_PTO_ROOT="${PTO_KERNELS_ROOT:-$REPO_ROOT/../pto_kernels}" +CANONICAL_PTO_FLAGS=( + -fenable-matrix + -I"$CANONICAL_PTO_ROOT/include" +) + +if [[ -n "${LINX_SYSROOT:-}" ]]; then + COMMON_FLAGS+=( + --sysroot="$LINX_SYSROOT" + -idirafter "$LINX_SYSROOT/usr/include" + ) +fi + +compile_cpp_language() { + "$COMPILER_DIR/clang++" "${COMMON_FLAGS[@]}" \ + "$SCRIPT_DIR/cpp_language.cpp" -o "$OUT_DIR/cpp_language.o" +} + +compile_tile_add() { + "$COMPILER_DIR/clang++" "${COMMON_FLAGS[@]}" "${CANONICAL_PTO_FLAGS[@]}" \ + "$SCRIPT_DIR/tile_add.cpp" -o "$OUT_DIR/tile_add.o" +} + +compile_gemm_tile() { + if [[ ! -f "$CANONICAL_PTO_ROOT/include/pto/linx/TileOps.hpp" ]]; then + echo "error: PTO_KERNELS_ROOT does not contain include/pto/linx/TileOps.hpp" >&2 + exit 2 + fi + "$COMPILER_DIR/clang++" "${COMMON_FLAGS[@]}" "${CANONICAL_PTO_FLAGS[@]}" \ + "$SCRIPT_DIR/gemm_tile.cpp" -o "$OUT_DIR/gemm_tile.o" +} + +compile_flash_attention() { + if [[ ! -f "$CANONICAL_PTO_ROOT/include/pto/linx/AutoModeKernels.hpp" ]]; then + echo "error: PTO_KERNELS_ROOT does not contain include/pto/linx/AutoModeKernels.hpp" >&2 + exit 2 + fi + "$COMPILER_DIR/clang++" "${COMMON_FLAGS[@]}" "${CANONICAL_PTO_FLAGS[@]}" \ + "$SCRIPT_DIR/flash_attention.cpp" -o "$OUT_DIR/flash_attention.o" +} + +case "$CASE" in + all) + compile_cpp_language + compile_tile_add + compile_gemm_tile + compile_flash_attention + ;; + cpp-language) compile_cpp_language ;; + tile-add) compile_tile_add ;; + gemm-tile) compile_gemm_tile ;; + flash-attention) compile_flash_attention ;; + *) + echo "error: unknown case '$CASE' (expected all, cpp-language, tile-add, gemm-tile, or flash-attention)" >&2 + exit 2 + ;; +esac + +echo "compiled documentation example(s) to $OUT_DIR" diff --git a/docs/examples/cpp_language.cpp b/docs/examples/cpp_language.cpp new file mode 100644 index 0000000..b367e5e --- /dev/null +++ b/docs/examples/cpp_language.cpp @@ -0,0 +1,40 @@ +template +concept StaticTileShape = (Rows > 0) && (Cols > 0); + +template + requires StaticTileShape +struct TileShape { + static constexpr int rows = Rows; + static constexpr int cols = Cols; + static constexpr int elements = Rows * Cols; +}; + +template +constexpr int vector_count() { + if constexpr (Shape::elements <= 256) { + return 1; + } else { + return (Shape::elements + 255) / 256; + } +} + +struct WorkSplit { + int begin; + int end; +}; + +constexpr WorkSplit split_rows(int core_id, int core_count, int rows) { + const int begin = rows * core_id / core_count; + const int end = rows * (core_id + 1) / core_count; + return {begin, end}; +} + +using ExampleTile = TileShape<16, 32>; +static_assert(ExampleTile::elements == 512); +static_assert(vector_count() == 2); + +extern "C" int cpp_language_probe(int core_id) { + const auto [begin, end] = split_rows(core_id, 4, 128); + const auto extent = [=]() constexpr { return end - begin; }; + return extent() + vector_count(); +} diff --git a/docs/examples/disassembly/flash_attention.s b/docs/examples/disassembly/flash_attention.s new file mode 100644 index 0000000..3db15cd --- /dev/null +++ b/docs/examples/disassembly/flash_attention.s @@ -0,0 +1,54 @@ +# Linx LLVM v0.57 block-template excerpt based on docs/examples/flash_attention.cpp. +# Repeated Q/K/V loads and QK/PV products are condensed to representative blocks. +# C.B.IOS shows the required shared-RHS lowering; compiler support is pending. +BSTART.TLOAD INT32 +B.DIM a5, 0, ->lb0 +B.DIM a5, 0, ->lb1 +B.DIM a5, 0, ->lb2 +B.ARG NORM.normal +B.IOR [a1,a4],[] +B.IOT last, ->t<4KB> + +BSTART.TLOAD INT32 +B.DIM a5, 0, ->lb0 +B.DIM a5, 0, ->lb1 +B.DIM a5, 0, ->lb2 +B.ARG NORM.normal +B.IOR [a2,a4],[] +C.B.IOS S#12 + +BSTART.TMATMUL INT32 +C.B.DIMI 8, ->lb0 +C.B.DIMI 8, ->lb1 +C.B.DIMI 8, ->lb2 +C.B.IOS S#12 +B.IOT t#8, last, ->acc<4KB> + +BSTART.ACCCVT INT32 +B.IOT last, ->m<4KB> + +BSTART.TLOAD INT32 +B.DIM a5, 0, ->lb0 +B.DIM a5, 0, ->lb1 +B.DIM a5, 0, ->lb2 +B.ARG NORM.normal +B.IOR [a3,a4],[] +C.B.IOS S#13 + +BSTART.TMATMUL INT32 +C.B.DIMI 8, ->lb0 +C.B.DIMI 8, ->lb1 +C.B.DIMI 8, ->lb2 +C.B.IOS S#13 +B.IOT m#2, last, ->acc<4KB> + +BSTART.ACCCVT INT32 +B.IOT last, ->m<4KB> + +BSTART.TSTORE INT32 +B.DIM a5, 0, ->lb0 +B.DIM a5, 0, ->lb1 +B.DIM a5, 0, ->lb2 +B.ARG NORM.normal +B.IOR [a0,a4],[] +B.IOT m#1, last, ->m<4KB> diff --git a/docs/examples/disassembly/gemm_tile.s b/docs/examples/disassembly/gemm_tile.s new file mode 100644 index 0000000..7f0236e --- /dev/null +++ b/docs/examples/disassembly/gemm_tile.s @@ -0,0 +1,24 @@ +# Linx LLVM v0.57 excerpt generated from docs/examples/gemm_tile.cpp. +BSTART.TLOAD INT32 +B.ARG NORM.normal +B.IOR [a1,a3],[] +B.IOT last, ->t<4KB> + +BSTART.TLOAD INT32 +B.ARG NORM.normal +B.IOR [a2,a3],[] +B.IOT last, ->t<4KB> + +BSTART.TMATMUL INT32 +C.B.DIMI 8, ->lb0 +C.B.DIMI 8, ->lb1 +C.B.DIMI 8, ->lb2 +B.IOT t#2, t#1, last, ->acc<4KB> + +BSTART.ACCCVT INT32 +B.IOT last, ->m<4KB> + +BSTART.TSTORE INT32 +B.ARG NORM.normal +B.IOR [a0,a3],[] +B.IOT m#1, last, ->m<4KB> diff --git a/docs/examples/disassembly/tile_add.s b/docs/examples/disassembly/tile_add.s new file mode 100644 index 0000000..e15a4de --- /dev/null +++ b/docs/examples/disassembly/tile_add.s @@ -0,0 +1,19 @@ +# Linx LLVM v0.57 excerpt generated from docs/examples/tile_add.cpp. +BSTART.TLOAD FP32 +B.DATR canon, FP64, Null, cmode0, rmode0 +B.IOT last, ->t<4KB> +B.IOR [a1,a3],[] + +BSTART.TLOAD FP32 +B.DATR canon, FP64, Null, cmode0, rmode0 +B.IOT last, ->t<4KB> +B.IOR [a2,a3],[] + +BSTART.TADD FP32 +B.DATR canon, FP64, Null, cmode0, rmode0 +B.IOT t#1, t#2, last, ->t<4KB> + +BSTART.TSTORE FP32 +B.DATR canon, FP64, Null, cmode0, rmode0 +B.IOT t#1, last, ->t<4KB> +B.IOR [a0,a3],[] diff --git a/docs/examples/flash_attention.cpp b/docs/examples/flash_attention.cpp new file mode 100644 index 0000000..cfe146e --- /dev/null +++ b/docs/examples/flash_attention.cpp @@ -0,0 +1,11 @@ +#include + +#include + +extern "C" void flash_attention_probe( + std::int32_t *out, + const std::int32_t *q, + const std::int32_t *k, + const std::int32_t *v) { + pto::linx::auto_mode::flash_attention_kernel_i32(q, k, v, out); +} diff --git a/docs/examples/gemm_tile.cpp b/docs/examples/gemm_tile.cpp new file mode 100644 index 0000000..d73613c --- /dev/null +++ b/docs/examples/gemm_tile.cpp @@ -0,0 +1,10 @@ +#include + +extern "C" void gemm_tile(int32_t *out, const int32_t *lhs, + const int32_t *rhs) { + constexpr unsigned kTileSize = 8; + auto t_lhs = pto::linx::tload(lhs); + auto t_rhs = pto::linx::tload(rhs); + auto t_out = pto::linx::tmatmul<8, 8, 8>(t_lhs, t_rhs); + pto::linx::tstore(out, t_out); +} diff --git a/docs/examples/tile_add.cpp b/docs/examples/tile_add.cpp new file mode 100644 index 0000000..011e479 --- /dev/null +++ b/docs/examples/tile_add.cpp @@ -0,0 +1,12 @@ +#include + +#include + +extern "C" void tile_add(std::int32_t *out, const std::int32_t *lhs, + const std::int32_t *rhs) { + constexpr unsigned kTileSize = 8; + auto t_lhs = pto::linx::tload(lhs); + auto t_rhs = pto::linx::tload(rhs); + auto t_out = pto::linx::tadd(t_lhs, t_rhs); + pto::linx::tstore(out, t_out); +} diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..507103c --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,25 @@ +# Glossary + +| Term | Meaning | +| --- | --- | +| PTO | Typed tile programming and instruction model used by one-level NPU kernels. | +| PTO intrinsic | Operation with defined operands, types, shape/layout constraints, value semantics, and memory effects. | +| PE | One of four independent processing elements in a core. | +| PE ID | Immutable value in `[0, 3]` returned by `get_thread_idx()`. | +| Core ID | Launch-defined value returned by `get_block_idx()`. | +| Logical tile | Typed value with shape, valid region, layout, location, and distribution. | +| PE-local tile | Tile-register version directly visible to one PE. | +| Shared tile register | Core-local, versioned tile value visible to all four PEs. | +| Shared region | Fixed quarter of a shared tile associated with one PE. | +| Defined mask | Immutable four-bit mask naming valid shared regions. | +| Tile ID | Compiler-assigned logical identifier for a tile value. | +| Tile version | One definition of a tile ID, bound immutably by consumers. | +| GlobalTensor | Non-owning typed view of a global-memory region. | +| Valid region | Logical rows and columns for which an operation defines values. | +| Group operation | Dynamic operation requiring compatible participation from all four PEs. | +| M-axis sharding | Matrix rows divided into four PE-local quarters. | +| TLSU | Tile load/store unit for global/local/shared movement forms. | +| PTO-AS | Target-independent textual form of a PTO operation. | +| Linx block | Target block template carrying tile IDs, dimensions, attributes, and memory operands. | +| `COMPILER_DIR` | Directory containing Linx Clang and LLVM binary tools. | +| `LINX_SYSROOT` | Linx target libc/runtime sysroot used for hosted links. | diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..cfba284 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,99 @@ +--- +hide: + - toc +--- + +
+LinxISA PTO 0.57 · one-level · four PEs + +# SuperNPUBench golden manual + +Program one core as a four-PE group with tile-ID dataflow, PE-local tile +versions, and versioned shared tile registers. + +
+ ACTIVE 0.57 contract + target: linx64-linx-none-elf + ISA: 111 PTO + 2 identity intrinsics +
+
+ +
+
+

Program four PEs

+

Start from one entry, branch by PE ID, partition tensors by core ID, and reconverge for group operations.

+ Open group execution → +
+
+

Browse PTO 0.57

+

Read only the workbook-selected operations, plus the two group identity intrinsics.

+ Open the 113-entry manual → +
+
+

Inspect one-level benchmarks

+

See complete source, reached kernel code, exact build variants, and the 0.57 PTO surface.

+ Open the one-level catalog → +
+
+ +## One model, five contracts + +
+
+ + + + + +
+
+

Four PEs start together, then execute independently

+

get_thread_idx() selects a PE path. get_block_idx() selects the core's tensor slice. Group operations require four compatible dynamic participants.

+

Group execution →

+
+ + + + +
+ +## Grouped matrix path + +
    +
  1. Core sliceget_block_idx()
    2. +
  2. PE quarterget_thread_idx()
    3. +
  3. Shared RHSTLOAD to shared tile
    4. +
  4. Group MMAfour PE-local result quarters
    5. +
+ +```cpp title="Architecture-level grouped matmul" linenums="1" +SharedTile shared_b; +if (get_thread_idx() == 0) { + TLOAD(shared_b, global_b); +} +TMATMUL(local_c, local_a, shared_b); +``` + +Continue with [group programming examples](tutorials/group-programming.md) for +grouped matmul, reduce-to-one, all-reduce, and neighbor exchange. + +!!! info "Compiler status" + Shared tile-register lowering is an architecture contract and is not yet a + required compiler feature. The one-level benchmark catalog remains the + compile-checked implementation surface. diff --git a/docs/intrinsics/assembly.md b/docs/intrinsics/assembly.md new file mode 100644 index 0000000..d99f38a --- /dev/null +++ b/docs/intrinsics/assembly.md @@ -0,0 +1,331 @@ +# PTO and Linx Assembly + + + +PTO-AS is the target-independent textual form of a PTO program. Intrinsic +entries show value-producing SSA and destination-passing forms where the +synchronized ISA revision defines them. + +## SSA Form + +```text +%dst = pto.tadd %src0, %src1 + : (!pto.tile<...>, !pto.tile<...>) -> !pto.tile<...> +``` + +An SSA operation creates a new logical result. Operand and result types carry +the tile shape, element representation, location, layout, and valid region. + +## Destination-Passing Form + +```text +pto.tadd ins(%src0, %src1 : !pto.tile_buf<...>, !pto.tile_buf<...>) + outs(%dst : !pto.tile_buf<...>) +``` + +Destination-passing form names input and output buffers explicitly. The +operation semantics are identical to the corresponding SSA form; only value +ownership and bufferization are explicit. + +## Compact PTO-AS Form + +Instruction pages also show a compact form: + +```text +%dst = tadd %src0, %src1 : !pto.tile<...> +``` + +The mnemonic, operand order, modifiers, and types MUST match the intrinsic +entry. Data dependencies are carried by logical tile IDs and their versions. + +## Types + +PTO assembly types distinguish scalar values, global tensor descriptors, +PE-local tiles, shared tiles, and tile buffers. Two values with the same byte +count are not interchangeable when their location or layout differs. + +## Memory Effects + +Pure tile arithmetic reads tile versions and writes a new tile version. Memory +operations declare global effects according to their intrinsic entries. The +compiler MAY reorder operations only when tile-version, memory, and +operation-specific ordering rules allow it. + +## Tile-ID Versioning + +The source program names logical tile values. The compiler assigns tile IDs, +and every destination definition creates a new version. Consumers bind to the +specific source versions selected during rename: + +```text +T#7.v3 = tadd T#2.v1, T#5.v4 +``` + +Physical tile-register allocation and out-of-order readiness are hardware and +compiler responsibilities. The source program does not wire completion +tokens. + +## Lowering to Linx Blocks + +The Linx backend lowers PTO operations to named block templates. A block has a +start header, payload instructions/descriptors, and a stop/end boundary. Block +encoding is a target concern; PTO-AS semantics remain defined by the intrinsic +operation and its typed operands. + +!!! info "Assembly provenance" + DavinciOO revision `e058afb10bdf630b1ef1ebfc30d727401cb5272c` does not contain a standalone + `isa/assembly/PTO-AS.md`. The PTO-AS section above is this manual's normalized + operation-level notation. The complete block-level section below is synchronized + from `isa/intrinsic/ASSEMBLY_SYNTAX.md` at that revision. + +## Authoritative Linx Block Assembly Conventions + +本文描述当前 intrinsic 文档中 block assembly 示例的规范写法。它继承 Linx assembler manual 的 block/header 语法，并收窄到 DavinciOO active PE-local tile/block intrinsic。 + +### Line Structure + +Linx assembler source 由一系列 assembly statement 组成，通常一行一条 statement。基本结构为： + +```asm +label: instruction /* comment */ +``` + +`label` 可作为 branch、block target 或符号表达式使用。`instruction` 可以是真实 ISA instruction，也可以是 assembler pseudo-instruction。规范注释形式使用 `/* ... */`。 + +### Block Structure + +一个 block header 从 `BSTART.*` 开始。后续 header instruction 逐条累积当前 block 的属性、维度、输入输出和依赖信息。 + +```asm +BSTART.TEPL TADD, FP16 +B.DATR PadValue.Null +B.DIM rM, 0, ->LB0 +B.DIM rN, 0, ->LB1 +B.IOT T#1, T#3, last, ->T<1KB> +``` + +对于当前 PE-local intrinsic，常见 block lowering 具有以下顺序。除 `BSTART.*` 和所需 operand binding 外，header 应按 opcode profile 需求发出；不消费或使用默认值的 header 可以省略。 + +1. `BSTART.*` 选择 block family、opcode profile 和 main dtype。 +2. `B.DATR` 设置可选 data attributes；若只使用默认属性可省略。 +3. `B.DIM` 设置 shape / loop dimension；只需声明该 profile 实际消费的 `LB`。 +4. `B.IOR` 设置需要的 GPR operand 或 dependency contract。 +5. 一条或多条 `B.IOT` 设置 Tile operand binding；最后一条必须带 `last`。 + +下一个 `BSTART.*` 表示下一个 block 的开始。 + +### Header-form Intrinsic Blocks + +当前 active PE-local intrinsic 页优先展示 header-form lowering。也就是说，示例中的 block 通常只有 `BSTART.*` 和若干 header instruction；实际计算语义由 `BSTART.*` 的 opcode profile 固定解释，而不是由文档内联一段可编程 block body。 + +Linx 的 general block model 支持 `B.TEXT