Skip to content

feat(ir/codegen): minimal host-prequant MXFP8 matmul path - #2147

Open
yanghaoran29 wants to merge 1 commit into
hw-native-sys:mainfrom
yanghaoran29:feat/mx-scale-data-path
Open

feat(ir/codegen): minimal host-prequant MXFP8 matmul path#2147
yanghaoran29 wants to merge 1 commit into
hw-native-sys:mainfrom
yanghaoran29:feat/mx-scale-data-path

Conversation

@yanghaoran29

@yanghaoran29 yanghaoran29 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Minimal host-prequant MXFP8 matmul on Ascend950 (AIC):

  • LeftScale / RightScale spaces + SoC + mx_layout load
  • Mat→Scale move with PendingScaleFill (flush at tget_scale_addr)
  • tile.tget_scale_addrpto.tget_scale_addr
  • tile.matmul_mxpto.tmatmul.mx
  • Data dtype FP8E4M3FN only; reject FP8E5M2 at matmul_mx only (reinterpret_view does not support FP8 in this PR)
  • Scale dtype FP8E8M0; host ZZ/NN scales

Depends on #2144. Refs #1975.

Docs: operators MX constraints (en/zh).

pto-isa (hardware / ISA)

Constraint Notes
Independent scale buffer Cube does not fold scale into Left/Right; TileType::ScaleLeft / ScaleRight ↔ PyPTO LeftScale / RightScale
Payload scale = float8_e8m0_t; this PR data = FP8E4M3FN only (reject FP8E5M2); K%32==0, fractal=32
Layout MX_A_* → row-major ZZ; MX_B_* → col-major NN; TLoadMxCube*
TMov CommonCheckMX allows uint8_t Mat → float8_e8m0 ScaleLeft/Right; canonical: ui8 Mat reshape then ui8→f8 Scale
bind-then-fill GetScaleAddr(Left/Right) before sidecar fill; provisional Mat/alloc addr is invalid
Alignment matches ISA tmatmul_mx: M↑16, K↑64, N↑32 (fp8)

PTOAS (dialect / EmitC)

Constraint Notes
Single loc=scaling no independent left/right_scale loc yet; both sides lower to loc=scaling, EmitC picks ScaleLeft/Right
dtype must be !pto.f8E8M0 ui8+scaling becomes Fixpipe TileType::Scaling; convert before Scale
No Mat↔Scaling treshape different loc; reshape stays on Mat (ui8), then tmov into scaling
Shape-matched Mat→Scale tmov flat [1,G] must treshape to [M,K/32] (or B-side shape) first
Order tget_scale_addr before Mat→scaling fill; PyPTO PendingScaleFill flushes at tget
#pto.layout / mx load mx_a_zz / mx_b_nn / …; this PR ST uses host ZZ/NN (AZZ2ZZ)
Coverage pto.tmatmul.mx + pto.tget_scale_addr

Test plan

  • UT: test_mx_scale_memspace, test_mx_ops (incl. reject E5M2), test_mx_ops_codegen
  • reinterpret_view does not support FP8 in this PR; E5M2 data is rejected at matmul_mx (tile_ops UT)
  • ST: test_matmul_mx_compiles_ptoas on a5/a5sim; device numeric on a5
  • Docs en/zh parity; no Co-authored-by on tip commit
  • Rebased onto latest main

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 58f12814-e9e1-4594-a3f6-178fa46bd3c3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds LeftScale and RightScale memory spaces, extends tile load/move and reinterpretation semantics, introduces deferred MX scale code generation, and adds PTOAS layout, synchronization, barrier, and ordering transformations.

Changes

MX scale integration

Layer / File(s) Summary
MX scale memory model
include/pypto/ir/*, python/bindings/modules/ir.cpp, python/pypto/..., src/ir/memref.cpp, src/backend/common/soc.cpp
Adds LeftScale and RightScale across memory-space enums, bindings, string conversion, layout inference, SoC routing, and reuse classification.
Tile operation contracts and inference
python/pypto/ir/op/tile_ops.py, python/pypto/language/op/tile_ops.py, src/ir/op/tile_ops/*, include/pypto/ir/reinterpret_view_semantics.h, tests/ut/ir/operators/test_tile_ops.py
Adds MX load layouts and move target shapes, validates static reshapes, preserves tile-view metadata, supports additional reinterpret dtypes, and tests FP4 shape preservation.
Deferred scale code generation
include/pypto/codegen/pto/pto_codegen.h, src/codegen/pto/*, src/backend/common/pto_ops_*.cpp
Tracks pending scale fills and valid-shape updates, emits specialized MX moves and loads/stores, handles FP8 reinterpret aliases, and tightens move elision conditions.
PTOAS ordering and output preparation
python/pypto/backend/_ptoas_preprocess.py, src/ir/transforms/canonicalize_io_order_pass.cpp, .gitignore
Adds MX layout rewrites, MTE3/TPUSH synchronization, MX barriers, FIFO-sticky ordering constraints, and local probe artifact ignores.

Estimated code review effort: 5 (Critical) | ~90 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PythonAPI
  participant TileIR
  participant PTOCodegen
  participant PTOAS
  PythonAPI->>TileIR: Emit tile.load or tile.move with MX metadata
  TileIR->>PTOCodegen: Deduce layout and generate PTO operations
  PTOCodegen->>PTOCodegen: Defer scale fill and valid-shape updates
  PTOCodegen->>PTOAS: Produce MX-aware PTOAS output
  PTOAS->>PTOAS: Insert flags, barriers, and FIFO ordering
Loading

Possibly related PRs

Poem

A bunny found scale buffers bright,
With tiled moves hopping right.
Shapes stayed true, barriers grew,
Flags kept the pipelines in view.
MX paths now sparkle and flow—
Hop hop hooray for PTO!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 41.07% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is clearly related to the PR’s main theme: adding a minimal host-prequant MXFP8 matmul path.
Description check ✅ Passed The description matches the changeset, covering MX scale spaces, mx_layout loads, deferred scale fills, and MX matmul support.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eceda96b8a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/backend/common/pto_ops_elementwise.cpp Outdated
Comment thread src/ir/transforms/canonicalize_io_order_pass.cpp Outdated
Comment thread python/pypto/ir/op/tile_ops.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
python/pypto/language/op/tile_ops.py (1)

357-408: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

mx_layout's documented "targets Mat by default" never takes effect. All three sites share one root cause: target_memory defaults to MemorySpace.Vec in both Python load() wrappers and is always forwarded as an explicit kwarg, so DeduceTileLoadType's "target_memory absent → default to Mat" fallback can never fire through the DSL. An MX load called without an explicit target_memory=MemorySpace.Mat silently produces a Vec-space TileType whose tile_view is nonetheless laid out for Mat (fractal=32, ZZ/NN row/col-major) — an internally inconsistent type.

  • python/pypto/language/op/tile_ops.py#L357-L408: change target_memory's default to None and resolve it to MemorySpace.Mat when mx_layout != "none" (else MemorySpace.Vec) before calling _ir_ops.load.
  • python/pypto/ir/op/tile_ops.py#L162-L224: apply the same None-default + mx_layout-aware resolution before building kwargs["target_memory"].
  • src/ir/op/tile_ops/memory.cpp#L146-L169: no change needed once the two Python sites are fixed — the existing fallback becomes reachable and correct.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/pypto/language/op/tile_ops.py` around lines 357 - 408, Update
python/pypto/language/op/tile_ops.py lines 357-408 and
python/pypto/ir/op/tile_ops.py lines 162-224 so load() uses a None target_memory
default, resolving to MemorySpace.Mat for non-"none" mx_layout and
MemorySpace.Vec otherwise before forwarding target_memory. Make no direct change
in src/ir/op/tile_ops/memory.cpp lines 146-169; its existing absent-target
fallback should become reachable and remain unchanged.
🧹 Nitpick comments (4)
src/ir/op/tile_ops/transform.cpp (1)

365-382: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

is_mx_scale_box keys only on layout/fractal, not memory space.

Any tile that happens to carry fractal=32 with matching row/row or col/col layouts takes the MX path, regardless of whether it is a Mat/LeftScale/RightScale tile. The added equal-byte-width check keeps this safe today, so this is a note rather than a defect — consider additionally gating on tile_type->memory_space_ being Mat/LeftScale/RightScale to keep the relaxation as narrow as the comment claims.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ir/op/tile_ops/transform.cpp` around lines 365 - 382, Narrow the
is_mx_scale_box condition to require tile_type->memory_space_ be Mat, LeftScale,
or RightScale in addition to the existing fractal and layout checks. Keep the
existing equal-byte-width validation and flat none_box handling unchanged.
python/pypto/ir/type.py (1)

38-59: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Make prefix matching order-independent instead of relying on dict insertion order.

The correctness of mem_leftscale_ vs mem_left_ currently hinges on the literal ordering of this dict, which a future alphabetical sort or refactor would silently break (a LeftScale memref would then infer MemorySpace.Left). Matching the longest prefix removes the hazard and keeps the comment purely informational.

♻️ Longest-prefix lookup
 def _infer_tile_memory_space_from_memref(memref: MemRef | None) -> MemorySpace | None:
     if memref is None:
         return None
-    for prefix, memory_space in _MEMREF_NAME_PREFIX_TO_SPACE.items():
+    for prefix, memory_space in sorted(
+        _MEMREF_NAME_PREFIX_TO_SPACE.items(), key=lambda kv: len(kv[0]), reverse=True
+    ):
         if memref.name_hint.startswith(prefix):
             return memory_space
     return None
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/pypto/ir/type.py` around lines 38 - 59, Update
_infer_tile_memory_space_from_memref to select the matching entry with the
longest prefix rather than relying on _MEMREF_NAME_PREFIX_TO_SPACE insertion
order. Preserve the None behavior for absent memrefs and unmatched names, while
ensuring mem_leftscale_ and mem_rightscale_ resolve to their specific memory
spaces regardless of dictionary ordering.
python/pypto/backend/_ptoas_preprocess.py (1)

61-79: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Docstring claims a "~8 line window" but the implementation tracks matches with unbounded lifetime.

mx_nd_vars entries are never evicted, so a TLOAD anywhere later in the same scan (not just within ~8 lines) whose source SSA matches any earlier MX_A_ND GlobalTensor gets a pipe_barrier(PIPE_ALL) inserted. Likely harmless in practice (SSA names are typically function-scoped/unique), but worth either implementing the described windowing or updating the comment to match the actual (global, per-file) behavior.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/pypto/backend/_ptoas_preprocess.py` around lines 61 - 79, Update the
preprocessing loop around mx_nd_vars so MX_A_ND variable entries are only
eligible for matching TLOAD statements within the documented preceding ~8
non-empty lines. Evict entries outside that window before checking each TLOAD,
while preserving the existing barrier insertion for valid nearby matches.
tests/ut/ir/operators/test_tile_ops.py (1)

2389-2401: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding coverage for the new tile.load(mx_layout=...) / tile.move(target_shape=...) validation paths.

This PR introduces several new CHECKs in DeduceTileLoadType (invalid mx_layout name, non-FP8E8M0/UINT8 dtype) and DeduceTileMoveType (target_shape element-count mismatch, non-static target_shape dims) with no direct unit tests in this file.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/ir/operators/test_tile_ops.py` around lines 2389 - 2401, Add direct
unit tests in this file for the new validation branches in DeduceTileLoadType
and DeduceTileMoveType: invalid mx_layout names, unsupported mx_layout dtypes,
target_shape element-count mismatches, and non-static target_shape dimensions.
Assert each case raises the expected CHECK/error while retaining coverage for
valid load and move behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/backend/common/pto_ops_datamove.cpp`:
- Around line 1219-1233: Update the alias-tile construction in the result buffer
handling block to derive valid_row and valid_col from
GetEffectiveTileView(*result_tile).valid_shape, falling back to
result_tile->shape_ when the effective dimensions are not constant, so partial
validity is preserved. Also add descriptive diagnostic messages to both
INTERNAL_CHECK_SPAN checks for the shape-rank and constant-dimension
requirements.
- Around line 1326-1333: Ensure deferred valid-shape operations registered by
RegisterPendingSetValidShape are consumed before function finalization clears
pending_scale_fills and pending_set_validshapes. Add a finalize-time flush using
TakePendingSetValidShape, or emit the pending set_validshape directly during
finalization, while preserving the existing deferred behavior for pending Scale
fills.

In `@src/backend/common/pto_ops_elementwise.cpp`:
- Around line 548-567: Replace the duplicated local replace_dim lambdas in the
scale and non-scale paths with one shared helper. Prefer deriving
mat_view_ui8_ty and view_ty through codegen::ExtractTileTypeInfo and
codegen::FormatTileBufTypeString, removing the manual replace_dim,
replace_token, and force_dyn_valid textual patching where applicable.
- Around line 568-582: Update the force_dyn_valid lambda to check whether each
find_first_of(",>", start) call returns std::string::npos before replacing the
dimension value. Leave the annotation unchanged when the terminator is missing,
for both v_row and v_col handling, while preserving the existing replacement
behavior when a terminator is found.
- Around line 604-616: Update the tile allocation branch around EmitTileAddr()
to support F8/e8m0 Mat sources with a known src_off during PTOAS planning. When
EmitTileAddr() is false, avoid CHECK(false) and either handle the planner case
separately or emit/alias the same SSA without an addr= operand; retain the
existing alloc_tile addr= path when tile addresses are emitted.

In `@src/backend/common/pto_ops_memory.cpp`:
- Around line 120-122: Validate mx_layout immediately after reading it in the
operation handling code: accept only "none" and the six supported layout names,
and reject any other value explicitly with the existing error mechanism before
computing or using is_mx_load. Preserve the current ND path for "none" and the
MX path for the supported names.
- Around line 227-252: Update the flat_into_2d predicate to require both
caller-provided offsets to be constant zero values before selecting the
flattened view path. Validate offsets_tuple elements alongside the existing
shape and dimension checks, so nonzero or nonconstant offsets use the standard
path; keep the existing zero offsets passed to EmitPartitionViewPTO only when
this predicate succeeds.

In `@src/ir/transforms/canonicalize_io_order_pass.cpp`:
- Around line 330-367: Restrict is_fifo_sticky so tile.move is sticky only for
MX-scale staging or V2C-related tpop results, not ordinary
target_memory=Left/Right Mat→Left/Right transfers. Preserve the existing
tile.tpush/tile.tpop handling, and use the available operation context or
metadata to distinguish cross-core/MX-scale boundary moves from regular
intra-core L1→L0 moves before building the successors chain.

---

Outside diff comments:
In `@python/pypto/language/op/tile_ops.py`:
- Around line 357-408: Update python/pypto/language/op/tile_ops.py lines 357-408
and python/pypto/ir/op/tile_ops.py lines 162-224 so load() uses a None
target_memory default, resolving to MemorySpace.Mat for non-"none" mx_layout and
MemorySpace.Vec otherwise before forwarding target_memory. Make no direct change
in src/ir/op/tile_ops/memory.cpp lines 146-169; its existing absent-target
fallback should become reachable and remain unchanged.

---

Nitpick comments:
In `@python/pypto/backend/_ptoas_preprocess.py`:
- Around line 61-79: Update the preprocessing loop around mx_nd_vars so MX_A_ND
variable entries are only eligible for matching TLOAD statements within the
documented preceding ~8 non-empty lines. Evict entries outside that window
before checking each TLOAD, while preserving the existing barrier insertion for
valid nearby matches.

In `@python/pypto/ir/type.py`:
- Around line 38-59: Update _infer_tile_memory_space_from_memref to select the
matching entry with the longest prefix rather than relying on
_MEMREF_NAME_PREFIX_TO_SPACE insertion order. Preserve the None behavior for
absent memrefs and unmatched names, while ensuring mem_leftscale_ and
mem_rightscale_ resolve to their specific memory spaces regardless of dictionary
ordering.

In `@src/ir/op/tile_ops/transform.cpp`:
- Around line 365-382: Narrow the is_mx_scale_box condition to require
tile_type->memory_space_ be Mat, LeftScale, or RightScale in addition to the
existing fractal and layout checks. Keep the existing equal-byte-width
validation and flat none_box handling unchanged.

In `@tests/ut/ir/operators/test_tile_ops.py`:
- Around line 2389-2401: Add direct unit tests in this file for the new
validation branches in DeduceTileLoadType and DeduceTileMoveType: invalid
mx_layout names, unsupported mx_layout dtypes, target_shape element-count
mismatches, and non-static target_shape dimensions. Assert each case raises the
expected CHECK/error while retaining coverage for valid load and move behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b40bd2d4-e7bc-49e4-86f4-443f542253e9

📥 Commits

Reviewing files that changed from the base of the PR and between bcaea33 and eceda96.

📒 Files selected for processing (26)
  • .gitignore
  • include/pypto/codegen/pto/pto_codegen.h
  • include/pypto/ir/memory_space.h
  • include/pypto/ir/pipe.h
  • include/pypto/ir/reinterpret_view_semantics.h
  • include/pypto/ir/tile_view_semantics.h
  • include/pypto/ir/transforms/utils/attrs.h
  • python/bindings/modules/ir.cpp
  • python/pypto/backend/_ptoas_preprocess.py
  • python/pypto/ir/op/tile_ops.py
  • python/pypto/ir/type.py
  • python/pypto/language/op/tile_ops.py
  • python/pypto/language/parser/type_resolver.py
  • python/pypto/pypto_core/ir.pyi
  • src/backend/common/pto_ops_datamove.cpp
  • src/backend/common/pto_ops_elementwise.cpp
  • src/backend/common/pto_ops_memory.cpp
  • src/backend/common/soc.cpp
  • src/codegen/pto/pto_codegen.cpp
  • src/codegen/pto/pto_type_utils.cpp
  • src/ir/memref.cpp
  • src/ir/op/tile_ops/memory.cpp
  • src/ir/op/tile_ops/transform.cpp
  • src/ir/transforms/canonicalize_io_order_pass.cpp
  • src/ir/transforms/memory_reuse_pass.cpp
  • tests/ut/ir/operators/test_tile_ops.py

Comment thread src/backend/common/pto_ops_datamove.cpp Outdated
Comment thread src/backend/common/pto_ops_datamove.cpp Outdated
Comment thread src/backend/common/pto_ops_elementwise.cpp Outdated
Comment thread src/backend/common/pto_ops_elementwise.cpp Outdated
Comment thread src/backend/common/pto_ops_elementwise.cpp Outdated
Comment thread src/backend/common/pto_ops_memory.cpp Outdated
Comment thread src/backend/common/pto_ops_memory.cpp Outdated
Comment thread src/ir/transforms/canonicalize_io_order_pass.cpp Outdated
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
- Scope tile.reshape layout inheritance to MX scale fractal=32 only so
  existing rmsnorm / mixed-matmul paths keep InferTileLayoutFromShape
  (fixes UT vector-split, ST treshape boxed↔none_box, pypto-lib-model).
- Drop drive-by Bias from IsCubeMemorySpace; revert unrelated .gitignore.
- Match FIFO sticky ops to registered tpush_to_*/tpop_from_* names.
- Default mx_layout loads to Mat; tighten mx_layout/flat-store checks.
- clang-tidy includes; add Ascend950 LeftScale/RightScale memspace UTs.
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
…e-sys#2147

Cover SoC mem_graph edges for MX scale sidecars alongside existing
capacity checks.
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, FP4/FP8E8M0 reinterpret_view aliases, and ptoas ND-flat
→ mx_a_zz preprocess. No MX quant/matmul ops yet.

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Interface alignment (memspace + scale data path):

pto-isa (A5 / MX):
  TileType::ScaleLeft / ScaleRight — L0A/L0B MX scale buffers
    (include/pto/common/type.hpp); payload float8_e8m0_t
  Layout::MX_A_ZZ / MX_A_ND / MX_A_DN — activation scale GM layouts;
    TLoadMxCube* (AZZ2ZZ / AND2ZZ / …); SFractal=32 row-major ZZ
  Layout::MX_B_NN / MX_B_ND / MX_B_DN — weight scale layouts;
    col-major NN for MX_B_*
  TMov CommonCheckMX — uint8_t Mat → float8_e8m0 ScaleLeft/Right
  GetScaleAddr(Left/Right) — bind Scale tile to L0 sidecar before fill
    (sidecar bind-then-fill; Mat→Mat tmov illegal on A5)

PTOAS (v0.48+ dialect / EmitC):
  loc=scaling + !pto.f8E8M0 → TileType::ScaleLeft|ScaleRight
    (ui8 + scaling wrongly maps to Fixpipe TileType::Scaling;
    LeftScale/RightScale share one MLIR loc `scaling` until distinct)
  #pto.layout<mx_a_zz|mx_a_nd|…|mx_b_nn|…> on make_tensor_view / tload
    → Layout::MX_*; InferPTOLayout prefers mx_a_zz/mx_b_nn for
    !pto.f8E8M0 Mat+fractal=32
  pto.tmov Mat→scaling — must follow tget_scale_addr (matches
    PTOA5NormalizeTMovPass); PyPTO defers fill until then
  EmitC preprocess: float8_e8m0/uint8 MX_A_ZZ → MX_A_ND so TLoad uses
    AND2ZZ (activation scales are ND-flat on GM); MTE3 drain before
    Vec TPUSH; pipe_barrier before MX_A_ND TLOAD after AIV store

PyPTO ↔ interface:
  MemorySpace::LeftScale  ↔ loc=scaling + !pto.f8E8M0 ↔ ScaleLeft
  MemorySpace::RightScale ↔ loc=scaling + !pto.f8E8M0 ↔ ScaleRight
  tile.load(..., mx_layout=mx_a_zz|…) ↔ #pto.layout<…> ↔ Layout::MX_*
  tile.move Mat→LeftScale/RightScale ↔ deferred tmov after GetScaleAddr
  FP8E8M0 / UINT8 scale payload ↔ float8_e8m0_t (1-byte MX exp)

Dependencies:
  - PyPTO: FP8E8M0 dtype (hw-native-sys#2144 / DataType::FP8E8M0)
  - PTOAS: v0.48+ (loc=scaling, MX layouts, NormalizeTMov)
  - pto-isa: A5 MX TileType + Layout + TLoadMx / TMov / GetScaleAddr

Also includes review/CI fixes for hw-native-sys#2147:
  - Scope tile.reshape layout inheritance to MX scale fractal=32 only
  - Stamp memory_space_ on tile.move only for LeftScale/RightScale
  - FIFO-sticky: registered tpush_to_*/tpop_from_* + Scale moves only
  - Drop drive-by Bias from IsCubeMemorySpace; do not change .gitignore
  - Default mx_layout loads to Mat; tighten mx_layout / flat-store checks
  - Ascend950 LeftScale/RightScale capacity and Mat→Scale path UTs
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 63aaca6 to 60c0546 Compare July 27, 2026 06:37
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, FP4/FP8E8M0 reinterpret_view aliases, and ptoas ND-flat
→ mx_a_zz preprocess. No MX quant/matmul ops yet.

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Also includes review fixes for hw-native-sys#2147:
  - Drop unused TakePending*/DeferTFree/TakeDeferredTFreeCount (flush API
    already lives in feat/mx-quant-ops and feat/mx-matmul-ops)
  - Assert no unflushed deferred scale fills at GenerateFunction end
  - Use INTERNAL_UNREACHABLE_SPAN for the unsupported f8 Mat→Scale path
  - Align _barrier_before_mx_a_nd_tload comment with no-distance-cutoff code

Co-authored-by: Cursor <cursoragent@cursor.com>
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 60c0546 to c3bd91b Compare July 27, 2026 07:38
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, FP4/FP8E8M0 reinterpret_view aliases, and ptoas ND-flat
→ mx_a_zz preprocess. No MX quant/matmul ops yet.

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Also includes review/CI fixes for hw-native-sys#2147:
  - Drop unused TakePending*/DeferTFree/TakeDeferredTFreeCount (flush API
    already lives in feat/mx-quant-ops and feat/mx-matmul-ops)
  - Assert no unflushed deferred scale fills at GenerateFunction end
  - Use INTERNAL_UNREACHABLE_SPAN for the unsupported f8 Mat→Scale path
  - Align _barrier_before_mx_a_nd_tload comment with no-distance-cutoff code
  - Narrow TileType.shape dims via isinstance(ConstInt) for pyright

Co-authored-by: Cursor <cursoragent@cursor.com>
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from c3bd91b to 5d9ba49 Compare July 27, 2026 07:55
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, FP4/FP8E8M0 reinterpret_view aliases, and ptoas ND-flat
→ mx_a_zz preprocess. No MX quant/matmul ops yet.

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Also includes review/CI fixes for hw-native-sys#2147:
  - Drop unused TakePending*/DeferTFree/TakeDeferredTFreeCount (flush API
    already lives in feat/mx-quant-ops and feat/mx-matmul-ops)
  - Assert no unflushed deferred scale fills at GenerateFunction end
  - Use INTERNAL_UNREACHABLE_SPAN for the unsupported f8 Mat→Scale path
  - Align _barrier_before_mx_a_nd_tload comment with no-distance-cutoff code
  - Narrow TileType.shape dims via isinstance(ConstInt) for pyright
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 5d9ba49 to ae356cc Compare July 27, 2026 08:20
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 27, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, FP4/FP8E8M0 reinterpret_view aliases, and ptoas ND-flat
→ mx_a_zz preprocess. No MX quant/matmul ops yet.

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Also includes review/CI fixes for hw-native-sys#2147:
  - Drop unused TakePending*/DeferTFree/TakeDeferredTFreeCount (flush API
    already lives in feat/mx-quant-ops and feat/mx-matmul-ops)
  - Assert no unflushed deferred scale fills at GenerateFunction end
  - Use INTERNAL_UNREACHABLE_SPAN for the unsupported f8 Mat→Scale path
  - Align _barrier_before_mx_a_nd_tload comment with no-distance-cutoff code
  - Narrow TileType.shape dims via isinstance(ConstInt) for pyright

     after rewrite; raise if PTOAS changed its layout spelling.

     count equals expected Vec-TPUSH count.
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch 2 times, most recently from 6de65d9 to 1a1c54e Compare July 27, 2026 11:32
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 28, 2026
Add Ascend950 LeftScale/RightScale memory spaces (L0A/L0B MX scale
buffers, SoC capacity, loc=scaling) and the scale data path: tile.load
mx_layout, Mat↔Scale move with target_shape and deferred fill until
tget_scale_addr, and FP8E8M0 reinterpret_view aliases. Reject FP8E5M2
reinterpret on Ascend950. No MX quant/matmul ops or MXFP4 aliases yet
(those land in hw-native-sys#2117 with ND rewrite / AIC PIPE_ALL barriers).

Phase 2 of the hw-native-sys#2117 split (includes former Phase 1.2). Depends on
FP8E8M0 (hw-native-sys#2144). Refs hw-native-sys#1975.

Also includes review/CI fixes for hw-native-sys#2147:
  - Drop unused TakePending*/DeferTFree/TakeDeferredTFreeCount (flush API
    already lives in feat/mx-quant-ops and feat/mx-matmul-ops)
  - Assert no unflushed deferred scale fills at GenerateFunction end
  - Use INTERNAL_UNREACHABLE_SPAN for the unsupported f8 Mat→Scale path
  - Narrow TileType.shape dims via isinstance(ConstInt) for pyright
  - Drop fragile PTOAS C++ regex post-process (MX_A_ZZ→ND, MTE3 drain,
    pipe_barrier); defer sync/layout fixes to hw-native-sys#2117
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 1a1c54e to 3dc95df Compare July 28, 2026 02:34
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 28, 2026
Treat FP4 tile shape/valid_shape as packed storage units (K/2) while scale
groups and matmul K-matching use logical element counts. Re-allow INT8↔FP4
reinterpret_view aliases for MXFP4 paths (kept out of hw-native-sys#2147).
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 3dc95df to 281311b Compare July 28, 2026 02:55
@yanghaoran29 yanghaoran29 changed the title feat(ir/codegen): MX scale memory spaces and tile data path feat(ir/codegen): minimal host-prequant MXFP8 matmul path Jul 28, 2026
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 28, 2026
Treat FP4 tile shape/valid_shape as packed storage units (K/2) while scale
groups and matmul K-matching use logical element counts. Re-allow INT8↔FP4
reinterpret_view aliases for MXFP4 paths (kept out of hw-native-sys#2147).
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 28, 2026
…-sys#2117

Re-introduce V2C/scale FIFO sticky (moved out of hw-native-sys#2147), dedupe TakePending
helpers after rebase, allow FP4/INT8 matmul_mx data while still rejecting
FP8E5M2, and document hw-native-sys#2117 ISA/PTOAS extension constraints (en/zh).
@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 281311b to e458aa5 Compare July 28, 2026 03:17
yanghaoran29 added a commit to yanghaoran29/pypto that referenced this pull request Jul 28, 2026
…-sys#2147

- src/ir/op/tile_ops/mx.cpp: drop unused "pypto/ir/type_inference.h";
  add <optional> for std::nullopt used at the TileType memref arg
- src/ir/transforms/canonicalize_io_order_pass.cpp: drop redundant
  <any> (already provided transitively by pypto/core/any_cast.h)
@Little-oil

Copy link
Copy Markdown
Contributor

我按当前 head 28141a64 复查了一遍。整体上,先打通 host-prequant MXFP8 的 Left/Right + ScaleLeft/ScaleRight -> Acc 路径是合理的;但以下问题建议在合入前处理。

阻塞问题

  1. Deferred tfree 会丢失 pipe id,并且第一个 tget_scale_addr 会排空所有 scale 的 free。

    PendingScaleFill 是按目标 scale SSA 保存的,但 DeferredTFree 只有一个全局 vector,而且只保存 core/split、不保存 id 或对应的 scale/source。只要存在任意 pending fill,所有 tfree 都会进入该 vector;第一个匹配的 tget_scale_addr 随后调用 TakeDeferredTFrees() 全量排空。

    我用两个显式 pipe(id=0/1)做了实际 pass/codegen 复现,生成顺序为:

    %scale_a = pto.tpop_from_aiv {id = 0, split = 0}
    %scale_b = pto.tpop_from_aiv {id = 1, split = 0}
    pto.tget_scale_addr ... outs(%scale_a_l0)
    pto.tmov ins(%scale_a) outs(%scale_a_l0)
    pto.tfree_from_aiv {split = 0}
    pto.tfree_from_aiv {split = 0}
    pto.tget_scale_addr ... outs(%scale_b_l0)
    pto.tmov ins(%scale_b) outs(%scale_b_l0)

    两次 free 都丢失了 id,且第二个 free 的释放时机早于 %scale_btmov。相关代码:
    crosscore.cpp#L267-L290
    pto_codegen.h#L439-L450
    datamove.cpp#L656-L672

    对当前 host-prequant baseline,更简单的做法是正常输出 tmov/tfree,交给 PTOAS v0.48 已有且默认启用的
    PTOA5NormalizeTMovPass
    把匹配的 tget_scale_addr 移到 tmov 前。若保留延迟机制,则至少需要按具体 pending fill 关联 id/core/split/source/dst,每个 tget 只能释放自己的 free,不能全局排空。

  2. matmul_mx 的 physical K、valid K 和 scale shape 校验与 PTOAS v0.48 不一致。

    当前使用 lhs valid K 做 %32==0,却没有检查 physical K 必须是 64 的正倍数,也没有分别校验 scale 的 physical shape/valid shape 或 lhs/rhs valid K。结果包括:

    • physical K=96 会通过 PyPTO,但被 PTOAS 拒绝;
    • physical K=64、valid K=48 会被 PyPTO 拒绝,但 PTOAS 使用 ceil(48/32)=2
    • scale 的 physical shape 错误可能被正确的 valid shape 掩盖。

    建议完全对齐 PTOAS verifier:physical K %64==0,physical/valid scale shape 分别按各自的 ceil(K/32) 校验,并比较两侧 valid K。

  3. 公开的四种 mx_layout 在当前锁定的 PTOAS v0.48 中无法解析。

    PyPTO 接受并原样输出六种 layout,但 PTOAS v0.48 的
    Layout enum
    只有 mx_a_zzmx_b_nnmx_a_nd/mx_a_dn/mx_b_nd/mx_b_dn 会在 MLIR parse 阶段失败。建议本 PR 先只开放、记录和测试 ZZ/NN,其他 layout 等工具链支持后再加入。

  4. tget_scale_addr 的输出 memory space 被固定成了 LeftScale

    这里的 .set_output_reuses_input(0) 只表示 MemRef 复用,不会覆盖 .set_output_memory(LeftScale)。因此 B 侧 RightScale -> tget_scale_addr 的结果会先被推成 LeftScale,随后又被消费者需求拉回 RightScale,产生错误分析/多余 move。建议改为 .set_output_memory_inherit_input() + .set_output_reuses_input(0),并校验 LeftScale↔LeftRightScale↔Right 配对。
    mx.cpp#L56-L69

  5. 新增的 MX ST 目前没有进入任何 CI job。

    test_matmul_mx.py 只允许 a5/a5sim;direct ST 强制 --platform=a2a3;a5sim job 又使用显式文件白名单且未包含该文件。因此当前全部绿色的 CI 并没有执行这个 PR 的 PTOAS compile test 或 A5 数值测试。建议把该文件加入 a5sim 和真机 A5 job。

建议拆出或简化

当前 PR 名称是 minimal host-prequant,但还包含未来 tquant/混合核路径才需要的 tile.move(target_shape)、generic flat-store、根据 UINT8/FP8E8M0 猜测 MX scale role、boxed reinterpret_view 以及未注册的 matmul_mx_acc/bias backend hook。这些路径当前 ST 都没有使用,建议留到后续 stacked PR,减少对通用 load/move/store/reinterpret_view 的影响范围。

另外,不建议因为 A5 MX matmul 不支持 E5M2,就让通用、跨后端的 tile.reinterpret_view 全局拒绝 E5M2;该限制放在 matmul_mx 或 A5 lowering 更合适。

总结:基础方向合理,但建议先解决以上 5 个阻塞问题,并把未来 tquant 脚手架拆出后再合入。

@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch from 28141a6 to 90de75e Compare July 28, 2026 08:15
@yanghaoran29

Copy link
Copy Markdown
Contributor Author

Fixed in 90de75e7 (squashed to a single commit on this PR):

  1. DeferredTFree — removed from this minimal host-prequant path (normal tmov/tfree; PTOAS PTOA5NormalizeTMovPass owns reorder).
  2. matmul_mx K/scale — physical K % 64 == 0; valid K any positive; scale physical/valid checked separately with ceil(K/32); lhs/rhs valid K compared.
  3. mx_layout — only mx_a_zz / mx_b_nn (PTOAS v0.48).
  4. tget_scale_addrset_output_memory_inherit_input() + LeftScale↔Left / RightScale↔Right pairing.
  5. CItest_matmul_mx.py added to system-tests-a5sim whitelist.
  6. Scaffold — dropped move(target_shape), MX flat-store, dtype fractal guessing, boxed reinterpret, and matmul_mx_acc/bias stubs.
  7. E5M2 — no longer rejected in generic reinterpret_view; still rejected at matmul_mx.

@yanghaoran29
yanghaoran29 force-pushed the feat/mx-scale-data-path branch 7 times, most recently from 15b5e71 to 5660c6f Compare July 29, 2026 01:30
Add a minimal Ascend950 MX block-scale GEMM data path for host-prequant
MXFP8: Left/Right data tiles plus LeftScale/RightScale → Acc.

IR / language
- Register tile.matmul_mx (FP8E4M3FN × FP8E8M0 → FP32 Acc) with physical
  alignment M%16 / K%64 / N%32 and separate physical/valid scale-group
  checks ceil(K/32) so PyPTO rejects cases that would fail PTOAS
  matmul_mx vs tget_scale_addr verifiers.
- Register tile.tget_scale_addr (DPS, in-place on dst_scale) for
  LeftScale↔Left / RightScale↔Right address bind.
- Extend tile.load with mx_layout={mx_a_zz,mx_b_nn} (Mat-only, fractal=32)
  and tile.move destinations LeftScale/RightScale (UINT8→FP8E8M0 for
  loc=scaling). Reject FP8E5M2 at matmul_mx only.
- Add MemorySpace::{LeftScale,RightScale}, Cube SoC budget (4KB each),
  Mat→scale mem-graph edges, and MemoryReuse L0 exemption.

Codegen
- Emit pto.tmatmul.mx / pto.tget_scale_addr; MX loads rebind
  make_tensor_view with #pto.layout<mx_*>; Left/RightScale map to
  MLIR loc=scaling.
- Emit Mat→scale pto.tmov in source order (PTOAS PTOA5NormalizeTMovPass
  reorders bind-before-fill). Drop unused matmul_mx_acc/bias hooks.
- Elide same-addr tile.move only when layout *and* shape match.

Correctness (mx_layout + InferTileMemorySpace)
- OpRegistry stamps target_memory=Mat into Call kwargs when mx_layout is
  set but target_memory was omitted, so type and kwargs stay in sync.
- InferFromOp forces Mat for mx_layout loads; Phase-3 rewrite preserves
  the DeduceTileLoadType fractal=32 TileView instead of rebuilding via
  GetImplicitTileView (ordinary Mat NZ would strip fractal and break
  TLoadMxCube*).

Tests / CI
- UT for IR validation, memspace layouts, kwargs stamping, Infer fractal
  preservation; codegen UT for tmov-before-tget ordering; a5/a5sim ST
  with host-prequant golden (wired into CI a5sim allowlist).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants