Skip to content

fix(toolchain): disable cce-simd-vf-fusion by default on A5 - #1474

Closed
lwDavid wants to merge 1 commit into
hw-native-sys:mainfrom
lwDavid:lwDavid/a5-disable-simd-vf-fusion
Closed

fix(toolchain): disable cce-simd-vf-fusion by default on A5#1474
lwDavid wants to merge 1 commit into
hw-native-sys:mainfrom
lwDavid:lwDavid/a5-disable-simd-vf-fusion

Conversation

@lwDavid

@lwDavid lwDavid commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Append --cce-simd-vf-fusion=false to the ccec compile flags for a5 / a5sim.
VF fusion can reorder same-V vector ops (expand / abs / sinkhorn) and breaks
DeepSeek-V4 correctness on Ascend950.

This restores the fix from 41fc8ba8, which was only ever on a side branch and
never reached main (git merge-base --is-ancestor 41fc8ba8 main → no).

Why it matters

Measured on a real Ascend 950 host, sweeping all 27 single-card DeepSeek V4-Pro
kernels in pypto-lib/models/deepseek/v4-pro/ serially:

kernels passing
without this flag 10 / 27
with this flag 26 / 27

(Everything else held fixed: same pypto, same pto-isa, same CANN, ptoas 0.48,
run serially on one locked card.)

The ~15 affected kernels compile and run cleanly — they just produce wrong
numbers. hc_pre's comb output, for example, comes back all-NaN. There is no
compile error and no runtime error, which is what makes this worth pinning down
with a test.

Scope

Deliberately limited to per-kernel incore compilation
(KernelCompiler.compile_incoreCCECToolchain.get_compile_flags).

It is not mirrored into the platform AICore build
(src/a5/platform/onboard/aicore/CMakeLists.txt), and should not be: the only VF
construct there is the __simd_vf__ meta anchor in simt_anchor.h, whose fusion
behaviour is exactly what makes bisheng tag the entry SIMD_SIMT_MIX_VF(4)
rather than SIMT_VF_ONLY(3). That file records that getting the classification
wrong produced "107000 param-invalid across the whole a5 ST suite". A comment now
says so at the flag site, so the asymmetry does not read as an oversight.

a2a3 / a2a3sim are untouched — the flag is behind a platform guard, and
CCECToolchain is only ever constructed with one of the four known platform
strings.

Tests

tests/ut/py/test_toolchain.py had no coverage of
CCECToolchain.get_compile_flags at all, and simd-vf-fusion appeared exactly
once in the repo. Added TestCCECToolchainCompileFlags asserting the flag is
present for a5/a5sim, absent for a2a3/a2a3sim, and present for both
aiv and aic core types. The constructor's ASCEND_HOME_PATH and compiler
existence probes are monkeypatched so the test needs no CANN install.

python -m pytest tests/ut/py/test_toolchain.py → 20 passed.

Note for anyone re-measuring

Compiled kernel binaries are not cached by compile flags — the non-external
branch of kernel_binary_cache_path keys on
incore_{func_id}_{core_type}_{stem}_{platform}.bin, and device_runner also
reuses a source.with_suffix('.o') sidecar. Reusing a build_output/ produced
before this change will silently re-run the VF-fused binary. Wipe build_output/
before comparing.

Open question for reviewers

--cce-simd-vf-fusion=false is a bare driver flag (same family as
--cce-aicore-only), not an -mllvm pass-through, so a ccec that does not
recognise it would hard-fail every A5 kernel compile with no way to opt out.
Worth gating on an env var following the existing
SIMPLER_ENABLE_PTO_SDMA_WORKSPACE idiom (default off = flag emitted)? Left out
here to keep the diff minimal and faithful to 41fc8ba8; happy to add it.

VF fusion can reorder same-V vector ops (expand/abs/sinkhorn) and break
DeepSeek-V4 correctness on Ascend950. Append --cce-simd-vf-fusion=false
for a5/a5sim ccec builds.

Without this, ~15 of the 27 single-card DeepSeek V4-Pro kernels compile
and run but produce wrong numbers on a real A5 device; with it they pass.
The failure mode is silent numerical corruption rather than a compile
error, so add a unit test pinning the flag on for a5/a5sim and off for
a2a3/a2a3sim -- tests/ut/py/test_toolchain.py previously had no coverage
of CCECToolchain.get_compile_flags at all.

Scope is deliberately limited to per-kernel incore compilation. The
platform AICore build (src/a5/platform/onboard/aicore/CMakeLists.txt)
must keep VF fusion enabled: its only VF construct is the __simd_vf__
meta anchor in simt_anchor.h, whose fusion behaviour is what makes
bisheng tag the entry SIMD_SIMT_MIX_VF(4).
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d04299aa-3ea6-468a-98ac-ce2873204c1a

📥 Commits

Reviewing files that changed from the base of the PR and between bdf6fed and 7b45bf4.

📒 Files selected for processing (2)
  • simpler_setup/toolchain.py
  • tests/ut/py/test_toolchain.py

📝 Walkthrough

Walkthrough

CCECToolchain.get_compile_flags() now disables SIMD VF fusion for a5 and a5sim. Unit tests verify behavior across platforms and both aiv and aic core types.

Changes

CCEC VF fusion control

Layer / File(s) Summary
Platform flag generation and validation
simpler_setup/toolchain.py, tests/ut/py/test_toolchain.py
CCECToolchain appends --cce-simd-vf-fusion=false for a5 and a5sim, while tests confirm it is absent for a2a3 and a2a3sim and applies to both supported core types.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Poem

A bunny hops through flags so bright,
“VF fusion off” for A5 tonight.
A2A3 keeps its setting true,
AIV and AIC share the cue.
Tests nibble bugs away—
Hip-hop, compile day!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly names the main change: disabling cce-simd-vf-fusion on A5.
Description check ✅ Passed The description is directly about the same A5 toolchain flag change and test coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch lwDavid/a5-disable-simd-vf-fusion

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lwDavid lwDavid self-assigned this Jul 25, 2026
@lwDavid lwDavid added the bug Something isn't working label Jul 25, 2026
@lwDavid lwDavid moved this to In Progress in pto project Jul 25, 2026
@lwDavid
lwDavid marked this pull request as draft July 25, 2026 08:43
@lwDavid

lwDavid commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

Closing — superseded by #1446, which is the better fix and already on main.

I had missed that #1446 landed on 2026-07-23. It enables -mllvm -cce-vf-aa-between-iters=true for A5, which makes VF fusion correct rather than turning it off, so the optimization is kept.

Verified on a real Ascend 950 host, sweeping all 27 single-card DeepSeek V4-Pro kernels serially (ptoas 0.48, same pypto and pto-isa in both arms):

simpler result
8cdb306c + this PR (fusion off) 26 / 27
d4071fe1 (#1446, fusion on + inter-iteration AA) 26 / 27

Identical, so there is no reason to disable fusion. The one remaining failure is
prefill_indexer score in both arms, which is an unrelated pre-existing bug.

For the record, one thing that is not fixed by either arm: prefill_sparse_attn
intermittently fails with AICPU 507018 (3 runs under #1446: PASS / FAIL / FAIL). It is
equally flaky with fusion disabled, so it is independent of the VF question.

The real gap on our side was that pypto pins runtime at 8cdb306c (07-20), which
predates both #1446 and #1423 (feat(a5/tmr): port allow_early_resolve / early-dispatch from a2a3). Bumping that pin is the correct fix and is being handled on the pypto side.

@lwDavid lwDavid closed this Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant