fix(toolchain): disable cce-simd-vf-fusion by default on A5 - #1474
fix(toolchain): disable cce-simd-vf-fusion by default on A5#1474lwDavid wants to merge 1 commit into
Conversation
VF fusion can reorder same-V vector ops (expand/abs/sinkhorn) and break DeepSeek-V4 correctness on Ascend950. Append --cce-simd-vf-fusion=false for a5/a5sim ccec builds. Without this, ~15 of the 27 single-card DeepSeek V4-Pro kernels compile and run but produce wrong numbers on a real A5 device; with it they pass. The failure mode is silent numerical corruption rather than a compile error, so add a unit test pinning the flag on for a5/a5sim and off for a2a3/a2a3sim -- tests/ut/py/test_toolchain.py previously had no coverage of CCECToolchain.get_compile_flags at all. Scope is deliberately limited to per-kernel incore compilation. The platform AICore build (src/a5/platform/onboard/aicore/CMakeLists.txt) must keep VF fusion enabled: its only VF construct is the __simd_vf__ meta anchor in simt_anchor.h, whose fusion behaviour is what makes bisheng tag the entry SIMD_SIMT_MIX_VF(4).
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthrough
ChangesCCEC VF fusion control
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Closing — superseded by #1446, which is the better fix and already on I had missed that #1446 landed on 2026-07-23. It enables Verified on a real Ascend 950 host, sweeping all 27 single-card DeepSeek V4-Pro kernels serially (ptoas 0.48, same pypto and pto-isa in both arms):
Identical, so there is no reason to disable fusion. The one remaining failure is For the record, one thing that is not fixed by either arm: The real gap on our side was that pypto pins |
Summary
Append
--cce-simd-vf-fusion=falseto thecceccompile flags fora5/a5sim.VF fusion can reorder same-V vector ops (expand / abs / sinkhorn) and breaks
DeepSeek-V4 correctness on Ascend950.
This restores the fix from
41fc8ba8, which was only ever on a side branch andnever reached
main(git merge-base --is-ancestor 41fc8ba8 main→ no).Why it matters
Measured on a real Ascend 950 host, sweeping all 27 single-card DeepSeek V4-Pro
kernels in
pypto-lib/models/deepseek/v4-pro/serially:(Everything else held fixed: same pypto, same pto-isa, same CANN, ptoas 0.48,
run serially on one locked card.)
The ~15 affected kernels compile and run cleanly — they just produce wrong
numbers.
hc_pre'scomboutput, for example, comes back all-NaN. There is nocompile error and no runtime error, which is what makes this worth pinning down
with a test.
Scope
Deliberately limited to per-kernel incore compilation
(
KernelCompiler.compile_incore→CCECToolchain.get_compile_flags).It is not mirrored into the platform AICore build
(
src/a5/platform/onboard/aicore/CMakeLists.txt), and should not be: the only VFconstruct there is the
__simd_vf__meta anchor insimt_anchor.h, whose fusionbehaviour is exactly what makes bisheng tag the entry
SIMD_SIMT_MIX_VF(4)rather than
SIMT_VF_ONLY(3). That file records that getting the classificationwrong produced "107000 param-invalid across the whole a5 ST suite". A comment now
says so at the flag site, so the asymmetry does not read as an oversight.
a2a3/a2a3simare untouched — the flag is behind a platform guard, andCCECToolchainis only ever constructed with one of the four known platformstrings.
Tests
tests/ut/py/test_toolchain.pyhad no coverage ofCCECToolchain.get_compile_flagsat all, andsimd-vf-fusionappeared exactlyonce in the repo. Added
TestCCECToolchainCompileFlagsasserting the flag ispresent for
a5/a5sim, absent fora2a3/a2a3sim, and present for bothaivandaiccore types. The constructor'sASCEND_HOME_PATHand compilerexistence probes are monkeypatched so the test needs no CANN install.
python -m pytest tests/ut/py/test_toolchain.py→ 20 passed.Note for anyone re-measuring
Compiled kernel binaries are not cached by compile flags — the non-external
branch of
kernel_binary_cache_pathkeys onincore_{func_id}_{core_type}_{stem}_{platform}.bin, anddevice_runneralsoreuses a
source.with_suffix('.o')sidecar. Reusing abuild_output/producedbefore this change will silently re-run the VF-fused binary. Wipe
build_output/before comparing.
Open question for reviewers
--cce-simd-vf-fusion=falseis a bare driver flag (same family as--cce-aicore-only), not an-mllvmpass-through, so accecthat does notrecognise it would hard-fail every A5 kernel compile with no way to opt out.
Worth gating on an env var following the existing
SIMPLER_ENABLE_PTO_SDMA_WORKSPACEidiom (default off = flag emitted)? Left outhere to keep the diff minimal and faithful to
41fc8ba8; happy to add it.