[Bug] bgemm (host_build_graph) intermittently produces garbage output in full sim-suite runs

[ChaoWao]

Platform

a2a3sim (Ascend 910B/C simulation)

Runtime Variant

host_build_graph

Description

tests/st/a2a3/host_build_graph/bgemm/test_bgemm.py::TestBgemmHostBuildGraph
intermittently fails its golden comparison with values that are not plausible
rounding error:

Golden mismatch on 'C': max_diff=0.004227787721902132,  rtol=0.001, atol=0.001
Golden mismatch on 'C': max_diff=1.850953679053646e+25, rtol=0.001, atol=0.001
Golden mismatch on 'C': max_diff=2.749136603892804e+29, rtol=0.001, atol=0.001
Golden mismatch on 'C': max_diff=3.220750303614268e+36, rtol=0.001, atol=0.001

Inputs are torch.randn(...) * 0.01, so the output magnitude is around 1e-2.
A max_diff of 1e25–1e36 is not accumulation drift — the comparison is reading
memory that was never written by this run, or was written by something else.

The failure appears only when the full suite runs; the case has never failed
in isolation. That points at cross-case interference or device-buffer reuse
rather than the GEMM kernel computing a wrong value. Likely the same class as
the scalar_data_test check tensor reading an uninitialised device slot, fixed
in #1449.

generate_args does not seed the RNG, so the exact inputs differ every run and
the failure is not bit-reproducible.

Steps to Reproduce

On a clean tree at the commit below (no local modifications):

rm -rf build/lib build/cache
python simpler_setup/build_runtimes.py --lib-dir build/lib --cache-dir build/cache \
    --platforms a2a3 a5 a2a3sim a5sim
pip install --no-build-isolation --force-reinstall --no-deps -e .

# Repeat — it reproduces roughly one run in two or three.
pytest examples tests/st --platform a2a3sim --device 0-15 -v \
    --pto-session-timeout 600 --require-pto-isa

Control that isolates it to the full-suite path:

# 6/6 passes, never reproduced
pytest tests/st/a2a3/host_build_graph/bgemm --platform a2a3sim --device 0-15 -q

Expected Behavior

C matches the golden within RTOL/ATOL (1e-3) on every run, and the suite is
deterministic with respect to case ordering and parallelism.

Actual Behavior

Roughly one full-suite run in two or three fails with max_diff between 1e25 and
1e36 — values that cannot come from fp32 accumulation of ~1e-2 inputs — while the
same case passes every time it is run alone.

Git Commit ID

247f561

CANN Version

8.5.T3 (/usr/local/Ascend/cann-9.0.0)

Driver Version

N/A — reproduces on the simulator, no driver involved

Host Platform

Linux (aarch64)

Additional Context

Found while validating PRs #1477 / #1478 / #1309. It is not caused by those
changes: the reproduction above is on unmodified main with build/lib and
build/cache deleted and rebuilt from scratch, which also rules out a stale or
mixed-vintage build.

Two notes for whoever picks this up:

• The first observed failure (max_diff=0.0042) looks like a tolerance-boundary
  miss and could mislead you into treating this as a tolerance-tuning problem.
  The 1e25–1e36 failures show it is not.
• Because the inputs are unseeded, adding a fixed seed would make the case
  reproducible but might also mask the window rather than fix it. Worth seeding
  for triage, not as the fix.

Related: #1449 (same class, scalar_data_test).