Fix: scalar_data_test check tensor reads uninitialized device slot by ChaoWao · Pull Request #1449 · hw-native-sys/simpler

ChaoWao · 2026-07-23T02:19:32Z

Summary

scalar_data_test intermittently fails st-onboard-a2a3 with a golden mismatch on check (max_diff≈9.4e-4, rtol/atol=1e-5). Root-caused on a2a3 silicon.

Root cause

The check output tensor is sized 10, but the orchestration writes only check[0..8] (9 values via set_tensor_data). Output-tensor slots are not seeded from the host buffer on device (confirmed on hardware: a 123.0 host sentinel reads back as 0.0), so the never-written check[9] reads whatever the device output buffer holds:

In isolation it's 0.0 → matches the zero golden → passes.
Under the full onboard suite (all tests share one device) it reads recycled buffer residue (~9.4e-4) → golden mismatch. This is why it's intermittent and survives re-runs.

Decisive evidence (a2a3, sentinel-seeded check):

mismatch on 'check': argmax_idx=9  actual=0.0  expected=123.0
actual  =[2.0,102.0,77.0,77.0,79.0,42.0,12.0,88.0,55.0, 0.0]
expected=[2.0,102.0,77.0,77.0,79.0,42.0,12.0,88.0,55.0, 123.0]

Fix

Size check to exactly the 9 written slots. Verified onboard on a2a3: a size-9 sentinel-seeded run passes (every slot overwritten). No golden change (it only sets 0..8); sim is bit-exact and unaffected. Unrelated to #1436 (this is a level=2 device-runtime test).

🤖 Generated with Claude Code

The `check` output tensor was sized 10, but the orchestration writes only check[0..8] via set_tensor_data (9 values). Output-tensor slots are not seeded from the host buffer on device, so the never-written check[9] read whatever the device output buffer held: 0.0 when the test runs in isolation (matching the zero golden), but recycled buffer residue (~9.4e-4) under the full onboard suite that shares one device — an intermittent golden mismatch on st-onboard-a2a3. Size `check` to exactly the 9 written slots. Verified onboard on a2a3: a size-9 sentinel-seeded run passes (all slots overwritten), and the failing slot-9 residue is gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gemini-code-assist · 2026-07-23T02:19:35Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai · 2026-07-23T02:19:50Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 00e88c95-89d3-4408-95f5-c635d46597d4

📥 Commits

Reviewing files that changed from the base of the PR and between 717d703 and d29aa94.

📒 Files selected for processing (1)

examples/a2a3/tensormap_and_ringbuffer/scalar_data_test/test_scalar_data.py

📝 Walkthrough

Walkthrough

The scalar data test’s check tensor now allocates nine float slots, matching the indices written by orchestration, with comments documenting the write range and avoiding an unused uninitialized slot.

Changes

Scalar data test

Layer / File(s)	Summary
Align check tensor allocation `examples/a2a3/tensormap_and_ringbuffer/scalar_data_test/test_scalar_data.py`	The `check` tensor allocation changes from 10 to 9 float slots, with comments documenting writes to indices `0..8`.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Poem

A bunny checks nine slots in line,
No tenth stray cell is left behind.
The writes now fit from zero to eight,
Clean little tensors, neatly great!
Hop, hop—tests align their fate.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main fix: resizing the check tensor to avoid reading an unwritten device slot.
Description check	✅ Passed	The description directly explains the intermittent failure, root cause, and the fix applied in this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Fixes hw-native-sys#1483 `tests/st/a2a3/host_build_graph/bgemm` declared its `C` argument `D.OUT`, but the task graph read-modify-writes it: `p_add.add_inout(c_view)` feeds `kernel_tile_add`, whose first `k` iteration `TLOAD`s `C` before any task has written it. The runtime stages only IN/INOUT tensors host->device (`runtime_maker.cpp`, "pure write-only OUTPUT buffers are never read by the kernel"), and since hw-native-sys#1365 it no longer zero-fills OUT buffers either. So `C` accumulated onto whatever the device allocation happened to hold. On sim that allocation is a plain `malloc` block on a process-lifetime worker: fresh zeros when the case runs alone, a previous case's bytes in a full-suite run — hence a golden mismatch of 1e25-1e36 that reproduced only under the full suite. Same class as hw-native-sys#1449. Declare `C` INOUT, matching the two sibling bgemm variants (`examples/a5/tensormap_and_ringbuffer/bgemm`, `examples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm`), which already tag their accumulator that way. Seed `C` with a non-zero base (0.25) so the staging is load-bearing and the case is deterministic: golden is now `C_init + A @ B`, which a zeroed or unstaged device buffer cannot match. Without the signature fix this fails every run instead of one in two or three — verified, it reproduces the reported 1e35 magnitude on the first attempt. Drop the `pytest.mark.skip` that quarantined the case: the underlying defect is fixed, so the case runs in CI again. Verified: bgemm passes 3/3 in isolation on a2a3sim and onboard on a2a3; the full `pytest examples tests/st --platform a2a3sim` suite is green 3/3 runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

) Fixes #1483 `tests/st/a2a3/host_build_graph/bgemm` declared its `C` argument `D.OUT`, but the task graph read-modify-writes it: `p_add.add_inout(c_view)` feeds `kernel_tile_add`, whose first `k` iteration `TLOAD`s `C` before any task has written it. The runtime stages only IN/INOUT tensors host->device (`runtime_maker.cpp`, "pure write-only OUTPUT buffers are never read by the kernel"), and since #1365 it no longer zero-fills OUT buffers either. So `C` accumulated onto whatever the device allocation happened to hold. On sim that allocation is a plain `malloc` block on a process-lifetime worker: fresh zeros when the case runs alone, a previous case's bytes in a full-suite run — hence a golden mismatch of 1e25-1e36 that reproduced only under the full suite. Same class as #1449. Declare `C` INOUT, matching the two sibling bgemm variants (`examples/a5/tensormap_and_ringbuffer/bgemm`, `examples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm`), which already tag their accumulator that way. Seed `C` with a non-zero base (0.25) so the staging is load-bearing and the case is deterministic: golden is now `C_init + A @ B`, which a zeroed or unstaged device buffer cannot match. Without the signature fix this fails every run instead of one in two or three — verified, it reproduces the reported 1e35 magnitude on the first attempt. Drop the `pytest.mark.skip` that quarantined the case: the underlying defect is fixed, so the case runs in CI again. Verified: bgemm passes 3/3 in isolation on a2a3sim and onboard on a2a3; the full `pytest examples tests/st --platform a2a3sim` suite is green 3/3 runs.

ChaoWao merged commit 6a36fd5 into hw-native-sys:main Jul 23, 2026
15 checks passed

ChaoWao deleted the fix-scalar-data-check-slot branch July 23, 2026 02:39

Little-oil mentioned this pull request Jul 23, 2026

Update: make dense dependency diagnostics debug-only #1448

Merged

8 tasks

This was referenced Jul 26, 2026

[Bug] bgemm (host_build_graph) intermittently produces garbage output in full sim-suite runs #1483

Closed

Fix: stage bgemm's C accumulator to the device (host_build_graph) #1488

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix: scalar_data_test check tensor reads uninitialized device slot - #1449

Fix: scalar_data_test check tensor reads uninitialized device slot#1449
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:fix-scalar-data-check-slot

ChaoWao commented Jul 23, 2026

Uh oh!

gemini-code-assist Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ChaoWao commented Jul 23, 2026

Summary

Root cause

Fix

Uh oh!

gemini-code-assist Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading