[Bug] Flaky ST: TestL3RemoteStore silently drops a peer remote_store under concurrent dist-system-tests runs

[Hzfengsy]

Component

Other (please specify in description) — distributed system tests (tests/st/distributed/) + the dist-system-tests CI job (.github/workflows/ci.yml). The failure is a silent numeric mismatch in the L3 pld.tile.remote_store path; it is not yet established whether the defect is in the distributed runtime or in CI job isolation.

Description

tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore intermittently fails with a data mismatch instead of an infra error — the ring/subview cross-rank remote_store produces wrong values rather than crashing.

It reproduces on unrelated branches at roughly the same wall-clock time and hits a different test method per run, which points at cross-run interference on the shared 2-card NPU host (HCCL ring state or device memory not isolated between concurrent dist-system-tests jobs) rather than a defect in either branch under test.

A silent wrong-value failure is worse than a crash: if it ever flakes the other way it could mask a real correctness regression in the L3 remote-store path.

Failure signature: a dropped write, not corrupted values

With SIZE = 64 and HALF = 32 (test_l3_remote_store.py:40-41), both recorded max diff values are exactly the maximum element of the expected tensor, which is the signature of a destination buffer left at its torch.zeros init value:

Run	Test	Expected slab	max(expected)	Reported max diff
30246833082	test_ring_shuffle	outputs[0] = inputs[1] = arange(100, 164)	163.0	163.0
30244762056	test_subview_halves	outputs[1] = cat(arange(0,32), arange(32,64))	63.0	63.0

In the test_subview_halves run the diff is 63.0 and not 163.0 — so rank 0's destination (expected_0 = arange(100, 164), max 163) did land correctly while rank 1's destination did not. That is one rank's peer write going missing entirely, not partial numeric corruption.

Caveat: this is inferred from the assertion's max diff arithmetic; the raw output tensors were not captured in the CI logs. Dumping outputs on failure would confirm it directly.

Why concurrent runs are possible

The workflow-level guard only serializes runs on the same ref:

# .github/workflows/ci.yml:10-12
concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Two different branches therefore run dist-system-tests concurrently. The job is currently in TEST MODE and anchors on the runner's own card rather than borrowing free cards via --device auto:

# .github/workflows/ci.yml:605-607
task-submit --device "$DEVICE_ID" --device-num 2 --timeout 3600 --max-time 900 \
  --run "cd $GITHUB_WORKSPACE/dist-checkout && source activate.sh && python -m pytest tests/st/distributed -v --device=\$TASK_DEVICE --ignore=tests/st/distributed/test_l2_multi_orch.py"

So there is no cross-branch mutual exclusion for the 2-card NPU set that these HCCL tests initialize a communicator over.

Steps to Reproduce

Observed twice on CI, ~25 min apart, on two unrelated branches, in the same test file:

1. Run 30246833082 (branch chore/ci-gate-jobs-behind-lint):

   FAILED tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_ring_shuffle
     - AssertionError: ring remote_store mismatch: max diff = 163.0
   

2. Run 30244762056 (branch issue-2134-ir-lower-trace):

   FAILED tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_subview_halves
     - AssertionError: subview remote_store mismatch: max diff = 63.0
   

Both runs were otherwise 1 failed, 56 passed, 22 skipped. Re-running the failed job alone passed, confirming the failure is not branch-related.

To reproduce deliberately, run two dist-system-tests jobs concurrently against the same NPU pair (or run the file in a loop while a second HCCL job holds the same cards):

source activate.sh
python -m pytest tests/st/distributed/test_l3_remote_store.py -v --device=<dev0>,<dev1> --count=10

Expected Behavior

TestL3RemoteStore::test_ring_shuffle and ::test_subview_halves pass deterministically on every run. Concurrent dist-system-tests jobs on the same host must not observe each other's HCCL ring state or device memory.

If cross-run interference is genuinely unavoidable on a shared host, the tests should fail loudly (comm-init / device-busy error) rather than silently returning a destination buffer that never received its peer write.

Actual Behavior

The cross-rank remote_store silently does not land on one rank; the destination stays at its zero-initialized value and only the final torch.allclose assertion catches it:

FAILED tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_ring_shuffle
  - AssertionError: ring remote_store mismatch: max diff = 163.0

FAILED tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_subview_halves
  - AssertionError: subview remote_store mismatch: max diff = 63.0

No HCCL error, no device error, no crash — only the numeric assertion.

Git Commit ID

460f856

NPU Kind

Ascend 910B

Host Platform

Linux (aarch64)

Additional Context

• Test source: tests/st/distributed/test_l3_remote_store.py (TestL3RemoteStore at :212, test_ring_shuffle at :215, test_subview_halves at :246; SIZE/HALF at :40-41).
• CI job: dist-system-tests at .github/workflows/ci.yml:541; run step at :597-607.
• Both tests build their program via ir.compile(..., distributed_config=DistributedConfig(device_ids=device_ids[:2], num_sub_workers=0)) and skip when fewer than 2 devices are available.
• Suggested next steps, roughly in order of cost:
  1. Dump the actual outputs tensor on assertion failure so the drop-vs-corrupt question is answered directly from CI logs rather than inferred.
  2. Add a concurrency guard so only one HCCL/NPU job touches a given card set at a time (e.g. a dedicated concurrency.group for dist-system-tests that is not keyed on github.ref, or flip the job to --device auto so task-submit lends disjoint cards).
  3. If the flake survives full isolation, the defect is in the L3 remote_store completion/ordering path rather than in CI.

Related: #1840 (flaky ST on a2a3 — different failure mode: hard AICore 507018 crash with device force-reset, single-card, test_dynamic_paged_attention).

[Hzfengsy]

Failing CI log

Source: run 30246833082 / job 89916678894 — dist-system-tests, runner npu-4, machine liteserver-hps-365a-00001, devices 10 and 11, 1 failed, 56 passed, 22 skipped in 309.79s.

Note: this is attempt 1. gh run view --job 89916678894 --log serves the attempt-2 (re-run) log instead, which passed and contains no pytest output — use gh api repos/hw-native-sys/pypto/actions/jobs/89916678894/logs to get the log below.

The assert False expansion confirms a dropped write, not corrupted values

pytest expanded torch.allclose(outputs, expected), so the actual tensors are in the log:

rank	outputs (actual)	expected	verdict
0	[0., 0., 0., …, 0.] (all zeros)	[100., 101., …, 163.]	peer write never landed
1	[0., 1., 2., …, 63.]	[0., 1., 2., …, 63.]	correct

Rank 1 received rank 0's input exactly; rank 0's destination stayed at its torch.zeros init value. So the failure is one rank's cross-rank remote_store going missing entirely, not partial numeric corruption — confirming directly what the issue body infers from the max diff arithmetic.

Correction to the issue body

The body states there is "no cross-branch mutual exclusion for the 2-card NPU set". That is wrong — the log shows task-submit takes an exclusive per-device lock before running:

[npu-lock] 获取设备 10 的锁 (无超时)...
[npu-lock] 已获取设备 10 的锁 (pid=1559119)
[npu-lock] 获取设备 11 的锁 (无超时)...
[npu-lock] 已获取设备 11 的锁 (pid=1559119)

The job waited 07:49:13 → 07:53:53 for that lock, held devices 10/11 exclusively for the whole pytest run, and released at 07:59:06. Device-level isolation is in place and working, so the "concurrent jobs share the same cards" mechanism in the body does not hold as written.

Log excerpt

任务已提交: task_20260727_154913_136536224935 (断开后可用 task-submit --wait task_20260727_154913_136536224935 重连)
等待任务执行: task_20260727_154913_136536224935 (Ctrl+C 终止任务)
[npu-lock] 获取设备 10 的锁 (无超时)...
[npu-lock] 已获取设备 10 的锁 (pid=1559119)
[npu-lock] 获取设备 11 的锁 (无超时)...
[npu-lock] 已获取设备 11 的锁 (pid=1559119)
...
tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_ring_shuffle FAILED [ 87%]
...
=================================== FAILURES ===================================
_____________________ TestL3RemoteStore.test_ring_shuffle ______________________

self = <test_l3_remote_store.TestL3RemoteStore object at 0xfffe65651780>
test_config = RunConfig(platform='a2a3', device_id=10, rtol=1e-05, atol=1e-05, strategy=<OptimizationStrategy.Default: 'Default'>, b..., ring_heap=None, ring_dep_pool=None, distributed_config=None, analyze_auto_scopes_for_deps=False, memory_planner=None)
device_ids = [10, 11]

    def test_ring_shuffle(self, test_config, device_ids):
        """Non-atomic overwrite: rank r pushes its input to peer (r + 1) % nranks."""
        if len(device_ids) < 2:
            pytest.skip(f"ring remote_store needs 2 devices, got {device_ids}")
    
        program = _build_ring_remote_store_program()
        compiled = ir.compile(
            program,
            platform=test_config.platform,
            distributed_config=DistributedConfig(
                device_ids=device_ids[:2],
                num_sub_workers=0,
            ),
        )
    
        inputs = torch.stack(
            [
                torch.arange(SIZE, dtype=torch.float32).reshape(1, SIZE),
                torch.arange(100.0, 100.0 + SIZE, dtype=torch.float32).reshape(1, SIZE),
            ]
        )
        outputs = torch.zeros((2, 1, SIZE), dtype=torch.float32)
    
        compiled(inputs, outputs)
    
        # outputs[r] = inputs[(r - 1) % nranks] → outputs[0] = inputs[1], etc.
        expected = torch.stack([inputs[1], inputs[0]])
>       assert torch.allclose(outputs, expected), (
            f"ring remote_store mismatch: max diff = {(outputs - expected).abs().max().item()}"
        )
E       AssertionError: ring remote_store mismatch: max diff = 163.0
E       assert False
E        +  where False = <built-in method allclose of type object at 0xffffb8fca128>(tensor([[[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]],\n\n        [[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,\n          14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.,\n          28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39., 40., 41.,\n          42., 43., 44., 45., 46., 47., 48., 49., 50., 51., 52., 53., 54., 55.,\n          56., 57., 58., 59., 60., 61., 62., 63.]]]), tensor([[[100., 101., 102., 103., 104., 105., 106., 107., 108., 109., 110.,\n          111., 112., 113., 114., 115., 116., 117., 118., 119., 120., 121.,\n          122., 123., 124., 125., 126., 127., 128., 129., 130., 131., 132.,\n          133., 134., 135., 136., 137., 138., 139., 140., 141., 142., 143.,\n          144., 145., 146., 147., 148., 149., 150., 151., 152., 153., 154.,\n          155., 156., 157., 158., 159., 160., 161., 162., 163.]],\n\n        [[  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,\n           11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,\n           22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.,  31.,  32.,\n           33.,  34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,  42.,  43.,\n           44.,  45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,  54.,\n           55.,  56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.]]]))
E        +    where <built-in method allclose of type object at 0xffffb8fca128> = torch.allclose

tests/st/distributed/test_l3_remote_store.py:242: AssertionError
...
=========================== short test summary info ============================
FAILED tests/st/distributed/test_l3_remote_store.py::TestL3RemoteStore::test_ring_shuffle - AssertionError: ring remote_store mismatch: max diff = 163.0
assert False
 +  where False = <built-in method allclose of type object at 0xffffb8fca128>(tensor([[[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n           0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]],\n\n        [[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,\n          14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.,\n          28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39., 40., 41.,\n          42., 43., 44., 45., 46., 47., 48., 49., 50., 51., 52., 53., 54., 55.,\n          56., 57., 58., 59., 60., 61., 62., 63.]]]), tensor([[[100., 101., 102., 103., 104., 105., 106., 107., 108., 109., 110.,\n          111., 112., 113., 114., 115., 116., 117., 118., 119., 120., 121.,\n          122., 123., 124., 125., 126., 127., 128., 129., 130., 131., 132.,\n          133., 134., 135., 136., 137., 138., 139., 140., 141., 142., 143.,\n          144., 145., 146., 147., 148., 149., 150., 151., 152., 153., 154.,\n          155., 156., 157., 158., 159., 160., 161., 162., 163.]],\n\n        [[  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,\n           11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,\n           22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.,  31.,  32.,\n           33.,  34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,  42.,  43.,\n           44.,  45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,  54.,\n           55.,  56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.]]]))
 +    where <built-in method allclose of type object at 0xffffb8fca128> = torch.allclose
======= 1 failed, 56 passed, 22 skipped, 2 warnings in 309.79s (0:05:09) =======
[npu-lock] 已释放设备 10 11 的锁