Skip to content

[Bug] A drained ring heap parked at a non-zero offset refuses a whole-span allocation #1469

Closed
#1514
Closed
[Bug] A drained ring heap parked at a non-zero offset refuses a whole-span allocation#1469
#1514
Labels
bugSomething isn't working

Description

@Leaf-Salix
Leaf-Salix
opened on Jul 25, 2026

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

PTO2TaskAllocator::try_bump_heap
(src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_ring_buffer.h) fails an
allocation of size S (with S <= heap_size_) when the ring is already
drained (heap_top_ == heap_tail_, used == 0) but parked at a non-zero
offset d — even though the whole ring is free. When heap_size_ = R lands in
[S, 2S): the tail arc has only R - d < S bytes, and wrapping to the head arc
is rejected by the strict > guard (d > S is false at d == S). The allocator
then latches orch_error_code=2 HEAP_RING_DEADLOCK with no live bytes in the
ring. This is a pure allocator-arithmetic defect — an empty ring failing to use
its own capacity — independent of any upper-layer workload.

Steps to Reproduce

Add a minimal ST (adds `tests/` only, no `src/` runtime change):
`tests/st/a2a3/tensormap_and_ringbuffer/heap_empty_ring_rebase/`. One 1 MiB
scratch (`[1024,256] float32`) is reused across two **consecutive scopes** (each
depth 1 → ring 1): iteration 1's `scope_end` reclaims T1, draining the ring to
`top == tail == S` (parked at offset S); iteration 2 allocates a same-size T2 →
hits the empty-ring wrap defect. Squeeze ring 1's heap to `1.5 x S = 1.5 MiB`
(inside `[S, 2S)`):


PTO2_RING_HEAP=268435456,1572864,268435456,268435456 \
  python -m pytest \
  tests/st/a2a3/tensormap_and_ringbuffer/heap_empty_ring_rebase/test_heap_empty_ring_rebase.py \
  -s -v -p no:xdist --platform a2a3 --device <N>


(On a shared box wrap it in `task-submit --device <N> --run "..."` with a per-run
`ASCEND_PROCESS_LOG_PATH`.)

Expected Behavior

A fully drained ring must accommodate any allocation with S <= heap_size_.
Iteration 2 should succeed and the test should pass with Y1[0] == Y2[0] == 42.

Actual Behavior

Iteration 2's allocation fails on the empty ring; the host reports:

[ERROR] validate_runtime_impl: PTO2 runtime failed: orch_error_code=2 sched_error_code=0 runtime_status=-2
[ERROR] validate_runtime_impl: error detail: orch_error_code=2 HEAP_RING_DEADLOCK -
        the ring task allocator ran out of both task slots and heap bytes, so no further task can be admitted

(Host-side 507018 ACL_ERROR_RT_AICPU_EXCEPTION; device-classified reason =
orch_error_code=2 HEAP_RING_DEADLOCK.)

Git Commit ID

6284040

CANN Version

9.0.0

Driver Version

26.0.rc1

Host Platform

Linux (aarch64)

Additional Context

Fix corroboration (same ST file, same PTO2_RING_HEAP, only the runtime
differs): the fixed branch passes with Y1[0] == Y2[0] == 42 (1 passed).
Since the fix fits both 1 MiB scratches into the 1.5 MiB ring, the ring is
demonstrably large enough — upstream's failure is not a capacity shortfall but
the empty-ring arithmetic refusing an allocation that fits.

Root cause: the failure path of try_bump_heap lacks an empty-ring rebase.
When top == tail (used == 0) is parked at a non-zero offset and neither arc
fits, the span should be rebased to heap_base_
(heap_top_ = alloc_size; heap_tail_ = 0;). The strict > wrap guard itself is
correct (needed for the full/empty ambiguity); the missing case is the empty
ring.

Suggested fix — add to the top >= tail failure path of try_bump_heap:

} else if (top == tail && alloc_size <= heap_size_) {
    // The ring is empty (top == tail, used == 0) but parked at a non-zero
    // offset; this allocation fits neither arc even though the whole ring is
    // free. Rebase to 0 to make the span contiguous.
    result = heap_base_;
    heap_top_ = alloc_size;
    heap_tail_ = 0;
}
lib-free regression test — test_task_allocator.cpp::HeapEmptyRingRebasesForContiguousAlloc 
TEST_F(TaskAllocatorTest, HeapEmptyRingRebasesForContiguousAlloc) {
    constexpr uint64_t S = 3000;  // > HEAP_SIZE/2: the ring holds one at a time
    auto r1 = allocator.alloc(S);
    ASSERT_FALSE(r1.failed());
    const uint64_t top_after = allocator.heap_top();

    // Reclaim task 0 fully: tail advances to its extent end == top, draining the
    // ring to empty (top == tail) while parked at a non-zero offset.
    consume_up_to(descriptors, last_alive, heap_buf, WINDOW_SIZE, 1, top_after);

    // The whole ring is free, so the next same-size allocation must succeed by
    // rebasing to the heap base.
    auto r2 = allocator.alloc(S);
    ASSERT_FALSE(r2.failed()) << "empty ring must place an allocation that fits its whole span";
    EXPECT_EQ(r2.packed_base, static_cast<void *>(heap_buf));
    EXPECT_EQ(allocator.heap_top(), top_after);
}

Reproduced-at-HEAD: dynamically reproduced on main HEAD 62840403 (real
a2a3, orch_error_code=2 HEAP_RING_DEADLOCK, 1 failed) as well as on fork point
8cdb306c. Consistently, pto_ring_buffer.h (containing try_bump_heap) is
byte-identical across 8cdb306c..62840403 — the 36 intervening commits
touch only pto_orchestrator.cpp (dense-dep diagnostics #1448, marker
consolidation #1410, dispatch marker #1411, early-dispatch #1405), none of the
allocator logic.

Reproduction range & fix: reproduced across 8cdb306c..HEAD 62840403, over
which pto_ring_buffer.h is byte-identical — so this is not a regression from
those commits. The introducing commit was not bisected; the missing empty-ring
branch in try_bump_heap is a long-standing gap in the allocator, not a recent
change. Fixed on our fork by fd82ae69 ("Fix: rebase a drained ring heap to its
base for contiguous reuse", applied to all three runtimes).

Reproduction — full ST sources

Drop these into tests/st/a2a3/tensormap_and_ringbuffer/heap_empty_ring_rebase/, then
run the command in Steps to Reproduce. No src/ change is required.

test_heap_empty_ring_rebase.py 
#!/usr/bin/env python3
# Copyright (c) PyPTO Contributors.
# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
# CANN Open Software License Agreement Version 2.0 (the "License").
# Please refer to the License for details. You may not use this file except in compliance with the License.
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
# See LICENSE in the root of the software repository for the full text of the License.
# -----------------------------------------------------------------------------------------------------------
"""heap_empty_ring_rebase: reuse one S-byte scratch across two scopes on a ring
sized [S, 2S).

Each iteration is its own scope on the same ring (scope depth 1 -> ring 1), so
iteration 1's scope_end reclaims T1 and drains the ring to top == tail == S
(empty, but parked at the non-zero offset S). Iteration 2 then allocates T2 of
the same S bytes: the request fits neither the tail arc (heap_size - S < S) nor
the wrapped head arc (strict wrap guard tail > S is false at tail == S), so a
drained ring refuses an allocation that fits its whole span.

Run ring 1's heap down to 1.5x the scratch via env so it lands in [S, 2S):

    PTO2_RING_HEAP=268435456,1572864,268435456,268435456 \
      python -m pytest .../test_heap_empty_ring_rebase.py \
      -s -v -p no:xdist --platform a2a3 --device <N>

Expected on a2a3 (verified): fixed simpler passes and Y1[0]==Y2[0]==42; upstream
8cdb306c fails with host "orch_error_code=2 HEAP_RING_DEADLOCK" over a ring that
the fix proves is spacious enough (both scratches fit by rebasing).
"""

import torch
from simpler.task_interface import ArgDirection as D

from simpler_setup import SceneTestCase, TaskArgsBuilder, Tensor, scene_test

SENTINEL = 42.0
INIT_VAL = -1.0


@scene_test(level=2, runtime="tensormap_and_ringbuffer")
class TestHeapEmptyRingRebase(SceneTestCase):
    """Draining a ring to a non-zero offset must not block a whole-span alloc."""

    RTOL = 0
    ATOL = 0

    CALLABLE = {
        "orchestration": {
            "source": "kernels/orchestration/heap_empty_ring_rebase_orch.cpp",
            "function_name": "aicpu_orchestration_entry",
            "signature": [D.INOUT, D.INOUT],
        },
        "incores": [
            {
                "func_id": 0,
                "name": "FILL_CONST",
                "source": "kernels/aic/kernel_write_const.cpp",
                "core_type": "aic",
                "signature": [D.OUT],
            },
            {
                "func_id": 1,
                "name": "COPY_FIRST",
                "source": "kernels/aic/kernel_copy_first.cpp",
                "core_type": "aic",
                "signature": [D.IN, D.INOUT],
            },
        ],
    }

    CASES = [
        {
            "name": "EmptyRingRebase",
            "platforms": ["a2a3sim", "a2a3"],
            "config": {"aicpu_thread_num": 2, "block_dim": 1},
            "params": {},
        },
    ]

    def generate_args(self, params):
        y1 = torch.full((16,), INIT_VAL, dtype=torch.float32)
        y2 = torch.full((16,), INIT_VAL, dtype=torch.float32)
        return TaskArgsBuilder(
            Tensor("y1", y1),
            Tensor("y2", y2),
        )

    def compute_golden(self, args, params):
        # FILL_CONST writes SENTINEL to each scratch tensor's [0]; COPY_FIRST
        # copies that to Y1[0] / Y2[0]. Both iterations must complete.
        args.y1[0] = SENTINEL
        args.y2[0] = SENTINEL


if __name__ == "__main__":
    SceneTestCase.run_module(__name__)
kernels/orchestration/heap_empty_ring_rebase_orch.cpp 
/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * heap_empty_ring_rebase: minimal repro of the empty-ring wrap-reuse deadlock.
 *
 * One scope reuses a single S-byte scratch tensor across two iterations. The
 * scope's ring is sized in [S, 2S): it holds exactly one S-byte tensor at a
 * time. Iteration 1 allocates T1 (at heap offset 0), fills it, and a consumer
 * copies T1[0] to external Y1 — fully draining the ring so top == tail == S,
 * i.e. the ring is empty but parked at the non-zero offset S. Iteration 2's
 * producer depends on iteration 1's consumer, so when T2 is allocated the ring
 * is already empty; yet the S-byte request fits neither the tail arc
 * (heap_size - S < S) nor the wrapped head arc (strict wrap guard tail > S is
 * false at tail == S). A drained ring therefore refuses an allocation that
 * fits its whole span.
 *
 * Args layout: [Y1, Y2]  (external outputs the host verifies)
 */

#include <cstdint>

#include "pto_orchestration_api.h"  // NOLINT(build/include_subdir)

#define FUNC_FILL_CONST 0
#define FUNC_COPY_FIRST 1

extern "C" {

__attribute__((visibility("default"))) PTO2OrchestrationConfig aicpu_orchestration_config(const L2TaskArgs &orch_args) {
    (void)orch_args;
    return PTO2OrchestrationConfig{
        .expected_arg_count = 2,
    };
}

__attribute__((visibility("default"))) void aicpu_orchestration_entry(const L2TaskArgs &orch_args) {
    const Tensor &ext_Y1 = orch_args.tensor(0).ref();
    const Tensor &ext_Y2 = orch_args.tensor(1).ref();

    // One S-byte scratch = [1024, 256] float32 = 1 MiB. The scope ring is sized
    // 1.5 MiB via PTO2_RING_HEAP, landing squarely in [S, 2S).
    uint32_t scratch_shapes[2] = {1024, 256};
    TensorCreateInfo scratch_ci(scratch_shapes, 2, DataType::FLOAT32);

    // Iteration 1 in its own scope: produce T1 at heap offset 0, consume it into
    // Y1, then close the scope so the ring reclaims T1 and drains to
    // top == tail == S (empty, parked at the non-zero offset S).
    PTO2_SCOPE() {
        L0TaskArgs fill1;
        fill1.add_output(scratch_ci);
        TaskOutputTensors t1_outs = rt_submit_aic_task(FUNC_FILL_CONST, fill1);
        const Tensor &t1 = t1_outs.get_ref(0);

        L0TaskArgs copy1;
        copy1.add_input(t1);
        copy1.add_inout(ext_Y1);
        rt_submit_aic_task(FUNC_COPY_FIRST, copy1);
    }

    // Iteration 2 in the next scope on the same ring: T2 is allocated after the
    // ring is drained at offset S, so the S-byte request fits neither the tail
    // arc nor the wrapped head arc even though the ring is entirely free.
    PTO2_SCOPE() {
        L0TaskArgs fill2;
        fill2.add_output(scratch_ci);
        TaskOutputTensors t2_outs = rt_submit_aic_task(FUNC_FILL_CONST, fill2);
        const Tensor &t2 = t2_outs.get_ref(0);

        L0TaskArgs copy2;
        copy2.add_input(t2);
        copy2.add_inout(ext_Y2);
        rt_submit_aic_task(FUNC_COPY_FIRST, copy2);
    }
}

}  // extern "C"
kernels/aic/kernel_write_const.cpp 
/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * Writes a fixed sentinel (42.0f) to args[0][0]. Used as the producer in the
 * dummy_task scene tests so a downstream consumer can verify the dependency
 * chain (producer -> ... -> dummy -> consumer) propagates the value intact.
 */

#include <cstdint>
#include <pto/pto-inst.hpp>

#include "tensor.h"

#ifndef __gm__
#define __gm__
#endif

#ifndef __aicore__
#define __aicore__ [aicore]  // NOLINT(whitespace/braces)
#endif

#include "intrinsic.h"

#ifdef PTO_CPUSTUB_HPP
#define dcci(...) \
    do {          \
    } while (0)
#endif
#ifndef SINGLE_CACHE_LINE
#define SINGLE_CACHE_LINE 0
#endif
#ifndef CACHELINE_OUT
#define CACHELINE_OUT 0
#endif

extern "C" __aicore__ void kernel_entry(__gm__ int64_t *args) {
    __gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[0]);
    __gm__ float *out = reinterpret_cast<__gm__ float *>(out_tensor->buffer.addr) + out_tensor->start_offset;

    out[0] = 42.0f;
    dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT);
}
kernels/aic/kernel_copy_first.cpp 
/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * Consumer kernel for the dummy_task scene tests: copies args[0][0] -> args[1][0].
 *
 * If the dummy_task in the middle of the chain has correctly waited on the
 * producer and notified this consumer, args[1][0] will equal the producer's
 * sentinel (42.0f). If the dependency chain broke or the dummy somehow ran a
 * kernel that clobbered args[0], the value will differ.
 */

#include <cstdint>
#include <pto/pto-inst.hpp>

#include "tensor.h"

#ifndef __gm__
#define __gm__
#endif

#ifndef __aicore__
#define __aicore__ [aicore]  // NOLINT(whitespace/braces)
#endif

#include "intrinsic.h"

#ifdef PTO_CPUSTUB_HPP
#define dcci(...) \
    do {          \
    } while (0)
#endif
#ifndef SINGLE_CACHE_LINE
#define SINGLE_CACHE_LINE 0
#endif
#ifndef CACHELINE_OUT
#define CACHELINE_OUT 0
#endif

extern "C" __aicore__ void kernel_entry(__gm__ int64_t *args) {
    __gm__ Tensor *in_tensor = reinterpret_cast<__gm__ Tensor *>(args[0]);
    __gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[1]);

    __gm__ float *in = reinterpret_cast<__gm__ float *>(in_tensor->buffer.addr) + in_tensor->start_offset;
    __gm__ float *out = reinterpret_cast<__gm__ float *>(out_tensor->buffer.addr) + out_tensor->start_offset;

    out[0] = in[0];
    dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT);
}

Relation to #429

Issue #429 (closed) also reported this try_bump_heap rejection at
tail == alloc_size, but proposed the opposite fix — relaxing the guard from
tail > alloc_size to >=. That relaxation is incorrect: permitting
tail == alloc_size places an allocation that makes top == tail, which this
allocator reads as empty (used == 0), reintroducing the full/empty ambiguity.
The strict > is therefore intentional — locked by
test_task_allocator.cpp::HeapWrapGuardRejectsTailEqualsAllocSize and
HeapLinearGapGuardRejectsExactFit. The correct fix keeps the strict > and
rebases a genuinely empty ring (top == tail, used == 0) to its base, as
shown above. #429's >= was never applied upstream — try_bump_heap still uses
strict > at HEAD 62840403 — and this empty-ring deadlock still reproduces
there. So this is not the same defect as #429's proposed change: same symptom and
function, opposite (and correct) fix, still unresolved at HEAD.

中文总结

try_bump_heap 在环排空但停在非零偏移、且 ring_size ∈ [S, 2S) 时,会拒绝一个本该装得下
整段的 S 分配(尾段 R-d<S 不够,回绕又被 strict > guard 挡),报 HEAP_RING_DEADLOCK——纯
分配器算术缺陷,空环没用满自己容量。最小 ST(一个 1 MiB scratch 在两个连续 scope 里复用,ring 压到
1.5 MiB)在 upstream main HEAD 62840403(及 fork point 8cdb306c)上死锁、在 https://github.com/Leaf-Salix/simpler/tree/fix/auto-scope-heapfree 修复分支 fd82ae69
上通过。修复=在 top==tail 失败路径
里 rebase 回 base;lib-free 回归测试见上方折叠块。

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions