[Bug] Heap reclaim coupled to the task watermark: a live zero-byte head freezes reclaim of later consumed extents

[Leaf-Salix]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

The heap reclaim front is tied to the task watermark last_task_alive. When the
oldest live task in a ring is a zero-byte extent (scope-gated, COMPLETED,
but still holding a lagging consumer), it pins last_task_alive; the heap bytes
of later, already-CONSUMED positive-extent tasks then cannot be
reclaimed — even though those bytes are physically reclaimable and the head task
itself owns no heap. The ring fills and reports HEAP_RING_DEADLOCK while
reclaimable space exists. This violates the most basic runtime contract —
consumed bytes are reclaimable — because a zero-byte live task owns no heap
and must not block retirement of the consumed bytes behind it.

Steps to Reproduce

Add a minimal ST (adds `tests/` only, no `src/` runtime change):
`tests/st/a2a3/tensormap_and_ringbuffer/heap_reclaim_watermark/`. Task graph (all
on ring 1):

1. A holder (AIC spin kernel, ~1.3 s) occupies one core to gate HC.
2. A **zero-extent dummy head H** and its consumer HC share **one scope** — H is
   scope-gated (alive) at the moment HC registers its dependency on H, so HC
   holds H's fanout; HC is also gated on the holder, so it cannot run while the
   holder spins → H stays `COMPLETED` but un-consumed, pinning ring 1's
   `last_task_alive` at task 0 (a zero-byte task).
3. Three filler producer/consumer pairs, each in its own scope, each producer
   allocating a 1 MiB scratch and each consumer copying `scratch[0]` to an
   external Y and fully consuming it. Ring 1 is squeezed to 2.5 MiB (~2 fillers).

(Key: HC must register its dependency **inside H's scope** — otherwise H, having
no consumer at submit time, auto-retires before HC is submitted and the explicit
dependency is skipped in `pto_orchestrator.cpp`, so H never pins.) Run:


PTO2_RING_HEAP=268435456,2621440,268435456,268435456 \
  python -m pytest \
  tests/st/a2a3/tensormap_and_ringbuffer/heap_reclaim_watermark/test_heap_reclaim_watermark.py \
  -s -v -p no:xdist --platform a2a3 --device <N>


(On a shared box wrap it in `task-submit --device <N> --run "..."` with a per-run
`ASCEND_PROCESS_LOG_PATH`.)

Expected Behavior

The third filler needs an earlier, already-consumed filler's bytes reclaimed;
those bytes are reclaimable, so the test should pass with every Y[0] == 42.

Actual Behavior

Upstream deadlocks; the device log shows:

FATAL: Task Allocator Deadlock - Heap Exhausted! ring=1
No reclaim progress for ~500 ms (25000000 cycles wall clock).
  Task ring 1: current=6, last_alive=0, active=6/16384 (0.0%)
  Heap ring 1: top=2097152, tail=0, size=2621440, used=2097152, available=524288
  Requested:  1048576 bytes
  Head task 0: state=1, consumers=0/1, scope_released=1

Host-side orch_error_code=2 HEAP_RING_DEADLOCK. last_alive=0 and tail=0 — the
reclaim front is pinned at task 0 (the zero-byte head); Head task 0: state=1 (COMPLETED), consumers=0/1, scope_released=1; used=2 MiB, available=0.5 MiB, Requested 1 MiB — two fillers' bytes are consumed but not
reclaimable; active=6/16384 (0.0%) — task slots are far from full, so this is
heap-bound (a reclaim defect), not a capacity limit.

Git Commit ID

6284040

CANN Version

9.0.0

Driver Version

26.0.rc1

Host Platform

Linux (aarch64)

Additional Context

Fix corroboration (same ST file, same PTO2_RING_HEAP, only the runtime
differs): the fixed branch passes with Y[0] == 42 (1 passed). Since the
fix runs all fillers through the 2.5 MiB ring, the ring is large enough —
upstream's failure is the reclaim front pinned by the zero-byte head, not
capacity.

Root cause: reclaim (update_heap_tail) historically advanced the heap tail
only up to last_task_alive; once that watermark is pinned by a not-fully-
consumed head, later consumed bytes cannot retire. A zero-extent head owns no
heap and should never participate in the heap-reclaim blocking decision.

Suggested fix: give the heap an independent reclaim cursor
(heap_reclaim_task_id_) decoupled from the task watermark: a zero-extent task
is never a stopping condition (skipped); positive extents stay strictly FIFO,
stopping at the first one that is neither covered by the watermark nor
CONSUMED; a consumed positive extent may retire ahead of the watermark
(consumed_ahead_of_watermark). See the fixed pto_ring_buffer.h::update_heap_tail.

lib-free regression test — test_task_allocator.cpp::ReplaysTask2639ZeroExtentHeadAndConsumedTask2640

TEST_F(TaskAllocatorTest, ReplaysTask2639ZeroExtentHeadAndConsumedTask2640) {
    constexpr int32_t INITIAL_TASK_ID = 2639;
    constexpr uint8_t RING_ID = 3;
    current_index.store(INITIAL_TASK_ID);
    last_alive.store(INITIAL_TASK_ID);
    allocator.init(
        descriptors.data(), WINDOW_SIZE, &current_index, &last_alive, heap_buf, HEAP_SIZE, &error_code,
        slot_states.data(), INITIAL_TASK_ID, RING_ID
    );

    auto zero_head = allocator.alloc(0);
    ASSERT_FALSE(zero_head.failed());
    ASSERT_EQ(zero_head.task_id, 2639);
    publish_allocation(descriptors, zero_head, RING_ID);
    slot_states[zero_head.slot].fanout_count = PTO2_FANOUT_SCOPE_BIT | 64;
    slot_states[zero_head.slot].fanout_refcount.store(PTO2_FANOUT_SCOPE_BIT | 51, std::memory_order_relaxed);
    slot_states[zero_head.slot].task_state.store(PTO2_TASK_COMPLETED, std::memory_order_release);

    auto consumed_bytes = allocator.alloc(HEAP_SIZE);
    ASSERT_FALSE(consumed_bytes.failed());
    ASSERT_EQ(consumed_bytes.task_id, 2640);
    publish_allocation(descriptors, consumed_bytes, RING_ID);
    slot_states[consumed_bytes.slot].task_state.store(PTO2_TASK_CONSUMED, std::memory_order_release);

    auto next = allocator.alloc(64);

    ASSERT_FALSE(next.failed());
    EXPECT_EQ(next.packed_base, static_cast<void *>(heap_buf));
    EXPECT_EQ(last_alive.load(std::memory_order_acquire), INITIAL_TASK_ID);
    EXPECT_EQ(allocator.heap_reclaim_task_id(), 2641);
    EXPECT_EQ(slot_states[zero_head.slot].task_state.load(std::memory_order_acquire), PTO2_TASK_COMPLETED);
}

Related lib-free tests: StopsAtFirstLivePositiveExtent,
ConcurrentBackpressureReclaimsWithoutLastAliveAdvance,
DecoupledReclaimAdvancesTailAcrossHeapWrap.

Real-workload corroboration (same defect, natural form): qwen3-14B prefill at
PTO2_RING_HEAP=4G x4 (16 GiB heap — capacity is clearly not the limit) deadlocks
on ring 3 under upstream — Head task 5807: state=1, consumers=52/64, scope_released=1, No reclaim progress ~500ms — while the fix PASS (142s).

Reproduced-at-HEAD: dynamically reproduced on main HEAD 62840403 (real
a2a3, orch_error_code=2 HEAP_RING_DEADLOCK; device dump identical — ring=1, last_alive=0, Head task 0: state=1, consumers=0/1, scope_released=1) as well as on
fork point 8cdb306c. Consistently, pto_ring_buffer.h (containing
update_heap_tail) is byte-identical across 8cdb306c..62840403, and the
three lines this repro's setup depends on — the zero-output auto-retire
(count_registrable_outputs == 0), the already-retired-producer dep skip
(dep_local_task_id < dep_last_task_alive), and the scope-gated fanout reference —
are unchanged; the 36 intervening commits touch only pto_orchestrator.cpp
(diagnostics #1448, marker consolidation #1410, dispatch marker #1411,
early-dispatch #1405).

Reproduction range & fix: reproduced across 8cdb306c..HEAD 62840403, over
which pto_ring_buffer.h is byte-identical — so this is not a regression from
those commits. The introducing commit was not bisected; update_heap_tail
advancing the heap tail off last_task_alive is a long-standing design in the
allocator, not a recent change. Fixed on our fork by c396b6e7 ("Fix: decouple
FIFO heap reclaim from task watermark", which introduces the independent
heap_reclaim_task_id_ cursor) [+ 611c79e9].

---

Reproduction — full ST sources

Drop these into tests/st/a2a3/tensormap_and_ringbuffer/heap_reclaim_watermark/, then
run the command in Steps to Reproduce. No src/ change is required.

test_heap_reclaim_watermark.py

#!/usr/bin/env python3
# Copyright (c) PyPTO Contributors.
# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
# CANN Open Software License Agreement Version 2.0 (the "License").
# Please refer to the License for details. You may not use this file except in compliance with the License.
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
# See LICENSE in the root of the software repository for the full text of the License.
# -----------------------------------------------------------------------------------------------------------
"""heap_reclaim_watermark: a zero-byte un-consumed head must not freeze reclaim
of later consumed heap bytes.

A holder task spins for the whole filler phase. A zero-extent dummy head H and
its consumer HC share one scope, so H is scope-gated (alive) when HC registers
its dependency on H; HC is also gated on the holder, so H stays COMPLETED but
un-consumed for the whole filler phase, pinning the ring task watermark at a
zero-byte task (the shape of a scope-gated barrier head with lagging consumers).
Three filler producer/consumer pairs (each 1 MiB scratch, each in its own scope)
then run on the same ring; with the ring sized to hold ~2 fillers, the third
filler needs an earlier filler's consumed bytes reclaimed -- but a runtime that
couples heap reclaim to the task watermark cannot free bytes past the pinned
zero-byte head, so it reports false heap exhaustion.

Run ring 1's heap down to ~2.5x the scratch via env so it holds ~2 fillers:

    PTO2_RING_HEAP=268435456,2621440,268435456,268435456 \
      python -m pytest .../test_heap_reclaim_watermark.py \
      -s -v -p no:xdist --platform a2a3 --device <N>

Expected on a2a3: fixed simpler passes and every Y[0]==42; upstream 8cdb306c
fails (heap deadlock) because the zero-byte head freezes reclaim of the
consumed filler bytes.
"""

import torch
from simpler.task_interface import ArgDirection as D

from simpler_setup import SceneTestCase, TaskArgsBuilder, Tensor, scene_test

SENTINEL = 42.0
INIT_VAL = -1.0
NUM_FILLERS = 3


@scene_test(level=2, runtime="tensormap_and_ringbuffer")
class TestHeapReclaimWatermark(SceneTestCase):
    """A zero-byte un-consumed reclaim head must not block later consumed bytes."""

    RTOL = 0
    ATOL = 0

    CALLABLE = {
        "orchestration": {
            "source": "kernels/orchestration/heap_reclaim_watermark_orch.cpp",
            "function_name": "aicpu_orchestration_entry",
            "signature": [D.INOUT, D.INOUT, D.INOUT, D.INOUT],
        },
        "incores": [
            {
                "func_id": 0,
                "name": "FILL_CONST",
                "source": "kernels/aic/kernel_write_const.cpp",
                "core_type": "aic",
                "signature": [D.OUT],
            },
            {
                "func_id": 1,
                "name": "COPY_FIRST",
                "source": "kernels/aic/kernel_copy_first.cpp",
                "core_type": "aic",
                "signature": [D.IN, D.INOUT],
            },
            {
                "func_id": 2,
                "name": "SPIN_HOLDER",
                "source": "kernels/aic/kernel_spin_holder.cpp",
                "core_type": "aic",
                "signature": [D.INOUT],
            },
        ],
    }

    CASES = [
        {
            "name": "ZeroByteHeadPinsReclaim",
            "platforms": ["a2a3"],
            "config": {"aicpu_thread_num": 4, "block_dim": 1},
            "params": {},
        },
    ]

    def generate_args(self, params):
        tensors = [Tensor(f"y{i}", torch.full((16,), INIT_VAL, dtype=torch.float32)) for i in range(NUM_FILLERS)]
        tensors.append(Tensor("yholder", torch.full((16,), INIT_VAL, dtype=torch.float32)))
        return TaskArgsBuilder(*tensors)

    def compute_golden(self, args, params):
        # Each filler's FILL_CONST writes SENTINEL to its scratch[0]; the
        # matching COPY_FIRST copies it to Y[i][0]. The holder writes SENTINEL to
        # Yholder[0]. All fillers and the holder must complete.
        for i in range(NUM_FILLERS):
            getattr(args, f"y{i}")[0] = SENTINEL
        args.yholder[0] = SENTINEL


if __name__ == "__main__":
    SceneTestCase.run_module(__name__)

kernels/orchestration/heap_reclaim_watermark_orch.cpp

/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * heap_reclaim_watermark: minimal repro of heap reclaim coupled to the task
 * watermark.
 *
 * A holder task spins on core 0 long enough for the whole filler phase. A
 * zero-extent dummy head H and its consumer HC are submitted in one scope, so H
 * is scope-gated (alive) at the moment HC registers its dependency on it -- HC
 * therefore holds H's fanout. HC is also gated on the holder, so it cannot run
 * until the spin finishes; H thus stays COMPLETED-but-un-consumed for the whole
 * filler phase, pinning the ring task watermark last_task_alive at a zero-byte
 * task (the shape of a scope-gated barrier head with lagging consumers).
 *
 * H is the ring's lowest task id. After it, NUM_FILLERS producer/consumer pairs
 * each run in their own scope on the SAME ring: the producer allocates an S-byte
 * scratch, the consumer copies scratch[0] into an external Y and fully consumes
 * it. With the ring sized to hold ~2 fillers, the later fillers need an earlier
 * filler's bytes reclaimed. Those bytes are CONSUMED and physically reclaimable,
 * but they are ahead of the watermark H pins. A runtime that couples heap
 * reclaim to last_task_alive cannot free them -> false heap exhaustion. A
 * runtime whose heap-reclaim cursor is decoupled skips the zero-byte head and
 * frees the consumed filler bytes.
 *
 * Args layout: [Y0, Y1, Y2, Y3, Yholder]  (external outputs the host verifies)
 */

#include <cstdint>

#include "pto_orchestration_api.h"  // NOLINT(build/include_subdir)

#define FUNC_FILL_CONST 0
#define FUNC_COPY_FIRST 1
#define FUNC_SPIN_HOLDER 2

// Ring holds ~2 fillers; the 3rd needs an earlier consumed filler reclaimed.
static constexpr int32_t NUM_FILLERS = 3;
// Must outlast the allocator's ~500 ms no-reclaim deadlock-detection window so a
// broken runtime latches the deadlock before HC releases H (~1.3 s on a3).
static constexpr uint64_t HOLDER_SPIN_ITERS = 1000000000;

extern "C" {

__attribute__((visibility("default"))) PTO2OrchestrationConfig aicpu_orchestration_config(const L2TaskArgs &orch_args) {
    (void)orch_args;
    return PTO2OrchestrationConfig{
        .expected_arg_count = 4,
    };
}

__attribute__((visibility("default"))) void aicpu_orchestration_entry(const L2TaskArgs &orch_args) {
    const Tensor *ys[NUM_FILLERS] = {
        &orch_args.tensor(0).ref(),
        &orch_args.tensor(1).ref(),
        &orch_args.tensor(2).ref(),
    };
    const Tensor &ext_Yholder = orch_args.tensor(3).ref();

    // S-byte filler scratch = [1024, 256] float32 = 1 MiB. The ring is sized
    // ~2.5 MiB via PTO2_RING_HEAP so it holds ~2 fillers at a time.
    uint32_t scratch_shapes[2] = {1024, 256};
    TensorCreateInfo scratch_ci(scratch_shapes, 2, DataType::FLOAT32);

    // Holder spins for the whole filler phase; HC is gated on it.
    L0TaskArgs holder_args;
    holder_args.add_inout(ext_Yholder);
    holder_args.add_scalar(HOLDER_SPIN_ITERS);
    PTO2TaskId holder_id = rt_submit_aic_task(FUNC_SPIN_HOLDER, holder_args).task_id();

    // H and HC share a scope so H is scope-gated (alive) when HC registers its
    // dependency on it; HC then holds H's fanout past scope close.
    PTO2TaskId h_id;
    PTO2_SCOPE() {
        L0TaskArgs h_args;
        h_id = rt_submit_dummy_task(h_args).task_id();

        L0TaskArgs hc_args;
        PTO2TaskId hc_deps[] = {h_id, holder_id};
        hc_args.set_dependencies(hc_deps, 2);
        rt_submit_dummy_task(hc_args);
    }

    for (int32_t i = 0; i < NUM_FILLERS; i++) {
        PTO2_SCOPE() {
            L0TaskArgs fill;
            fill.add_output(scratch_ci);
            TaskOutputTensors fill_outs = rt_submit_aic_task(FUNC_FILL_CONST, fill);
            const Tensor &t = fill_outs.get_ref(0);

            L0TaskArgs copy;
            copy.add_input(t);
            copy.add_inout(*ys[i]);
            rt_submit_aic_task(FUNC_COPY_FIRST, copy);
        }
    }
}

}  // extern "C"

kernels/aic/kernel_spin_holder.cpp

/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * Spins for args[1] iterations, then writes 42.0f to args[0][0]. Holds its
 * AICore long enough that a dependent (gated on this task) stays un-run while
 * other tasks make progress.
 *
 * Args: args[0] = output Tensor* (INOUT), args[1] = spin_iters.
 */

#include <cstdint>
#include <pto/pto-inst.hpp>

#include "tensor.h"

#ifndef __gm__
#define __gm__
#endif

#ifndef __aicore__
#define __aicore__ [aicore]  // NOLINT(whitespace/braces)
#endif

#include "intrinsic.h"

#ifdef PTO_CPUSTUB_HPP
#define dcci(...) \
    do {          \
    } while (0)
#endif
#ifndef SINGLE_CACHE_LINE
#define SINGLE_CACHE_LINE 0
#endif
#ifndef CACHELINE_OUT
#define CACHELINE_OUT 0
#endif

extern "C" __aicore__ void kernel_entry(__gm__ int64_t *args) {
    __gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[0]);
    __gm__ float *out = reinterpret_cast<__gm__ float *>(out_tensor->buffer.addr) + out_tensor->start_offset;

    int32_t spin_iters = static_cast<int32_t>(args[1]);
    volatile int32_t acc = 0;
    for (int32_t i = 0; i < spin_iters; i++) {
        acc++;  // volatile keeps the loop; ++ avoids int32 overflow UB for large counts
    }
    (void)acc;

    out[0] = 42.0f;
    dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT);
}

kernels/aic/kernel_write_const.cpp

/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * Writes a fixed sentinel (42.0f) to args[0][0]. Used as the producer in the
 * dummy_task scene tests so a downstream consumer can verify the dependency
 * chain (producer -> ... -> dummy -> consumer) propagates the value intact.
 */

#include <cstdint>
#include <pto/pto-inst.hpp>

#include "tensor.h"

#ifndef __gm__
#define __gm__
#endif

#ifndef __aicore__
#define __aicore__ [aicore]  // NOLINT(whitespace/braces)
#endif

#include "intrinsic.h"

#ifdef PTO_CPUSTUB_HPP
#define dcci(...) \
    do {          \
    } while (0)
#endif
#ifndef SINGLE_CACHE_LINE
#define SINGLE_CACHE_LINE 0
#endif
#ifndef CACHELINE_OUT
#define CACHELINE_OUT 0
#endif

extern "C" __aicore__ void kernel_entry(__gm__ int64_t *args) {
    __gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[0]);
    __gm__ float *out = reinterpret_cast<__gm__ float *>(out_tensor->buffer.addr) + out_tensor->start_offset;

    out[0] = 42.0f;
    dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT);
}

kernels/aic/kernel_copy_first.cpp

/*
 * Copyright (c) PyPTO Contributors.
 * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
 * CANN Open Software License Agreement Version 2.0 (the "License").
 * Please refer to the License for details. You may not use this file except in compliance with the License.
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
 * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
 * See LICENSE in the root of the software repository for the full text of the License.
 * -----------------------------------------------------------------------------------------------------------
 */

/**
 * Consumer kernel for the dummy_task scene tests: copies args[0][0] -> args[1][0].
 *
 * If the dummy_task in the middle of the chain has correctly waited on the
 * producer and notified this consumer, args[1][0] will equal the producer's
 * sentinel (42.0f). If the dependency chain broke or the dummy somehow ran a
 * kernel that clobbered args[0], the value will differ.
 */

#include <cstdint>
#include <pto/pto-inst.hpp>

#include "tensor.h"

#ifndef __gm__
#define __gm__
#endif

#ifndef __aicore__
#define __aicore__ [aicore]  // NOLINT(whitespace/braces)
#endif

#include "intrinsic.h"

#ifdef PTO_CPUSTUB_HPP
#define dcci(...) \
    do {          \
    } while (0)
#endif
#ifndef SINGLE_CACHE_LINE
#define SINGLE_CACHE_LINE 0
#endif
#ifndef CACHELINE_OUT
#define CACHELINE_OUT 0
#endif

extern "C" __aicore__ void kernel_entry(__gm__ int64_t *args) {
    __gm__ Tensor *in_tensor = reinterpret_cast<__gm__ Tensor *>(args[0]);
    __gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[1]);

    __gm__ float *in = reinterpret_cast<__gm__ float *>(in_tensor->buffer.addr) + in_tensor->start_offset;
    __gm__ float *out = reinterpret_cast<__gm__ float *>(out_tensor->buffer.addr) + out_tensor->start_offset;

    out[0] = in[0];
    dcci(&out[0], SINGLE_CACHE_LINE, CACHELINE_OUT);
}

中文总结

堆回收前沿被绑在 task 水位 last_task_alive;当队头是一个零 extent、scope-gated、COMPLETED
但有滞后 consumer 的 task,它钉住水位,后面已消费的正 extent 字节就回收不了,明明有空间却报
HEAP_RING_DEADLOCK。最小 ST 用一个自旋 holder 门控 HC、H 与 HC 同 scope(让 H 在 HC 登记依赖时
还 scope-gated 存活,否则会 auto-retire 钉不住),再用 3 个 1 MiB filler 填 2.5 MiB 环:upstream
main HEAD 62840403(及 fork point 8cdb306c)死锁(last_alive=0, Head task 0 zero-extent, consumers=0/1),修复分支 c396b6e7 通过。修复=给 heap 独立的、与水位解耦的回收 cursor,跳过零字节队头;lib-free 回归测试见上方折叠块,
真实 workload 佐证见 qwen prefill PTO2_RING_HEAP=4GB。

[ChaoZheng109]

Not a bug — expected backstop behavior on a slow-but-progressing op, not a structural deadlock

Thanks for the detailed repro. After tracing the allocator and submit path, this is not a reclaim defect or a real deadlock — it is the 500 ms heap-spin backstop firing on a deliberately-constructed stall (1.3 s holder) that is longer than the backstop. Forward progress is possible here; the watchdog just declares it dead first.

Why it always drains given enough time

The heap allocator's deadlock backstop is 500 ms (No reclaim progress for ~500 ms in the log). The repro parks a holder that spins for 1.3 s. So the backstop fires at 500 ms while the holder is still spinning. Raise the backstop past 1.3 s and the whole chain completes:

holder finishes → HC runs → HC consumes H → H retires → last_task_alive advances → heap_tail advances → the third filler allocates.

Per our own triage rule (running-onboard.md): a backstop firing on work that is merely slow/stalled is not a proven structural deadlock. The FATAL: Task Allocator Deadlock label here is the wall-clock backstop, not a capacity/head-of-line proof.

Why the "consumed bytes are reclaimable" framing doesn't hold

Reclaim is in-order by design — the task/heap/fanin pools share a single last_task_alive watermark. H does pin the watermark, but only because it is genuinely still live: consumers=0/1, i.e. HC has not consumed it yet. Retiring a task that still has a pending consumer is not permitted; that is in-order semantics, not a broken contract. And the pin only exists because H and HC share a scope — put them in separate scopes and H auto-retires (no consumer registered at submit time), so the situation never arises.

The circular-deadlock variant is unreachable (the important part)

The one thing that would be a real bug is a circular wait: HC can't run ← heap reclaim stuck ← H can't retire ← HC hasn't consumed H. It is structurally impossible, and the ordering in submit_task_common is why:

1. prepare_task runs first (pto_orchestrator.cpp:806) and blocks in allocator.alloc() (:521) until HC's own slot + heap are allocated.
2. Only after that does HC register its dependency on H — append_fanin_or_fail → prod_state->fanout_count++ (:313, called from :890/:908) — which is the action that pins H.

So the only way H can pin the reclaim watermark is via a consumer that has already cleared the blocking alloc. Combined with the invariant that heap is allocated exclusively at orchestrator submit time (never on the worker at exec time — see pto_ring_buffer.h:70), a pinning consumer needs zero further heap allocation and zero further orchestrator progress to run. Its only remaining blockers are compute/core dependencies (here, the holder). The heap-blocked orchestrator can therefore only stall tasks submitted after HC — never HC itself, the one task that unpins H. The first link of the cycle never closes.

The same argument extends to HC's whole dependency closure: you can only depend on an already-submitted producer, so every producer of HC is already submitted-and-allocated too; their execution needs no further alloc.

Conclusion

• The filed repro is a timeout race (500 ms backstop < 1.3 s stall), i.e. expected behavior.
• The genuinely-dangerous circular variant is unreachable by construction (alloc-before-pin ordering + submit-time-only heap allocation).
• Recommended handling for real workloads: isolate independent work in separate scopes (so a zero-byte head auto-retires), or size the backstop above the workload's real stall.

Closing as not-a-bug.

[ChaoZheng109]

中文说明(方便阅览):非 bug,是预期内的 backstop 行为,不是结构性死锁

感谢详细的复现。跟踪了分配器和提交路径后,结论是:这不是回收缺陷,也不是真死锁,而是 500ms 堆自旋 backstop 撞上了一个被人为构造得比它更长的 stall(1.3s 的 holder)。前进本来是能发生的,只是看门狗先把它判死了。

为什么给够时间总能排空

堆分配器的死锁 backstop 是 500ms(日志里的 No reclaim progress for ~500 ms)。复现里 holder 自旋 1.3s,于是 backstop 在 500ms 先炸,而此时 holder 还在转。把 backstop 调过 1.3s,整条链会跑通:

holder 结束 → HC 能跑 → HC 消费 H → H 退休 → last_task_alive 前进 → heap_tail 前进 → 第三个 filler 拿到堆

按仓库自己的 triage 规则(running-onboard.md):backstop 触发在"仅仅是慢/被 stall 的 op"上,不构成已被证明的结构性死锁。这里的 FATAL: Task Allocator Deadlock 是墙钟 backstop,不是容量/head-of-line 的证明。

为什么"consumed 字节可回收"的说法不成立

回收本就是按序设计的——task/heap/fanin 三个池共享同一个 last_task_alive 水位线。H 确实钉住了水位线,但那是因为它真的还活着:consumers=0/1,即 HC 还没消费它。退休一个仍有未完成消费者的任务本就是不允许的,这是 in-order 语义,不是契约被破坏。而且这个 pin 只有在 H、HC 同 scope 时才成立——放到不同 scope,H 提交时无消费者会自动退休,这个局面根本不会出现。

环形死锁变体不可达(最关键的部分)

唯一真会是 bug 的,是这样一个环:HC 跑不起来 ← 堆回收卡住 ← H 退不了休 ← HC 没消费 H。它在结构上不可能成立,原因就在 submit_task_common 的顺序:

1. prepare_task 先执行(pto_orchestrator.cpp:806),在 allocator.alloc()(:521)里自旋直到 HC 自己的 slot + 堆分配到手。
2. 之后才登记 HC 对 H 的依赖 —— append_fanin_or_fail → prod_state->fanout_count++(:313,由 :890/:908 调用),这才是"钉住 H"的动作。

也就是说,H 能钉住水位线的唯一途径,是一个已经越过阻塞分配点的消费者。再加上一条不变量:堆只在 orchestrator 提交期分配,执行期(worker 上)不再申请任何堆(见 pto_ring_buffer.h:70)——所以钉住 H 的那个消费者要跑起来,不需要再申请堆、也不需要 orchestrator 再前进一步,只剩计算/核依赖(本例就是 holder)。因此堆阻塞的 orchestrator 只能卡住 HC 之后提交的任务,永远卡不到 HC 本身,而 HC 正是解开 H 的那个任务。环的第一环永远闭合不了。

同样的推理适用于 HC 的整个依赖闭包:只能依赖已提交的生产者,所以 HC 的每个生产者也都已提交并分配完成,它们的执行不需要再分配。

结论

• 现有复现是超时竞争(500ms backstop < 1.3s stall),属预期行为。
• 真正危险的环形变体因"先 alloc 后 pin"的顺序 + "堆仅提交期分配"的不变量而结构性不可达。
• 实际负载的建议处理:用不同 scope 隔离独立工作(让零字节头任务自动退休),或把 backstop 调到长于负载的真实 stall。

按 non-bug 关闭。