Large-value read hangs forever when the registered-memory segment pool is momentarily exhausted

[thweetkomputer]

Summary

A very-large-value read (or Floor resolving to a large value) can hang forever if the shared GlobalRegisteredMemory segment pool is transiently empty at the moment the read needs a segment. The request never completes (SetDone never fires), the coroutine stack / RootMeta handle / mapping snapshots stay pinned, NumActive() stays > 0 so the shard busy-polls and Stop() cannot join.

Mechanism

ReadTask::Read (src/tasks/read_task.cpp:108) and ReadTask::Floor (src/tasks/read_task.cpp:240) pass a bare yield callback into GetLargeValue:

auto yield_fn = [this]() { Yield(); };

which reaches GlobalRegisteredMemory::GetSegment (include/global_registered_memory.h:118):

while (true) {
    auto result = TryGetSegment();
    if (result.first) return result;
    if (evict_func_) evict_func_();   // evict_func_ is never installed
    yield();                          // <- bare KvTask::Yield()
}

KvTask::Yield() is a raw boost::context switch back to the shard loop that does not re-enqueue the task on any ready queue and leaves status_ == Ongoing. The shard only resumes tasks it pops from a ready queue; FinishIo cannot fire (no inflight IO at GetSegment); Resume() no-ops for Ongoing tasks. So once the task yields here it is orphaned — it is never woken again, even after another task Recycle()s a segment.

The background segment-allocation path issues the identical GetSegment with YieldToLowPQ() instead (src/tasks/background_write.cpp:396), confirming the read path simply lost the re-schedule.

Reachability

The zero-copy read arm that hits this (IoStringBuffer destination / Floor always passes &large_value_) is only reachable with a caller-provided external GlobalRegisteredMemory pool shared with the networking layer (the legacy zero-copy deployment). Pinned mode rejects that arm. So it needs the external-pool deployment plus a transient pool drain under concurrent large-value reads.

Proposed fix

The correct fix is a wake-on-release mechanism, not a busy-poll:

• When GetSegment finds the pool empty, the caller should register as a waiter and suspend, instead of spinning yield()/YieldToLowPQ().
• Recycle() (which returns a segment to the pool) should wake one registered waiter, re-entering it through the owning shard's ready queue.

Note the shard-thread-affinity invariant: Recycle() can be called from a foreign thread (the networking layer holds the same pool), so the wakeup must hand the task back to its shard's ready queue rather than resuming it on the releasing thread — the same discipline used for cloud/standby completions.

A minimal stop-gap (swap the bare Yield() for YieldToLowPQ() to match background_write) would remove the lost-wakeup hang, but it busy-polls the low-priority queue until a segment frees. The wake-on-release design avoids that spin and is the intended fix.