Skip to content

fix(mesh): stop application delivery from stalling the read loop - #22

Merged
ForeverInLaw merged 1 commit into
mainfrom
fix/delivery-never-stalls-the-reader
Jul 28, 2026
Merged

fix(mesh): stop application delivery from stalling the read loop#22
ForeverInLaw merged 1 commit into
mainfrom
fix/delivery-never-stalls-the-reader

Conversation

@ForeverInLaw

Copy link
Copy Markdown
Contributor

Delivery to the application is a synchronous callback that decrypts and writes to disk. It was fed from a single shared queue drained by one goroutine, and deliverLocal's send into it blocked. A file transfer's chunks filled the queue, the send blocked, readPeer stopped reading, and the transport's 256-packet inbound buffer overflowed — discarding whatever arrived next, with no record anywhere.

Both symptoms were measured on a live pair today:

  • transfers stuck forever at 63% / 32% / 0% — a lost chunk is never re-sent;
  • sessions dying at ~37s on a healthy link — the packets thrown away behind the stall include the pings a peer counts six of before giving up.
stream_drops = 36, all on the default stream

The comment already sitting on that counter described this exact failure; nothing had ever acted on it.

What changes

One queue and one worker per channel:

  • a slow transfer on one channel can no longer delay control traffic on another — that is the only reason ordering needs to be per-channel rather than global;
  • the enqueue never blocks. A dropped message on one channel is a bounded, counted loss (__local_delivery_dropped__); a stalled reader is an unbounded, invisible one that takes the whole session with it.

Note the dispatchSem this replaces for the delivery path was inert: a single goroutine acquired and released it inside one iteration, so it never permitted any concurrency at all.

Tests

Two properties, both verified to fail against a blocking enqueue (checked by temporarily restoring the blocking send):

  • enqueueing returns even when the application is wedged and the queue is past its depth;
  • a stalled blob channel does not hold up a control message behind it.

mesh, gossip and transport suites pass; build and vet clean.

Delivery to the application is a synchronous callback that decrypts and writes
to disk. It was fed from a single shared queue drained by one goroutine, and
deliverLocal's send into it blocked. So a file transfer's chunks filled the
queue, the send blocked, readPeer stopped reading, and the transport's
256-packet inbound buffer overflowed — discarding whatever arrived next.

Both of the symptoms that produces were measured on a live pair today:

  - transfers that stick forever at 63% / 32% / 0%, because a lost chunk is
    never re-sent; and
  - sessions dying at ~37s on a healthy link, because the packets thrown away
    behind the stall include the pings a peer counts six of before giving up.

    stream_drops = 36, all on the default stream

Delivery now runs one queue and one worker per channel:

  - a slow transfer on one channel can no longer delay control traffic on
    another, which is the only reason ordering needs to be per-channel rather
    than global;
  - the enqueue never blocks. A dropped message on one channel is a bounded,
    counted loss (__local_delivery_dropped__); a stalled reader is an unbounded
    and invisible one, and it takes the session with it.

The dispatchSem it replaces for this path was inert: one goroutine acquired and
released it within a single iteration, so it never permitted any concurrency.

Tests pin both properties, and both fail against a blocking enqueue.
@ForeverInLaw
ForeverInLaw merged commit 1a12bcb into main Jul 28, 2026
2 checks passed
@ForeverInLaw
ForeverInLaw deleted the fix/delivery-never-stalls-the-reader branch July 28, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant