fix(mesh): stop application delivery from stalling the read loop - #22
Merged
Conversation
Delivery to the application is a synchronous callback that decrypts and writes
to disk. It was fed from a single shared queue drained by one goroutine, and
deliverLocal's send into it blocked. So a file transfer's chunks filled the
queue, the send blocked, readPeer stopped reading, and the transport's
256-packet inbound buffer overflowed — discarding whatever arrived next.
Both of the symptoms that produces were measured on a live pair today:
- transfers that stick forever at 63% / 32% / 0%, because a lost chunk is
never re-sent; and
- sessions dying at ~37s on a healthy link, because the packets thrown away
behind the stall include the pings a peer counts six of before giving up.
stream_drops = 36, all on the default stream
Delivery now runs one queue and one worker per channel:
- a slow transfer on one channel can no longer delay control traffic on
another, which is the only reason ordering needs to be per-channel rather
than global;
- the enqueue never blocks. A dropped message on one channel is a bounded,
counted loss (__local_delivery_dropped__); a stalled reader is an unbounded
and invisible one, and it takes the session with it.
The dispatchSem it replaces for this path was inert: one goroutine acquired and
released it within a single iteration, so it never permitted any concurrency.
Tests pin both properties, and both fail against a blocking enqueue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivery to the application is a synchronous callback that decrypts and writes to disk. It was fed from a single shared queue drained by one goroutine, and
deliverLocal's send into it blocked. A file transfer's chunks filled the queue, the send blocked,readPeerstopped reading, and the transport's 256-packet inbound buffer overflowed — discarding whatever arrived next, with no record anywhere.Both symptoms were measured on a live pair today:
The comment already sitting on that counter described this exact failure; nothing had ever acted on it.
What changes
One queue and one worker per channel:
__local_delivery_dropped__); a stalled reader is an unbounded, invisible one that takes the whole session with it.Note the
dispatchSemthis replaces for the delivery path was inert: a single goroutine acquired and released it inside one iteration, so it never permitted any concurrency at all.Tests
Two properties, both verified to fail against a blocking enqueue (checked by temporarily restoring the blocking send):
mesh, gossip and transport suites pass; build and vet clean.