Skip to content

[Test] No test asserts that a large message does not delay unrelated traffic on the same connection #1385

Description

@pathosDev

Use case

The cluster transport is one TCP connection per peer with a length-prefixed frame decoder. Frames are read and dispatched in arrival order, and there is no separate lane for large payloads — grep for chunking, multiplexing or lane separation across src/cluster/Transport.ts and src/cluster/Protocol.ts finds only TCP read chunks, which is a different thing.

So a single large message occupies the connection for as long as it takes to transfer, and everything behind it waits. That includes gossip and heartbeats.

The consequence is not a slow message; it is a false failure detection. Heartbeats delayed behind a multi-megabyte payload look exactly like a peer that has stopped responding, and the phi-accrual detector will eventually mark the peer unreachable. A downing strategy then acts on that. A large but entirely legitimate message can therefore take a healthy node out of the cluster.

Nothing tests this. There is a frame size cap, which bounds the worst case, and tests/unit/Protocol.test.ts asserts that decode cost stays linear in the bytes received — both good, and neither is the question. The question is whether unrelated traffic on the same connection is delayed, and it has never been asked.

Proposed shape

  • A test that sends a message near the frame cap between two nodes and asserts that heartbeat or gossip traffic in the same window is not delayed beyond the failure-detector threshold.
  • Failing that assertion is a design finding, not a test bug. If head-of-line blocking is real at the current cap, the options are a lower cap, a separate lane for control traffic, or chunking — all larger changes, and all better decided with a measurement than without one.
  • The measurement should come first regardless: what payload size, at what link speed, actually delays a heartbeat past the threshold with the default settings? That number determines whether this is urgent or theoretical.

This also connects to #1023: injecting latency would let the test construct the condition deterministically rather than relying on a payload large enough to be slow in real time.

Acceptance

  • A test measures whether control traffic is delayed behind a large payload on the same connection.
  • The payload size at which a heartbeat is delayed past the unreachability threshold is recorded on this issue.
  • If blocking is confirmed, a follow-up issue captures the design decision.

Verification status

Confirmed by reading. src/cluster/Transport.ts was read around its read path: onData pushes chunks into a per-connection decoder and dispatches complete frames in order. grep -niE "chunk|multiplex|headOfLine|priorityLane" over src/cluster/Transport.ts and src/cluster/Protocol.ts returns only TCP-level read-chunk references. No test in tests/multi-node/ or tests/unit/ sends a large payload and measures unrelated traffic. Related: #1023.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions