test: make client.tcl CLIENT KILL deterministic against connection-registration race - #532
Conversation
client.tcl intermittently failed in the 3-node cluster log-replay phase with "ERR No such client" on `r client kill $addr`. The test opens a second connection, reads its addr via CLIENT INFO, then kills it from the `r` connection. CLIENT KILL / LIST / INFO and SELECT are all DirectCommands that run on the connected node with no auto_redirect, so both connections live on node 0 and CLIENT KILL scans node 0's own connection map. The race is that a freshly accepted brpc connection can already answer its own commands (via the io_uring recv ring) before Acceptor::OnNewConnectionsUntilEAGAIN inserts its socket into _socket_map -- the map CLIENT LIST / CLIENT KILL enumerate (brpc documents this "funny race condition"). When CLIENT KILL wins that race it matches zero sockets and returns "No such client". The window is sub-millisecond normally but widens under the CPU starvation of the log-replay phase (three nodes on core_number=2), which is why it flaked there while passing the other 8 cluster runs in the same job. EloqKV's CLIENT KILL only supports the old-style addr:port form, so killing by id is not an option without a server change. Instead, wait until the target addr is actually visible in CLIENT LIST -- the same map CLIENT KILL scans -- before killing it, so once the addr appears the kill is deterministic. This keeps cluster coverage rather than restricting the cases to single-node. Verified on a scratch single-node server: all four client.tcl tests pass, including the two later CLIENT KILL-by-addr sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughClient kill tests now wait for target addresses to appear in ChangesClient kill test synchronization
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
tests/unit/eloq/client.tclintermittently fails in CI's 3-node cluster log-replay phase with[exception]: Executing test client: ERR No such client.onr client kill $addr(observed on PR #527, run 29137305901, jobRelWithDebInfo-ELOQDSS_ELOQSTORE-arm64). In that same job the file ran 9 times in cluster mode — 8 passed, only the log-replay run failed — so this is a race, not a cluster incompatibility, and it is independent of that PR's changes.Root cause
The test opens a second connection, reads its
addrviaCLIENT INFO, then kills it from therconnection.CLIENT KILL/CLIENT LIST/CLIENT INFOandSELECTare allDirectCommands — they execute on the connected node with noauto_redirect/ key routing. So both connections live on node 0, andCLIENT KILLscans node 0's own connection map. "Target landed on another node" is ruled out;$addris also stable (nothing touches the killed connection betweenCLIENT INFOand the kill).Acceptor::OnNewConnectionsUntilEAGAINinserts its socket into_socket_map— the map thatCLIENT LIST/CLIENT KILLenumerate (brpc documents this "funny race condition" inacceptor.cpp). WhenCLIENT KILLwins that race it matches zero sockets →killed_ == 0→ERR No such client.core_number=2), which is why it flaked only there.Fix
EloqKV's
CLIENT KILLonly supports the old-styleaddr:portform (killing by id would need a server change), so the fix waits until the targetaddris actually visible inCLIENT LIST— the same_socket_mapthatCLIENT KILLscans — before killing it. Once the addr appears, the kill is deterministic. A smallkill_client_by_addrhelper replaces the three racyr client kill $addrsites. This keeps cluster coverage instead of restricting the cases to single-node, and follows the same "wait on the real condition, don't sleep/race" approach as the earlier pubsub flake fix (e5ca361).Verification
Ran the real harness against a scratch single-node server: all four
client.tcltests pass, including the two laterCLIENT KILL-by-addr sites (CLIENT ID, CLIENT INFO, CLIENT LIST, CLIENT KILL). No product code changes.🤖 Generated with Claude Code
Summary by CodeRabbit