Fix data race in round_robin --job-retries test#470
Merged
esabol merged 1 commit intoJul 13, 2026
Conversation
_job_retry_TEST() started the worker on a background thread but read limit.count()/limit.expected() for its assertions right after gearman_client_do() returned, without joining that thread first. The worker thread can still be executing job_retry_WORKER() (writing limit->_count) at that point, racing the main thread's read with no mutex, atomic, or join between them. ThreadSanitizer confirmed this: 29/30 runs of the --job-retries=10 collection flagged a data race between Limit::count() (main thread, round_robin.cc:339) and Limit::increment() (worker thread, round_robin.cc:262), matching the intermittent "Assertion 'uint32_t(limit.expected())' != 'uint32_t(limit.count())'" failure reported in gearman#469. Call handle->shutdown() (which signals and joins the worker thread) before checking the counters, mirroring the pattern already used in round_robin_epoch_does_not_block_regular_TEST() in the same file. Verified: 50/50 clean runs under ThreadSanitizer after the fix (was 29/30 failing before), and 10/10 clean runs in a normal build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Alexei Pastuchov <info@maximka.de>
Member
|
I'm happy to merge this. Thank you for the analysis and helping to make the tests more robust! |
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #469 — the intermittent
t/round_robinfailure onround_robin.--job-retries=10.GEARMAN_FATAL(
Assertion 'uint32_t(limit.expected())' != 'uint32_t(limit.count())').This is a bug in the test, not in gearmand itself.
_job_retry_TEST()(
tests/round_robin.cc) starts the worker on a background thread viatest_worker_start(), then readslimit.count()/limit.expected()forits assertions immediately after
gearman_client_do()returns — withoutever joining that background thread first. The worker thread can still be
executing
job_retry_WORKER()(which writeslimit->_count) at that exactmoment, racing the main thread's read with no mutex, atomic, or join tying
the two together.
ThreadSanitizer confirms this directly: 29 of 30 runs of the
--job-retries=10collection reportedwhich lines up exactly with the intermittent CI failure.
Fix
Call
handle->shutdown()(which signals and joins the worker thread)before checking the counters — the same pattern already used a little
further up in this file, in
round_robin_epoch_does_not_block_regular_TEST().Test plan
-fsanitize=thread; ran the--job-retries=10collection30x before the fix — 29/30 runs hit the data race.
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com