Skip to content

Fix data race in round_robin --job-retries test#470

Merged
esabol merged 1 commit into
gearman:masterfrom
p-alik:fix-issue-469-round-robin-retry-race
Jul 13, 2026
Merged

Fix data race in round_robin --job-retries test#470
esabol merged 1 commit into
gearman:masterfrom
p-alik:fix-issue-469-round-robin-retry-race

Conversation

@p-alik

@p-alik p-alik commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #469 — the intermittent t/round_robin failure on
round_robin.--job-retries=10.GEARMAN_FATAL
(Assertion 'uint32_t(limit.expected())' != 'uint32_t(limit.count())').

This is a bug in the test, not in gearmand itself. _job_retry_TEST()
(tests/round_robin.cc) starts the worker on a background thread via
test_worker_start(), then reads limit.count() / limit.expected() for
its assertions immediately after gearman_client_do() returns — without
ever joining that background thread first. The worker thread can still be
executing job_retry_WORKER() (which writes limit->_count) at that exact
moment, racing the main thread's read with no mutex, atomic, or join tying
the two together.

ThreadSanitizer confirms this directly: 29 of 30 runs of the
--job-retries=10 collection reported

WARNING: ThreadSanitizer: data race
  Read of size 4 by main thread:
    Limit::count() tests/round_robin.cc:272
    _job_retry_TEST tests/round_robin.cc:339
  Previous write of size 4 by thread T2:
    Limit::increment() tests/round_robin.cc:262
    job_retry_WORKER tests/round_robin.cc:315

which lines up exactly with the intermittent CI failure.

Fix

Call handle->shutdown() (which signals and joins the worker thread)
before checking the counters — the same pattern already used a little
further up in this file, in round_robin_epoch_does_not_block_regular_TEST().

Test plan

  • Built with -fsanitize=thread; ran the --job-retries=10 collection
    30x before the fix — 29/30 runs hit the data race.
  • Same build, 50x after the fix — 0/50 races, 0 failures.
  • Normal (non-TSan) build, 10x — 0 failures.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

_job_retry_TEST() started the worker on a background thread but read
limit.count()/limit.expected() for its assertions right after
gearman_client_do() returned, without joining that thread first. The
worker thread can still be executing job_retry_WORKER() (writing
limit->_count) at that point, racing the main thread's read with no
mutex, atomic, or join between them.

ThreadSanitizer confirmed this: 29/30 runs of the --job-retries=10
collection flagged a data race between Limit::count() (main thread,
round_robin.cc:339) and Limit::increment() (worker thread,
round_robin.cc:262), matching the intermittent
"Assertion 'uint32_t(limit.expected())' != 'uint32_t(limit.count())'"
failure reported in gearman#469.

Call handle->shutdown() (which signals and joins the worker thread)
before checking the counters, mirroring the pattern already used in
round_robin_epoch_does_not_block_regular_TEST() in the same file.
Verified: 50/50 clean runs under ThreadSanitizer after the fix (was
29/30 failing before), and 10/10 clean runs in a normal build.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Alexei Pastuchov <info@maximka.de>
@esabol

esabol commented Jul 13, 2026

Copy link
Copy Markdown
Member

I'm happy to merge this. Thank you for the analysis and helping to make the tests more robust!

@esabol
esabol merged commit 8365521 into gearman:master Jul 13, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Intermittent t/round_robin test failure during 2.0.0 release testing

2 participants