Intermittent CI test failures due to port-reservation race in libtest::Server::start()

[p-alik]

Summary

Investigating the follow-up flakiness noted on #469 / #470 (t/protocol failed on PR #470's own CI run, t/worker failed on the post-merge run) turned up a separate, pre-existing root cause unrelated to the round-robin retry fix in #470.

This is not limited to round_robin — it's a race in the shared libtest test harness that can intermittently hit any test that starts a gearmand server, and it predates #469/#470. For example, a scheduled run from 2026-07-11 (before #469 was even filed) shows t/vector failing on attempt 1 and passing on retry:
https://github.com/gearman/gearmand/actions/runs/29146585525/job/86529099800

Other examples seen this week (all passed on automatic retry, which is why the overall runs show green):

• t/protocol on alpine-3.19 gcc-13, PR Fix data race in round_robin --job-retries test #470 run: https://github.com/gearman/gearmand/actions/runs/29280197849/job/86919231598
• t/worker on alpine-3.20 gcc-13, post-merge run: https://github.com/gearman/gearmand/actions/runs/29281319528/job/86922952926

Three unrelated test binaries, three different platforms, none touching retry/round-robin logic.

Root cause

libtest/port.cc's get_free_port() reserves an ephemeral port properly: it binds a throwaway socket to it and keeps that socket open, which prevents any other process on the same machine from binding to that port for as long as the socket stays open.

But libtest/server.cc:232, in Server::start(), releases that reservation before actually starting the server:

libtest::release_port(_port);      // line 232 — releases the reservation

Application::error_t ret;
if (Application::SUCCESS != (ret= _app.run()))   // line 235 — THEN forks/execs gearmand

Between release_port() and the moment the newly forked/exec'd gearmand process actually calls bind() (fork/exec latency + arg parsing + its own startup path), the port is free for anyone to grab. Under make check -j, many test binaries run concurrently as separate processes, each independently calling get_free_port(). If another test process's get_free_port() call lands in that window, it can grab the same port number, and this test's gearmand then fails to bind (or, worse, some other test's server ends up on the port this test thinks it owns). The specific symptom (which test fails, connection-refused vs. wrong-response vs. timeout) depends on which two tests collided, which is why it's a different victim each time.

This gets more likely on slower/busier CI runners (alpine containers, older Ubuntu images, contended shared runners), which matches the observed pattern.

Suggested fix direction

Don't release the port reservation until the child gearmand process is confirmed to be listening (there's already a Wait/pid-file poll loop a few lines below _app.run() in Server::start() that could be extended to gate the release). Needs a look at Application::run()'s fork/exec path to make sure there isn't a reason the socket has to be released first (e.g. if the child needs the exact port free before it can bind at all, rather than inheriting the reservation).

Related

• Intermittent t/round_robin test failure during 2.0.0 release testing #469 (original round_robin flake report — different, already-fixed bug)
• Fix data race in round_robin --job-retries test #470 (the round_robin fix — unaffected by this issue)