Flaky RNG C++ Tests

[ZachPipes]

(Claude-Assisted)

What's wrong

The C++ tests run under CTest now; they never did before. Three assertions in the RNG blocks fail at random, roughly 18% of runs combined.

File	Line	Assertion	Tolerance	Sigma	Measured (800 runs)	Theory
test_discrete_uniform.cpp	104	variance	1e-2	2.0	5.25%	4.6%
test_geometric.cpp	117	variance	0.15	2.2	2.38%	2.8%
test_uniform.cpp	100	mean	5e-3	2.9	0.25%	0.4%

Combined that's ~7.7% of full-suite runs. The other ten cases: 0 failures in 200 runs each.

Why

Each block draws N=250000 samples and checks the mean or variance against theory with a fixed absolute tolerance. Those tolerances are round numbers rather than multiples of the estimator's standard error, so they land around 2 sigma and fail at roughly the rate a 2 sigma bound fails.

Worked example: geometric variance at p=0.25 is 12.0 with SE 0.068, so 0.15 is 2.2 sigma. The mean assertion on the line above uses the same 0.15 against SE 0.0069 — 21.7 sigma — which is why it never trips.

The samplers themselves are fine. Every measured rate agrees with ideal-RNG theory, and uniform_sample matches a numpy reference of the same estimator (0.25% vs 0.30%).

Fix

(Claude-suggested)

Size each tolerance from the estimator's standard error, roughly 7 sigma or more. These values ran 400 times with no failures:

• test_geometric.cpp:117 0.15 -> 0.5
• test_discrete_uniform.cpp:104 1e-2 -> 0.05
• test_uniform.cpp:100 5e-3 -> 0.02

A real defect would present as a systematic offset, which these still catch.

Separately, the samplers use thread_local std::mt19937 rng{std::random_device{}()}. Adding a seedable overload would make these tests deterministic and let the tolerances stay tight.

The other ten test files use the same _sample() block shape and are currently passing, but their tolerances haven't been checked against their standard errors.

Meanwhile

python-distro.yml runs ctest --repeat until-pass:3, which drops the flake to ~0.05%. A real regression is reproducible and still fails all three attempts. Remove the flag when this is fixed.

Repro

Linux/macOS:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target fastdist_tests --parallel
ctest --test-dir build --output-on-failure

Windows. The VS generator is multi-config, so CMAKE_BUILD_TYPE is ignored; pick the config at build time instead:

cmake -S . -B build
cmake --build build --target fastdist_tests --config Release --parallel
ctest --test-dir build -C Release --output-on-failure

Each run reseeds, so one run is one sample:

Bash:

for i in $(seq 1 300); do ./build/fastdist_tests >/dev/null 2>&1 || echo FAIL; done | wc -l

Powershell:

 $f=0; 1..300 | %{ & .\build\Release\fastdist_tests.exe *> $null; if($LASTEXITCODE -ne 0){$f++} }; "$f / 300 failed"

Measured 23/300 (7.7%) on MSVC/Windows, Release. GCC on Ubuntu will differ.