test: run suite in parallel with pytest-xdist#680
Merged
Conversation
Add pytest-xdist (latest stable, 3.8.0) to the dev extras and default pytest to 4 workers via addopts. Session-scoped fixtures are per-worker under xdist, so each worker starts its own Postgres container and template database — the speedup comes from a few extra Postgres servers rather than aggressive process-level concurrency. Per-test databases already use UUID names, so the suite is parallel-safe as-is. Use `-n 0` to run serially.
janbjorge
force-pushed
the
claude/pytest-xdist-integration-4pe3mk
branch
from
e4a4ff0 to
f0ee256
Compare
The postgres image logs "database system is ready to accept connections" twice: once from the initdb bootstrap server (unix socket only) and once from the real server. The wait strategy matched the first occurrence, so fixtures connected before TCP was listening and got connection resets. With 4 workers running initdb concurrently on a 4-vCPU runner the bootstrap window stretches to seconds, making the race near-certain; the failed session fixture is then cached by pytest, erroring every DB test in that worker. Wait for the second occurrence, as testcontainers-java does. Also bound per-container memory (shared_buffers 128MB, max_wal_size 512MB, tmpfs capped at 1g) since each xdist worker now runs its own container with its data dir on tmpfs (RAM).
LogMessageWaitStrategy in testcontainers 4.13 ignores its `times` argument, so the previous fix still matched the initdb bootstrap server's ready line and fixtures connected before TCP was listening. Anchor the wait on the bootstrap shutdown line followed by the real ready line instead, and retry the session fixture's first connection for up to a minute as belt and braces.
The bootstrap server's log output is split across stdout and stderr, so both the "ready to accept connections" line and the shutdown+ready sequence are ambiguous stream-wise. "PostgreSQL init process complete" appears exactly once, on stdout, after the bootstrap server is gone; the remaining gap until the real server listens on TCP is bridged by the connection retry in the migrated_db fixture.
Replace log-line matching and the fixture-level connection retry with a docker healthcheck probing pg_isready over TCP, mirroring the service container setup in ci.yml. The initdb bootstrap server only listens on the unix socket, so the container cannot report healthy before the real server accepts TCP connections — readiness is decided by the infrastructure instead of log heuristics or client retries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pytest-xdist>=3.8.0(latest stable) to thedevextras and updateuv.lock.addopts = "--durations=15 -n 4"inpyproject.toml.CREATE DATABASE ... TEMPLATE), so the suite is parallel-safe with no fixture changes.AGENTS.md, including-n 0to run serially and-n <N>to change the worker count.Timing (full suite, 777 passed / 11 skipped):
-n 0)Testing
make checkpassed (ruff check, ruff format, lint-imports, mypy, full pytest suite)-n 0) and with the new 4-worker default against Postgres 16 — identical resultsChecklist
Generated by Claude Code