Test suite flakiness: 22 issues across race conditions, timing, and resource management

[mkysel]

Summary

Systematic analysis of all 135 _test.go files identified 22 distinct flakiness patterns that can cause intermittent CI failures, test hangs, and fatal panics.

Breakdown: 9 CRITICAL · 8 HIGH · 5 MEDIUM

---

CRITICAL — Likely Causing Failures Today

1. Concurrent map access without mutex — originator_stream_test.go

Lines: pkg/sync/originator_stream_test.go:162, 197, 282, 341, 378, 433, 504

lastSequenceIds maps are written by goroutines and read in require.Eventually polling loops without synchronization. Go's runtime will panic:

fatal error: concurrent map read and map write

Fix: Wrap with sync.RWMutex or use sync.Map.

---

2. Channel reads without timeout — can hang forever

• pkg/registry/notifier_test.go:62-76 — for v := range ch blocks forever if channel never closes
• pkg/indexer/common/log_handler_test.go:99-107 — unbuffered done channel; goroutine orphaned on timeout, may panic on double-close
• pkg/sync/originator_stream_test.go — direct <-ch reads without select/timeout

Fix: Always use select with case <-time.After(...) or test-scoped context deadline.

---

3. Package-level state mutation without synchronization

• pkg/tracing/tracing_test.go:13-19 — apmEnabled (package-level bool) set/restored via t.Cleanup() with no mutex. Parallel tests race on this value.
• pkg/api/message/subscribe_test.go:33-50 — allRows (package-level slice) re-assigned in setupTest(). Parallel tests overwrite each other's data.

Fix: Protect with sync.Mutex or restructure so each test owns its own data.

---

4. Ultra-tight require.Eventually timeouts (500ms for DB ops)

• pkg/indexer/storer/groupMessage_test.go — 50ms tick / 500ms timeout
• pkg/indexer/storer/identityUpdate_test.go — 50ms tick / 500ms timeout

A single slow query or GC pause under CI load causes failure.

Fix: Increase to 5s+ timeout.

---

5. Goroutines spawned without lifecycle management

pkg/sync/originator_stream_test.go:158-160, 193-195, 276, 333, 376, 431, 500, 572, 634

go dbStorerInstance.Start() called without WaitGroup, done channel, or t.Cleanup. Tests exit before goroutine finishes → resource leaks, stale goroutines writing to closed DB connections.

Fix: Use t.Cleanup() to cancel context and wait for goroutine completion.

---

6. Unfired barrier channel — goroutine leak

pkg/blockchain/noncemanager/manager_test.go:264-300

Barrier channel created but never closed. All 50 goroutines block forever on <-barrier.

Fix: Add close(barrier) to release goroutines.

---

HIGH — Frequent Intermittent Failures

7. time.Sleep as synchronization (~50+ instances)

Most problematic:

File	Sleep	Waiting For
pkg/payerreport/store_test.go:185	1ms	DB write
pkg/payerreport/workers/settlement_test.go:516	1ms	Async op
pkg/blockchain/noncemanager/redis/manager_test.go:52,182	1ms	Redis ops
pkg/blockchain/noncemanager/manager_test.go:303	1ms	Concurrent nonce detection
pkg/db/sequences_test.go:58,78	10ms	DB locking
pkg/db/payer_test.go:68	5ms	Concurrent payer creation
pkg/tracing/tracing_test.go:82,107	100ms	50ms TTL expiry (only 2x margin)
pkg/blockchain/oracle/oracle_test.go:101	300ms	250ms stale threshold (50ms margin)

Fix: Replace with require.Eventually or event-driven synchronization.

---

8. Sleeps coupled to internal polling constants

File	Pattern
pkg/api/message/subscribe_topics_test.go:94	Sleep(SubscribeWorkerPollTime + 100ms)
pkg/api/metadata/cursor_test.go:99	Sleep(SubscribeWorkerPollTime + 100ms)
pkg/api/message/subscribe_test.go:620	Sleep(4 * SubscribeWorkerPollTime)

Fix: Use require.Eventually instead of coupling to implementation internals.

---

9. reflect.DeepEqual on proto messages

pkg/server/server_test.go:178-180, 200-202 — proto structs have unexported internal fields that cause spurious inequality. Note: pkg/api/message/publish_test.go:84 correctly uses proto.Equal().

Fix: Replace with proto.Equal().

---

10. Unsynchronized counter in flaky query helper

pkg/db/subscription_test.go:115-125 — numQueries++ in a closure without synchronization creates a data race when called concurrently by the polling mechanism.

Fix: Use atomic.Int32.

---

11. Random sleep in test — non-reproducible

pkg/blockchain/oracle/oracle_test.go:84 — time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond). Failures cannot be reproduced.

Fix: Use fixed sleep or event-driven sync.

---

12. Unseeded rand in test inputs

• pkg/api/message/subscribe_test.go:391 — rand.Intn(high - low)
• pkg/db/gateway_envelope_batch_test.go:59,152 — rand.Intn(10) + 1 for batch size

Different behavior every run; failures non-reproducible.

Fix: Use fixed seed or deterministic inputs.

---

13. Port reuse race

pkg/db/pgx_test.go:150-176 — listener opened on :0, port captured, listener closed, then port reused. Between close and rebind another process can claim the port.

Fix: Pass the listener directly instead of close-and-rebind.

---

14. gRPC listener leak

pkg/api/payer/publish_test.go:264-283 — net.Listen + grpc.NewServer without t.Cleanup(s.GracefulStop). Listener leaks if test fails before goroutine starts.

Fix: Add t.Cleanup(func() { s.GracefulStop() }).

---

MEDIUM — Occasional Failures

15. context.Background() instead of test-scoped context

Widespread (~20+ files), especially pkg/prune/prune_test.go (13 occurrences). Test timeouts don't propagate; hung operations hang forever.

Fix: Use t.Context() (Go 1.24+).

---

16. Missing t.Cleanup for servers/workers

• pkg/blockchain/rpcMultiClient_test.go — HTTP test servers
• pkg/sync/syncworker_test.go — workers
• pkg/api/server_test.go — server shutdown

---

17. Database result ordering assumptions

• pkg/prune/prune_test.go:194,240 — SELECT without ORDER BY
• pkg/db/gateway_envelope_batch_test.go:362 — payer_id query without ordering
• pkg/api/message/subscribe_test.go — notification order assumed deterministic

Fix: Add ORDER BY or sort results before comparison.

---

18. Tight time.After deadlines in select blocks

File	Timeout	What It Guards
pkg/indexer/common/log_handler_test.go:105,166	1s	Async DB work
pkg/api/message/publish_worker_test.go:408	2s	Worker exit

Fix: Use 10s+ for I/O-bound operations in CI.

---

19. Hardcoded Redis address shared across all tests

pkg/testutils/redis/redis.go:14 — localhost:6379 DB 15. All Redis tests compete for the same instance.

Fix: Use testcontainers or verify availability before tests.

---

20. Concurrent Anvil container startup race

pkg/testutils/anvil/anvil_test.go:26-49 — 10 concurrent containers with 5s timeout and no backoff.

---

21. Connection pool exhaustion under parallel load

pkg/testutils/store.go:70-74 — default pool (25 conns) shared across parallel tests.

---

22. Transaction lock race in prune tests

pkg/prune/prune_test.go:203-235 — test holds lock and runs executor with no guaranteed timeout behavior.

---

Top 10 Flakiest Files

Rank	File	Issues
1	pkg/sync/originator_stream_test.go	Concurrent map access, orphaned goroutines, no lifecycle mgmt
2	pkg/api/message/subscribe_test.go	Shared global state, ordering assumptions, unseeded rand
3	pkg/tracing/tracing_test.go	Package-level state race, tight TTL margins
4	pkg/indexer/common/log_handler_test.go	1s timeout, unbuffered done channel, goroutine leak
5	pkg/registry/notifier_test.go	Unbounded channel read, 100 unsynced goroutines
6	pkg/server/server_test.go	reflect.DeepEqual on protos, lifecycle races
7	pkg/blockchain/noncemanager/manager_test.go	Unfired barrier, 1ms sleeps
8	pkg/blockchain/oracle/oracle_test.go	Random sleep, 50ms timing margin
9	pkg/db/subscription_test.go	Unsynchronized counter, context lifecycle
10	pkg/db/gateway_envelope_batch_test.go	Unseeded rand, missing ORDER BY

Suggested Fix Priority

Immediate (prevents panics and hangs):

1. Mutex on lastSequenceIds map in originator_stream_test.go
2. Timeouts on all bare channel reads
3. close(barrier) in nonce manager test
4. Mutex on apmEnabled in tracing tests
5. atomic.Int32 for numQueries in subscription_test.go
6. proto.Equal instead of reflect.DeepEqual in server_test.go

Short-term (reduces CI flakiness):
7. Replace all 1ms sleeps with require.Eventually
8. Increase 500ms Eventually timeouts to 5s+
9. t.Cleanup for all goroutines/servers started in tests
10. Fix port reuse race in pgx_test.go

Medium-term (improves reliability):
11. Migrate context.Background() → t.Context() across test suite
12. Replace proto require.Equal with protocmp.Transform()
13. Remove sleep coupling to SubscribeWorkerPollTime
14. Sort results before slice comparisons in query tests
15. Fixed seed or deterministic inputs for rand in tests

🤖 Generated with Claude Code

[neekolas]

Fix the 6 "Immediate" items highlighted at the bottom

[xmtp-coder-tasks]

Task completed.

[mkysel]

Observed CI Failures on main (Mar 13 – Apr 7, 2026)

Analyzed 15 failed Test runs on main. Of those, 12 had test failures (3 had expired logs). Every single failure maps to issues identified in this report.

Failure Summary

#	Date	Test	Package	Run	Root Cause (from report)
1	Apr 7	(build failure)	pkg/api/message	24099151259	Build error (undefined: queries.InsertGatewayEnvelopeParams) — not flakiness
2	Apr 6	TestCreateServer	pkg/server	24053592826	Condition never satisfied at server_test.go:185 — #9 reflect.DeepEqual on protos + timing
3	Apr 3	TestCreateServer	pkg/server	23948465543	Condition never satisfied at server_test.go:163 — same as above
4	Apr 3	TestCreateServer	pkg/server	23948430138	Condition never satisfied at server_test.go:163 — same as above
5	Mar 31	TestCreateServer	pkg/server	23821953756	Condition never satisfied at server_test.go:185 — same as above
6	Mar 26	TestSubscribeEnvelopesByTopic	pkg/api/message	23617865241	DB connection context canceled during subscription poll — #15 context.Background(), #7 timing
7	Mar 24	TestSubscribeVariableEnvelopesPerOriginator	pkg/api/message	23497364512	DB connection context canceled during poll — #15, #7
8	Mar 22	TestPublishGroupMessageConcurrent	pkg/blockchain	23398548984	timed out + context deadline exceeded on Anvil HTTP — #17 container startup race, #7 timing
9	Mar 20	TestEnvelopesGenerator	pkg/stress	23335641609	Byte-level mismatch in generated envelopes — #12 unseeded rand / non-determinism
10	Mar 19	TestSubscribeEnvelopesByTopic	pkg/api/message	23306072608	Same as #6 — #15, #7
11	Mar 17	TestCreateServer	pkg/server	23204089421	Same as #2 — #9, timing
12	Mar 16	TestCreateServer	pkg/server	23154220174	Same as #2 — #9, timing
13	Mar 16	TestSimultaneousSubscriptions	pkg/api/message	23144638724	Should be true at subscribe_test.go:175 — #3 shared global state, #7 timing, #8 sleep coupling
14	Mar 13	TestSubscribeVariableEnvelopesPerOriginator	pkg/api/message	23047808475	DB context canceled — #15, #7
15	Mar 13	TestSubscribeVariableEnvelopesPerOriginator	pkg/api/message	23047782735	DB context canceled — #15, #7

Failure Frequency by Test

Test	Failures	Packages
TestCreateServer	6	pkg/server
TestSubscribeVariableEnvelopesPerOriginator	3	pkg/api/message
TestSubscribeEnvelopesByTopic	2	pkg/api/message
TestSimultaneousSubscriptions	1	pkg/api/message
TestPublishGroupMessageConcurrent	1	pkg/blockchain
TestEnvelopesGenerator	1	pkg/stress

Root Cause Mapping

TestCreateServer (6 failures) — The require.Eventually at lines 163/185 uses reflect.DeepEqual on proto messages (issue #9). reflect.DeepEqual can return false for semantically equal protos due to internal fields. Combined with cross-node sync timing, this is the #1 source of CI failures on main.

TestSubscribe* tests (6 failures combined) — All show failed to connect to postgres: context canceled during subscription polling. The subscribe workers use context.Background() (issue #15) and rely on time.Sleep-based synchronization (issue #7). Under CI load, the DB connections are canceled before subscriptions complete.

TestPublishGroupMessageConcurrent (1 failure) — context deadline exceeded on Anvil container HTTP call. Maps to Anvil startup race (issue #17) and tight timeouts (issue #7).

TestEnvelopesGenerator (1 failure) — Byte-level proto mismatch suggests non-deterministic serialization or unseeded randomness (issue #12).

Highest-Impact Fixes

Fixing just 3 issues would have prevented all 14 test failures (excluding the build error):

1. Replace reflect.DeepEqual with proto.Equal in server_test.go → prevents 6 TestCreateServer failures
2. Use test-scoped contexts in subscribe tests + increase timeouts → prevents 6 subscribe test failures
3. Increase Anvil container timeout + use proto.Equal in stress tests → prevents remaining 2 failures

[xmtp-coder-agent]

Implementation Spec — Immediate items (1–6)

Working on this. Scope is narrowly the six "Immediate" items from the issue body, per direction above. HIGH/MEDIUM items are deferred.

Fix plan

1. lastSequenceIds mutex (pkg/sync/originator_stream.go + _test.go): Add a sync.Mutex field to originatorStream guarding the map, lock it in validateEnvelope, and expose a LastSequenceID(uint32) uint64 accessor for tests to read from require.Eventually goroutines.
2. Bare channel reads / leaky waiters:
   • pkg/registry/notifier_test.go: CountChannel gains a stop signal wired through t.Cleanup so its reader goroutine can't outlive the test.
   • pkg/indexer/common/log_handler_test.go: Bump the 1s time.After to 10s and track the waiter goroutine via WaitGroup + t.Cleanup.
3. close(barrier) in pkg/blockchain/noncemanager/manager_test.go: Already present at line 320 — verification only. Will add a clarifying comment so this doesn't regress.
4. apmEnabled mutex (pkg/tracing/tracing.go + _test.go): Add a package-private sync.RWMutex; migrate reads/writes through it (including Start, Stop, IsEnabled, SetEnabledForTesting). Tests stop flipping the bool directly and go through SetEnabledForTesting.
5. numQueries → atomic.Int32 (pkg/db/subscription_test.go): Swap the unsynchronized counter in flakyEnvelopesQuery.
6. proto.Equal in pkg/server/server_test.go: Replace both reflect.DeepEqual calls with proto.Equal, drop the reflect import.

Validation

• go test -race across the six affected packages + nonce manager.
• dev/lint-fix.

Opening a draft PR against this issue once these are in.

[mkysel]

Make a new PR and get rid of the sleeps in tests