Flaky CI Failure: TestMigrator panic on context cancel during cleanup
Workflow: Test
Failed run: https://github.com/xmtp/xmtpd/actions/runs/27297373695
Commit: 74e5a9f
Failed jobs: Test (Node)
Summary
The TestMigrator test in pkg/migrator failed with a FATAL log in the reader goroutine while running the gateway_envelopes_last_id sub-test. The reader attempts to get migration progress after the context has already been cancelled during teardown, causing a fatal-level panic that crashes the test binary. This is a race between test cleanup (context cancellation + container teardown) and the reader goroutine still trying to reconnect to the database.
Error Details
FATAL migrator.reader migrator/worker.go:129 failed to get migration progress {"table": "key_packages", "error": "failed to connect to `user=postgres database=test_newrawdb_fggucgimppeq`:\n\t127.0.0.1:8765 (localhost): failed to receive message: context canceled\n\t[::1]:8765 (localhost): dial error: dial tcp [::1]:8765: operation was canceled"}
github.com/xmtp/xmtpd/pkg/migrator.(*Worker).StartReader.func1
/home/runner/work/xmtpd/xmtpd/pkg/migrator/worker.go:129
github.com/xmtp/xmtpd/pkg/tracing.PanicWrap
/home/runner/work/xmtpd/xmtpd/pkg/tracing/tracing.go:237
github.com/xmtp/xmtpd/pkg/tracing.GoPanicWrap.func1
/home/runner/work/xmtpd/xmtpd/pkg/tracing/tracing.go:299
runtime/pprof.Do
/opt/hostedtoolcache/go/1.26.4/x64/src/runtime/pprof/runtime.go:57
FAIL github.com/xmtp/xmtpd/pkg/migrator 12.142s
Analysis
The failure is in pkg/migrator/worker.go:129 inside StartReader. After the migration completes (migration service stopped is logged), the sub-test TestMigrator/gateway_envelopes_last_id starts. During this phase, the context is cancelled and the testcontainers postgres container is torn down. A reader goroutine still alive at this point tries to connect to the now-gone database and hits a context-canceled error.
PanicWrap in pkg/tracing/tracing.go converts this error into a panic/fatal, which propagates and kills the test. The root cause is a timing/cleanup race: the reader goroutine isn't fully stopped before the database container is removed. This is a test-code failure (not infrastructure), likely triggered under load (race test mode with -race).
Reported by Flaky Failure Watcher
Flaky CI Failure: TestMigrator panic on context cancel during cleanup
Workflow: Test
Failed run: https://github.com/xmtp/xmtpd/actions/runs/27297373695
Commit: 74e5a9f
Failed jobs: Test (Node)
Summary
The
TestMigratortest inpkg/migratorfailed with aFATALlog in the reader goroutine while running thegateway_envelopes_last_idsub-test. The reader attempts to get migration progress after the context has already been cancelled during teardown, causing a fatal-level panic that crashes the test binary. This is a race between test cleanup (context cancellation + container teardown) and the reader goroutine still trying to reconnect to the database.Error Details
Analysis
The failure is in
pkg/migrator/worker.go:129insideStartReader. After the migration completes (migration service stoppedis logged), the sub-testTestMigrator/gateway_envelopes_last_idstarts. During this phase, the context is cancelled and the testcontainers postgres container is torn down. A reader goroutine still alive at this point tries to connect to the now-gone database and hits a context-canceled error.PanicWrapinpkg/tracing/tracing.goconverts this error into a panic/fatal, which propagates and kills the test. The root cause is a timing/cleanup race: the reader goroutine isn't fully stopped before the database container is removed. This is a test-code failure (not infrastructure), likely triggered under load (race test mode with-race).Reported by Flaky Failure Watcher