Stabilize the live-Postgres CI gate (#118 follow-up)

[stephane-segning]

Summary

• crates/sms-auth/tests/rbac_layer2_live_postgres.rs — adds ca653a1's per-binary TEST_MUTEX pattern, missing since this file's own PR (RBAC layer 2: permission/scope enforcement (#24) #112) landed the day after ca653a1 established it.
• crates/sms-worker/tests/live_postgres.rs — same pattern added; this file predates ca653a1 by months and was never updated, protected only by being #[ignore]d and never run in CI until Add a testcontainers-backed live-Postgres harness; wire it into CI (#102) #118.
• crates/sms-test-support/src/lib.rs — stale-schema detection now keys on a content fingerprint of schema/migrations/postgres/**, not on whether public.messages merely exists.
• crates/sms-worker/tests/claim_live_postgres.rs — test-only fix to two_concurrent_claimers_never_both_win_the_same_row's assertion (folded in from a parallel investigation into the same CI job — see Scope).

Intent

Source of truth: the three-way CI failure this branch was created to diagnose — #118's new live-Postgres suites (sms-test-support) job failing on a different test almost every run, three runs producing three different failures in three different test binaries. This PR addresses the one assigned to this investigation directly (sms-auth::rbac_layer2_live_postgres::provider_write_route_denies_a_token_that_is_correctly_scoped_for_something_else, failing with duplicate key value violates unique constraint "pg_type_typname_nsp_index" inside rotate_signing_key), plus a second, previously-undetected gap in the same class found while auditing every live suite for it, plus the harness's own stale-schema-detection bug named explicitly in the investigation brief.

Scope

• In scope: the pg_type collision's root cause and fix, an audit of every one of the 14 live suites for the same missing-mutex shape, the harness's stale-schema-detection bug, and (per an explicit scope hand-off partway through this investigation, once a parallel investigation into a different CI failure on this same job concluded the claim-loop code itself was sound) a test-only assertion fix in claim_live_postgres.rs.
• Out of scope, deliberately: no production code changed anywhere in this PR — crates/sms-worker/src/claim.rs was read and, briefly, locally and temporarily edited to sanity-check the new assertion's sensitivity to a real regression, then fully reverted (git diff on that file is empty in this PR). Also out of scope: the fixed test-harness container name's cross-process-interference footgun, found live during this investigation when a concurrently-running parallel investigation's own test runs collided with this one on the same container — flagged as a follow-up task rather than folded into this PR, since it's a local-multi-session concern, not a CI-gate reliability one (CI never runs two concurrent cargo test --workspace -- --ignored invocations against the same container).

Verification

The pg_type root cause, and how it was proven: ca653a1 (#102) established a per-binary tokio::sync::Mutex every test in a live suite acquires for its whole body, to serialize a binary's own concurrently-running tests against Postgres's own first-use pg_type catalog race. Two files never got this pattern: rbac_layer2_live_postgres.rs (added in #112, dated 2026-08-03 11:05 — ten hours after ca653a1 landed at 01:19 the same day, so the pattern already existed and simply wasn't applied) and sms-worker/tests/live_postgres.rs (added in #92, months before ca653a1 existed, and never revisited since — protected only by being #[ignore]d, and #118 is the first time anything ran it in CI). Confirmed via git log/grep across every *_live*.rs file in the workspace: every other multi-test live suite has TEST_MUTEX; these two didn't. Fixed by applying the identical, already-proven mechanism — no new mitigation invented.

Reproduction attempts (honest reporting, not a clean repro): could not reproduce the pg_type collision locally despite 9 escalating attempts — default settings, 32x oversubscribed --test-threads, the harness container throttled to --cpus 0.5 and --cpus 0.25, and full-workspace cold sweeps at each throttle level. This is consistent with the investigation brief's own framing ("It reproduces in CI, not on this fast machine") and with ca653a1's own finding that this race is real but low-probability, needing conditions this fast, many-core dev machine doesn't reliably hit. The fix's correctness rests on it being the exact, already-validated ca653a1 mechanism applied to files that were missing it — not on a fresh local reproduction.

The stale-schema bug, proven end to end: built a baseline template with the old harness code (operator_prefix_rules: 14 rows, matching AGENTS.md's documented seed count). Edited 0002_bootstrap/up.sql to add a 15th row (a throwaway probe, reverted before committing — git diff on that file is empty in this PR) without touching 0001_init. Reran a live suite with the old code: bug reproduced — row count silently stayed at 14, cargo test still reported ok, exactly the "harness runs tests against a schema older than the working tree" defect described in the brief. Reran the identical scenario with the fix: row count correctly became 15, the COMMENT ON DATABASE fingerprint changed, no manual just test-live-clean needed — the harness self-healed on its very next call.

claim_live_postgres.rs: rebuilt and ran the full 7-test suite 3 consecutive times with the real (unmodified) claim.rs — all green. Separately, temporarily removed .if_match(self.version) from the one queued -> routed write path, ran the target test 3 times (all still passed — a genuinely narrow race window on this machine, consistent with the pg_type non-repro pattern above), then fully reverted the change (confirmed via git diff showing zero delta) before proceeding. CAS soundness itself (240 concurrent attempts, exactly one winner per round, byte-identical if_match between the pinned 0.6.7 and upstream 0.7.5) was independently established by the parallel investigation this fix was handed off from.

Full-suite stability: just check, just lint (fmt + clippy -D warnings), just parity, ./ci/assert-no-raw-sqlx.sh, just test all green. Three consecutive full cargo test --workspace -- --ignored sweeps — two against a warm/reused container, one against a freshly test-live-clean'd cold container — all green, 37/37 test-result groups, 0 failures, 0 flakes.

Container safety: every container operation in this investigation was scoped to the dev.vsms.test-harness=true label (verified via docker inspect before touching it), never by image or bare name. Final docker ps -a container count and names match this machine's pre-investigation baseline exactly (the 11 pre-existing tenant-provisioner-* containers, untouched, plus the one vsms-test-harness-postgres).

Screenshots/Evidence

N/A — backend test-infrastructure fixes, no UI surface. Log excerpts and exact commands are in this PR's commit message and available on request; the key numbers are summarized in Verification above.

Risk Assessment

Low. No production code path changes (confirmed via git diff --stat against main touching only crates/sms-auth/tests/, crates/sms-worker/tests/, and crates/sms-test-support/src/lib.rs — the latter is test-infrastructure-only, allowlisted under R1 as the "migrations" exception). The pg_type fix is the lowest-risk kind of change possible: applying an already-shipped, already-proven mitigation to files that were missing it, not inventing a new one. The stale-schema fix changes when the harness rebuilds its template (more often, on genuine content drift, never on a no-op rerun) — proven both ways (stale-repros, fix-heals) against a real container, not just reasoned about.

AI Usage Declaration

This PR was implemented by Claude Code (Claude Sonnet 5) under my direction: diagnosing the pg_type collision's root cause via git history and cross-file auditing, fixing the harness's stale-schema-detection bug, and folding in a hand-off fix to a parallel investigation's test assertion.

• I reviewed the diagnosis (both files' git history relative to ca653a1, confirmed via git log/grep, not assumed) and the fingerprint mechanism's design (why COMMENT ON DATABASE/pg_shdescription rather than an in-schema marker table — it must never be copied into per-binary scratch databases via CREATE DATABASE ... TEMPLATE).
• I confirmed the stale-schema fix against a real container in both directions — reproduced the bug with the old code, then proved the fix heals it — rather than trusting the diagnosis alone, per this repo's own house rule that a green cargo test (or a plausible-sounding fix) is not to be trusted without live verification.
• I take responsibility for the accuracy of every iteration count and test-pass count above; they are copied from this session's actual command output, not estimated. Where I could not reproduce the pg_type race locally, I reported that honestly rather than claiming false confidence.

Reviewer Focus

• Whether COMMENT ON DATABASE / pg_shdescription is the right place to stamp the migration fingerprint, versus (for example) a dedicated tracking table in the admin postgres database — the tradeoff considered was: a table would need explicit cleanup reasoning of its own, whereas pg_shdescription is cluster metadata that Postgres's own dropdb() cleans up for free and is structurally guaranteed to never leak into a per-binary scratch database.
• Whether the two missing-mutex fixes are complete — I audited every *_live*.rs/*_live_postgres.rs file in the workspace (14 total, matching the harness's own documented count) for the shape "2+ #[tokio::test]s, no TEST_MUTEX" and found exactly these two; worth a second pass in case the audit missed a naming variant.
• The follow-up task flagged separately (not in this PR) about the harness's fixed container name causing cross-process interference between concurrent cargo test -- --ignored invocations — confirmed live during this investigation, when a parallel investigation's own runs collided with this one. Worth deciding whether it belongs in this PR's scope after all, or stays a follow-up.

[changeset-bot]

⚠️ No Changeset found

Latest commit: a959f2a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR