Skip to content

feat: add the cross-machine wake for Postgres — LISTEN/NOTIFY#23

Merged
rodrigobnogueira merged 1 commit into
mainfrom
feat/postgres-wake
Jul 19, 2026
Merged

feat: add the cross-machine wake for Postgres — LISTEN/NOTIFY#23
rodrigobnogueira merged 1 commit into
mainfrom
feat/postgres-wake

Conversation

@rodrigobnogueira

Copy link
Copy Markdown
Contributor

What

The third and final wake tier: workers on other machines get the wake through the one thing they already share — the database. Postgres LISTEN/NOTIFY, Postgres-only by nature (SQLite/MySQL keep the WakeSocket tier; for SQLite that's the whole story anyway).

// producer — pg_notify rides the enqueue transaction: delivered ON COMMIT, dropped on rollback
new PostgresOutboxStore({ wakeChannel: 'outbox_wake' })

// worker — a dedicated (non-pooled) LISTEN connection feeds the loop's waker
const listener = new PostgresWakeListener({
  connect: () => new pg.Client({ connectionString, keepAlive: true }),
  channel: 'outbox_wake',
  waker,
});
listener.start(); // reconnects on drops; await listener.stop() on shutdown

The commit-atomicity is the elegant part: because pg_notify runs inside the caller's transaction, the wake needs no post-commit discipline — Postgres holds it until commit and drops it on rollback, so the signal is atomic with the event becoming visible. Best-effort contract as always: notifications missed during a reconnect gap are not recovered; polling stays the delivery backstop.

Hardening (from a 2-agent adversarial review before this PR)

The review traced two serious issues that are fixed here, with regression tests:

  • The pg end-during-connect deadlock. client.end() during an in-flight connect() means the connect promise never settles (pg skips its connect callback once _ending is set) — an un-raced await connect() would hang stop() forever, blocking graceful shutdown until SIGKILL. The session now parks on the connection's end/error events first and races connect() against them.
  • The 63-byte channel limit. Beyond NAMEDATALEN - 1, LISTEN silently truncates but pg_notify raises — and since it rides the enqueue transaction, a 64+-char "valid-looking" channel would abort the caller's business transaction on every enqueue. assertValidWakeChannel now enforces the cap (verified live against Postgres 16 by the reviewer).
  • Also: a throwing custom WakeSignal routes to onError instead of crashing the worker from inside pg's socket-data handler; reconnectDelayMs is validated (0/NaN would hot-loop a PG outage); documented factories set keepAlive: true so a half-open LISTEN socket (NAT/firewall idle eviction) is detected by the OS instead of parking forever.

The review also refuted the NOTIFY-flood DoS concern: OutboxWaker latches, so N notifications collapse to one follow-up tick — no amplification beyond what the (already-authenticated) DB user could do directly.

Tests / gates

  • 20 hermetic listener specs over scriptable fake connections (reconnect on error/clean-end/refused-connect/failed-LISTEN, stop during hung connect, stop during backoff, throwing waker with/without onError, channel + delay validation, quoting incl. case-sensitivity) + pglite observing a real pg_notify from enqueue.
  • Gated real-Postgres round-trip: delivered on commit, dropped on rollback — ran live (Docker): 135/135 hermetic, 4/4 integration (PG + MySQL), 0 skipped.
  • Coverage 100% on wake.ts and outbox-store.ts; complexity ≤15; core typecheck clean.
  • Docs: README wake-tier recipes, website/docs/api-reference.md (also closes the 0.4.0 gap — WorkerLoopOptions.waker and the wake tiers were undocumented on the site), CHANGELOG.

Note: Release and security checks is expected red on the pre-existing website (docusaurus) advisories — unrelated; the published-tarball audit is green.

No version bump; next release (0.5.0) ships this.

Completes the wake tiers: OutboxWaker (in-process, 0.4.0), WakeSocket pair
(same machine, 0.4.0), and now LISTEN/NOTIFY through the database for workers
on other machines.

Producer side: new PostgresOutboxStore({ wakeChannel }) piggybacks pg_notify
on the enqueue transaction — Postgres delivers it ON COMMIT and drops it on
rollback, so the wake is atomic with the event becoming visible and needs no
post-commit discipline. Worker side: PostgresWakeListener holds a dedicated
(non-pooled) LISTEN connection feeding the loop's OutboxWaker, reconnecting
with a fixed delay on drops. Best-effort by the same contract as every tier:
missed notifications are not recovered — polling remains the delivery
backstop.

Hardening from an adversarial review pass:
- channels: identifier allow-list + the 63-byte Postgres limit (beyond it
  LISTEN silently truncates while pg_notify RAISES — aborting the caller's
  BUSINESS transaction on every enqueue) + double-quoting so LISTEN stays
  case-aligned with pg_notify's exact-string channel
- the pg end-during-connect deadlock: a client ended mid-connect never
  settles connect(); the session parks on the connection's end/error events
  FIRST and races connect against them, so stop() cannot hang
- a throwing custom WakeSignal routes to onError instead of crashing the
  worker from inside pg's socket-data handler
- reconnectDelayMs validated (0/negative/NaN would hot-loop a PG outage)
- documented factories set keepAlive: true — a LISTEN socket is pure-receive,
  so without TCP keepalives a half-open death is never detected

Tests: 20 hermetic listener specs over scriptable fake connections + pglite
pg_notify observation + a gated real-Postgres round-trip proving
delivered-on-commit / dropped-on-rollback. 100% coverage; complexity <=15.
Docs: README wake tiers, api-reference (incl. the 0.4.0 waker gap), CHANGELOG.
@github-actions

Copy link
Copy Markdown

📊 Coverage Coverage Report

Metric PR Base Diff
Statements ████████████████████ 2070/2070 (100%) 1841/1841 (100%) ⚪ 0%
Branches ████████████████████ 279/279 (100%) 236/236 (100%) ⚪ 0%
Functions ████████████████████ 88/88 (100%) 78/78 (100%) ⚪ 0%
Lines ████████████████████ 2070/2070 (100%) 1841/1841 (100%) ⚪ 0%
🧾 Changed files
File Statements Branches Diff
messaging/dialects/postgres/outbox-store.ts ████████████████████ 137/137 (100%) 15/15 (100%) ⚪ 0%
messaging/dialects/postgres/wake.ts ████████████████████ 205/205 (100%) 40/40 (100%) 🆕 new

Updated for d5701d2 | Compared against base branch

@github-actions

Copy link
Copy Markdown

⏱️ Performance Report

✅ Tests 135 passed, 0 failed, 0 skipped
🧪 Suites 32
⏱️ Total step time 256.00s 🔴 +42.00s
⚙️ Test execution 247.69s 🔴 +41.30s
🐢 Slowest test suites
# Suite Tests Duration vs Base
1 PostgresWakeListener 17 27.89s -
2 runWorkerLoop 13 22.19s 🔴 +618ms
3 WakeSocketServer + WakeSocketClient 9 15.45s 🔴 +362ms
4 OutboxClaimer (publish outcomes) 9 15.32s 🔴 +81ms
5 SqliteOutboxStore 8 13.86s 🔴 +935ms
6 KafkaInboxConsumer 8 13.68s 🔴 +417ms
7 OutboxWaker 7 12.09s 🔴 +814ms
8 MysqlOutboxStore 7 11.74s ⚪ ~0
9 InProcessOutboxTransport 5 8.44s ⚪ ~0
10 SqliteInboxStore 5 8.39s ⚪ ~0
🐌 Slowest individual tests
# Suite Test Duration vs Base
1 PostgresOutboxStore wakeChannel PostgresOutboxStore wakeChannel enqueue fires pg_notify on the configured channel (observed via LISTEN) 3.49s -
2 OutboxProducer (atomic enqueue) OutboxProducer (atomic enqueue) enqueue commits the outbox row with the business write 2.63s 🔴 +70ms
3 SqliteOutboxStore SqliteOutboxStore claimBatch reclaims a stuck processing row past the timeout 2.21s 🔴 +684ms
4 OutboxProducer (atomic enqueue) OutboxProducer (atomic enqueue) a throw rolls back BOTH the outbox row and the business write 1.94s 🔴 +61ms
5 OutboxRegistry OutboxRegistry a second register for the same topic throws (wiring bug, not silent replace) 1.93s 🔴 +175ms
6 PostgresOutboxStore PostgresOutboxStore markCompleted / retry / markFailed transition the row 1.92s 🔴 +149ms
7 OutboxWaker OutboxWaker wait() resolves on the backstop timer when nothing wakes it 1.92s 🔴 +185ms
8 runWorkerLoop runWorkerLoop an abort during the idle sleep wakes it and removes the listener 1.90s 🔴 +175ms
9 KafkaInboxConsumer KafkaInboxConsumer acks (logs) a duplicate without re-running the side effect 1.90s 🔴 +258ms
10 SqliteOutboxStore SqliteOutboxStore markCompleted, retry, markFailed transition the row 1.89s 🔴 +241ms
11 PostgresInboxStore PostgresInboxStore runOnce processes a fresh key (async side effect) 1.87s 🔴 +130ms
12 OutboxWaker OutboxWaker works without a signal: both the backstop and notify paths 1.83s 🔴 +310ms
13 OutboxWaker OutboxWaker an abort during the wait wakes it and removes the listener 1.83s 🔴 +280ms
14 in-process wiring through Nest DI in-process wiring through Nest DI the registry injects into the transport and handlers register on module init 1.83s 🔴 +177ms
15 OutboxClaimer (publish outcomes) OutboxClaimer (publish outcomes) tick publishes a pending row and marks it completed 1.82s ⚪ ~0

Updated for d5701d2 | Compared against base branch

@github-actions

Copy link
Copy Markdown

🧠 Cognitive Complexity Report

Metric PR Base Diff
Total complexity 79 66 🔴 +13
Max function complexity 12 12 ⚪ 0
Functions measured 39 30 ⚪ +9
🧩 Most complex functions
Function Location Complexity Status
runWorkerLoop packages/messaging/outbox-worker.ts:34 12 🟢 OK
KafkaInboxConsumer.consume packages/messaging/adapters/kafka/kafka-inbox-consumer.ts:67 5 🟢 OK
PostgresWakeListener.#supervise packages/messaging/dialects/postgres/wake.ts:122 5 🟢 OK
WakeSocketServer.listen packages/messaging/wake-socket.ts:52 5 🟢 OK
OutboxClaimer.onPublishError packages/messaging/outbox-claimer.service.ts:84 4 🟢 OK
deriveDedupKey packages/messaging/wire-contract.ts:55 4 🟢 OK
MysqlInboxStore.runOnce packages/messaging/dialects/mysql/inbox-store.ts:43 3 🟢 OK
PostgresInboxStore.runOnce packages/messaging/dialects/postgres/inbox-store.ts:37 3 🟢 OK
SqliteInboxStore.runOnce packages/messaging/dialects/sqlite/inbox-store.ts:39 3 🟢 OK
KafkaInboxConsumer.readKey packages/messaging/adapters/kafka/kafka-inbox-consumer.ts:90 2 🟢 OK
🧾 Changed files
File Total Max Functions Total diff
messaging/dialects/postgres/outbox-store.ts 3 1 3 🔴 +2
messaging/dialects/postgres/wake.ts 11 5 7 🆕 new

🧭 Cognitive complexity is reported as a review signal, not a merge gate. Prefer small, intention-revealing refactors when complexity rises.


Updated for d5701d2 | Compared against base branch

@rodrigobnogueira
rodrigobnogueira merged commit ac90566 into main Jul 19, 2026
6 of 7 checks passed
@rodrigobnogueira
rodrigobnogueira deleted the feat/postgres-wake branch July 19, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant