Skip to content

Catch pg client connection errors + add synchronous crash logging#397

Merged
CCheng00 merged 1 commit into
mainfrom
fix-pg-client-error-guard
Jul 15, 2026
Merged

Catch pg client connection errors + add synchronous crash logging#397
CCheng00 merged 1 commit into
mainfrom
fix-pg-client-error-guard

Conversation

@CCheng00

Copy link
Copy Markdown
Contributor

Prevents the recurring idle-in-transaction crash from killing the process. When Postgres terminates a checked-out connection (e.g. a transaction held open past idle_in_transaction_session_timeout while awaiting a slow Twilio/Teams send), the client emits an 'error' event. Previously nothing listened for it on a checked-out client (pool.on('error') only covers idle pool clients), so it became an unhandled 'error' -> uncaughtException -> process exit(1) and a 2-3 min outage while ECS replaced the task.

  • db.js: attach client.on('error') in pool.on('connect') (once per physical connection) so the termination is logged ("Database client connection error: ...") and handled instead of crashing the process. The in-flight request still fails and rolls back, but the process stays up.
  • crashLogging.js + index.js: install process-level uncaughtException / unhandledRejection handlers that write the stack synchronously to stderr (fs.writeSync), so any future fatal error is captured in CloudWatch even if Sentry's async delivery is lost on exit.

This is a safety net that stops the outages regardless of which code path still holds a send inside a transaction; the remaining in-transaction sends in eventHandlers.js will be moved out separately.

Prevents the recurring idle-in-transaction crash from killing the process.
When Postgres terminates a checked-out connection (e.g. a transaction held
open past idle_in_transaction_session_timeout while awaiting a slow
Twilio/Teams send), the client emits an 'error' event. Previously nothing
listened for it on a checked-out client (pool.on('error') only covers idle
pool clients), so it became an unhandled 'error' -> uncaughtException ->
process exit(1) and a 2-3 min outage while ECS replaced the task.

- db.js: attach client.on('error') in pool.on('connect') (once per physical
  connection) so the termination is logged ("Database client connection
  error: ...") and handled instead of crashing the process. The in-flight
  request still fails and rolls back, but the process stays up.
- crashLogging.js + index.js: install process-level uncaughtException /
  unhandledRejection handlers that write the stack synchronously to stderr
  (fs.writeSync), so any future fatal error is captured in CloudWatch even
  if Sentry's async delivery is lost on exit.

This is a safety net that stops the outages regardless of which code path
still holds a send inside a transaction; the remaining in-transaction sends
in eventHandlers.js will be moved out separately.
@CCheng00
CCheng00 merged commit 9c8e8df into main Jul 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant