Skip to content

Fix Windows embedded Postgres command completion and owned teardown - #2544

Open
darionco wants to merge 2 commits into
bastani-inc:mainfrom
darionco:fix/windows-embedded-postgres-completion-teardown
Open

Fix Windows embedded Postgres command completion and owned teardown#2544
darionco wants to merge 2 commits into
bastani-inc:mainfrom
darionco:fix/windows-embedded-postgres-completion-teardown

Conversation

@darionco

@darionco darionco commented Aug 19, 2026

Copy link
Copy Markdown

Fix Windows embedded Postgres command completion and owned teardown

Summary

On Windows, runLocalCommand captured stdout/stderr with piped handles and resolved on the child's close event. pg_ctl start daemonizes Postgres; inherited pipe handles remain open in the Postgres process, so close does not fire at the pg_ctl process boundary even after the server is ready. This blocks embedded-Postgres provisioning.

This patch resolves local commands on the child exit event, resumes both streams without waiting for EOF, and preserves bounded stdout/stderr tails, exit-code semantics, and error handling.

It also makes embedded Postgres teardown explicit for clusters started by the current process: record the data directory and postmaster PID after readiness, stop with pg_ctl -m fast -w, wait for the captured process and TCP listener to disappear, and wire provider shutdown through DBOS lifecycle shutdown. Processes that attach to an already-running cluster do not claim ownership or stop it.

Scope

  • packages/workflows/src/durable/local-command.ts: general inherited-stdio completion fix (closeexit).
  • packages/workflows/src/durable/dbos-embedded-postgres.ts: process-local cluster ownership and graceful stop/wait.
  • packages/workflows/src/durable/dbos-local-postgres.ts: embedded-provider shutdown hook.
  • packages/workflows/src/durable/dbos-lifecycle.ts: invoke local-provider shutdown after DBOS shutdown.

Evidence — achieved

  • Real Windows reproduction: pg_ctl exits at its process boundary while the daemonized Postgres child retains the stdio pipes; close is delayed until the daemon exits, while exit fires when pg_ctl exits.
  • Exact-head weigent CI run 32311811708, Windows proof job 96256153352: embedded proof passed and job cleanup reported zero orphan Postgres processes.
  • Consecutive exact-head rerun of 32311811708, Windows proof job 96258157451: embedded proof passed in 51.1s and job cleanup reported zero orphan Postgres processes.
  • The proof provisions the bundled cluster, runs a durable task/stage workflow, shuts down, and removes the sandbox data directory with force disabled.

Tradeoff disclosure

Resolving on exit and resuming the streams avoids the Windows inherited-handle deadlock, but a trailing diagnostic chunk emitted between the final data event and exit may not be captured. This is intentional: waiting for close is the failure mode being fixed, and callers retain bounded diagnostics already received before process exit.

Regression-test disclosure

The end-to-end Windows embedded-Postgres proof currently lives in the downstream weigent repository, where it exercises the carried patch and verifies provisioning, durable execution, shutdown, and force-disabled removal. We can add an Atomic-side regression test in the upstream harness if preferred.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Greptile Summary

Embedded Postgres shutdown now follows the postmaster PID recorded by the session that started it. A focused check confirmed that shutdown completes when that PID no longer matches postmaster.pid, even if an unrelated process is listening on port 5439.

Confidence Score: 5/5

No blocking failure remains.

The shared-port teardown path was exercised with a live unrelated listener and a changed postmaster PID; the current implementation completed immediately rather than waiting for the port to close.

T-Rex T-Rex Logs

What T-Rex did

  • Captured the before state of the postmaster identity shutdown harness, showing shutdown=timed-out, elapsedMs=802, and listener 127.0.0.1:5439 (pidFile=5035).
  • Captured the after state, showing shutdown=completed, elapsedMs=0, and listener 127.0.0.1:5439 (pidFile=5014).
  • Confirmed the harness imports instrumented copies only and that no product source was modified.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (2): Last reviewed commit: "Address greptile P1: gate cluster-shutdo..." | Re-trigger Greptile

On Windows, runLocalCommand resolved on the child 'close' event; pg_ctl
start daemonizes Postgres, which inherits the stdio pipe handles and keeps
them open, so 'close' never fires at the pg_ctl boundary even after the
server is ready — blocking embedded-Postgres provisioning. Resolve on
'exit' (resuming streams without waiting for EOF), and add owned-cluster
teardown (record dataDir + postmaster PID after readiness, pg_ctl -m fast
-w stop, wait for process + listener to close, wired through DBOS lifecycle
shutdown). Processes that attach to an already-running cluster do not stop it.
Comment thread packages/workflows/src/durable/dbos-embedded-postgres.ts
@darionco

Copy link
Copy Markdown
Author

Tracking issue for this implementation: #2547.

Addressing reviewer feedback:

  • greptile P1 (shutdown wait keys on port reachability, dbos-embedded-postgres.ts:133) — acknowledged and valid in the class we are fixing. Once the owned postmaster PID is unavailable, an unrelated listener on 127.0.0.1:5439 would keep clusterStillRunning true indefinitely. Fix in progress: gate the completion wait on process ownership (the captured PID) rather than raw port reachability, treating the owned cluster as down when its PID is gone, with a bounded wait as fallback. This is consistent with the PR's root-cause fix (own the PID, not the port). Update with a re-verified Windows proof green to follow.

Will loop on any further reviewer feedback until there is none addressable and CI is green.

…p port-reachability fallback

Once the owned postmaster PID is gone, clusterStillRunning no longer consults
TCP on 127.0.0.1:5439, so an unrelated listener can no longer keep DBOS
shutdown pending indefinitely. Wait now returns false immediately when the
recorded postmaster.pid no longer matches the captured PID.

Refs bastani-inc#2547
@darionco

Copy link
Copy Markdown
Author

Loop complete — no addressable reviewer feedback remaining and CI is green.

  • Greptile P1 (shutdown wait keyed on port reachability) — resolved in 82157ee5: clusterStillRunning now gates purely on postmaster PID ownership and no longer consults TCP once the owned PID is gone. Greptile re-review passed.
  • Windows verification at exact head 734cddf (mirrored by 82157ee5): two consecutive clean proof greens — the embedded-Postgres Windows proof ("provisions, runs a durable workflow, shuts down, and releases v18 for removal") passes, full suite 523 passed / 8 skipped, and no Terminate orphan process lines (zero leaked postmaster/data-dir).
  • Tracking issue: Windows embedded Postgres: durable command completion never fires and owned-cluster teardown leaks postmaster + EBUSY data dir #2547.

Ready for review/merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant