Skip to content

fix: clear stale postmaster.pid before starting postgres (33.0.6:3) - #120

Closed
MattDHill wants to merge 1 commit into
masterfrom
fix/postgres-stale-lockfile
Closed

fix: clear stale postmaster.pid before starting postgres (33.0.6:3)#120
MattDHill wants to merge 1 commit into
masterfrom
fix/postgres-stale-lockfile

Conversation

@MattDHill

@MattDHill MattDHill commented Jul 27, 2026

Copy link
Copy Markdown
Member

Note: this PR originally claimed to explain the "Timed out waiting for postgres,upgrade" report. Service logs have since disproved that. The change is retained as hardening for a real gap, but it is not the fix for that bug — see Correction below. Do not merge this expecting it to resolve that ticket.

Change

getBaseDaemons is the only place in the stack that starts PostgreSQL without first clearing a stale postmaster.pid:

  • the SDK's Backups.withPgDumpstartPg does it before each pg_ctl start (lib/backup/Backups.ts)
  • this package's 0.3.5x migration relocatePostgres does it (startos/versions/current.ts)
  • the daemon path did not

PostgreSQL writes postmaster.pid into PGDATA while running and removes it on a clean exit. A stop that never completes leaves it behind. On the next start PostgreSQL reads the PID it names and aborts with FATAL: lock file "postmaster.pid" already exists if that PID is alive. Each chain build runs in a fresh PID namespace with a fresh, low PID assignment, so the recorded PID can be live and owned by an unrelated process — the guard misfires.

Adds a pg-recover oneshot, with postgres gaining requires: ['pg-recover']. Placing it in getBaseDaemons covers setupMain, install init and update init in one place.

Removal is unconditional, which is safe because nothing else can hold the data directory: backups only run once the service is stopped (service/mod.rs — "the backup never starts before the service is stopped"), both init paths run with the service stopped, and the chain reconciler fully awaits healthDaemon.term() and subcontainer.destroy() before starting a replacement entry. It runs as root so ownership can never block the unlink and wedge the chain on the new oneshot.

Correction

The originating report was a user stuck updating with Timed out waiting for postgres,upgrade,__RUN_UNTIL_SUCCESS. I attributed it to a stale lock file. The service logs show a different failure:

Error: Database is uninitialized and superuser password is not specified.
Error: docker-entrypoint.sh exited with code 1

That is the postgres entrypoint's docker_verify_minimum_env, which only runs when $PGDATA/PG_VERSION is absent. PostgreSQL is not failing to open an existing cluster — there is no cluster there at all, and upgradeNextcloud passes postgresEnv without POSTGRES_PASSWORD (unlike the install path), so the entrypoint refuses to initialize one and exits 1 on a loop until the 30-minute timeout.

The logs also show the user is migrating from 32.0.11:0, not 33.0.5:1. Their first attempt ran the real 0.3.5x migration — migrateNextcloud's recursive chmod walk — for 53 minutes across 54,400 directories before being SIGTERM'd mid-walk. Every attempt after that skips the 0.3.5x block entirely in ~3 seconds, so the Postgres cluster is never relocated out of the Debian 17/main layout while the migration believes it has already run.

Root cause is therefore in the 0.3.5x migration: its completion marker (start9/config.yaml) and the work it gates are not committed atomically, so an interruption leaves the two permanently disagreeing. That needs a separate fix, tracked separately.

Scope

  • Does not fix the report above.
  • Does not re-run if Postgres crash-restarts within a single container lifetime — that case self-heals, since Postgres correctly identifies its own stale file in the same PID namespace.
  • No evidence this gap has bitten a user yet; it is hardening against a mechanism the SDK and the migration both already defend against.

Test plan

  1. Install 33.0.6:3 on a StartOS 0.4.0 box and confirm Nextcloud starts, the web UI loads, and you can log in.
  2. Confirm normal restart is unaffected: stop and start the service from the StartOS UI; it should come back healthy.
  3. Reproduce the unclean shutdown:
    • With the service running, hard-stop the box (or force-stop the service) so Postgres is SIGKILLed rather than shut down cleanly.
    • Confirm postmaster.pid is still present: start-cli package attach nextcloud -n postgres-sub -- ls /var/lib/postgresql/data/postmaster.pid
    • Start the service. It should reach healthy rather than sitting on "starting" indefinitely; the logs should show Postgres performing crash recovery, with no lock file "postmaster.pid" already exists.
  4. Backups still work end to end: run a backup, then restore it to a fresh install and confirm files and users come back.

…33.0.6:3

PostgreSQL writes postmaster.pid into PGDATA while running and removes it
on a clean exit. A stop that never completes leaves it behind, and on the
next start Postgres aborts with

  FATAL: lock file "postmaster.pid" already exists

if the PID recorded there is alive. Every chain build gets a fresh PID
namespace with a fresh, low PID assignment, so that PID is quite likely to
be live and owned by an unrelated process — the guard misfires and the
daemon never comes up.

Nothing surfaces when it happens: the postgres ready check maps any
non-zero pg_isready to 'loading' with display: null, and Daemon.runLoop
restarts the process forever on a backoff. In setupMain that shows as a
service stuck starting. In the update-time upgrade chain added in 33.0.6:0
it shows as runUntilSuccess exhausting its 30-minute budget and failing
init with "Timed out waiting for postgres,upgrade,__RUN_UNTIL_SUCCESS",
which then rolls the update back — so the update fails identically on
every retry.

getBaseDaemons is the only place in the stack that starts Postgres without
this prelude. The SDK's Backups.withPgDump does it before each pg_ctl start
(lib/backup/Backups.ts), and the 0.3.5x migration's relocatePostgres does
it too — the daemon path was simply missed.

Removal is unconditional, which is safe because nothing else can hold the
data directory: backups only run once the service is stopped, both init
paths run with the service stopped, and the chain reconciler fully terms an
entry before starting its replacement. Runs as root so ownership can never
block the unlink and wedge the chain on the new oneshot.

Placing it in getBaseDaemons covers setupMain, install init and update init
in one place.
@MattDHill

Copy link
Copy Markdown
Member Author

Superseded by #121, which folds this change in as its second commit.

Two reasons to consolidate rather than land both: they conflict mechanically (both touch utils.ts and versions/current.ts, and both bump 33.0.6:3), and this PR's original rationale did not survive the evidence — the user's service logs show Database is uninitialized and superuser password is not specified, an empty PGDATA, not a stale lockfile. That root cause is fixed in #121's first commit.

The change itself is retained because the mechanism is real — Postgres treats SIGTERM as a smart shutdown and the SDK kills at 60s, so an unclean stop can strand postmaster.pid — but it now travels as clearly-labelled hardening with no claimed incident behind it, in its own commit so it can be dropped independently.

@MattDHill MattDHill closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant