fix: clear stale postmaster.pid before starting postgres (33.0.6:3) - #120
fix: clear stale postmaster.pid before starting postgres (33.0.6:3)#120MattDHill wants to merge 1 commit into
Conversation
…33.0.6:3 PostgreSQL writes postmaster.pid into PGDATA while running and removes it on a clean exit. A stop that never completes leaves it behind, and on the next start Postgres aborts with FATAL: lock file "postmaster.pid" already exists if the PID recorded there is alive. Every chain build gets a fresh PID namespace with a fresh, low PID assignment, so that PID is quite likely to be live and owned by an unrelated process — the guard misfires and the daemon never comes up. Nothing surfaces when it happens: the postgres ready check maps any non-zero pg_isready to 'loading' with display: null, and Daemon.runLoop restarts the process forever on a backoff. In setupMain that shows as a service stuck starting. In the update-time upgrade chain added in 33.0.6:0 it shows as runUntilSuccess exhausting its 30-minute budget and failing init with "Timed out waiting for postgres,upgrade,__RUN_UNTIL_SUCCESS", which then rolls the update back — so the update fails identically on every retry. getBaseDaemons is the only place in the stack that starts Postgres without this prelude. The SDK's Backups.withPgDump does it before each pg_ctl start (lib/backup/Backups.ts), and the 0.3.5x migration's relocatePostgres does it too — the daemon path was simply missed. Removal is unconditional, which is safe because nothing else can hold the data directory: backups only run once the service is stopped, both init paths run with the service stopped, and the chain reconciler fully terms an entry before starting its replacement. Runs as root so ownership can never block the unlink and wedge the chain on the new oneshot. Placing it in getBaseDaemons covers setupMain, install init and update init in one place.
|
Superseded by #121, which folds this change in as its second commit. Two reasons to consolidate rather than land both: they conflict mechanically (both touch The change itself is retained because the mechanism is real — Postgres treats SIGTERM as a smart shutdown and the SDK kills at 60s, so an unclean stop can strand |
Change
getBaseDaemonsis the only place in the stack that starts PostgreSQL without first clearing a stalepostmaster.pid:Backups.withPgDump→startPgdoes it before eachpg_ctl start(lib/backup/Backups.ts)relocatePostgresdoes it (startos/versions/current.ts)PostgreSQL writes
postmaster.pidintoPGDATAwhile running and removes it on a clean exit. A stop that never completes leaves it behind. On the next start PostgreSQL reads the PID it names and aborts withFATAL: lock file "postmaster.pid" already existsif that PID is alive. Each chain build runs in a fresh PID namespace with a fresh, low PID assignment, so the recorded PID can be live and owned by an unrelated process — the guard misfires.Adds a
pg-recoveroneshot, withpostgresgainingrequires: ['pg-recover']. Placing it ingetBaseDaemonscoverssetupMain, install init and update init in one place.Removal is unconditional, which is safe because nothing else can hold the data directory: backups only run once the service is stopped (
service/mod.rs— "the backup never starts before the service is stopped"), both init paths run with the service stopped, and the chain reconciler fully awaitshealthDaemon.term()andsubcontainer.destroy()before starting a replacement entry. It runs asrootso ownership can never block the unlink and wedge the chain on the new oneshot.Correction
The originating report was a user stuck updating with
Timed out waiting for postgres,upgrade,__RUN_UNTIL_SUCCESS. I attributed it to a stale lock file. The service logs show a different failure:That is the postgres entrypoint's
docker_verify_minimum_env, which only runs when$PGDATA/PG_VERSIONis absent. PostgreSQL is not failing to open an existing cluster — there is no cluster there at all, andupgradeNextcloudpassespostgresEnvwithoutPOSTGRES_PASSWORD(unlike the install path), so the entrypoint refuses to initialize one and exits 1 on a loop until the 30-minute timeout.The logs also show the user is migrating from 32.0.11:0, not 33.0.5:1. Their first attempt ran the real 0.3.5x migration —
migrateNextcloud's recursive chmod walk — for 53 minutes across 54,400 directories before being SIGTERM'd mid-walk. Every attempt after that skips the 0.3.5x block entirely in ~3 seconds, so the Postgres cluster is never relocated out of the Debian17/mainlayout while the migration believes it has already run.Root cause is therefore in the 0.3.5x migration: its completion marker (
start9/config.yaml) and the work it gates are not committed atomically, so an interruption leaves the two permanently disagreeing. That needs a separate fix, tracked separately.Scope
Test plan
33.0.6:3on a StartOS 0.4.0 box and confirm Nextcloud starts, the web UI loads, and you can log in.postmaster.pidis still present:start-cli package attach nextcloud -n postgres-sub -- ls /var/lib/postgresql/data/postmaster.pidlock file "postmaster.pid" already exists.