Move queue state out of analysis JSON into real columns

[kasnder]

Summary

The apps.analysis JSON column previously encoded four different things at once:

• NULL = queued
• {success:false, logs:'Processing in progress', timestamp} = worker lock
• {success:false, reason, retryable} = failure record
• anything else = completed analysis payload

This PR moves queue/lock/failure state into real columns and drives all scheduling off them, while keeping apps.analysis as the untouched raw analyser payload.

State machine

New columns on apps (migration 007):

• status text NOT NULL DEFAULT 'queued' CHECK IN (queued, processing, analysed, failed)
• processing_started timestamptz — when the current processing lock was taken
• failure_reason text, failure_retryable boolean — populated only for failed

Transitions:

• addApp inserts a row → queued (column default).
• nextApp picks a candidate (queued, expired processing lock, stale analysed, or retryable failed) with FOR UPDATE SKIP LOCKED, then sets status='processing', processing_started=NOW(). This lock is non-destructive — it no longer overwrites analysis, so the last good result stays visible on the website during a refetch.
• updateAnalysis writes the new payload and sets status='analysed' or status='failed' (+ failure_reason/failure_retryable) via the pure, unit-tested deriveAnalysisState helper. It still writes the raw payload to apps.analysis and still snapshots history into app_analyses (ON CONFLICT DO NOTHING).

Migrations & safety

Migrations run on deploy before the new code boots, and must tolerate the currently deployed code running briefly against the migrated schema. All three are additive/backward-compatible:

• 007 — additive columns with defaults + a backfill from the existing JSON state (order matters: the processing indicator also carries success=false, so it is classified first). Malformed processing timestamps are regex-validated and fall back to NOW() so a bad value can't abort the migration. Partial indexes back the queue scan. Old code ignores these columns.
• 008 — json→jsonb and timestamp→timestamptz (values assumed UTC via AT TIME ZONE 'UTC'). The pg node driver treats json/jsonb and timestamp/timestamptz equivalently on both read and write, so the still-running old code keeps working.
• 009 — enforce case-insensitive bundle IDs: merge rows differing only by case (dedupe colliding app_analyses rows around the (appid, analysed) unique index before repointing history to the keeper), then a unique index on lower(appid). No appid values are rewritten, so the existing validated apps_appid_valid CHECK still holds.

Migration ordering 007 → 008 → 009 is enforced by the runner's sorted-filename ledger.

During the brief old-code window a row may end up with status='queued' but an old Processing in progress JSON marker (or vice-versa); this self-heals on the next nextApp and is harmless (idempotent, SKIP LOCKED).

API freeze — Raspberry Pi analyser requires NO changes

The analyser talks to this server only over HTTP. All endpoints it uses are byte-identical in shape:

• /queue still returns app.appid (nextApp returns {appid}).
• /uploadAnalysis and /reportAnalysisFailure still flow through updateAnalysis, which returns the same pg result (RETURNING appid, details, analysed) and still writes the failure JSON to apps.analysis so the website's failure display keeps working.
• /ping unchanged.

Scripts

queue-refetch, priority-report, and queue-status now classify state via the status columns. queue-refetch also resets status='queued' and clears the lock/failure columns when queueing a refetch.

Tests

npm test (node --test): 22/22 pass, including new deriveAnalysisState coverage. No local Postgres was available, so the migration SQL was validated by careful review rather than execution.

🤖 Generated with Claude Code

[kasnder]

Independent review of the subagent's work — three issues found and fixed in df880c2:

1. Branch was cut before Run migrations on startup for hands-off Railway deploys #10 merged, so the PR as pushed would have silently reverted the migrate-on-start package.json change. Rebased onto current main (clean; this branch never touches package.json).
2. Failure-retry regression: the old JSON predicate only retried retryable failures once analysisversion was outdated or the analysis passed the staleness cutoff. The port made status='failed' AND failure_retryable immediately eligible, so a popular, persistently failing app would be re-handed to the analyser in a tight loop and starve the queue (failures are recorded with the current version via /reportAnalysisFailure). Restored the version/staleness gate in nextApp, priority-report, and queue-status (×2), and shaped the failed partial index to match.
3. addApp conflict target: after 009's unique(lower(appid)), a race between case-variants of the same bundle ID would raise a unique violation not covered by ON CONFLICT (appid). Switched to a bare ON CONFLICT DO NOTHING.

Verified clean: 007 backfill CASE order (processing-before-failed), timestamp regex fallback, 008 UTC reinterpretation, 009's dedupe-before-repoint around the (appid, analysed) unique index (handles loser-vs-loser and loser-vs-keeper collisions; NULL analysed rows can't collide), API shapes for the Pi unchanged, tests 22/22.

Remaining known risk (unchanged from the PR description): migrations reviewed but not executed against a real Postgres — recommend eyeballing the Railway deploy logs when this merges.