ci: seed preview database locally then bulk-import to Turso#1144
Open
busbyk wants to merge 1 commit into
Open
Conversation
The preview deploy seeded each PR's fresh Turso DB by running seed:standalone with DATABASE_URI pointed at remote Turso. Payload's seed does thousands of per-row create() calls, so against a remote libSQL database every row is a network round-trip (~289s on ubuntu-latest, and long enough on higher-latency runners to lose the connection mid-seed). Seed into a local SQLite file instead, then bulk-import that file into the fresh Turso preview DB in a single operation: - Seed step overrides NODE_ENV off `production` so the SQLite adapter runs in push mode and builds the current schema directly from the Payload config — the same fast path the CI build job uses. Running migrations from zero isn't viable here: the backfill_document_references data migration reads collections whose block tables a later migration adds, so the query hits `no such table` on a fresh DB. - Import with `turso db create --from-dump` (a sqlite3 .dump) rather than `--from-file`, which requires WAL mode and mis-parses valid WAL databases on sqlite3 >= 3.50 (tursodatabase/turso-cli#1030). The end state is identical — a fully seeded Turso preview DB — but the seed/import drops from ~5 min to seconds and no longer depends on sustained low-latency round-trips, so it works on any runner. Refs #1143 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Preview deployment: https://busbykxpreview-seed-local-import.preview.avy-fx.org |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
previewdeploy seeded each PR's fresh Turso (libSQL) database by runningpnpm seed:standalonewithDATABASE_URIpointed at remote Turso. Payload's seed performs thousands of individualcreate()calls, and against a remote database each one is a network round-trip — so the step is latency-bound (~289s onubuntu-latest, and long enough on higher-latency runners to lose the connection mid-seed).This seeds into a local SQLite file instead, then bulk-imports that file into the fresh Turso preview DB in a single operation. The end state is identical — a fully seeded Turso preview DB — but the seed/import drops from ~5 min to seconds and no longer depends on sustained low-latency round-trips, so it's runner-agnostic.
Related Issues
Refs #1143
Key Changes
file:./preview-seed.dband overridesNODE_ENVoffproductionso the SQLite adapter runs in push mode and builds the current schema directly from the Payload config — the same fast path the CIbuildjob uses.20260505_..._backfill_document_referencesdata migration reads collections via the Local API, whose query is built from the current config and references block tables (e.g.pages_blocks_form_embed) that a later migration adds — so on a fresh DB the query hitsno such tableand aborts. The old remote-Turso path would hit this too.turso db create --from-dump(asqlite3 .dump), reordered to run after the local seed.--from-dumpis used rather than--from-file/db import, which require WAL mode and mis-parse valid WAL databases on sqlite3 ≥ 3.50 (tursodatabase/turso-cli#1030); a plain SQL dump is runner-agnostic.Run database migrations+Seed the database). Vercel env wiring, build, deploy, and aliasing are unchanged — the deployed preview still points at the populated Turso DB.runs-on: ubuntu-latest; moving preview onto a Blacksmith runner is the trivial follow-up called out as out-of-scope in Speed up preview seeding: seed locally, then bulk-import to Turso #1143.How to test
The preview workflow runs on this PR (
opened/synchronize), so its own preview deployment is the end-to-end check: after the Preview environment is approved, confirmhttps://<ref>.preview.avy-fx.orgcomes up fully seeded (tenants, users, pages) and that theSeed a local SQLite database+Create the preview database from the seeded filesteps complete in well under a minute.Local verification performed while developing:
sqlite3 .dump(~1.5 MB) reloaded into a fresh DB preserves all 248 tables and identical row counts — a faithful proxy for Turso's--from-dump, which replays the same SQL. (A live Turso round-trip wasn't run locally; the PR's preview run exercises it for real.)Screenshots / Demo video
n/a
Migration Explanation
No schema/migration changes — workflow-only.
Future enhancements / Questions
previewjob (now that seeding is fast and runner-agnostic) — trivial follow-up per Speed up preview seeding: seed locally, then bulk-import to Turso #1143.🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.