Migration numbering collides across parallel branches: two migrations numbered 26, one silently never applies

[granludo]

What happened

Two branches independently claimed migration number 26:

• integration/kbserver-lamb-dev-main (Integrate Knowledge Stores, Library Manager & RAG citations into dev (#337, reconciled with #425) #456) — _migration_26 creates the knowledge_stores and kb_content_links tables.
• feature/creator-api-keys — _migration_26 creates the api_keys table for the OpenAI-compatible facade.

The runner applies only migrations above the recorded schema version. So whichever branch touches a database first records version 26, and the other branch's migration 26 is skipped forever on that database. Its tables never appear, and nothing reports an error — the schema version claims to be current.

This is not hypothetical. On the shared dev database the api_keys migration ran on 2026-07-08 at 16:09. LAMB_api_keys exists, LAMB_knowledge_stores does not, and the schema version reads 26. Every attempt to create a Knowledge Store failed against a table that was never created — reported, unhelpfully, as a name collision (filed separately).

migrations.py merges cleanly between the two branches, because they add functions at different line ranges. Git cannot see that two functions with the same number are a conflict, so neither review nor CI catches it.

Immediate fix

Applied on integration/456-merge: api_keys keeps 26 (it is already applied in the wild) and Knowledge Stores moves to 27, which applies cleanly on a database that has seen either branch. Version 26 is left as a documented gap.

The real problem

Sequential integers do not survive parallel branches. The next two people to add a migration in the same week will hit this again, and the failure is silent — a missing table, no error, and an error message pointing somewhere else entirely.

Worth considering:

1. Non-sequential identifiers — timestamp or ULID-based migration ids with a recorded set of applied ids rather than a high-water mark. This is what makes the problem structurally impossible rather than merely unlikely.
2. A CI guard — fail the build when two migration functions share a number, or when LATEST_VERSION does not match the highest defined migration. Cheap, catches the collision at PR time.
3. A startup assertion — after migrations run, verify the tables each migration claims to create actually exist, and log loudly if not. This would have surfaced the problem months before a user did.

At minimum (2) should land before more parallel feature branches carry migrations.

[granludo]

Unpacking this — the collision is three-way, not two-way

Surveying every live branch's migrations.py:

Branch	LATEST_VERSION	What it defines above 25
dev / aac-refactoring / next	25	—
integration/kbserver-lamb-dev-main (#456, Noa)	26	26 = knowledge_stores + kb_content_links
feature/creator-api-keys	26	26 = api_keys
lamba_refactor_merge_kvcache_v2_with_cost_management (#459, Francesc)	29	26 = lti_activities columns · 27 = knowledge_stores + kb_content_links · 28 = model_pricing cache columns · 29 = cost columns + backfill

So three different migrations currently claim number 26 — knowledge stores, api_keys, and the lti_activities columns. Whichever reaches a given database first wins, and the other two are skipped on that database permanently, silently, with the schema version reporting "up to date".

Two further observations that matter more than the count:

Francesc's branch already renumbered knowledge stores to 27. He carries Noa's work (56 commits mentioning it) on an independent lineage — git merge-base --is-ancestor confirms neither branch descends from the other — and in doing so assigned knowledge stores to 27. The fix applied here (26 → 27) lands on the same number by coincidence, which is convenient but is exactly the kind of luck a numbering scheme should not depend on.

Nothing accounts for api_keys in Francesc's sequence. His 26 is lti_activities. When feature/creator-api-keys eventually merges, its migration 26 will be skipped on any database that has already seen his 26 — the same failure, one branch later.

Blast radius

The good news is that it is still ahead of us, not behind us. dev is at 25, so any deployment tracking dev — including production instances — is at schema version ≤ 25 and has not yet been exposed. The corruption we hit exists only on the shared development database, which saw feature/creator-api-keys on 2026-07-08 and is therefore stuck at 26 = api_keys with the knowledge-store tables absent.

That gives a clear deadline: the numbering must be reconciled before #456, #459, or feature/creator-api-keys merges anywhere that a production database will follow. Whichever merges first silently defines 26 for every database downstream, and the losers never run again. Since all three are queued to land on the same integration branch, the reconciliation has to happen once, deliberately, across all three — not branch by branch.

Why the current design cannot survive this

_get_current_version takes MAX(version) — a high-water mark, not a set. The consequences follow directly:

1. A migration whose number is below the high-water mark never runs, whatever it contains.
2. Gaps are unrecoverable: once the mark passes a number, that number is dead on that database.
3. A missing migration method is logged and skipped (continue), so a gap does not stop the run — the mark still advances past it on the next successful migration.
4. Nothing verifies afterwards that a migration produced what it claimed. The tables it was supposed to create are simply absent, and the first symptom is a runtime error somewhere unrelated — in our case a 409 about a name.

Sequential integers assigned by hand are a coordination protocol between humans. With four people on four branches and agents generating code, that protocol has already failed once.

Recommended reconciliation, in order

1. Decide the 26–30 sequence once, across all three branches, and apply it to each: something like 26 = lti_activities, 27 = knowledge_stores, 28 = model_pricing cache, 29 = cost columns, 30 = api_keys. Francesc's branch already matches for 26–29; only api_keys needs to move.
2. Add a CI guard: fail when two migration methods share a number, and when LATEST_VERSION does not equal the highest defined method. Both are a few lines and would have caught this at PR time.
3. Add a post-migration assertion: after the run, check that the tables each migration claims to create exist; log loudly otherwise. This turns a silent skip into a visible failure at startup rather than a mystery 409 weeks later.
4. Then consider replacing sequential integers with timestamp or ULID identifiers and a recorded set of applied ids. This makes the whole class of failure structurally impossible instead of merely guarded against. It is the real fix; 1–3 are what make the current release safe.

[granludo]

The checkpoint harness found a hole in the agreed numbering — a gap can never be filled

Built a migration checkpoint harness (testing/migrations/checkpoint.py) that creates a database at a chosen schema version the way a real install does, then upgrades a copy and reports what the chain actually produced. Marc's point, and it is the right one: re-running a migration that already ran proves nothing, because the guards make it a no-op. The case that breaks is a deployment that has never seen the new migrations.

Running it against a v25 baseline — which is where dev and therefore production sit today — through the renumbered chain:

upgrade: v25 -> v30
  new tables (2): LAMB_kb_content_links, LAMB_knowledge_stores
  new columns on LAMB_lti_activities: activity_type, lis_outcome_service_url, setup_config
  new columns on LAMB_model_pricing: cache_read_per_1m, cache_write_per_1m, cached_input_per_1m, requires_explicit_cache
  new columns on LAMB_assistant_usage_totals: cache_read_tokens_total, cache_write_tokens_total, cached_prompt_tokens_total, non_cached_prompt_tokens_total
  new columns on LAMB_usage_logs: cost_usd
  WARNING: versions never applied on this database: [26]

The upgrade works. But look at the warning.

Version 26 — api_keys — is left as a gap, and on this runner a gap is permanent. _get_current_version returns MAX(version), and the loop is range(current_version + 1, LATEST_VERSION + 1). Once a database reaches 30, current_version is 30 and nothing below it is ever considered again. So when feature/creator-api-keys later merges with its migration 26, that database will report itself up to date and the api_keys table will never be created — the exact failure this issue was opened about, reproduced by the fix for it.

This is only invisible because the number happens to be missing rather than wrong. The agreed assignment is fine as an ordering; it is the runner that cannot express it.

Two ways out, and I would take the second:

1. Move api_keys above the ceiling — make it 31 instead of 26. Works, costs nothing, and is pure luck-management: it holds only until the next pair of branches interleaves.
2. Make the runner use the applied set instead of a high-water mark. Iterate over every defined migration whose number is not present in schema_version, in ascending order, rather than everything above the maximum. Perhaps ten lines. It makes gaps fillable, so branches merging in any order converge on the same schema, and it removes the entire class of failure rather than dodging this instance of it.

With (2) in place, the numbering agreed here needs no further care: whichever branch lands first, the others' migrations still run when they arrive.

[granludo]

Fixed on aac-refactoring: the runner now tracks the applied set

apply_all compared against MAX(version) and ran everything above it, so a migration numbered below that mark could never run — whatever it contained, however the gap arose. A database that skipped one was stuck with it permanently while reporting itself up to date.

It now reads the set of applied versions and runs every migration the code defines that is not in that set, ascending, up to LATEST_VERSION. Versions are discovered from the methods rather than assumed contiguous, because branches developed in parallel leave gaps. LATEST_VERSION still acts as a ceiling, so building a database at a chosen version keeps working.

Consequence: branches carrying migrations can now merge in any order and converge on the same schema. A migration that arrives late still runs. The numbering agreed on this issue — 26 api_keys, 27 knowledge_stores, 28–30 from the KV-cache branch — no longer depends on merge order to be correct.

Verified rather than argued. Using the checkpoint harness: on a v25 database with migration 20's record deleted and the model_pricing table it creates dropped — a migration that genuinely never ran, below the high-water mark — the runner re-applies it, the table returns with its 10 seeded rows, and the version is recorded. The old runner would not have looked at it. Backend suite: 138 passed.

Still worth adding, and cheap: a CI check that fails when two migration methods share a number. The runner now survives collisions, but two migrations wearing the same number still means one of them is lost at merge time.