Skip to content

feat(storage): replicate messages + read receipts across the fleet (uuid-keyed incremental sync, interim shared storage)#11

Open
andrei-hasna wants to merge 4 commits into
mainfrom
feat/fleet-comms-shared-storage
Open

feat(storage): replicate messages + read receipts across the fleet (uuid-keyed incremental sync, interim shared storage)#11
andrei-hasna wants to merge 4 commits into
mainfrom
feat/fleet-comms-shared-storage

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Why

Fleet comms Phase 0 blocker (todos task 1e723ce4, tracking 843cfb62, strategy wf_09ba3f51-f00): conversations is a per-machine SQLite island. messages and message_read_receipts sit in SYNC_EXCLUDED, so a message posted on spark01 is invisible on apple01 — every comms design assumption (durable channel record, [FREEZE] visibility, ack receipts) is invalid until they replicate.

They were excluded for a real reason: SQLite messages.id is per-machine AUTOINCREMENT — ids collide fleet-wide, so the generic full-table pk-upsert engine would corrupt them.

What

  • New src/lib/message-sync.ts — uuid-keyed incremental append-only replication of messages + message_read_receipts through a shared hub Postgres. reply_to and receipt message_id translate through the parent message's uuid on each side; per-machine cursors live in the local never-synced _message_sync_state table.
  • v1 semantics (documented in docs/FLEET-SHARED-STORAGE-INTERIM.md): append-only; on re-sync content wins by newer edited_at, read_at/pinned_at set-once; receipts hold the cursor back until their message replicates; TIMESTAMPTZ params sent zone-explicit so hub session TZ can never shift instants. Known limitation: steady-state edits/deletes/pins do NOT re-send (no change log yet — v2 follow-up; secret-purge must run per machine until then).
  • Wiring: default table set for storage push/pull/sync now includes the two tables (--no-messages opts out); storage status shows cursors/counts; syncPush/syncPull route message tables to the new engine.
  • db.ts uuid repair pass on open — long-lived DBs carry NULL uuids from an older migration that added the column without backfill (627 on spark01); uuid is the replication key so it must be total + unique.
  • A1 coherence: module is marked CUTOVER: gate off with the rest of the legacy sync engine; the pure-remote flip lane removes it (see docs/CUTOVER-RUNBOOK.md — unchanged).

Verification

  • bun test: 979 pass / 0 fail (16 new tests: round trip, bidirectional exchange, reply_to remapping across id spaces, edit LWW, receipt translation + holdback, uuid backfill idempotency, TZ normalization). tsc --noEmit clean.
  • Live: on-prem hub (spark01 PG16, db hasna_conversations), spark01 + spark02 configured, cross-machine canary read-back — evidence in todos task 1e723ce4 comments.

Rollout / rollback

spark01 + spark02 first; apple01/03/06 staged (runbook included). Rollback = delete storage/config.json (runtime is still local SQLite; hub down never blocks an agent). NO AWS involved (gate 97610c99 closed).

@andrei-hasna
feat(storage): replicate messages + read receipts via uuid-keyed incr…
59a4e20 
…emental sync (interim fleet shared storage)

Conversations was a per-machine SQLite island: messages and
message_read_receipts were excluded from the storage sync surface, so a
message posted on one machine was invisible everywhere else (fleet comms
Phase 0 blocker, todos 1e723ce4 / tracking 843cfb62).

- NEW src/lib/message-sync.ts: uuid-keyed incremental append-only
  replication of messages + message_read_receipts through the shared hub
  Postgres. Per-machine integer ids collide fleet-wide, so reply_to and
  receipt message_id translate through the parent message uuid on each
  side; per-machine cursors live in local never-synced _message_sync_state.
- v1 conflict semantics: append-only; on re-sync content wins by newer
  edited_at, read_at/pinned_at are set-once. Steady-state edits/deletes do
  not re-send (documented; v2 change-log follow-up). Receipts hold the
  cursor back until their message replicates. TIMESTAMPTZ params are sent
  zone-explicit and normalized back to naive-UTC text on pull.
- storage push/pull/sync default table set now includes messages +
  message_read_receipts; --no-messages opts out; storage status shows
  message-sync cursors. syncPush/syncPull route message tables to the new
  engine (generic pk-upsert engine still never touches them).
- db.ts: idempotent uuid repair on open (backfill NULL/duplicate uuids +
  unique index) — 627 NULL-uuid rows observed on spark01 from an older
  migration that added the column without backfill.
- docs/FLEET-SHARED-STORAGE-INTERIM.md: on-prem hub architecture, staged
  satellite runbook (spark02 done; apple01/03/06 staged), rollback.
  Marked CUTOVER: gate off with the rest of the legacy sync engine (A1).
@andrei-hasna
fix(storage): heal pull-cursor races with an id re-scan margin; make …
2ec7e6d 
…storage subpath test machine-independent

- Hub ids are assigned at INSERT but visible at COMMIT: with two machines
  pushing concurrently, a row can appear behind an already-advanced pull
  cursor and be skipped forever. pullMessages now re-scans a fixed id
  margin (200) behind the saved cursor each run; the WHERE-gated conflict
  clause makes the overlap idempotent and free. Regression test simulates
  the late-visible row.
- storage-sync.test.ts asserted getStorageDatabaseUrl() is null, which
  read the machine's real storage/config.json — fails on any machine with
  remote storage configured (as fleet machines now are). Force local mode
  via env in the test; also cover the new message-sync exports.
@andrei-hasna

andrei-hasna commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Adversarial self-review findings (reconciled in 2ec7e6d) — for the independent reviewer:

  1. Concurrent-push/pull id-visibility race (CONFIRMED, fixed): hub ids are assigned at INSERT but visible at COMMIT, so a puller running while two machines push could advance its cursor past a not-yet-visible row and skip it permanently. Fixed with a 200-id re-scan margin behind the saved pull cursor (idempotent via the WHERE-gated conflict clause) + regression test simulating a late-visible row.
  2. Test isolation (CONFIRMED, fixed): storage-sync.test.ts asserted getStorageDatabaseUrl() is null, which reads the machine's real ~/.hasna/conversations/storage/config.json — fails on any machine with remote storage configured (fleet machines now are). Forced local mode via env in the test. Full suite re-run with live config present: 980 pass / 0 fail.
  3. Known v1 limitations (documented, accepted): append-only steady state (edits/deletes/pins/DM-read don't re-send — secret purge must run per machine until a v2 change log); channel_subscriptions.since_message_id syncs raw per-machine cursors (pre-existing legacy engine behavior); first sync re-pushes pulled rows as hub-side no-ops (pull-then-push with cursor 0 — one-time cost, observed on spark02's initial sync).

Live evidence so far: hub landed exactly 24,403 (spark01) + 11,295 (spark02) = 35,698 messages — zero uuid collisions across machines; 627 legacy NULL uuids repaired on spark01 by the new open-time repair pass. Cross-machine canary read-back running now; results in todos 1e723ce4.

@andrei-hasna
fix(storage): propagate agent removal through sync via tombstones
aa6eff4 
The legacy pk-upsert engine is append-only, so 'agents remove' on one
machine was resurrected by the next storage pull. This silently undid the
2026-07-06 supervised registry purge (579 -> 98 agents rebounded to 583;
todos bc244f4d x 1e723ce4, confirmed by both rollout adversaries).

- new _sync_agent_tombstones table (SQLite lazy-create + PG migration 3)
- removePresence() records a tombstone; renameAgent() tombstones the old
  name, bumps last_seen_at, and clears a stale tombstone on the new name
- storage push uploads tombstones and deletes remote agent_presence rows
  older than their tombstone; storage pull downloads tombstones and
  reconciles local rows the same way (including rows the same pull just
  resurrected). Re-registered/heartbeating agents always outlive tombstones
- tombstone timestamps use the legacy engine's raw-naive convention so
  comparisons stay internally consistent (documented caveat)
- runbook: mandatory pre-cutover registry groom per machine + semantics
- 13 new tests (full suite 992 pass / 0 fail)

Task: 843cfb62 (fleet comms rollout, fixer stream); refs bc244f4d, 1e723ce4
@andrei-hasna
fix(storage): presence sync converges on agent identity with last-see…
36e42ac 
…n-wins guard

The presence upsert conflict target was (agent, project_id) but both stores
enforce uniqueness on agent alone; a name reused with a different project_id
on another machine took the INSERT path, violated
idx_agent_presence_agent_unique, and aborted the entire presence table sync
(spark02's registry push never converged during the 2026-07-06 cutover, and
the re-purge convergence hit the same wall in both directions).

- conflict target for agent_presence is now the agent name
- new per-table UPDATE_GUARDS: presence updates only apply when
  excluded.last_seen_at > agent_presence.last_seen_at, so a stale replica
  can never regress fresher presence on push OR pull
- test DDLs mirror production's idx_agent_presence_agent_unique; 2 new tests
  (994 pass / 0 fail)

Task: 843cfb62 (fleet comms rollout, fixer stream); refs bc244f4d, 1e723ce4
@andrei-hasna andrei-hasna mentioned this pull request Jul 23, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrei-hasna