feat(server): idempotent bulk message ingest + count for backfill parity#14
Open
andrei-hasna wants to merge 6 commits into
Open
feat(server): idempotent bulk message ingest + count for backfill parity#14andrei-hasna wants to merge 6 commits into
andrei-hasna wants to merge 6 commits into
Conversation
… parity
Adds POST /v1/messages/bulk so the authoritative local message store can be
backfilled into the cloud Postgres to full parity without duplicates:
- Preserves each message's uuid and created_at (and every scalar field) so the
cloud copy mirrors local history rather than a batch of "now"-stamped rows.
- Idempotent via ON CONFLICT (uuid) DO NOTHING — re-running a batch inserts 0.
- Returns { requested, inserted, skipped, total } where total is the
authoritative post-insert count, the API-visible parity signal.
- Guards with the conversations:write scope; caps batches at 2000.
- Adds GET /v1/messages?count=1 to read the total (filters honoured) without
paging through every row (list is capped at 500).
- pg-migrations: idempotent CREATE UNIQUE INDEX on messages(uuid) to guarantee
the ON CONFLICT target exists even on tables where uuid was added via ALTER.
- openapi + regenerated SDK (19 ops); api.test.ts covers idempotency, scope,
validation, and count.
…ildable) HEAD bumped @hasna/contracts to ^0.5.0 (for the client cloud-store path added in #13) but never regenerated bun.lock, which still pins 0.4.1 — so `bun install --frozen-lockfile` (the ECS Docker build) cannot resolve and the image is unbuildable. The serve process imports only @hasna/contracts/auth (present in 0.4.1) and the vendored storage-kit (self-contained, KIT_VERSION 0.4.1); it never imports the client-only cloud-store.ts. The deployed prod image already runs 0.4.1 auth, so existing API keys stay valid. Pinning to ^0.4.1 restores a coherent, buildable state for the server deploy. The client cloud-store.ts (which needs contracts 0.5.x `client/storage`) is part of the separate client-flip workstream: it will move to 0.5.x once contracts 0.5.x is published and the lockfile regenerated.
The CodeBuild fleet image builder pulls the base image anonymously from docker.io, whose shared-IP rate limit (HTTP 429 Too Many Requests on oven/bun:1) reliably fails the build. Switch the base to AWS's credential-free ECR Public mirror of the Debian Official Image and install a pinned Bun via the official installer. oven/bun:1 is itself Debian-based, so the runtime is functionally equivalent while removing the docker.io dependency.
Contributor
Author
Shipped + backfill verified live (self_hosted, acct 789877399345, us-east-1)Deployed: Migration: one-shot Auth safety: existing API key still returns HTTP 200 on Backfill (local messages.db → cloud, idempotent):
Build path self-healed: CodeBuild's anonymous docker.io pull of Note: this backfill closes the message split-brain (cloud was ~9). The client read-path flip across machines remains a separate, atomically-gated step. |
…rver read/receipt endpoints Client (cloud-store): add cloud-routed readMessages, searchMessages, readDigest, markRead/markReadByIds/markAllRead/markChannelRead/markSessionRead, markUnreadByIds, listUnreadCounts, editMessage, pin/unpinMessage, getPinnedMessages, recordReadReceiptsBatch, getReadReceipts. Each dispatches to https://conversations.hasna.xyz/v1 with the bearer key when self_hosted, else falls through to the local SQLite store (fully reversible). Coerce Postgres bigint ids to numbers in parseMessage so cloud rows match the Message shape. Wire the MCP tools (messaging, channels), the MCP notification poller, and the CLI commands (read/channel/search/digest/mark/edit/pin/pinned/watch) to the routed cloud-store facade so bin/mcp.js and bin/index.js actually hit the cloud (previously 0 cloud refs in the MCP bundle). Refactor readDigest into a storage-agnostic assembleDigest packer so the local and cloud digest paths share one byte-budget algorithm. Server (api.ts): extend GET /v1/messages with since, since_id, order, offset, unread_only, q (ILIKE search), threads_only, mentions_only, include_reply_counts; add POST /v1/messages/read + /unread, GET /v1/messages/unread-counts, GET /v1/messages/pinned, GET/POST /v1/messages/:id/receipts, POST /v1/messages/:id/pin|unpin, and PATCH /v1/messages/:id (edit). All additive and backward-compatible. Bump to 0.5.0.
…ilter - MCP advanced.ts and CLI locks.ts lock-conflict auto-DMs now route through cloud-store sendMessage (self_hosted -> cloud API, else local), removing the last message-write split-brain when the fleet is flipped. - server GET /v1/messages now honours ?uuid= for exact membership checks so backfill parity can be measured against the cloud uuid set (was ignored). - read-only advanced tools (thread replies, mentions, message-read-status, getMessagesForAgent) still read local: documented residual, need new cloud endpoints; non-corrupting (read-only).
Report cloud-served message/unread counts and the API URL when HASNA_CONVERSATIONS_* self_hosted routing is active, instead of the stale local sqlite path. Falls back to the local db when not in cloud mode. Adds cloudStatus()/cloudApiUrl() helpers to cloud-store.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
POST /v1/messages/bulkandGET /v1/messages?count=1to the self_hosted conversations API so the authoritative local message store can be backfilled into the cloud Postgres to full parity.Why
The fleet cutover exposed a split-brain: cloud had ~9 messages vs ~36k local. The only write path was
POST /v1/messages, which can't setuuid/created_at(so it's neither idempotent nor history-preserving) and inserts one row per request. This adds a proper backfill path.How
uuid+created_at(and all scalar fields).ON CONFLICT (uuid) DO NOTHING— re-running a batch inserts 0.{ requested, inserted, skipped, total };totalis the authoritative post-insert count.conversations:writescope required; batches capped at 2000.GET /v1/messages?count=1reads the total without paging (list caps at 500).messages(uuid)to guarantee the conflict target.Verification
bun run typecheckcleanbun test src/server/api.test.ts→ 13 passbun test src/lib/pg-migrations.test.ts→ 10 pass