diff --git a/AGENTS.md b/AGENTS.md index 0e4343e..0aae991 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,8 @@ # Contents -- `src/` – the deterministic local iMessage and Contacts readers, normalized - corpus and metrics, private SQLite store, profile parser, Agent Skill - installer, and `messagelikeme` CLI. +- `src/` – the deterministic local iMessage, Contacts, and private source-bundle + readers, normalized corpus and metrics, private SQLite store, profile parser, + Agent Skill installer, and `messagelikeme` CLI. - `schema/` – public versioned JSON Schemas for deterministic artifacts and agent-authored profiles. - `docs/` – public methodology, evidence limits, research review, and prior-art @@ -25,7 +25,7 @@ - Use Bun 1.3.14 and run `bun run check` before handing off a change. Do not add another package manager or lockfile. - Keep the public description exact: “A local-first CLI and Agent Skill for - studying your private iMessage history and drafting messages that sound like + studying private messaging history and drafting messages that sound like you.” - Keep the public repository independently buildable. Do not reference another source repository, private packages, sibling paths, private fixtures, or @@ -33,6 +33,12 @@ - Keep `chat.db` authoritative and ingestion read-only, query-only, ownership-checked, schema-validated, and bounded. Never modify Messages, contacts, attachments, or SQLite sidecars. +- Treat a `message-like-me.local-message-bundle` as an untrusted, private, + versioned directory boundary. Require its fixed inventory, canonical UTF-8, + owner-only modes, bounded records, artifact digests, and manifest digest. + Never let bundle absence erase retained history unless a future contract + explicitly declares authoritative coverage; apply explicit deletions and + tombstones separately. - Treat AddressBook databases as optional label-enrichment sources. Isolate every database plus WAL or journal before SQLite opens it, validate contact entities and property owners dynamically, read only names and exact @@ -54,7 +60,8 @@ - Keep the command name `messagelikeme`, the repository and package name `message-like-me`, and the Agent Skill name `message-like-me`. Treat `messagelikeme.com` as an informational project page, never as a data plane. -- Keep CLI commands namespaced as `ingest imessage|contacts`, +- Keep CLI commands namespaced as `ingest imessage|contacts|bundle`, + `sources list|show`, `contacts list|show|resolve`, `inspect tempo|sessions`, `study prepare`, `profile apply|show|export`, plus `init`, `context`, `skill`, and `doctor`. Machine-readable commands support diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a3245f..fbd19a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +## 0.3.0 + +- Add strict private version-one source-bundle ingestion for local Beeper + exports produced through Wrench, including account and network provenance, + replies, edits, deletions, attachments, reactions, and tombstones. +- Namespace corpus ownership by source so native iMessage and multiple provider + accounts coexist. Bounded snapshot absence retains prior history, explicit + terminal state suppresses evidence, and later reappearance restores it. +- Add `sources list` and `sources show` with active message, conversation, + reaction, undated-reaction, completeness, and warning health. +- Require Wrench 0.13.0 or newer for its direct official Beeper CLI path, + sequential per-account progress, retained-shard validation, and atomic + seven-file publication. +- Partition sessions, bursts, and response episodes by conversation, preserve + truncated text bubbles as tempo evidence, and count undated reactions without + inventing timestamps or exposing raw provider reaction values in aggregate + output. +- Upgrade existing version-two stores in place while retaining conversations, + profiles, study packets, and evidence provenance. + ## 0.2.0 - Aggregate conservatively matched direct threads into one AddressBook person diff --git a/README.md b/README.md index 8112bb7..0208ef6 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,14 @@ # Message Like Me -**A local-first CLI and Agent Skill for studying your private iMessage history -and drafting messages that sound like you.** +**A local-first CLI and Agent Skill for studying private messaging history and +drafting messages that sound like you.** -Message Like Me turns a local Messages database into deterministic conversation -metrics, bounded study packets, and reusable style profiles. Its Agent Skill -teaches Codex, Claude, and other coding agents how to interpret those local -artifacts and draft unsent replies in your voice. +Message Like Me turns private local messaging history into deterministic +conversation metrics, bounded study packets, and reusable style profiles. It +reads native iMessage history and strict local source bundles, including +multi-account Beeper exports produced through Wrench. Its Agent Skill teaches +Codex, Claude, and other coding agents how to interpret those local artifacts +and draft unsent replies in your voice. The CLI does not call an AI service, authenticate with a product account, send messages, or operate Messages. The agent already running the skill supplies the @@ -23,7 +25,7 @@ Message Like Me requires Bun 1.3.14 or newer. Install the immutable public release from GitHub, then install the Agent Skill: ```sh -bun add --global github:hraness/message-like-me#v0.2.0 +bun add --global github:hraness/message-like-me#v0.3.0 messagelikeme skill install ``` @@ -40,7 +42,7 @@ messagelikeme skill path Message Like Me is distributed directly through GitHub and is not published to npm. -## Start with your local history +## Start with private local history Initialize the private data store and inspect its location: @@ -79,6 +81,58 @@ copy of the database and its transactional sidecars, and opens only that copy with SQLite. It does not change Messages, `chat.db`, or its sidecars. macOS may require permission for the terminal or agent host to read Messages data. +To study accounts connected through Beeper, install or update to +[Wrench 0.13.0 or newer](https://github.com/hraness/wrench/releases), then ask +it to create a new private Message Like Me bundle: + +```sh +wrench beeper export-message-like-me \ + --auth \ + --output /absolute/private/path/beeper-bundle \ + --json +``` + +The optional `--limit-chats`, `--limit-messages`, and `--max-participants` +flags lower the export bounds. The output path must be a normalized absolute +path to a directory that does not already exist. Wrench calls the pinned +[official Beeper CLI](https://github.com/beeper/cli) directly. It enumerates +the connected account realm, invokes `export --no-attachments` once per +account in deterministic order, and reports the account ordinal, elapsed-time +heartbeats, and cumulative validated chat and message counts on stderr. It +retains each private raw shard until it can atomically publish the complete +mode-`0700` seven-file bundle with mode-`0600` files. + +The export does not use the separate +[Beeper Desktop API MCP project](https://github.com/beeper/desktop-api-mcp). +The CLI path supplies the bounded account snapshots and local files needed for +hash validation, deterministic conversion, crash recovery, and atomic +publication. Provider URLs and credentials are excluded. Message Like Me does +not receive the Beeper credential and does not call Beeper or Wrench itself. + +Ingest the finished directory, then inspect its redacted source health: + +```sh +messagelikeme ingest bundle --input /absolute/private/path/beeper-bundle --json +messagelikeme sources list --json +messagelikeme sources show --json +``` + +The importer verifies the fixed version-one inventory, canonical UTF-8 NDJSON, +record and byte bounds, owner-only permissions, artifact digests, and manifest +digest before changing the store. One bundle may contain several connected +accounts and networks; each becomes a separate source namespace. Native +iMessage and prior bundle sources remain alongside it. + +The complete interchange, integrity, identity, and reimport laws are in the +[version-one local message bundle contract](docs/local-message-bundle-v1.md). + +Beeper exports describe bounded local observations. A later bounded export +that omits an older record does not delete retained history. Explicit deletion, +removal, replacement, and tombstone records suppress their target, and a later +reappearance restores it. Older snapshots cannot overwrite newer state. Use +`sources show --private --json` only when you deliberately need the +private provider account and source metadata. + Optionally enrich and join direct conversations with private identities from macOS Contacts: @@ -96,16 +150,17 @@ messagelikeme ingest contacts \ --json ``` -Contacts ingest may run before or after iMessage ingest. It reads only bounded -name, email, and phone fields from a stable private copy. Exact normalized -email or phone handles can join several one-to-one iMessage, SMS, and email -threads for the same AddressBook person into one analysis scope. Existing -conversation IDs remain aliases for that person scope. Shared handles remain -ambiguous, local phone numbers never gain a guessed country code, unmatched -threads stay separate, and groups are never collapsed to one person. Contact -labels have their own revision, so a rename does not stale a messaging-style -profile. `messagelikeme doctor` reports local aggregate state without asking -for an account or credential. +Contacts ingest may run before or after any message source. It reads only +bounded name, email, and phone fields from a stable private copy. Exact +normalized email or E.164 phone handles can join several one-to-one threads +for the same AddressBook person into one analysis scope. A bundle conversation +is eligible only when the producer positively marks its direct participant +roster complete. Existing conversation IDs remain aliases for that person +scope. Shared handles remain ambiguous, local phone numbers never gain a +guessed country code, unmatched threads stay separate, and groups are never +collapsed to one person. Contact labels have their own revision, so a rename +does not stale a messaging-style profile. `messagelikeme doctor` reports local +aggregate state without asking for an account or credential. ## Inspect behavior without exposing prose @@ -124,8 +179,15 @@ outgoing turns, within-session response latency, single-message versus multi-message replies, surface prose features, multi-point response contexts, reactions, and explicit reply use. Incoming messages establish what you were responding to; they are never counted as examples of your writing style. -Session and burst gaps are configurable seconds and are recorded with each -result. They are segmentation choices, not universal facts about conversation. +Sessions, bursts, and response episodes never cross a source conversation +boundary. Person scopes spanning several apps expose a sorted `services` +breakdown instead of hiding the mixed-channel evidence behind a null service. +Reactions with no provider timestamp still contribute to reaction counts and +direction, but never to temporal metrics. Raw provider reaction values remain +private; ordinary metrics and drafting context expose only fixed-size counts, +direction, datedness, and the outgoing reaction ratio. Session and burst gaps +are configurable seconds and are recorded with each result. They are +segmentation choices, not universal facts about conversation. Pass `--private` to `contacts list` or `contacts show` only when you need to resolve a pseudonymous contact to its local private label or participants. @@ -244,6 +306,9 @@ Run `messagelikeme --help` for the checked grammar. The public surfaces are: messagelikeme init [--json] messagelikeme ingest imessage [--database PATH] [--json] messagelikeme ingest contacts [--addressbook PATH] [--json] +messagelikeme ingest bundle --input ABS_PATH [--json] +messagelikeme sources list [--private] [--json] +messagelikeme sources show SOURCE_ID [--private] [--json] messagelikeme contacts list [--min-outgoing N] [--limit N] [--private] [--json] messagelikeme contacts show CONTACT_ID [--private] [--json] messagelikeme contacts resolve QUERY --private [--limit N] [--json] @@ -271,10 +336,13 @@ Place global `--data-dir PATH` before the command. - The original `chat.db` and AddressBook databases remain authoritative. SQLite opens only stable private copies, never the source files or sidecars. +- Source bundles remain private caller-owned inputs. Import verifies their + fixed inventory, canonical bytes, digests, bounds, and owner-only modes. - The normalized corpus, profiles, and installation key stay in a private local store with owner-only permissions. -- Stable contact, conversation, and message IDs are derived with a private - per-install HMAC key. Pseudonymous IDs are not encryption. +- Stable source, contact, participant, conversation, message, and reaction IDs + are derived with a private per-install HMAC key. Pseudonymous IDs are not + encryption. - Aggregate commands omit bodies and private labels. Study and evaluation packets are bounded, explicit body-bearing exports. - Message text never goes to a Message Like Me server. There is no service, @@ -312,9 +380,9 @@ bun install --frozen-lockfile --ignore-scripts bun run check ``` -Tests use synthetic Messages and AddressBook databases plus synthetic -conversations. Never add a real message, handle, group title, attachment, -contact record, private path, or derived profile to a fixture. +Tests use synthetic Messages and AddressBook databases plus synthetic source +bundles and conversations. Never add a real message, handle, group title, +attachment, contact record, private path, or derived profile to a fixture. The canonical repository is [`hraness/message-like-me`](https://github.com/hraness/message-like-me). diff --git a/SECURITY.md b/SECURITY.md index be1a95f..a36a5a4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -11,10 +11,12 @@ boundary, observed result, and a reproduction built from synthetic data. ## Private-data boundary -Message Like Me reads private iMessage history to derive local analysis. The +Message Like Me reads private messaging history to derive local analysis. The following values are sensitive even when they do not contain an obvious name: - the source Messages and AddressBook databases and their SQLite sidecars; +- local message bundles, manifests, connected-account metadata, and provider + provenance; - contact names, email addresses, and phone numbers; - message bodies, timestamps, reply links, tapbacks, and attachment metadata; - contact, participant, conversation, and group metadata; @@ -54,6 +56,36 @@ provenance. Missing or unsupported text remains unavailable rather than being guessed. Reply targets and tapbacks remain separate from prose so they cannot silently become authored style evidence. +## Local message bundle ingestion + +`messagelikeme ingest bundle` accepts only a normalized absolute path to a +current-user-owned physical mode-`0700` directory. The version-one directory +contains exactly `manifest.json` and six mode-`0600` canonical UTF-8 NDJSON +artifacts. Files must be regular, singly linked, owner-controlled, stable while +read, and free of symbolic-link traversal. + +The importer validates the manifest before allocating for its artifacts. It +caps one line at 2 MiB, the complete bundle at 500,000 records and 512 MiB, and +connected accounts at 128. It streams each artifact, rejects invalid UTF-8, +requires canonical JSON plus final newlines, and verifies exact record counts, +bytes, SHA-256 artifact digests, and the canonical manifest projection digest. +These checks detect malformed or changed local input. They do not establish +that the provider data is truthful or complete. + +The accepted privacy declaration permits attachment metadata only and requires +provider URLs and credentials to be excluded. The bundle may still contain +message bodies, names, handles, timestamps, account identifiers, and graph +coordinates. Keep it under the same controls as the normalized store, and do +not place it in Git, logs, issues, packages, or ordinary agent context. + +Each connected account is stored in its own per-install HMAC namespace. +Bounded, truncated, and unknown source absence never deletes retained history. +Explicit tombstones and terminal message or reaction state suppress their +validated targets. A later matching record can clear suppression, while an +older or conflicting equal-time snapshot is rejected. `sources list` is +redacted. `sources show --private` deliberately reveals provider account and +source metadata. + ## Contacts enrichment Contacts enrichment is optional. The reader discovers populated @@ -83,7 +115,8 @@ a prose profile. ## Local identifiers -Contact, conversation, and message identifiers are derived with an HMAC key +Source, contact, participant, conversation, message, and reaction identifiers +are derived with an HMAC key created for one local installation. They reduce accidental disclosure and keep stable local references without storing handles in ordinary views. They are not anonymization against an attacker who can read the local corpus or key. @@ -95,8 +128,11 @@ it. ## Inspection and study packets Aggregate contact, session, tempo, and surface-style views omit message bodies -and private labels by default. `--private` deliberately reveals local private -identity fields. Use it only when the current task needs that mapping. +and private labels by default. Raw provider reaction values also remain private; +aggregate and drafting-context views expose only fixed-size reaction counts, +direction, datedness, and the outgoing reaction ratio. `--private` deliberately +reveals local private identity fields. Use it only when the current task needs +that mapping. `contacts resolve QUERY --private` performs bounded exact matching against private labels. It does not do prefix, substring, phonetic, or fuzzy matching, diff --git a/dist/cli-xby0v0et.js b/dist/cli-mxxakdqk.js similarity index 87% rename from dist/cli-xby0v0et.js rename to dist/cli-mxxakdqk.js index 177e0d5..1b38977 100644 --- a/dist/cli-xby0v0et.js +++ b/dist/cli-mxxakdqk.js @@ -19,11 +19,12 @@ function prettyJson(value) { // src/types.ts var CORPUS_SCHEMA_VERSION = 1; -var METRICS_SCHEMA_VERSION = 1; +var METRICS_SCHEMA_VERSION = 2; var PROFILE_SCHEMA_VERSION = 2; var LEGACY_PROFILE_SCHEMA_VERSION = 1; var STUDY_PACKET_SCHEMA_VERSION = 2; var EVALUATION_PACKET_SCHEMA_VERSION = 1; var CONTACTS_SCHEMA_VERSION = 1; +var MESSAGE_BUNDLE_SCHEMA_VERSION = 1; -export { canonicalJson, sha256, prettyJson, CORPUS_SCHEMA_VERSION, METRICS_SCHEMA_VERSION, PROFILE_SCHEMA_VERSION, LEGACY_PROFILE_SCHEMA_VERSION, STUDY_PACKET_SCHEMA_VERSION, EVALUATION_PACKET_SCHEMA_VERSION, CONTACTS_SCHEMA_VERSION }; +export { canonicalJson, sha256, prettyJson, CORPUS_SCHEMA_VERSION, METRICS_SCHEMA_VERSION, PROFILE_SCHEMA_VERSION, LEGACY_PROFILE_SCHEMA_VERSION, STUDY_PACKET_SCHEMA_VERSION, EVALUATION_PACKET_SCHEMA_VERSION, CONTACTS_SCHEMA_VERSION, MESSAGE_BUNDLE_SCHEMA_VERSION }; diff --git a/dist/cli.js b/dist/cli.js index 71290c8..663bf0e 100755 --- a/dist/cli.js +++ b/dist/cli.js @@ -5,17 +5,18 @@ import { CORPUS_SCHEMA_VERSION, EVALUATION_PACKET_SCHEMA_VERSION, LEGACY_PROFILE_SCHEMA_VERSION, + MESSAGE_BUNDLE_SCHEMA_VERSION, METRICS_SCHEMA_VERSION, PROFILE_SCHEMA_VERSION, STUDY_PACKET_SCHEMA_VERSION, canonicalJson, prettyJson, sha256 -} from "./cli-xby0v0et.js"; +} from "./cli-mxxakdqk.js"; // src/commands.ts -import { lstat as lstat3 } from "fs/promises"; -import { isAbsolute as isAbsolute4, resolve as resolve5 } from "path"; +import { lstat as lstat4 } from "fs/promises"; +import { isAbsolute as isAbsolute5, resolve as resolve6 } from "path"; // src/errors.ts var EXIT_CODES = { @@ -53,6 +54,7 @@ var VALUE_OPTIONS = new Set([ "burst-gap", "data-dir", "database", + "input", "limit", "min-outgoing", "output", @@ -134,6 +136,12 @@ function rejectUnused(parsed, allowedOptions, allowedFlags) { } } +// src/bundle.ts +import { createHash, createHmac as createHmac2 } from "crypto"; +import { constants as fsConstants2, createReadStream } from "fs"; +import { lstat, open, readdir, realpath } from "fs/promises"; +import { isAbsolute as isAbsolute2, join as join2, resolve as resolve2 } from "path"; + // src/contacts.ts import { Database } from "bun:sqlite"; import { createHmac } from "crypto"; @@ -815,12 +823,1152 @@ function readMacOSContacts(path, options) { }); } +// src/bundle.ts +var MAX_MANIFEST_BYTES = 1024 * 1024; +var MAX_RECORDS = 500000; +var MAX_RECORD_BYTES = 2 * 1024 * 1024; +var MAX_TOTAL_BYTES = 512 * 1024 * 1024; +var MAX_ACCOUNTS = 128; +var MAX_IDENTIFIER_BYTES2 = 1024; +var MAX_SHORT_TEXT_BYTES = 8 * 1024; +var MAX_BODY_BYTES = 1024 * 1024; +var MAX_PARTICIPANTS = 1e4; +var MAX_ATTACHMENTS = 256; +var MAX_WARNINGS = 128; +var ARTIFACTS = Object.freeze([ + Object.freeze({ path: "accounts.ndjson", kind: "account" }), + Object.freeze({ path: "participants.ndjson", kind: "participant" }), + Object.freeze({ path: "conversations.ndjson", kind: "conversation" }), + Object.freeze({ path: "messages.ndjson", kind: "message" }), + Object.freeze({ path: "reactions.ndjson", kind: "reaction" }), + Object.freeze({ path: "tombstones.ndjson", kind: "tombstone" }) +]); +function object(value, label) { + if (value === null || typeof value !== "object" || Array.isArray(value) || Object.getPrototypeOf(value) !== Object.prototype && Object.getPrototypeOf(value) !== null) + throw new CliError("invalid-data", `${label} must be a plain object`); + return value; +} +function exactKeys(value, keys, label) { + const expected = [...keys].sort(); + const observed = Object.keys(value).sort(); + if (expected.length !== observed.length || observed.some((key, index) => key !== expected[index])) + throw new CliError("invalid-data", `${label} must contain exactly: ${keys.join(", ")}`); +} +function boundedText2(value, label, maximum) { + if (typeof value !== "string" || Buffer.byteLength(value, "utf8") > maximum || value.includes("\x00")) { + throw new CliError("invalid-data", `${label} must be NUL-free text within ${maximum} UTF-8 bytes`); + } + return value; +} +function nullableText(value, label, maximum) { + return value === null ? null : boundedText2(value, label, maximum); +} +function identifier(value, label) { + const result = boundedText2(value, label, MAX_IDENTIFIER_BYTES2); + if (result.length === 0 || /[\u0000-\u001f\u007f]/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a non-empty identifier without ASCII controls`); + } + return result; +} +function nullableIdentifier(value, label) { + return value === null ? null : identifier(value, label); +} +function token(value, label, maximum = 128) { + const result = boundedText2(value, label, maximum); + if (!/^[a-z0-9](?:[a-z0-9._+-]*[a-z0-9])?$/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a lowercase categorical token`); + } + return result; +} +function version(value, label) { + const result = boundedText2(value, label, 128); + if (!/^[A-Za-z0-9](?:[A-Za-z0-9._+-]*[A-Za-z0-9])?$/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a bounded version token`); + } + return result; +} +function oneOf(value, values, label) { + if (typeof value !== "string" || !values.includes(value)) { + throw new CliError("invalid-data", `${label} must be one of: ${values.join(", ")}`); + } + return value; +} +function integer2(value, label, maximum = Number.MAX_SAFE_INTEGER) { + if (!Number.isSafeInteger(value) || value < 0 || value > maximum) { + throw new CliError("invalid-data", `${label} must be a non-negative safe integer`); + } + return value; +} +function nullableInteger(value, label) { + return value === null ? null : integer2(value, label); +} +function boolean(value, label) { + if (typeof value !== "boolean") + throw new CliError("invalid-data", `${label} must be boolean`); + return value; +} +function nullableBoolean(value, label) { + return value === null ? null : boolean(value, label); +} +function timestamp(value, label) { + const result = boundedText2(value, label, 64); + const date = new Date(result); + if (!Number.isFinite(date.getTime()) || date.toISOString() !== result) { + throw new CliError("invalid-data", `${label} must be a canonical UTC timestamp`); + } + return result; +} +function nullableTimestamp(value, label) { + return value === null ? null : timestamp(value, label); +} +function digest(value, label) { + const result = boundedText2(value, label, 64); + if (!/^[a-f0-9]{64}$/u.test(result)) + throw new CliError("invalid-data", `${label} must be lowercase SHA-256`); + return result; +} +function array(value, label, maximum) { + if (!Array.isArray(value) || value.length > maximum) { + throw new CliError("invalid-data", `${label} must contain at most ${maximum} items`); + } + return value; +} +function identifiers(value, label, maximum) { + const result = array(value, label, maximum).map((item, index) => identifier(item, `${label}[${index}]`)); + if (new Set(result).size !== result.length) + throw new CliError("invalid-data", `${label} repeats an ID`); + return Object.freeze(result); +} +function parseProvenance(value, label) { + const record = object(value, label); + exactKeys(record, ["providerId", "providerRevision", "observedAt", "connectedAccountProviderId"], label); + return Object.freeze({ + providerId: identifier(record.providerId, `${label}.providerId`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`), + observedAt: timestamp(record.observedAt, `${label}.observedAt`), + connectedAccountProviderId: identifier(record.connectedAccountProviderId, `${label}.connectedAccountProviderId`) + }); +} +function parseCommon(record, kind, extraKeys, label) { + exactKeys(record, ["schemaVersion", "kind", "id", "accountId", "network", "provenance", ...extraKeys], label); + if (record.schemaVersion !== 1 || record.kind !== kind) { + throw new CliError("invalid-data", `${label} has the wrong schemaVersion or kind`); + } + return Object.freeze({ + schemaVersion: 1, + kind, + id: identifier(record.id, `${label}.id`), + accountId: identifier(record.accountId, `${label}.accountId`), + network: token(record.network, `${label}.network`, 64), + provenance: parseProvenance(record.provenance, `${label}.provenance`) + }); +} +function parseAccount(record, label) { + const common = parseCommon(record, "account", ["displayName", "handle", "selfParticipantId"], label); + if (common.id !== common.accountId || common.provenance.providerId !== common.provenance.connectedAccountProviderId) { + throw new CliError("invalid-data", `${label} does not establish one connected account realm`); + } + return Object.freeze({ + ...common, + kind: "account", + displayName: nullableText(record.displayName, `${label}.displayName`, MAX_SHORT_TEXT_BYTES), + handle: nullableText(record.handle, `${label}.handle`, MAX_SHORT_TEXT_BYTES), + selfParticipantId: identifier(record.selfParticipantId, `${label}.selfParticipantId`) + }); +} +function parseParticipant(record, label) { + const common = parseCommon(record, "participant", ["displayName", "handle", "isSelf"], label); + return Object.freeze({ + ...common, + kind: "participant", + displayName: nullableText(record.displayName, `${label}.displayName`, MAX_SHORT_TEXT_BYTES), + handle: nullableText(record.handle, `${label}.handle`, MAX_SHORT_TEXT_BYTES), + isSelf: boolean(record.isSelf, `${label}.isSelf`) + }); +} +function parseConversation(record, label) { + const common = parseCommon(record, "conversation", [ + "type", + "title", + "participantIds", + "participantsComplete", + "startedAt", + "lastMessageAt" + ], label); + const startedAt = nullableTimestamp(record.startedAt, `${label}.startedAt`); + const lastMessageAt = nullableTimestamp(record.lastMessageAt, `${label}.lastMessageAt`); + if (startedAt !== null && lastMessageAt !== null && startedAt > lastMessageAt) { + throw new CliError("invalid-data", `${label}.startedAt must not follow lastMessageAt`); + } + return Object.freeze({ + ...common, + kind: "conversation", + type: oneOf(record.type, ["direct", "group", "channel", "unknown"], `${label}.type`), + title: nullableText(record.title, `${label}.title`, MAX_SHORT_TEXT_BYTES), + participantIds: identifiers(record.participantIds, `${label}.participantIds`, MAX_PARTICIPANTS), + participantsComplete: nullableBoolean(record.participantsComplete, `${label}.participantsComplete`), + startedAt, + lastMessageAt + }); +} +function parseReply(value, label) { + if (value === null) + return null; + const record = object(value, label); + exactKeys(record, ["messageId", "providerId"], label); + return Object.freeze({ + messageId: record.messageId === null ? null : identifier(record.messageId, `${label}.messageId`), + providerId: identifier(record.providerId, `${label}.providerId`) + }); +} +function parseEdit(value, sentAt, label) { + if (value === null) + return null; + const record = object(value, label); + if (record.kind === "in-place") { + exactKeys(record, ["kind", "editedAt", "providerRevision"], label); + const editedAt2 = timestamp(record.editedAt, `${label}.editedAt`); + if (editedAt2 < sentAt) + throw new CliError("invalid-data", `${label} precedes the message`); + return Object.freeze({ + kind: "in-place", + editedAt: editedAt2, + providerRevision: identifier(record.providerRevision, `${label}.providerRevision`) + }); + } + if (record.kind !== "replacement") { + throw new CliError("invalid-data", `${label}.kind must be in-place or replacement`); + } + exactKeys(record, [ + "kind", + "replacesMessageId", + "replacesProviderId", + "editedAt", + "providerRevision" + ], label); + const replacesMessageId = record.replacesMessageId === null ? null : identifier(record.replacesMessageId, `${label}.replacesMessageId`); + const replacesProviderId = identifier(record.replacesProviderId, `${label}.replacesProviderId`); + const editedAt = timestamp(record.editedAt, `${label}.editedAt`); + if (editedAt < sentAt) + throw new CliError("invalid-data", `${label} precedes the message`); + return Object.freeze({ + kind: "replacement", + replacesMessageId, + replacesProviderId, + editedAt, + providerRevision: identifier(record.providerRevision, `${label}.providerRevision`) + }); +} +function parseDeletion(value, label) { + if (value === null) + return null; + const record = object(value, label); + exactKeys(record, ["state", "observedAt", "providerRevision"], label); + return Object.freeze({ + state: oneOf(record.state, ["revoked", "deleted-for-me", "revoked-and-deleted-for-me"], `${label}.state`), + observedAt: timestamp(record.observedAt, `${label}.observedAt`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`) + }); +} +function parseAttachments(value, label) { + return Object.freeze(array(value, label, MAX_ATTACHMENTS).map((item, index) => { + const itemLabel = `${label}[${index}]`; + const record = object(item, itemLabel); + exactKeys(record, ["kind", "mimeType", "name", "sizeBytes"], itemLabel); + const name = nullableText(record.name, `${itemLabel}.name`, MAX_SHORT_TEXT_BYTES); + if (name !== null && (name === "." || name === ".." || name.includes("/") || name.includes("\\"))) { + throw new CliError("invalid-data", `${itemLabel}.name must not be a path`); + } + return Object.freeze({ + kind: oneOf(record.kind, ["audio", "document", "image", "link", "sticker", "video", "unknown"], `${itemLabel}.kind`), + mimeType: nullableText(record.mimeType, `${itemLabel}.mimeType`, 256), + name, + sizeBytes: nullableInteger(record.sizeBytes, `${itemLabel}.sizeBytes`) + }); + })); +} +function parseMessage(record, label) { + const common = parseCommon(record, "message", [ + "conversationId", + "senderParticipantId", + "direction", + "sentAt", + "sortKey", + "body", + "bodyTruncated", + "replyTo", + "edit", + "deletion", + "attachments" + ], label); + const sentAt = timestamp(record.sentAt, `${label}.sentAt`); + const deletion = parseDeletion(record.deletion, `${label}.deletion`); + const body = nullableText(record.body, `${label}.body`, MAX_BODY_BYTES); + if (deletion !== null && body !== null) { + throw new CliError("invalid-data", `${label}.body must be null for a deleted message`); + } + return Object.freeze({ + ...common, + kind: "message", + conversationId: identifier(record.conversationId, `${label}.conversationId`), + senderParticipantId: record.senderParticipantId === null ? null : identifier(record.senderParticipantId, `${label}.senderParticipantId`), + direction: oneOf(record.direction, ["incoming", "outgoing", "unknown"], `${label}.direction`), + sentAt, + sortKey: identifier(record.sortKey, `${label}.sortKey`), + body, + bodyTruncated: nullableBoolean(record.bodyTruncated, `${label}.bodyTruncated`), + replyTo: parseReply(record.replyTo, `${label}.replyTo`), + edit: parseEdit(record.edit, sentAt, `${label}.edit`), + deletion, + attachments: parseAttachments(record.attachments, `${label}.attachments`) + }); +} +function parseReaction(record, label) { + const common = parseCommon(record, "reaction", [ + "messageId", + "messageProviderId", + "participantId", + "body", + "reactedAt", + "state" + ], label); + return Object.freeze({ + ...common, + kind: "reaction", + messageId: record.messageId === null ? null : identifier(record.messageId, `${label}.messageId`), + messageProviderId: identifier(record.messageProviderId, `${label}.messageProviderId`), + participantId: record.participantId === null ? null : identifier(record.participantId, `${label}.participantId`), + body: boundedText2(record.body, `${label}.body`, MAX_SHORT_TEXT_BYTES), + reactedAt: nullableTimestamp(record.reactedAt, `${label}.reactedAt`), + state: oneOf(record.state, ["active", "removed"], `${label}.state`) + }); +} +function parseTombstone(record, label) { + const common = parseCommon(record, "tombstone", [ + "entityKind", + "entityId", + "entityProviderId", + "deletedAt", + "scope", + "providerRevision" + ], label); + return Object.freeze({ + ...common, + kind: "tombstone", + entityKind: oneOf(record.entityKind, ["conversation", "message", "reaction"], `${label}.entityKind`), + entityId: record.entityId === null ? null : identifier(record.entityId, `${label}.entityId`), + entityProviderId: identifier(record.entityProviderId, `${label}.entityProviderId`), + deletedAt: timestamp(record.deletedAt, `${label}.deletedAt`), + scope: oneOf(record.scope, ["remote", "local", "unknown"], `${label}.scope`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`) + }); +} +function parseRecord(value, kind, label) { + const record = object(value, label); + switch (kind) { + case "account": + return parseAccount(record, label); + case "participant": + return parseParticipant(record, label); + case "conversation": + return parseConversation(record, label); + case "message": + return parseMessage(record, label); + case "reaction": + return parseReaction(record, label); + case "tombstone": + return parseTombstone(record, label); + } +} +function parseArtifact(value, index) { + const expected = ARTIFACTS[index]; + const label = `manifest.artifacts[${index}]`; + const record = object(value, label); + exactKeys(record, ["path", "mediaType", "recordKind", "records", "bytes", "sha256"], label); + if (record.path !== expected.path || record.mediaType !== "application/x-ndjson" || record.recordKind !== expected.kind) + throw new CliError("invalid-data", `${label} does not match the fixed artifact inventory`); + return Object.freeze({ + path: expected.path, + mediaType: "application/x-ndjson", + recordKind: expected.kind, + records: integer2(record.records, `${label}.records`, MAX_RECORDS), + bytes: integer2(record.bytes, `${label}.bytes`, MAX_TOTAL_BYTES), + sha256: digest(record.sha256, `${label}.sha256`) + }); +} +function parseManifest(value) { + const record = object(value, "manifest"); + exactKeys(record, [ + "schemaVersion", + "format", + "source", + "provider", + "timestamps", + "completeness", + "warnings", + "privacy", + "counts", + "artifacts", + "integrity" + ], "manifest"); + if (record.schemaVersion !== MESSAGE_BUNDLE_SCHEMA_VERSION || record.format !== "message-like-me.local-message-bundle") { + throw new CliError("invalid-data", "Manifest has an unsupported schemaVersion or format"); + } + const source = object(record.source, "manifest.source"); + exactKeys(source, ["id", "version"], "manifest.source"); + if (source.id !== "beeper-local") + throw new CliError("invalid-data", "manifest.source.id must be beeper-local"); + const provider = object(record.provider, "manifest.provider"); + exactKeys(provider, ["id", "version"], "manifest.provider"); + if (provider.id !== "beeper") + throw new CliError("invalid-data", "manifest.provider.id must be beeper"); + const timestamps = object(record.timestamps, "manifest.timestamps"); + exactKeys(timestamps, ["startedAt", "finishedAt", "createdAt"], "manifest.timestamps"); + const startedAt = timestamp(timestamps.startedAt, "manifest.timestamps.startedAt"); + const finishedAt = timestamp(timestamps.finishedAt, "manifest.timestamps.finishedAt"); + const createdAt = timestamp(timestamps.createdAt, "manifest.timestamps.createdAt"); + if (startedAt > finishedAt || finishedAt > createdAt) { + throw new CliError("invalid-data", "Manifest timestamps are not monotonic"); + } + const completeness = object(record.completeness, "manifest.completeness"); + exactKeys(completeness, ["kind", "reason", "observedFrom", "observedThrough"], "manifest.completeness"); + const observedFrom = nullableTimestamp(completeness.observedFrom, "manifest.completeness.observedFrom"); + const observedThrough = nullableTimestamp(completeness.observedThrough, "manifest.completeness.observedThrough"); + if (observedFrom !== null && observedThrough !== null && observedFrom > observedThrough) { + throw new CliError("invalid-data", "Manifest completeness bounds are reversed"); + } + const warnings = array(record.warnings, "manifest.warnings", MAX_WARNINGS).map((value2, index) => token(value2, `manifest.warnings[${index}]`)); + if (new Set(warnings).size !== warnings.length) + throw new CliError("invalid-data", "Manifest warnings repeat"); + const privacy = object(record.privacy, "manifest.privacy"); + exactKeys(privacy, ["classification", "attachments", "providerUrls", "credentials"], "manifest.privacy"); + if (privacy.classification !== "private-local" || privacy.attachments !== "metadata-only" || privacy.providerUrls !== "excluded" || privacy.credentials !== "excluded") + throw new CliError("invalid-data", "Manifest privacy guarantees are unsupported"); + const counts = object(record.counts, "manifest.counts"); + exactKeys(counts, ARTIFACTS.map(({ kind }) => kind), "manifest.counts"); + const parsedCounts = Object.fromEntries(ARTIFACTS.map(({ kind }) => [ + kind, + integer2(counts[kind], `manifest.counts.${kind}`, MAX_RECORDS) + ])); + if (parsedCounts.account > MAX_ACCOUNTS) { + throw new CliError("invalid-data", `Manifest exceeds the ${MAX_ACCOUNTS}-account safety bound`); + } + if (!Array.isArray(record.artifacts) || record.artifacts.length !== ARTIFACTS.length) { + throw new CliError("invalid-data", "Manifest must list the fixed six artifacts"); + } + const artifacts = Object.freeze(record.artifacts.map(parseArtifact)); + let totalRecords = 0; + let totalBytes = 0; + for (const artifact of artifacts) { + if (artifact.records !== parsedCounts[artifact.recordKind]) { + throw new CliError("invalid-data", `${artifact.path} count disagrees with manifest.counts`); + } + totalRecords += artifact.records; + totalBytes += artifact.bytes; + } + if (totalRecords > MAX_RECORDS || totalBytes > MAX_TOTAL_BYTES) { + throw new CliError("invalid-data", "Manifest exceeds the bundle record or byte bound"); + } + const integrity = object(record.integrity, "manifest.integrity"); + exactKeys(integrity, ["algorithm", "bundleSha256"], "manifest.integrity"); + if (integrity.algorithm !== "sha256") + throw new CliError("invalid-data", "Manifest integrity algorithm is unsupported"); + const result = Object.freeze({ + schemaVersion: 1, + format: "message-like-me.local-message-bundle", + source: Object.freeze({ id: "beeper-local", version: version(source.version, "manifest.source.version") }), + provider: Object.freeze({ id: "beeper", version: version(provider.version, "manifest.provider.version") }), + timestamps: Object.freeze({ startedAt, finishedAt, createdAt }), + completeness: Object.freeze({ + kind: oneOf(completeness.kind, ["bounded-local", "truncated", "unknown"], "manifest.completeness.kind"), + reason: completeness.reason === null ? null : token(completeness.reason, "manifest.completeness.reason"), + observedFrom, + observedThrough + }), + warnings: Object.freeze(warnings), + privacy: Object.freeze({ + classification: "private-local", + attachments: "metadata-only", + providerUrls: "excluded", + credentials: "excluded" + }), + counts: Object.freeze(parsedCounts), + artifacts, + integrity: Object.freeze({ algorithm: "sha256", bundleSha256: digest(integrity.bundleSha256, "manifest.integrity.bundleSha256") }) + }); + const { integrity: _integrity, ...projection } = result; + if (sha256(canonicalJson(projection)) !== result.integrity.bundleSha256) { + throw new CliError("invalid-data", "Manifest bundle SHA-256 does not match its canonical projection"); + } + return result; +} +function sameFile2(left, right) { + return left.dev === right.dev && left.ino === right.ino; +} +async function bundleDirectory(path) { + if (!isAbsolute2(path) || resolve2(path) !== path) { + throw new CliError("unsafe-path", "Bundle input must be a normalized absolute path"); + } + const before = await lstat(path); + if (!before.isDirectory() || before.isSymbolicLink() || (before.mode & 511) !== 448 || typeof process.getuid === "function" && before.uid !== process.getuid()) + throw new CliError("unsafe-path", "Bundle input must be a current-user-owned mode-0700 physical directory"); + const physical = await realpath(path); + if (physical !== path) + throw new CliError("unsafe-path", "Bundle input path must not traverse a symbolic link"); + const after = await lstat(physical); + if (!sameFile2(before, after)) + throw new CliError("unsafe-path", "Bundle directory changed while resolving"); + const expected = ["manifest.json", ...ARTIFACTS.map(({ path: artifactPath }) => artifactPath)].sort(); + const entries = (await readdir(physical)).sort(); + if (entries.length !== expected.length || entries.some((entry, index) => entry !== expected[index])) { + throw new CliError("invalid-data", "Bundle directory does not contain exactly the version-one inventory"); + } + return physical; +} +async function openPrivateFile(path, maximumBytes, allowEmpty) { + const handle = await open(path, fsConstants2.O_RDONLY | fsConstants2.O_NOFOLLOW); + try { + const before = await handle.stat({ bigint: true }); + if (!before.isFile() || before.nlink !== 1n || before.size > BigInt(maximumBytes) || !allowEmpty && before.size < 1n || (before.mode & 0o777n) !== 0o600n || typeof process.getuid === "function" && before.uid !== BigInt(process.getuid())) + throw new CliError("unsafe-path", `${path} must be a private physical file within its bound`); + return { handle, before }; + } catch (error) { + await handle.close(); + throw error; + } +} +async function assertFileUnchanged(path, handle, before) { + const after = await handle.stat({ bigint: true }); + if (before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs || before.ctimeNs !== after.ctimeNs) + throw new CliError("unsafe-path", `${path} changed while it was read`); +} +async function closeReadHandle(handle) { + try { + await handle.close(); + } catch (error) { + if (error.code !== "EBADF") + throw error; + } +} +function decodeUtf8(bytes, label) { + try { + return new TextDecoder("utf-8", { fatal: true }).decode(bytes); + } catch (error) { + throw new CliError("invalid-data", `${label} is not valid UTF-8`, { cause: error }); + } +} +async function readManifest(path) { + const opened = await openPrivateFile(path, MAX_MANIFEST_BYTES, false); + try { + const bytes = Uint8Array.from(await opened.handle.readFile()); + await assertFileUnchanged(path, opened.handle, opened.before); + let value; + try { + value = JSON.parse(decodeUtf8(bytes, "manifest.json")); + } catch (error) { + throw new CliError("invalid-data", "manifest.json is not valid UTF-8 JSON", { cause: error }); + } + const manifest = parseManifest(value); + if (!Buffer.from(`${canonicalJson(manifest)} +`, "utf8").equals(Buffer.from(bytes))) { + throw new CliError("invalid-data", "manifest.json must use canonical JSON with one final newline"); + } + return Object.freeze({ bytes, manifest }); + } finally { + await closeReadHandle(opened.handle); + } +} +async function readArtifact(root, artifact) { + const path = join2(root, artifact.path); + const opened = await openPrivateFile(path, artifact.bytes, true); + const hash = createHash("sha256"); + const records = []; + let totalBytes = 0; + let pending = Buffer.alloc(0); + let endedWithNewline = false; + try { + const stream = createReadStream(path, { + fd: opened.handle.fd, + autoClose: false, + start: 0, + highWaterMark: 64 * 1024 + }); + for await (const value of stream) { + const chunk = Buffer.from(value); + hash.update(chunk); + totalBytes += chunk.byteLength; + if (totalBytes > artifact.bytes) + throw new CliError("invalid-data", `${artifact.path} exceeds manifest bytes`); + pending = pending.length === 0 ? chunk : Buffer.concat([pending, chunk]); + let newline = pending.indexOf(10); + while (newline >= 0) { + const line = pending.subarray(0, newline); + pending = pending.subarray(newline + 1); + endedWithNewline = true; + if (line.byteLength < 1 || line.byteLength + 1 > MAX_RECORD_BYTES) { + throw new CliError("invalid-data", `${artifact.path} contains a blank or oversized record`); + } + let parsed; + try { + parsed = JSON.parse(decodeUtf8(line, `${artifact.path} record`)); + } catch (error) { + throw new CliError("invalid-data", `${artifact.path} contains invalid UTF-8 JSON`, { cause: error }); + } + const normalized = parseRecord(parsed, artifact.recordKind, `${artifact.path}:${records.length + 1}`); + if (!Buffer.from(canonicalJson(normalized), "utf8").equals(line)) { + throw new CliError("invalid-data", `${artifact.path} records must use canonical JSON`); + } + records.push(normalized); + if (records.length > artifact.records) { + throw new CliError("invalid-data", `${artifact.path} exceeds its manifest record count`); + } + newline = pending.indexOf(10); + } + if (pending.byteLength + 1 > MAX_RECORD_BYTES) { + throw new CliError("invalid-data", `${artifact.path} contains an oversized record`); + } + if (pending.length > 0) + endedWithNewline = false; + } + await assertFileUnchanged(path, opened.handle, opened.before); + } finally { + await closeReadHandle(opened.handle); + } + if (pending.byteLength !== 0 || artifact.records > 0 && !endedWithNewline) { + throw new CliError("invalid-data", `${artifact.path} must end every record with a newline`); + } + if (totalBytes !== artifact.bytes || records.length !== artifact.records || hash.digest("hex") !== artifact.sha256) + throw new CliError("invalid-data", `${artifact.path} does not match its manifest integrity`); + return Object.freeze(records); +} +function hmacKey(value) { + const key = typeof value === "string" ? new TextEncoder().encode(value) : value; + if (!(key instanceof Uint8Array) || key.byteLength < 16 || key.byteLength > 1024) { + throw new CliError("invalid-data", "Bundle HMAC key must contain 16 through 1024 bytes"); + } + return Uint8Array.from(key); +} +function hmac2(key, namespace, value) { + return createHmac2("sha256", key).update(`message-like-me\x00bundle-${namespace}\x00`, "utf8").update(value, "utf8").digest("hex"); +} +function compareCodeUnits(left, right) { + return left < right ? -1 : left > right ? 1 : 0; +} +function recordMap(records, label) { + const result = new Map; + for (const record of records) { + if (result.has(record.id)) + throw new CliError("invalid-data", `${label} repeats a bundle-local ID`); + result.set(record.id, record); + } + return result; +} +function groupByAccount(records) { + const grouped = new Map; + for (const record of records) { + const values = grouped.get(record.accountId) ?? []; + values.push(record); + grouped.set(record.accountId, values); + } + return grouped; +} +function attachmentProvenance(messageId, attachments) { + return Object.freeze(attachments.map((attachment, index) => ({ + id: `${messageId}:attachment:${index + 1}`, + kind: attachment.kind, + mimeType: attachment.mimeType, + fileName: attachment.name, + bytes: attachment.sizeBytes + }))); +} +function reactionTimelineCoordinate(localReactionId) { + return `\x1Freaction-timeline:${localReactionId}`; +} +function normalizeBundle(manifest, manifestSha256, records, key) { + const accounts = records.account; + const participants = records.participant; + const conversations = records.conversation; + const messages = records.message; + const reactions = records.reaction; + const tombstones = records.tombstone; + if (accounts.length > MAX_ACCOUNTS) { + throw new CliError("invalid-data", `Bundle exceeds the ${MAX_ACCOUNTS}-account safety bound`); + } + const accountById = recordMap(accounts, "accounts"); + recordMap(participants, "participants"); + recordMap(conversations, "conversations"); + const messageRecordById = recordMap(messages, "messages"); + const reactionById = recordMap(reactions, "reactions"); + recordMap(tombstones, "tombstones"); + for (const [kind, values] of [ + ["account", accounts], + ["participant", participants], + ["conversation", conversations], + ["message", messages], + ["reaction", reactions], + ["tombstone", tombstones] + ]) { + const providerCoordinates = new Set; + for (const record of values) { + const coordinate = `${record.accountId}\x00${record.provenance.providerId}`; + if (providerCoordinates.has(coordinate)) { + throw new CliError("invalid-data", `${kind} records repeat a provider identity within one account`); + } + providerCoordinates.add(coordinate); + } + } + for (const record of [...participants, ...conversations, ...messages, ...reactions, ...tombstones]) { + const account = accountById.get(record.accountId); + if (account === undefined || account.network !== record.network || account.provenance.connectedAccountProviderId !== record.provenance.connectedAccountProviderId) + throw new CliError("invalid-data", "A record does not match its connected account realm"); + } + const participantsByAccount = groupByAccount(participants); + const conversationsByAccount = groupByAccount(conversations); + const messagesByAccount = groupByAccount(messages); + const reactionsByAccount = groupByAccount(reactions); + const tombstonesByAccount = groupByAccount(tombstones); + const result = []; + const sourceIds = new Set; + for (const account of accounts) { + const accountParticipants = participantsByAccount.get(account.id) ?? []; + const participantById = new Map(accountParticipants.map((participant) => [participant.id, participant])); + const self = participantById.get(account.selfParticipantId); + if (self === undefined || !self.isSelf || accountParticipants.filter(({ isSelf }) => isSelf).length !== 1) { + throw new CliError("invalid-data", "An account must have exactly one matching self participant"); + } + const accountConversations = conversationsByAccount.get(account.id) ?? []; + const conversationParticipantIds = new Map(accountConversations.map((conversation) => [ + conversation.id, + new Set(conversation.participantIds) + ])); + for (const conversation of accountConversations) { + for (const participantId of conversation.participantIds) { + if (!participantById.has(participantId)) { + throw new CliError("invalid-data", "A conversation references an unknown participant"); + } + } + if (conversation.type === "direct" && conversation.participantsComplete === true && (conversation.participantIds.length !== 2 || !conversation.participantIds.includes(account.selfParticipantId) || conversation.participantIds.filter((participantId) => participantById.get(participantId)?.isSelf === false).length !== 1)) { + throw new CliError("invalid-data", "A complete direct conversation must contain one self and one non-self participant"); + } + } + const conversationById = new Map(accountConversations.map((conversation) => [conversation.id, conversation])); + const namespace = [ + manifest.provider.id, + account.provenance.connectedAccountProviderId, + self.provenance.providerId + ].join("\x00"); + const sourceId = `source_${hmac2(key, "source", namespace)}`; + if (sourceIds.has(sourceId)) { + throw new CliError("invalid-data", "Connected accounts repeat a stable source realm"); + } + sourceIds.add(sourceId); + const conversationLocalIds = new Map(accountConversations.map((conversation) => [ + conversation.id, + `conversation_${hmac2(key, "conversation", `${namespace}\x00${conversation.provenance.providerId}`)}` + ])); + const participantLocalIds = new Map(accountParticipants.map((participant) => [ + participant.id, + `participant_${hmac2(key, "participant", `${namespace}\x00${participant.provenance.providerId}`)}` + ])); + const normalizedConversations = accountConversations.map((conversation) => { + const known = conversation.participantIds.flatMap((id) => { + const participant = participantById.get(id); + return participant === undefined ? [] : [participant]; + }); + const peers = known.filter(({ isSelf }) => !isSelf); + const completeDirectPeer = conversation.type === "direct" && conversation.participantsComplete === true && peers.length === 1 ? peers[0] : null; + const canonicalHandle = completeDirectPeer?.handle === null || completeDirectPeer === null ? null : normalizeContactHandle(completeDirectPeer.handle); + return Object.freeze({ + id: conversationLocalIds.get(conversation.id), + sourceKey: conversation.provenance.providerId, + privateLabel: conversation.title, + service: account.network, + participantCount: conversation.type === "direct" ? 1 : peers.length, + participantIds: Object.freeze(peers.map((participant) => participantLocalIds.get(participant.id))), + privateParticipants: canonicalHandle === null ? Object.freeze([]) : Object.freeze([canonicalHandle.normalizedValue]), + group: conversation.type !== "direct" + }); + }); + const accountMessages = messagesByAccount.get(account.id) ?? []; + const messageById = new Map(accountMessages.map((message) => [message.id, message])); + const messageByProviderId = new Map(accountMessages.map((message) => [ + message.provenance.providerId, + message + ])); + const replacementTargets = new Map; + const replacerByTarget = new Map; + for (const message of accountMessages) { + if (!conversationById.has(message.conversationId)) { + throw new CliError("invalid-data", "A message references an unknown conversation"); + } + if (message.senderParticipantId !== null && !participantById.has(message.senderParticipantId)) { + throw new CliError("invalid-data", "A message references an unknown sender participant"); + } + const sender = message.senderParticipantId === null ? null : participantById.get(message.senderParticipantId); + const conversation = conversationById.get(message.conversationId); + if (sender !== null && (message.direction === "outgoing" && !sender.isSelf || message.direction === "incoming" && sender.isSelf)) + throw new CliError("invalid-data", "A message direction conflicts with its sender identity"); + if (sender !== null && conversation.participantsComplete === true && !conversationParticipantIds.get(conversation.id).has(sender.id)) + throw new CliError("invalid-data", "A message sender is outside its complete conversation roster"); + if (message.replyTo !== null) { + const localTarget = message.replyTo.messageId === null ? undefined : messageRecordById.get(message.replyTo.messageId); + if (message.replyTo.messageId !== null && (localTarget === undefined || localTarget.accountId !== account.id || localTarget.provenance.providerId !== message.replyTo.providerId)) + throw new CliError("invalid-data", "A message reply has mismatched target coordinates"); + const providerTarget = messageByProviderId.get(message.replyTo.providerId); + const target = localTarget ?? providerTarget; + if (message.replyTo.providerId === message.provenance.providerId || target !== undefined && target.conversationId !== message.conversationId) + throw new CliError("invalid-data", "A message reply has an invalid conversation target"); + } + if (message.edit?.kind === "replacement") { + const localTarget = message.edit.replacesMessageId === null ? undefined : messageRecordById.get(message.edit.replacesMessageId); + if (message.edit.replacesMessageId !== null && (localTarget === undefined || localTarget.accountId !== account.id || localTarget.provenance.providerId !== message.edit.replacesProviderId)) + throw new CliError("invalid-data", "A message edit has mismatched replacement coordinates"); + const providerTarget = messageByProviderId.get(message.edit.replacesProviderId); + const target = localTarget ?? providerTarget; + if (message.edit.replacesProviderId === message.provenance.providerId || target !== undefined && target.conversationId !== message.conversationId) + throw new CliError("invalid-data", "A message edit has an invalid replacement target"); + if (replacerByTarget.has(message.edit.replacesProviderId)) { + throw new CliError("invalid-data", "A message version has multiple replacements"); + } + replacerByTarget.set(message.edit.replacesProviderId, message.provenance.providerId); + replacementTargets.set(message.id, Object.freeze({ + target, + externalId: message.edit.replacesProviderId + })); + } + } + const editEdges = new Map([...replacementTargets.entries()].map(([messageId, target]) => [ + messageById.get(messageId).provenance.providerId, + target.externalId + ])); + const completedEditNodes = new Set; + for (const start of editEdges.keys()) { + if (completedEditNodes.has(start)) + continue; + const seen = new Set; + const chain = []; + let current = start; + while (current !== undefined && !completedEditNodes.has(current)) { + if (seen.has(current)) + throw new CliError("invalid-data", "Message replacement edits contain a cycle"); + seen.add(current); + chain.push(current); + current = editEdges.get(current); + } + for (const node of chain) + completedEditNodes.add(node); + } + const analyzableMessages = accountMessages.filter(({ direction }) => direction !== "unknown").sort((left, right) => compareCodeUnits(left.conversationId, right.conversationId) || compareCodeUnits(left.sortKey, right.sortKey) || compareCodeUnits(left.sentAt, right.sentAt) || compareCodeUnits(left.provenance.providerId, right.provenance.providerId)); + const normalizedMessages = []; + const messageProvenance = []; + const localMessageIds = new Map; + const localReactionIds = new Map; + const timelineReactionIds = new Set; + for (const [index, message] of analyzableMessages.entries()) { + const localId = `message_${hmac2(key, "message", `${namespace}\x00${message.provenance.providerId}`)}`; + localMessageIds.set(message.id, localId); + const body = message.bodyTruncated === true || message.deletion !== null ? null : message.body; + normalizedMessages.push(Object.freeze({ + id: localId, + sourceRowId: index + 1, + sourceGuid: message.provenance.providerId, + conversationId: conversationLocalIds.get(message.conversationId), + sentAt: message.sentAt, + direction: message.direction, + body, + bodySource: body === null ? "unavailable" : "text", + kind: body !== null || message.bodyTruncated === true ? "text" : message.attachments.length > 0 ? "attachment" : "unknown", + replyToSourceGuid: message.replyTo?.providerId ?? null, + editedAt: message.edit?.editedAt ?? null, + retractedAt: message.deletion?.observedAt ?? null, + service: account.network, + attachmentCount: message.attachments.length + })); + messageProvenance.push(Object.freeze({ + messageId: localId, + externalId: message.provenance.providerId, + providerSortKey: message.sortKey, + replyToExternalId: message.replyTo?.providerId ?? null, + attachments: attachmentProvenance(localId, message.attachments), + metadata: message + })); + } + const accountReactions = reactionsByAccount.get(account.id) ?? []; + for (const reaction of accountReactions) { + localReactionIds.set(reaction.id, `message_${hmac2(key, "reaction", `${namespace}\x00${reaction.provenance.providerId}`)}`); + } + const reactionFacts = []; + for (const reaction of accountReactions) { + const localTarget = reaction.messageId === null ? undefined : messageRecordById.get(reaction.messageId); + if (reaction.messageId !== null && (localTarget === undefined || localTarget.accountId !== account.id || localTarget.provenance.providerId !== reaction.messageProviderId)) + throw new CliError("invalid-data", "A reaction has mismatched target coordinates"); + if (reaction.participantId !== null && !participantById.has(reaction.participantId)) { + throw new CliError("invalid-data", "A reaction references an unknown participant"); + } + const target = localTarget ?? messageByProviderId.get(reaction.messageProviderId); + const participant = reaction.participantId === null ? null : participantById.get(reaction.participantId); + const targetConversationId = target === undefined ? null : conversationLocalIds.get(target.conversationId) ?? null; + if (target !== undefined && participant !== null) { + const targetConversation = conversationById.get(target.conversationId); + if (targetConversation.participantsComplete === true && !conversationParticipantIds.get(targetConversation.id).has(participant.id)) + throw new CliError("invalid-data", "A reaction participant is outside its complete conversation roster"); + } + const localId = localReactionIds.get(reaction.id); + reactionFacts.push(Object.freeze({ + id: localId, + externalId: reaction.provenance.providerId, + targetExternalId: reaction.messageProviderId, + conversationId: targetConversationId, + direction: participant === null ? null : participant.isSelf ? "outgoing" : "incoming", + body: reaction.body, + reactedAt: reaction.reactedAt, + state: reaction.state + })); + if (reaction.state !== "active" || reaction.reactedAt === null || reaction.participantId === null) + continue; + if (participant === null || target === undefined || targetConversationId === null) + continue; + timelineReactionIds.add(reaction.id); + const timelineCoordinate = reactionTimelineCoordinate(localId); + normalizedMessages.push(Object.freeze({ + id: localId, + sourceRowId: normalizedMessages.length + 1, + sourceGuid: timelineCoordinate, + conversationId: targetConversationId, + sentAt: reaction.reactedAt, + direction: participant.isSelf ? "outgoing" : "incoming", + body: null, + bodySource: "unavailable", + kind: "reaction", + replyToSourceGuid: reaction.messageProviderId, + editedAt: null, + retractedAt: null, + service: account.network, + attachmentCount: 0 + })); + messageProvenance.push(Object.freeze({ + messageId: localId, + externalId: timelineCoordinate, + providerSortKey: null, + replyToExternalId: reaction.messageProviderId, + attachments: Object.freeze([]), + metadata: reaction + })); + } + const reactionFactByExternal = new Map(reactionFacts.map((fact) => [fact.externalId, fact])); + const auxiliaryRecords = [ + { kind: "account", id: account.provenance.providerId, record: account }, + ...accountParticipants.map((participant) => ({ + kind: "participant", + id: participant.provenance.providerId, + record: participant + })), + ...accountReactions.map((reaction) => ({ + kind: "reaction", + id: reaction.provenance.providerId, + record: reaction + })), + ...(tombstonesByAccount.get(account.id) ?? []).map((tombstone) => ({ + kind: "tombstone", + id: tombstone.provenance.providerId, + record: tombstone + })), + ...accountMessages.filter(({ direction }) => direction === "unknown").map((message) => ({ + kind: "excluded-message", + id: message.provenance.providerId, + record: message + })) + ]; + const accountTombstones = tombstonesByAccount.get(account.id) ?? []; + const deletions = accountTombstones.map((tombstone) => { + const entityId = tombstone.entityId; + let localEntityId = null; + if (entityId !== null) { + if (tombstone.entityKind === "conversation") { + const target = conversationById.get(entityId); + if (target === undefined) { + throw new CliError("invalid-data", "A tombstone references an unknown local conversation"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched conversation identity"); + } + localEntityId = conversationLocalIds.get(entityId) ?? null; + } else if (tombstone.entityKind === "message") { + const target = messageRecordById.get(entityId); + if (target === undefined || target.accountId !== account.id) { + throw new CliError("invalid-data", "A tombstone references an unknown local message"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched message identity"); + } + localEntityId = localMessageIds.get(entityId) ?? null; + } else if (tombstone.entityKind === "reaction") { + const target = reactionById.get(entityId); + if (target === undefined || target.accountId !== account.id) { + throw new CliError("invalid-data", "A tombstone references an unknown local reaction"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched reaction identity"); + } + localEntityId = localReactionIds.get(entityId) ?? null; + } + } + return Object.freeze({ + entityKind: tombstone.entityKind, + localEntityId, + externalId: tombstone.entityProviderId, + deletedAt: tombstone.deletedAt, + reason: "tombstone" + }); + }); + for (const [messageId, replacement] of replacementTargets) { + const message = messageById.get(messageId); + deletions.push(Object.freeze({ + entityKind: "message", + localEntityId: replacement.target === undefined ? null : localMessageIds.get(replacement.target.id) ?? null, + externalId: replacement.externalId, + deletedAt: message.edit.editedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId), + reason: "replacement" + })); + } + for (const message of accountMessages) { + if (message.deletion === null) + continue; + deletions.push(Object.freeze({ + entityKind: "message", + localEntityId: localMessageIds.get(message.id) ?? null, + externalId: message.provenance.providerId, + deletedAt: message.deletion.observedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId), + reason: "tombstone" + })); + } + for (const message of accountMessages) { + if (message.direction !== "unknown") + continue; + deletions.push(Object.freeze({ + entityKind: "message", + localEntityId: null, + externalId: message.provenance.providerId, + deletedAt: message.provenance.observedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId), + reason: "explicit-exclusion" + })); + } + for (const reaction of accountReactions) { + if (timelineReactionIds.has(reaction.id)) + continue; + const fact = reactionFactByExternal.get(reaction.provenance.providerId); + deletions.push(Object.freeze({ + entityKind: reaction.state === "removed" ? "reaction" : "reaction-timeline", + localEntityId: localReactionIds.get(reaction.id), + externalId: reaction.provenance.providerId, + deletedAt: reaction.provenance.observedAt, + ...fact.conversationId === null ? {} : { expectedConversationId: fact.conversationId }, + reason: reaction.state === "removed" ? "tombstone" : "explicit-exclusion" + })); + } + const sourceWarnings = [...manifest.warnings]; + const unknownDirections = accountMessages.filter(({ direction }) => direction === "unknown").length; + const undatedReactions = accountReactions.filter(({ reactedAt }) => reactedAt === null).length; + if (unknownDirections > 0) + sourceWarnings.push(`unknown-direction-messages:${unknownDirections}`); + if (undatedReactions > 0) + sourceWarnings.push(`undated-reactions:${undatedReactions}`); + const accountTimelineBounds = [ + ...accountMessages.map(({ sentAt }) => sentAt), + ...accountReactions.flatMap(({ reactedAt }) => reactedAt === null ? [] : [reactedAt]) + ].sort(compareCodeUnits); + const accountObservedFrom = accountTimelineBounds[0] ?? null; + const accountObservedThrough = accountTimelineBounds.at(-1) ?? null; + const revisionHash = createHash("sha256"); + const revisionHeader = canonicalJson({ + schemaVersion: 1, + source: manifest.source, + provider: manifest.provider, + completeness: manifest.completeness, + warnings: manifest.warnings + }); + revisionHash.update(`${revisionHeader.length}:`, "utf8").update(revisionHeader, "utf8"); + for (const [kind, values] of [ + ["account", [account]], + ["participant", accountParticipants], + ["conversation", accountConversations], + ["message", accountMessages], + ["reaction", accountReactions], + ["tombstone", accountTombstones] + ]) { + revisionHash.update(`${kind.length}:${kind}`, "utf8"); + for (const record of values) { + const encoded = canonicalJson(record); + revisionHash.update(`${Buffer.byteLength(encoded, "utf8")}:`, "utf8").update(encoded, "utf8"); + } + } + const revision = revisionHash.digest("hex"); + result.push(Object.freeze({ + source: Object.freeze({ + id: sourceId, + kind: "bundle", + provider: manifest.provider.id, + network: account.network, + accountId: account.provenance.connectedAccountProviderId, + externalId: account.provenance.connectedAccountProviderId, + revision, + generatedAt: manifest.timestamps.createdAt, + producer: manifest.source, + coverage: Object.freeze({ + history: manifest.completeness.kind === "unknown" ? "unknown" : "bounded", + observedFrom: accountObservedFrom, + observedTo: accountObservedThrough, + kind: manifest.completeness.kind, + reason: manifest.completeness.reason + }), + manifestSha256, + identity: Object.freeze({ account, selfParticipantProviderId: self.provenance.providerId }), + warnings: Object.freeze(sourceWarnings) + }), + conversations: Object.freeze(normalizedConversations), + conversationProvenance: Object.freeze(accountConversations.map((conversation) => ({ + conversationId: conversationLocalIds.get(conversation.id), + externalId: conversation.provenance.providerId, + metadata: conversation + }))), + messages: Object.freeze(normalizedMessages), + messageProvenance: Object.freeze(messageProvenance), + reactionFacts: Object.freeze(reactionFacts), + auxiliaryRecords: Object.freeze(auxiliaryRecords), + deletions: Object.freeze(deletions) + })); + } + return Object.freeze(result); +} +async function readMessageBundle(path, options) { + const key = hmacKey(options.hmacKey); + const root = await bundleDirectory(path); + const manifestResult = await readManifest(join2(root, "manifest.json")); + const manifest = manifestResult.manifest; + const manifestSha256 = sha256(manifestResult.bytes); + const parsedRecords = []; + for (const artifact of manifest.artifacts) + parsedRecords.push(await readArtifact(root, artifact)); + const records = Object.fromEntries(manifest.artifacts.map((artifact, index) => [ + artifact.recordKind, + parsedRecords[index] + ])); + return Object.freeze({ + schemaVersion: MESSAGE_BUNDLE_SCHEMA_VERSION, + manifestSha256, + sources: normalizeBundle(manifest, manifestSha256, records, key) + }); +} + // src/imessage.ts import { Database as Database2 } from "bun:sqlite"; -import { createHash, createHmac as createHmac2 } from "crypto"; +import { createHash as createHash2, createHmac as createHmac3 } from "crypto"; import { chmodSync as chmodSync2, - constants as fsConstants2, + constants as fsConstants3, copyFileSync as copyFileSync2, lstatSync as lstatSync2, mkdirSync as mkdirSync2, @@ -829,8 +1977,8 @@ import { rmSync as rmSync2 } from "fs"; import { homedir as homedir2, tmpdir as tmpdir2 } from "os"; -import { basename as basename2, isAbsolute as isAbsolute2, join as join2, resolve as resolve2 } from "path"; -var DEFAULT_IMESSAGE_DATABASE = join2(homedir2(), "Library", "Messages", "chat.db"); +import { basename as basename2, isAbsolute as isAbsolute3, join as join3, resolve as resolve3 } from "path"; +var DEFAULT_IMESSAGE_DATABASE = join3(homedir2(), "Library", "Messages", "chat.db"); var APPLE_EPOCH_MILLISECONDS = Date.UTC(2001, 0, 1); var DEFAULT_MAX_DATABASE_BYTES = 16 * 1024 * 1024 * 1024; var MAX_CONFIGURABLE_DATABASE_BYTES = 64 * 1024 * 1024 * 1024; @@ -879,12 +2027,12 @@ function stableJson(value) { return `{${Object.keys(record).sort().map((key) => `${JSON.stringify(key)}:${stableJson(record[key])}`).join(",")}}`; } function sha2562(value) { - return createHash("sha256").update(value).digest("hex"); + return createHash2("sha256").update(value).digest("hex"); } -function hmac2(key, namespace, value) { - return createHmac2("sha256", key).update(`message-like-me\x00${namespace}\x00`, "utf8").update(value, "utf8").digest("hex"); +function hmac3(key, namespace, value) { + return createHmac3("sha256", key).update(`message-like-me\x00${namespace}\x00`, "utf8").update(value, "utf8").digest("hex"); } -function hmacKey(value) { +function hmacKey2(value) { const key = typeof value === "string" ? new TextEncoder().encode(value) : value; if (!(key instanceof Uint8Array) || key.byteLength < 16 || key.byteLength > 1024) { throw new Error("iMessage HMAC key must contain 16 through 1024 bytes"); @@ -901,20 +2049,20 @@ function boundedInteger2(value, fallback, minimum, maximum, label) { function ownedByCurrentUser(stats) { return typeof process.getuid !== "function" || stats.uid === BigInt(process.getuid()); } -function sameFile2(left, right) { +function sameFile3(left, right) { return left.dev === right.dev && left.ino === right.ino; } function inspectSource(path, maximumBytes) { - if (!isAbsolute2(path)) + if (!isAbsolute3(path)) return fail2("path must be absolute"); - const requested = resolve2(path); + const requested = resolve3(path); const requestedStats = lstatSync2(requested, { bigint: true }); if (!requestedStats.isFile() || requestedStats.isSymbolicLink() || requestedStats.nlink !== 1n || !ownedByCurrentUser(requestedStats) || requestedStats.size < 1n || requestedStats.size > BigInt(maximumBytes)) { return fail2("must be one current-user-owned regular non-symlink file within the configured size bound"); } const physicalPath = realpathSync2(requested); const physicalStats = lstatSync2(physicalPath, { bigint: true }); - if (!sameFile2(requestedStats, physicalStats)) { + if (!sameFile3(requestedStats, physicalStats)) { return fail2("changed identity while its path was resolved"); } return Object.freeze({ path: physicalPath, stats: physicalStats }); @@ -935,7 +2083,7 @@ function validateSidecar2(path, stats, maximumBytes) { } function snapshotMembers2(source, maximumBytes) { const current = inspectSource(source.path, maximumBytes); - if (!sameFile2(source.stats, current.stats)) + if (!sameFile3(source.stats, current.stats)) return fail2("changed identity before its snapshot was isolated"); const members = [{ suffix: "", path: current.path, stats: current.stats }]; for (const suffix of ["-wal", "-journal"]) { @@ -959,25 +2107,25 @@ function snapshotMembers2(source, maximumBytes) { function sameSnapshotMembers(left, right) { return left.length === right.length && left.every((member, index) => { const other = right[index]; - return other !== undefined && member.suffix === other.suffix && sameFile2(member.stats, other.stats) && member.stats.size === other.stats.size && member.stats.mtimeNs === other.stats.mtimeNs && member.stats.ctimeNs === other.stats.ctimeNs; + return other !== undefined && member.suffix === other.suffix && sameFile3(member.stats, other.stats) && member.stats.size === other.stats.size && member.stats.mtimeNs === other.stats.mtimeNs && member.stats.ctimeNs === other.stats.ctimeNs; }); } function isolateSource2(source, maximumBytes) { const temporaryRoot = tmpdir2(); - if (!isAbsolute2(temporaryRoot)) + if (!isAbsolute3(temporaryRoot)) return fail2("requires an absolute temporary directory"); - const temporaryDirectory = mkdtempSync2(join2(temporaryRoot, "message-like-me-source-")); + const temporaryDirectory = mkdtempSync2(join3(temporaryRoot, "message-like-me-source-")); chmodSync2(temporaryDirectory, 448); try { for (let attempt = 0;attempt < SOURCE_SNAPSHOT_ATTEMPTS; attempt += 1) { const before = snapshotMembers2(source, maximumBytes); - const attemptDirectory = join2(temporaryDirectory, `attempt-${attempt}`); + const attemptDirectory = join3(temporaryDirectory, `attempt-${attempt}`); mkdirSync2(attemptDirectory, { mode: 448 }); let copyFailedForRace = false; try { for (const member of before) { - const destination = join2(attemptDirectory, `${basename2(source.path)}${member.suffix}`); - copyFileSync2(member.path, destination, fsConstants2.COPYFILE_EXCL | fsConstants2.COPYFILE_FICLONE); + const destination = join3(attemptDirectory, `${basename2(source.path)}${member.suffix}`); + copyFileSync2(member.path, destination, fsConstants3.COPYFILE_EXCL | fsConstants3.COPYFILE_FICLONE); chmodSync2(destination, 384); } } catch (error) { @@ -991,7 +2139,7 @@ function isolateSource2(source, maximumBytes) { if (!copyFailedForRace && sameSnapshotMembers(before, after)) { return Object.freeze({ source: Object.freeze({ path: source.path, stats: before[0].stats }), - path: join2(attemptDirectory, basename2(source.path)), + path: join3(attemptDirectory, basename2(source.path)), temporaryDirectory }); } @@ -1260,7 +2408,7 @@ function loadHandles(database, key) { rowId, id, service, - participantId: hmac2(key, "participant", `${service ?? ""}\x00${id}`) + participantId: hmac3(key, "participant", `${service ?? ""}\x00${id}`) })); } return result; @@ -1296,7 +2444,7 @@ function loadChats(database, schema, handles, key) { const participants = [...handleIds.get(rowId) ?? new Set].sort((left, right) => left - right).map((handleId) => handles.get(handleId)).filter((handle) => handle !== undefined); const services = [...new Set(participants.map((participant) => participant.service).filter((service) => service !== null))].sort(); const conversation = Object.freeze({ - id: hmac2(key, "conversation", sourceKey), + id: hmac3(key, "conversation", sourceKey), sourceKey, privateLabel: privateLabel2, service: declaredService === null || declaredService === "" ? services.length === 1 ? services[0] : null : declaredService, @@ -1429,7 +2577,7 @@ function aggregateWarnings(counts, hasAttachmentJoin) { return Object.freeze(warnings); } function readIMessageDatabase(path, options) { - const key = hmacKey(options.hmacKey); + const key = hmacKey2(options.hmacKey); const maximumDatabaseBytes = boundedInteger2(options.maxDatabaseBytes, DEFAULT_MAX_DATABASE_BYTES, 1, MAX_CONFIGURABLE_DATABASE_BYTES, "maxDatabaseBytes"); const maximumMessages = boundedInteger2(options.maxMessages, DEFAULT_MAX_MESSAGES, 1, MAX_CONFIGURABLE_MESSAGES, "maxMessages"); const maximumBodyBytes = boundedInteger2(options.maxBodyBytes, DEFAULT_MAX_BODY_BYTES, 1, MAX_CONFIGURABLE_BODY_BYTES, "maxBodyBytes"); @@ -1475,7 +2623,7 @@ function readIMessageDatabase(path, options) { const joins = loadChatJoins(database, first, last); const attachments = loadAttachmentCounts(database, schema, first, last); for (const row of page) { - const id = hmac2(key, "message", row.sourceGuid); + const id = hmac3(key, "message", row.sourceGuid); if (messageIds.has(id)) return fail2("contains duplicate message GUIDs"); if (row.isSpam === 1 || row.isCorrupt === 1) { @@ -1580,7 +2728,7 @@ function readIMessageDatabase(path, options) { } // src/metrics.ts -import { createHash as createHash2 } from "crypto"; +import { createHash as createHash3 } from "crypto"; var DEFAULT_SESSION_GAP_SECONDS = 8 * 60 * 60; var DEFAULT_BURST_GAP_SECONDS = 5 * 60; var DEFAULT_STUDY_LIMIT = 12; @@ -1592,8 +2740,8 @@ var MAX_STUDY_MESSAGES_PER_DIRECTION = 64; var DEFAULT_MAX_STUDY_PACKET_BODY_BYTES = 256 * 1024; var MAX_STUDY_PACKET_BODY_BYTES = 1024 * 1024; var MAX_GAP_SECONDS = 30 * 24 * 60 * 60; -function digest(namespace, parts) { - const hash = createHash2("sha256"); +function digest2(namespace, parts) { + const hash = createHash3("sha256"); hash.update(`message-like-me\x00${namespace}\x00`, "utf8"); for (const part of parts) hash.update(`${part.length}:`, "utf8").update(part, "utf8"); @@ -1684,7 +2832,7 @@ function sessionsFor(messages, corpusRevision, contactId, gapSeconds) { const incomingCount = group.filter(({ message }) => message.direction === "incoming").length; const outgoingCount = group.length - incomingCount; return Object.freeze({ - id: digest("session", [corpusRevision, contactId, String(index), ...group.map(({ message }) => message.id)]), + id: digest2("session", [corpusRevision, contactId, String(index), ...group.map(({ message }) => message.id)]), startedAt: first.message.sentAt, endedAt: last.message.sentAt, durationSeconds: round((last.milliseconds - first.milliseconds) / 1000, 3), @@ -1725,7 +2873,7 @@ function burstsFor(messages, sessions, corpusRevision, contactId, burstGapSecond const textBodies = bodies(block.messages); result.push(Object.freeze({ metric: Object.freeze({ - id: digest("burst", [corpusRevision, contactId, session.id, ...messageIds]), + id: digest2("burst", [corpusRevision, contactId, session.id, ...messageIds]), sessionId: session.id, startedAt: first.message.sentAt, endedAt: last.message.sentAt, @@ -1807,7 +2955,7 @@ function responsesFor(bursts, corpusRevision, contactId) { const incomingIds = Object.freeze(incoming.messages.map(({ message }) => message.id)); const outgoingIds = Object.freeze(outgoing.messages.map(({ message }) => message.id)); result.push(Object.freeze({ - id: digest("response", [corpusRevision, contactId, ...incomingIds, "->", ...outgoingIds]), + id: digest2("response", [corpusRevision, contactId, ...incomingIds, "->", ...outgoingIds]), startedAt: incoming.messages[0].message.sentAt, incomingMessageIds: incomingIds, outgoingMessageIds: outgoingIds, @@ -1898,14 +3046,41 @@ function tempoMetrics(messages, responses) { multiQuestionEpisodes: responses.filter((response) => response.incomingQuestions > 1).length }); } -function reactionMetrics(messages) { - const reactions = messages.filter(({ message }) => message.kind === "reaction" && message.retractedAt === null); - const outgoing = reactions.filter(({ message }) => message.direction === "outgoing").length; - const outgoingActions = messages.filter(({ message }) => message.direction === "outgoing" && timelineEligible(message)).length; +function reactionMetrics(messages, facts) { + const legacy = messages.filter(({ message }) => message.kind === "reaction" && message.retractedAt === null).map(({ message }) => ({ + id: message.id, + externalId: message.sourceGuid, + targetExternalId: message.replyToSourceGuid ?? message.sourceGuid, + conversationId: message.conversationId, + direction: message.direction, + body: "unknown", + reactedAt: message.sentAt, + state: "active" + })); + const merged = new Map(legacy.map((fact) => [fact.id, fact])); + for (const fact of facts ?? []) + merged.set(fact.id, fact); + const source = [...merged.values()]; + const ids = new Set; + const reactions = source.filter((fact, index) => { + if (typeof fact.id !== "string" || fact.id.length === 0 || ids.has(fact.id) || fact.direction !== null && fact.direction !== "incoming" && fact.direction !== "outgoing" || typeof fact.body !== "string" || fact.state !== "active" && fact.state !== "removed") + throw new Error(`reactionFacts[${index}] is invalid`); + if (fact.reactedAt !== null) + canonicalTimestamp(fact.reactedAt, `reactionFacts[${index}].reactedAt`); + ids.add(fact.id); + return fact.state === "active"; + }); + const outgoing = reactions.filter(({ direction }) => direction === "outgoing").length; + const incoming = reactions.filter(({ direction }) => direction === "incoming").length; + const unknownDirection = reactions.length - outgoing - incoming; + const outgoingActions = messages.filter(({ message }) => message.kind !== "reaction" && message.direction === "outgoing" && timelineEligible(message)).length + outgoing; return Object.freeze({ total: reactions.length, - incoming: reactions.length - outgoing, + incoming, outgoing, + unknownDirection, + dated: reactions.filter(({ reactedAt }) => reactedAt !== null).length, + undated: reactions.filter(({ reactedAt }) => reactedAt === null).length, outgoingReactionRatio: ratio(outgoing, outgoingActions) }); } @@ -1922,9 +3097,26 @@ function analyzeContact(messages, corpusRevision, contactId, options = {}) { throw new Error("burstGapSeconds cannot exceed sessionGapSeconds"); } const ordered = orderedMessages(messages); - const sessions = sessionsFor(ordered, corpusRevision, contactId, sessionGapSeconds); - const burstRecords = burstsFor(ordered, sessions, corpusRevision, contactId, burstGapSeconds); - const responses = responsesFor(burstRecords, corpusRevision, contactId); + const byConversation = new Map; + for (const row of ordered) { + const rows = byConversation.get(row.message.conversationId) ?? []; + rows.push(row); + byConversation.set(row.message.conversationId, rows); + } + const sessions = []; + const burstRecords = []; + const responses = []; + for (const conversationId of [...byConversation.keys()].sort((left, right) => left.localeCompare(right, "en-US"))) { + const rows = Object.freeze(byConversation.get(conversationId)); + const conversationSessions = sessionsFor(rows, corpusRevision, contactId, sessionGapSeconds); + const conversationBursts = burstsFor(rows, conversationSessions, corpusRevision, contactId, burstGapSeconds); + sessions.push(...conversationSessions); + burstRecords.push(...conversationBursts); + responses.push(...responsesFor(conversationBursts, corpusRevision, contactId)); + } + sessions.sort((left, right) => left.startedAt.localeCompare(right.startedAt, "en-US") || left.id.localeCompare(right.id, "en-US")); + burstRecords.sort((left, right) => left.metric.startedAt.localeCompare(right.metric.startedAt, "en-US") || left.metric.id.localeCompare(right.metric.id, "en-US")); + responses.sort((left, right) => left.startedAt.localeCompare(right.startedAt, "en-US") || left.id.localeCompare(right.id, "en-US")); return Object.freeze({ schemaVersion: METRICS_SCHEMA_VERSION, corpusRevision, @@ -1937,11 +3129,11 @@ function analyzeContact(messages, corpusRevision, contactId, options = {}) { textMessageCount: ordered.filter(({ message }) => message.retractedAt === null && message.kind === "text" && message.body !== null).length, sessionGapSeconds, burstGapSeconds, - sessions, + sessions: Object.freeze(sessions), bursts: Object.freeze(burstRecords.map(({ metric }) => metric)), - responses, + responses: Object.freeze(responses), tempo: tempoMetrics(ordered, responses), - reactions: reactionMetrics(ordered), + reactions: reactionMetrics(ordered, options.reactionFacts), surface: surfaceMetrics(ordered) }); } @@ -2166,6 +3358,9 @@ function aggregateStudyMetrics(metrics) { total: metrics.reactions.total, incoming: metrics.reactions.incoming, outgoing: metrics.reactions.outgoing, + unknownDirection: metrics.reactions.unknownDirection, + dated: metrics.reactions.dated, + undated: metrics.reactions.undated, outgoingReactionRatio: metrics.reactions.outgoingReactionRatio }), surface: Object.freeze({ @@ -2283,7 +3478,7 @@ function buildEvaluationPackets(messages, metrics, options) { emittedBodyBytes += candidate.bodyBytes; } const caseIds = selected.map(({ example }) => example.id); - const evaluationId = digest("evaluation", [ + const evaluationId = digest2("evaluation", [ metrics.corpusRevision, evidenceRevision, metrics.contactId, @@ -2352,46 +3547,46 @@ import { randomBytes } from "crypto"; import { chmod, link, - lstat, + lstat as lstat2, mkdir, - open, + open as open2, readFile, - realpath, + realpath as realpath2, stat, unlink } from "fs/promises"; import { homedir as homedir3, platform } from "os"; -import { basename as basename3, dirname as dirname2, isAbsolute as isAbsolute3, join as join3, resolve as resolve3 } from "path"; +import { basename as basename3, dirname as dirname2, isAbsolute as isAbsolute4, join as join4, resolve as resolve4 } from "path"; function defaultDataDirectory() { const override = process.env.XDG_DATA_HOME; if (override !== undefined && override.trim() !== "") { - if (!isAbsolute3(override)) { + if (!isAbsolute4(override)) { throw new CliError("unsafe-path", "XDG_DATA_HOME must be absolute"); } - return join3(resolve3(override), "message-like-me"); + return join4(resolve4(override), "message-like-me"); } if (platform() === "darwin") { - return join3(homedir3(), "Library", "Application Support", "Message Like Me"); + return join4(homedir3(), "Library", "Application Support", "Message Like Me"); } - return join3(homedir3(), ".local", "share", "message-like-me"); + return join4(homedir3(), ".local", "share", "message-like-me"); } function dataPaths(explicit) { - if (explicit !== undefined && !isAbsolute3(explicit)) { + if (explicit !== undefined && !isAbsolute4(explicit)) { throw new CliError("unsafe-path", "Data directory must be absolute"); } - const root = explicit === undefined ? defaultDataDirectory() : resolve3(explicit); - if (!isAbsolute3(root)) + const root = explicit === undefined ? defaultDataDirectory() : resolve4(explicit); + if (!isAbsolute4(root)) throw new CliError("unsafe-path", "Data directory must be absolute"); return { root, - database: join3(root, "message-like-me.sqlite3"), - installKey: join3(root, "install.key"), - packets: join3(root, "study-packets") + database: join4(root, "message-like-me.sqlite3"), + installKey: join4(root, "install.key"), + packets: join4(root, "study-packets") }; } async function existingType(path) { try { - return await lstat(path); + return await lstat2(path); } catch (error) { if (error.code === "ENOENT") return null; @@ -2414,26 +3609,26 @@ async function ensurePrivateDirectory(path) { throw new CliError("unsafe-path", `${path} must be a directory`); } await mkdir(path, { recursive: true, mode: 448 }); - const after = await lstat(path); + const after = await lstat2(path); if (after.isSymbolicLink() || !after.isDirectory()) { throw new CliError("unsafe-path", `${path} is not a physical directory`); } await assertOwned(path); await chmod(path, 448); - return realpath(path); + return realpath2(path); } async function initializeDataPaths(paths) { const physicalRoot = await ensurePrivateDirectory(paths.root); - const physicalPackets = await ensurePrivateDirectory(join3(physicalRoot, "study-packets")); + const physicalPackets = await ensurePrivateDirectory(join4(physicalRoot, "study-packets")); return { root: physicalRoot, - database: join3(physicalRoot, basename3(paths.database)), - installKey: join3(physicalRoot, basename3(paths.installKey)), + database: join4(physicalRoot, basename3(paths.database)), + installKey: join4(physicalRoot, basename3(paths.installKey)), packets: physicalPackets }; } async function assertPrivateRegularFile(path) { - const metadata = await lstat(path); + const metadata = await lstat2(path); if (metadata.isSymbolicLink() || !metadata.isFile()) { throw new CliError("unsafe-path", `${path} must be a physical regular file`); } @@ -2452,7 +3647,7 @@ async function loadOrCreateInstallKey(path) { } const key = randomBytes(32); try { - const handle = await open(path, "wx", 384); + const handle = await open2(path, "wx", 384); try { await handle.writeFile(`${key.toString("hex")} `, "utf8"); @@ -2471,7 +3666,7 @@ async function loadOrCreateInstallKey(path) { } async function privateOutputDirectory(path) { await mkdir(path, { recursive: true, mode: 448 }); - const requested = await lstat(path); + const requested = await lstat2(path); if (requested.isSymbolicLink() || !requested.isDirectory()) { throw new CliError("unsafe-path", `${path} must be a physical directory`); } @@ -2479,12 +3674,12 @@ async function privateOutputDirectory(path) { if ((requested.mode & 63) !== 0) { throw new CliError("unsafe-path", `${path} must already have private permissions; refusing to change a caller-owned directory`); } - return realpath(path); + return realpath2(path); } async function syncDirectory(path) { let handle = null; try { - handle = await open(path, "r"); + handle = await open2(path, "r"); await handle.sync(); } catch (error) { const code = error.code; @@ -2496,12 +3691,12 @@ async function syncDirectory(path) { } } async function atomicWritePrivate(path, bytes) { - const parent = await privateOutputDirectory(dirname2(resolve3(path))); - const destination = join3(parent, basename3(path)); - const temporary = join3(parent, `.${basename3(path)}.${process.pid}.${randomBytes(8).toString("hex")}.tmp`); + const parent = await privateOutputDirectory(dirname2(resolve4(path))); + const destination = join4(parent, basename3(path)); + const temporary = join4(parent, `.${basename3(path)}.${process.pid}.${randomBytes(8).toString("hex")}.tmp`); let published = false; try { - const handle = await open(temporary, "wx", 384); + const handle = await open2(temporary, "wx", 384); try { await handle.writeFile(bytes); await handle.chmod(384); @@ -2531,16 +3726,16 @@ async function atomicWritePrivate(path, bytes) { } // src/profile.ts -import { constants as fsConstants3 } from "fs"; -import { open as open2 } from "fs/promises"; +import { constants as fsConstants4 } from "fs"; +import { open as open3 } from "fs/promises"; var MAX_PROFILE_FILE_BYTES = 4 * 1024 * 1024; -function object(value, label) { +function object2(value, label) { if (value === null || typeof value !== "object" || Array.isArray(value)) { throw new CliError("invalid-data", `${label} must be an object`); } return value; } -function exactKeys(value, keys, label) { +function exactKeys2(value, keys, label) { const expected = new Set(keys); for (const key of Object.keys(value)) { if (!expected.has(key)) @@ -2579,7 +3774,7 @@ function isoTimestamp(value, label) { function nullableIsoTimestamp(value, label) { return value === null ? null : isoTimestamp(value, label); } -function digest2(value, label) { +function digest3(value, label) { const parsed = text(value, label, 64); if (!/^[a-f0-9]{64}$/u.test(parsed)) { throw new CliError("invalid-data", `${label} must be lowercase SHA-256`); @@ -2599,8 +3794,8 @@ function confidenceLevel(value, label) { return value; } function parseStyleProfileV1(value) { - const root = object(value, "profile"); - exactKeys(root, [ + const root = object2(value, "profile"); + exactKeys2(root, [ "schemaVersion", "contactId", "corpusRevision", @@ -2624,8 +3819,8 @@ function parseStyleProfileV1(value) { if (!/^[a-f0-9]{64}$/u.test(packetSha256)) { throw new CliError("invalid-data", "profile.packetSha256 must be lowercase SHA-256"); } - const prose = object(root.prose, "profile.prose"); - exactKeys(prose, [ + const prose = object2(root.prose, "profile.prose"); + exactKeys2(prose, [ "register", "capitalization", "punctuation", @@ -2636,18 +3831,18 @@ function parseStyleProfileV1(value) { "closings", "notablePatterns" ], "profile.prose"); - const tempo = object(root.tempo, "profile.tempo"); - exactKeys(tempo, [ + const tempo = object2(root.tempo, "profile.tempo"); + exactKeys2(tempo, [ "defaultBundle", "singleLongMessage", "multipleMessages", "responseTiming", "followUps" ], "profile.tempo"); - const replies = object(root.replies, "profile.replies"); - exactKeys(replies, ["usage", "useWhen", "avoidWhen"], "profile.replies"); - const confidence = object(root.confidence, "profile.confidence"); - exactKeys(confidence, ["overall", "limitations"], "profile.confidence"); + const replies = object2(root.replies, "profile.replies"); + exactKeys2(replies, ["usage", "useWhen", "avoidWhen"], "profile.replies"); + const confidence = object2(root.confidence, "profile.confidence"); + exactKeys2(confidence, ["overall", "limitations"], "profile.confidence"); if (!Array.isArray(root.contexts) || root.contexts.length > 32) { throw new CliError("invalid-data", "profile.contexts must contain at most 32 items"); } @@ -2686,8 +3881,8 @@ function parseStyleProfileV1(value) { avoidWhen: textArray(replies.avoidWhen, "profile.replies.avoidWhen") }, contexts: root.contexts.map((item, index) => { - const context = object(item, `profile.contexts[${index}]`); - exactKeys(context, [ + const context = object2(item, `profile.contexts[${index}]`); + exactKeys2(context, [ "when", "incomingPattern", "responseStrategy", @@ -2713,8 +3908,8 @@ function parseStyleProfileV1(value) { }; } function parseStyleProfileV2(value) { - const root = object(value, "profile"); - exactKeys(root, [ + const root = object2(value, "profile"); + exactKeys2(root, [ "schemaVersion", "contactId", "corpusRevision", @@ -2735,10 +3930,10 @@ function parseStyleProfileV2(value) { throw new CliError("invalid-data", `profile.schemaVersion must be ${PROFILE_SCHEMA_VERSION}`); } const contactId = text(root.contactId, "profile.contactId", 128); - const corpusRevision = digest2(root.corpusRevision, "profile.corpusRevision"); - const packetSha256 = digest2(root.packetSha256, "profile.packetSha256"); - const evidence = object(root.evidence, "profile.evidence"); - exactKeys(evidence, [ + const corpusRevision = digest3(root.corpusRevision, "profile.corpusRevision"); + const packetSha256 = digest3(root.packetSha256, "profile.packetSha256"); + const evidence = object2(root.evidence, "profile.evidence"); + exactKeys2(evidence, [ "evidenceRevision", "firstMessageAt", "lastMessageAt", @@ -2763,8 +3958,8 @@ function parseStyleProfileV2(value) { if (after !== null && before !== null && after >= before) { throw new CliError("invalid-data", "profile.evidence.after must be earlier than before"); } - const prose = object(root.prose, "profile.prose"); - exactKeys(prose, [ + const prose = object2(root.prose, "profile.prose"); + exactKeys2(prose, [ "register", "capitalization", "punctuation", @@ -2775,18 +3970,18 @@ function parseStyleProfileV2(value) { "closingPatterns", "notablePatterns" ], "profile.prose"); - const tempo = object(root.tempo, "profile.tempo"); - exactKeys(tempo, [ + const tempo = object2(root.tempo, "profile.tempo"); + exactKeys2(tempo, [ "defaultBundle", "singleLongMessage", "multipleMessages", "responseTiming", "followUps" ], "profile.tempo"); - const replies = object(root.replies, "profile.replies"); - exactKeys(replies, ["usage", "useWhen", "avoidWhen"], "profile.replies"); - const confidence = object(root.confidence, "profile.confidence"); - exactKeys(confidence, [ + const replies = object2(root.replies, "profile.replies"); + exactKeys2(replies, ["usage", "useWhen", "avoidWhen"], "profile.replies"); + const confidence = object2(root.confidence, "profile.confidence"); + exactKeys2(confidence, [ "overall", "prose", "tempo", @@ -2801,8 +3996,8 @@ function parseStyleProfileV2(value) { throw new CliError("invalid-data", "profile.claims must contain at most 64 items"); } const contexts = root.contexts.map((item, index) => { - const context = object(item, `profile.contexts[${index}]`); - exactKeys(context, [ + const context = object2(item, `profile.contexts[${index}]`); + exactKeys2(context, [ "when", "incomingPattern", "responseStrategy", @@ -2820,8 +4015,8 @@ function parseStyleProfileV2(value) { }; }); const claims = root.claims.map((item, index) => { - const claim = object(item, `profile.claims[${index}]`); - exactKeys(claim, [ + const claim = object2(item, `profile.claims[${index}]`); + exactKeys2(claim, [ "dimension", "statement", "basis", @@ -2856,7 +4051,7 @@ function parseStyleProfileV2(value) { packetSha256, analyzedAt: isoTimestamp(root.analyzedAt, "profile.analyzedAt"), evidence: { - evidenceRevision: digest2(evidence.evidenceRevision, "profile.evidence.evidenceRevision"), + evidenceRevision: digest3(evidence.evidenceRevision, "profile.evidence.evidenceRevision"), firstMessageAt, lastMessageAt, messageCount: nonNegativeInteger(evidence.messageCount, "profile.evidence.messageCount"), @@ -2906,7 +4101,7 @@ function parseStyleProfileV2(value) { }; } function parseStyleProfile(value) { - const root = object(value, "profile"); + const root = object2(value, "profile"); if (root.schemaVersion === LEGACY_PROFILE_SCHEMA_VERSION) return parseStyleProfileV1(root); if (root.schemaVersion === PROFILE_SCHEMA_VERSION) @@ -2916,7 +4111,7 @@ function parseStyleProfile(value) { async function readStyleProfile(path) { let parsed; try { - const handle = await open2(path, fsConstants3.O_RDONLY | fsConstants3.O_NOFOLLOW); + const handle = await open3(path, fsConstants4.O_RDONLY | fsConstants4.O_NOFOLLOW); try { const before = await handle.stat(); const privateMode = (before.mode & 63) === 0; @@ -2957,13 +4152,13 @@ async function readStyleProfile(path) { } // src/skill-install.ts -import { cp, lstat as lstat2, mkdir as mkdir2, realpath as realpath2, rm } from "fs/promises"; +import { cp, lstat as lstat3, mkdir as mkdir2, realpath as realpath3, rm } from "fs/promises"; import { homedir as homedir4 } from "os"; -import { dirname as dirname3, join as join4, resolve as resolve4 } from "path"; +import { dirname as dirname3, join as join5, resolve as resolve5 } from "path"; import { fileURLToPath } from "url"; async function exists(path) { try { - await lstat2(path); + await lstat3(path); return true; } catch (error) { if (error.code === "ENOENT") @@ -2972,25 +4167,25 @@ async function exists(path) { } } function bundledSkillPath() { - return resolve4(dirname3(fileURLToPath(import.meta.url)), "../skills/message-like-me"); + return resolve5(dirname3(fileURLToPath(import.meta.url)), "../skills/message-like-me"); } function targetRoot(target, scope, projectDirectory) { const directory = target === "codex" ? ".codex" : target === "claude" ? ".claude" : ".agents"; - return scope === "user" ? join4(homedir4(), directory, "skills") : join4(resolve4(projectDirectory), directory, "skills"); + return scope === "user" ? join5(homedir4(), directory, "skills") : join5(resolve5(projectDirectory), directory, "skills"); } async function installSkill(options) { const source = bundledSkillPath(); if (!await exists(source)) throw new CliError("not-found", `Bundled skill is missing at ${source}`); - const sourceMetadata = await lstat2(source); + const sourceMetadata = await lstat3(source); if (sourceMetadata.isSymbolicLink() || !sourceMetadata.isDirectory()) { throw new CliError("unsafe-path", "Bundled skill must be a physical directory"); } const root = targetRoot(options.target, options.scope, options.projectDirectory ?? process.cwd()); await mkdir2(root, { recursive: true, mode: 448 }); - const destination = join4(root, "message-like-me"); + const destination = join5(root, "message-like-me"); if (await exists(destination)) { - const metadata = await lstat2(destination); + const metadata = await lstat3(destination); if (metadata.isSymbolicLink()) { throw new CliError("unsafe-path", `Refusing to replace symbolic link ${destination}`); } @@ -3000,27 +4195,46 @@ async function installSkill(options) { await rm(destination, { recursive: true, force: true }); } await cp(source, destination, { recursive: true, errorOnExist: true }); - return realpath2(destination); + return realpath3(destination); } // src/store.ts import { Database as Database3 } from "bun:sqlite"; +import { createHash as createHash4 } from "crypto"; import { closeSync, - constants as fsConstants4, + constants as fsConstants5, fchmodSync, fstatSync, lstatSync as lstatSync3, openSync } from "fs"; -var STORE_SCHEMA_VERSION = 2; +var STORE_SCHEMA_VERSION = 3; var PERSON_SCOPE_PREFIX = "person_"; +var IMESSAGE_SOURCE_ID = "source_imessage_local"; var SCHEMA = ` PRAGMA foreign_keys = ON; CREATE TABLE IF NOT EXISTS metadata ( key TEXT PRIMARY KEY, value TEXT NOT NULL ) STRICT; + CREATE TABLE IF NOT EXISTS corpus_sources ( + id TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('imessage', 'bundle')), + provider TEXT NOT NULL, + network TEXT, + account_id TEXT, + external_id TEXT NOT NULL, + input_revision TEXT NOT NULL, + revision TEXT NOT NULL, + generated_at TEXT, + producer_json TEXT NOT NULL, + coverage_json TEXT NOT NULL, + manifest_sha256 TEXT, + identity_json TEXT NOT NULL, + warnings_json TEXT NOT NULL, + ingested_at TEXT NOT NULL + ) STRICT; CREATE TABLE IF NOT EXISTS conversations ( id TEXT PRIMARY KEY, source_key TEXT NOT NULL, @@ -3031,6 +4245,15 @@ var SCHEMA = ` private_participants_json TEXT NOT NULL, is_group INTEGER NOT NULL CHECK (is_group IN (0, 1)) ) STRICT; + CREATE TABLE IF NOT EXISTS conversation_sources ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS conversation_sources_lookup + ON conversation_sources(source_id, conversation_id); CREATE TABLE IF NOT EXISTS messages ( id TEXT PRIMARY KEY, source_row_id INTEGER NOT NULL, @@ -3051,6 +4274,54 @@ var SCHEMA = ` CREATE INDEX IF NOT EXISTS messages_conversation_time ON messages(conversation_id, sent_at, source_row_id, id); CREATE INDEX IF NOT EXISTS messages_source_guid ON messages(source_guid); + CREATE TABLE IF NOT EXISTS message_provenance ( + message_id TEXT PRIMARY KEY REFERENCES messages(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + reply_to_external_id TEXT, + attachments_json TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS message_provenance_source + ON message_provenance(source_id, message_id); + CREATE TABLE IF NOT EXISTS corpus_reaction_facts ( + id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + external_id TEXT NOT NULL, + target_external_id TEXT NOT NULL, + conversation_id TEXT REFERENCES conversations(id) ON DELETE SET NULL, + direction TEXT CHECK (direction IN ('incoming','outgoing')), + body TEXT NOT NULL, + reacted_at TEXT, + state TEXT NOT NULL CHECK (state IN ('active','removed')), + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS corpus_reaction_facts_source + ON corpus_reaction_facts(source_id,conversation_id,id); + CREATE TABLE IF NOT EXISTS corpus_source_records ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('account','participant','reaction','tombstone','excluded-message') + ), + external_id TEXT NOT NULL, + record_json TEXT NOT NULL, + PRIMARY KEY (source_id, kind, external_id) + ) WITHOUT ROWID, STRICT; + CREATE TABLE IF NOT EXISTS corpus_source_suppressions ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('conversation','message','reaction','reaction-timeline','participant','account') + ), + local_id TEXT NOT NULL, + external_id TEXT NOT NULL, + suppressed_at TEXT NOT NULL, + reason TEXT NOT NULL CHECK ( + reason IN ('authoritative-absence','tombstone','explicit-exclusion','replacement','reappeared') + ), + suppressed INTEGER NOT NULL CHECK (suppressed IN (0,1)), + PRIMARY KEY (source_id, kind, local_id) + ) WITHOUT ROWID, STRICT; CREATE TABLE IF NOT EXISTS study_packets ( sha256 TEXT PRIMARY KEY, contact_id TEXT NOT NULL, @@ -3107,6 +4378,82 @@ var SCHEMA = ` CREATE INDEX IF NOT EXISTS conversation_contact_labels_lookup ON conversation_contact_labels(normalized_label, conversation_id); `; +var SOURCE_SCHEMA = ` + CREATE TABLE IF NOT EXISTS corpus_sources ( + id TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('imessage', 'bundle')), + provider TEXT NOT NULL, + network TEXT, + account_id TEXT, + external_id TEXT NOT NULL, + input_revision TEXT NOT NULL, + revision TEXT NOT NULL, + generated_at TEXT, + producer_json TEXT NOT NULL, + coverage_json TEXT NOT NULL, + manifest_sha256 TEXT, + identity_json TEXT NOT NULL, + warnings_json TEXT NOT NULL, + ingested_at TEXT NOT NULL + ) STRICT; + CREATE TABLE IF NOT EXISTS conversation_sources ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS conversation_sources_lookup + ON conversation_sources(source_id, conversation_id); + CREATE TABLE IF NOT EXISTS message_provenance ( + message_id TEXT PRIMARY KEY REFERENCES messages(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + reply_to_external_id TEXT, + attachments_json TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS message_provenance_source + ON message_provenance(source_id, message_id); + CREATE TABLE IF NOT EXISTS corpus_reaction_facts ( + id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + external_id TEXT NOT NULL, + target_external_id TEXT NOT NULL, + conversation_id TEXT REFERENCES conversations(id) ON DELETE SET NULL, + direction TEXT CHECK (direction IN ('incoming','outgoing')), + body TEXT NOT NULL, + reacted_at TEXT, + state TEXT NOT NULL CHECK (state IN ('active','removed')), + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS corpus_reaction_facts_source + ON corpus_reaction_facts(source_id,conversation_id,id); + CREATE TABLE IF NOT EXISTS corpus_source_records ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('account','participant','reaction','tombstone','excluded-message') + ), + external_id TEXT NOT NULL, + record_json TEXT NOT NULL, + PRIMARY KEY (source_id, kind, external_id) + ) WITHOUT ROWID, STRICT; + CREATE TABLE IF NOT EXISTS corpus_source_suppressions ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('conversation','message','reaction','reaction-timeline','participant','account') + ), + local_id TEXT NOT NULL, + external_id TEXT NOT NULL, + suppressed_at TEXT NOT NULL, + reason TEXT NOT NULL CHECK ( + reason IN ('authoritative-absence','tombstone','explicit-exclusion','replacement','reappeared') + ), + suppressed INTEGER NOT NULL CHECK (suppressed IN (0,1)), + PRIMARY KEY (source_id, kind, local_id) + ) WITHOUT ROWID, STRICT; +`; var CONTACT_SCOPE_SCHEMA = ` CREATE TABLE IF NOT EXISTS conversation_contact_scopes ( conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, @@ -3313,7 +4660,15 @@ function personScope(database, addressBookContactId) { SELECT association.conversation_id FROM conversation_contact_scopes association JOIN conversations conversation ON conversation.id=association.conversation_id + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id WHERE association.contact_id=? AND conversation.is_group=0 + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) ORDER BY association.conversation_id `, addressBookContactId); if (rows.length === 0) @@ -3332,7 +4687,15 @@ function analysisScope(database, contactId) { return personScope(database, addressBookContactId); } } - const conversation = get(database, "SELECT id FROM conversations WHERE id=?", contactId); + const conversation = get(database, `SELECT conversation.id FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id + WHERE conversation.id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + )`, contactId); if (conversation === null) return null; const matched = get(database, ` @@ -3350,27 +4713,53 @@ function analysisScope(database, contactId) { function messageRowsForScope(database, scope, exactConversationId, window = UNBOUNDED_EVIDENCE_WINDOW) { if (exactConversationId !== undefined) { return all(database, ` - SELECT * FROM messages WHERE conversation_id=? - AND (? IS NULL OR sent_at>=?) AND (? IS NULL OR sent_at=?) AND (? IS NULL OR message.sent_at=?) AND (? IS NULL OR message.sent_at=?) AND (? IS NULL OR sent_at=?) AND (? IS NULL OR message.sent_at row.reacted_at === null ? window.after === null && window.before === null : (window.after === null || row.reacted_at >= window.after) && (window.before === null || row.reacted_at < window.before)).map((row) => ({ + id: row.id, + externalId: row.external_id, + targetExternalId: row.target_external_id, + conversationId: row.conversation_id, + direction: row.direction, + body: row.body, + reactedAt: row.reacted_at, + state: row.state + })); +} function scopeEvidenceRevision(database, scope, exactConversationId, window = UNBOUNDED_EVIDENCE_WINDOW) { const conversationIds = exactConversationId === undefined ? scope.conversationIds : Object.freeze([exactConversationId]); const messages = messageRowsForScope(database, scope, exactConversationId, window).map(corpusMessage); - return sha256(canonicalJson(window.after === null && window.before === null ? { + const reactions = reactionFactsForScope(database, scope, window); + return sha256(canonicalJson(reactions.length > 0 ? { + schemaVersion: 3, + scopeId: scope.id, + conversationIds, + evidenceWindow: window, + messages, + reactions + } : window.after === null && window.before === null ? { schemaVersion: 1, scopeId: scope.id, conversationIds, @@ -3422,9 +4848,25 @@ function scopeMessageCounts(database, scope) { coalesce(sum(CASE WHEN message.direction='outgoing' THEN 1 ELSE 0 END),0) AS outgoing_count`; const row = scope.kind === "person" ? get(database, `${select} FROM messages message + JOIN message_provenance provenance ON provenance.message_id=message.id JOIN conversation_contact_scopes association ON association.conversation_id=message.conversation_id - WHERE association.contact_id=?`, scope.addressBookContactId) : get(database, `${select} FROM messages message WHERE message.conversation_id=?`, scope.conversationIds[0]); + WHERE association.contact_id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + )`, scope.addressBookContactId) : get(database, `${select} + FROM messages message + JOIN message_provenance provenance ON provenance.message_id=message.id + WHERE message.conversation_id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + )`, scope.conversationIds[0]); return { firstMessageAt: row?.first_message_at ?? null, lastMessageAt: row?.last_message_at ?? null, @@ -3463,11 +4905,49 @@ function backfillLegacyEvidence(database) { } } } +function backfillLegacySource(database) { + const conversations = get(database, "SELECT count(*) AS value FROM conversations")?.value ?? 0; + const assigned = get(database, "SELECT count(*) AS value FROM conversation_sources")?.value ?? 0; + if (assigned !== 0 && assigned !== conversations) { + throw new CliError("invalid-data", "Local store has partially assigned corpus source ownership"); + } + if (conversations === 0 || assigned === conversations) + return; + const revision = scalarText(database, "corpus_revision"); + if (revision === null || !/^[a-f0-9]{64}$/u.test(revision)) { + throw new CliError("invalid-data", "Legacy local store has no valid corpus revision"); + } + const identity = scalarText(database, "source_identity") ?? canonicalJson({ migrated: true }); + const warnings = scalarText(database, "warnings") ?? canonicalJson([]); + const ingestedAt = scalarText(database, "ingested_at") ?? "1970-01-01T00:00:00.000Z"; + database.query(` + INSERT INTO corpus_sources( + id,kind,provider,network,account_id,external_id,input_revision,revision,generated_at, + producer_json,coverage_json,manifest_sha256,identity_json,warnings_json,ingested_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + `).run(IMESSAGE_SOURCE_ID, "imessage", "apple", null, null, "local-imessage", revision, revision, null, canonicalJson({ id: "message-like-me", version: "legacy" }), canonicalJson({ history: "complete-current-local", observedFrom: null, observedTo: null }), null, identity, warnings, ingestedAt); + database.exec(` + INSERT INTO conversation_sources(conversation_id,source_id,external_id,metadata_json) + SELECT id,'${IMESSAGE_SOURCE_ID}',source_key,'{}' FROM conversations; + `); + const rows = all(database, ` + SELECT id,source_guid,reply_to_source_guid,attachment_count + FROM messages ORDER BY id + `); + const insert = database.query(` + INSERT INTO message_provenance( + message_id,source_id,external_id,reply_to_external_id,attachments_json,metadata_json + ) VALUES (?,?,?,?,?,?) + `); + for (const row of rows) { + insert.run(row.id, IMESSAGE_SOURCE_ID, row.source_guid, row.reply_to_source_guid, canonicalJson({ count: row.attachment_count, detailsAvailable: false }), canonicalJson({ migrated: true })); + } +} function initializeStoreSchema(database) { const existingStore = tableExists(database, "metadata"); - const version = userVersion(database); - if (version > STORE_SCHEMA_VERSION) { - throw new CliError("invalid-data", `Local store schema ${version} is newer than supported schema ${STORE_SCHEMA_VERSION}`); + const version2 = userVersion(database); + if (version2 > STORE_SCHEMA_VERSION) { + throw new CliError("invalid-data", `Local store schema ${version2} is newer than supported schema ${STORE_SCHEMA_VERSION}`); } if (!existingStore) { database.exec(SCHEMA); @@ -3479,6 +4959,7 @@ function initializeStoreSchema(database) { throw new CliError("invalid-data", `Local store is missing required table ${table}`); } } + database.exec(SOURCE_SCHEMA); transaction(database, () => { database.exec(CONTACT_SCOPE_SCHEMA); database.exec(` @@ -3494,12 +4975,13 @@ function initializeStoreSchema(database) { addColumn(database, "study_packets", "evidence_json TEXT"); addColumn(database, "profiles", "scope_id TEXT"); addColumn(database, "profiles", "evidence_revision TEXT"); + backfillLegacySource(database); backfillLegacyEvidence(database); database.exec(`PRAGMA user_version=${STORE_SCHEMA_VERSION}`); }); database.exec(SCHEMA); } -function rebuildConversationLabels(database, hmacKey2) { +function rebuildConversationLabels(database, hmacKey3) { database.exec("DELETE FROM conversation_contact_labels; DELETE FROM conversation_contact_scopes;"); const contacts = new Map(all(database, `SELECT id,private_label,normalized_label,label_basis,contacts_revision FROM addressbook_contacts ORDER BY id`).map((row) => [row.id, row])); @@ -3512,7 +4994,17 @@ function rebuildConversationLabels(database, hmacKey2) { owners.set(key, values); } const conversations = all(database, ` - SELECT id,private_participants_json FROM conversations WHERE is_group=0 ORDER BY id + SELECT conversation.id,conversation.private_participants_json + FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id + WHERE conversation.is_group=0 AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) + ORDER BY conversation.id `); const insertScope = database.query(`INSERT INTO conversation_contact_scopes( conversation_id,contact_id,contacts_revision @@ -3528,10 +5020,10 @@ function rebuildConversationLabels(database, hmacKey2) { let matchedWithoutLabel = 0; for (const conversation of conversations) { const normalizedHandles = stringArray(conversation.private_participants_json, `conversation ${conversation.id} participants`).map(normalizeContactHandle).filter((handle) => handle !== null); - if (normalizedHandles.length > 0 && owners.size > 0 && hmacKey2 === undefined) { + if (normalizedHandles.length > 0 && owners.size > 0 && hmacKey3 === undefined) { throw new CliError("internal", "The installation key is required to rebuild contact labels"); } - const keys = hmacKey2 === undefined ? new Set : new Set(normalizedHandles.map((handle) => `${handle.kind}\x00${contactHandleMatchId(hmacKey2, handle)}`)); + const keys = hmacKey3 === undefined ? new Set : new Set(normalizedHandles.map((handle) => `${handle.kind}\x00${contactHandleMatchId(hmacKey3, handle)}`)); if (keys.size > 0) eligibleConversations += 1; const candidates = new Set; @@ -3597,7 +5089,7 @@ function hardenDatabaseFiles(path) { ]) { let descriptor; try { - descriptor = openSync(candidate, fsConstants4.O_RDONLY | fsConstants4.O_NOFOLLOW); + descriptor = openSync(candidate, fsConstants5.O_RDONLY | fsConstants5.O_NOFOLLOW); } catch (error) { const code = error.code; if (!required && code === "ENOENT") @@ -3621,6 +5113,225 @@ function hardenDatabaseFiles(path) { } } } +function globalCorpusRevision(database) { + const sources = all(database, "SELECT id,kind,input_revision,revision FROM corpus_sources ORDER BY id"); + if (sources.length === 0) + return null; + if (sources.length === 1 && sources[0].id === IMESSAGE_SOURCE_ID && sources[0].kind === "imessage") + return sources[0].input_revision; + return sha256(canonicalJson({ + schemaVersion: 1, + sources: sources.map(({ id, kind, revision }) => ({ id, kind, revision })) + })); +} +function sourceStateRevision(database, sourceId) { + const hash = createHash4("sha256"); + hash.update("message-like-me\x00stored-source-state-v1\x00", "utf8"); + const append = (kind, row) => { + const encoded = canonicalJson(row); + hash.update(`${kind.length}:${kind}${encoded.length}:`, "utf8").update(encoded, "utf8"); + }; + const source = get(database, ` + SELECT kind,provider,network,account_id,external_id,producer_json, + coverage_json,warnings_json + FROM corpus_sources WHERE id=? + `, sourceId); + if (source === null) + throw new CliError("internal", `Missing corpus source ${sourceId}`); + append("source", source); + for (const row of database.query(` + SELECT conversation.id,conversation.source_key,conversation.private_label, + conversation.service,conversation.participant_count, + conversation.participant_ids_json,conversation.private_participants_json, + conversation.is_group + FROM conversation_sources ownership + JOIN conversations conversation ON conversation.id=ownership.conversation_id + WHERE ownership.source_id=? + ORDER BY ownership.external_id,conversation.id + `).iterate(sourceId)) + append("conversation", row); + for (const row of database.query(` + SELECT message.id,message.source_row_id,message.source_guid,message.conversation_id, + message.sent_at,message.direction,message.body,message.body_source,message.kind, + message.reply_to_source_guid,message.edited_at,message.retracted_at,message.service, + message.attachment_count,provenance.external_id, + provenance.reply_to_external_id,provenance.attachments_json + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + ORDER BY provenance.external_id,message.id + `).iterate(sourceId)) + append("message", row); + for (const row of database.query(` + SELECT id,external_id,target_external_id,conversation_id,direction,body,reacted_at,state + FROM corpus_reaction_facts WHERE source_id=? ORDER BY external_id,id + `).iterate(sourceId)) + append("reaction-fact", row); + for (const row of database.query(` + SELECT kind,local_id,external_id,reason FROM corpus_source_suppressions + WHERE source_id=? AND suppressed=1 ORDER BY kind,local_id + `).iterate(sourceId)) + append("suppression", row); + return hash.digest("hex"); +} +function compareCodeUnits2(left, right) { + return left < right ? -1 : left > right ? 1 : 0; +} +function storedProviderSortKey(row) { + const parsed = parsedJson(row.metadata_json, `Message ${row.id} provenance`); + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) + return null; + const record = parsed; + const value = "providerSortKey" in record ? record.providerSortKey : record.sortKey; + return typeof value === "string" ? value : null; +} +function rerankBundleMessages(database, sourceId) { + const rows = all(database, ` + SELECT message.id,message.conversation_id,message.sent_at,message.kind, + provenance.external_id,provenance.metadata_json + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + ORDER BY message.conversation_id,message.id + `, sourceId); + const byConversation = new Map; + for (const value of rows) { + const row = Object.freeze({ ...value, provider_sort_key: storedProviderSortKey(value) }); + const values = byConversation.get(row.conversation_id) ?? []; + values.push(row); + byConversation.set(row.conversation_id, values); + } + const update = database.query("UPDATE messages SET source_row_id=? WHERE id=?"); + for (const values of byConversation.values()) { + for (const [index, row] of values.entries()) + update.run(-(index + 1), row.id); + values.sort((left, right) => { + const leftReaction = left.kind === "reaction"; + const rightReaction = right.kind === "reaction"; + if (leftReaction !== rightReaction) + return leftReaction ? 1 : -1; + if (!leftReaction) { + const sort = compareCodeUnits2(left.provider_sort_key ?? left.external_id, right.provider_sort_key ?? right.external_id); + if (sort !== 0) + return sort; + } + return compareCodeUnits2(left.sent_at, right.sent_at) || compareCodeUnits2(left.external_id, right.external_id) || compareCodeUnits2(left.id, right.id); + }); + for (const [index, row] of values.entries()) + update.run(index + 1, row.id); + } +} +function setCorpusRevision(database) { + const revision = globalCorpusRevision(database); + if (revision === null) { + database.query("DELETE FROM metadata WHERE key='corpus_revision'").run(); + return null; + } + database.query(` + INSERT INTO metadata(key,value) VALUES ('corpus_revision',?) + ON CONFLICT(key) DO UPDATE SET value=excluded.value + `).run(revision); + return revision; +} +function validSourceDescriptor(source) { + if (source.id !== IMESSAGE_SOURCE_ID && !/^source_[a-f0-9]{64}$/u.test(source.id) || source.kind !== "imessage" && source.kind !== "bundle" || source.provider.length < 1 || Buffer.byteLength(source.provider, "utf8") > 256 || !/^[a-f0-9]{64}$/u.test(source.revision) || source.externalId.length < 1 || Buffer.byteLength(source.externalId, "utf8") > 4096 || source.warnings.length > 130) + throw new CliError("invalid-data", `Corpus source ${source.id} is invalid`); + canonicalTimestampOrNull(source.generatedAt, `Corpus source ${source.id} generatedAt`); + if (source.kind === "bundle" && source.generatedAt === null) { + throw new CliError("invalid-data", `Bundle source ${source.id} requires generatedAt`); + } + canonicalTimestampOrNull(source.coverage.observedFrom, `Corpus source ${source.id} observedFrom`); + canonicalTimestampOrNull(source.coverage.observedTo, `Corpus source ${source.id} observedTo`); + if (source.coverage.observedFrom !== null && source.coverage.observedTo !== null && source.coverage.observedFrom > source.coverage.observedTo) + throw new CliError("invalid-data", `Corpus source ${source.id} has invalid coverage bounds`); + if (source.coverage.history !== "complete-current-local" && source.coverage.history !== "bounded" && source.coverage.history !== "unknown") + throw new CliError("invalid-data", `Corpus source ${source.id} has invalid history coverage`); + if (source.coverage.kind !== undefined && (source.coverage.kind.length < 1 || Buffer.byteLength(source.coverage.kind, "utf8") > 128 || /\p{Cc}/u.test(source.coverage.kind)) || source.coverage.reason !== undefined && source.coverage.reason !== null && (source.coverage.reason.length < 1 || Buffer.byteLength(source.coverage.reason, "utf8") > 128 || /\p{Cc}/u.test(source.coverage.reason))) + throw new CliError("invalid-data", `Corpus source ${source.id} has invalid coverage metadata`); + if (source.manifestSha256 !== null && !/^[a-f0-9]{64}$/u.test(source.manifestSha256)) + throw new CliError("invalid-data", `Corpus source ${source.id} has an invalid manifest digest`); + if (source.producer.id.length < 1 || source.producer.version.length < 1 || Buffer.byteLength(source.producer.id, "utf8") > 256 || Buffer.byteLength(source.producer.version, "utf8") > 256) + throw new CliError("invalid-data", `Corpus source ${source.id} has invalid producer identity`); + for (const warning of source.warnings) { + if (Buffer.byteLength(warning, "utf8") > 1024 || warning.includes("\x00")) { + throw new CliError("invalid-data", `Corpus source ${source.id} has an invalid warning`); + } + } +} +function validateSourceSnapshot(snapshot) { + validSourceDescriptor(snapshot.source); + if (snapshot.conversations.length > 2000000 || snapshot.messages.length > 2000000 || (snapshot.reactionFacts?.length ?? 0) > 2000000 || snapshot.conversationProvenance.length !== snapshot.conversations.length || snapshot.messageProvenance.length !== snapshot.messages.length) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} exceeds its result bounds`); + const conversationIds = new Set(snapshot.conversations.map(({ id }) => id)); + if (conversationIds.size !== snapshot.conversations.length) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} repeats conversation IDs`); + } + const conversationProvenance = new Map(snapshot.conversationProvenance.map((value) => [value.conversationId, value])); + if (conversationProvenance.size !== snapshot.conversationProvenance.length || [...conversationIds].some((id) => !conversationProvenance.has(id))) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid conversation provenance`); + const externalConversations = new Set; + for (const provenance of snapshot.conversationProvenance) { + if (provenance.externalId.length < 1 || Buffer.byteLength(provenance.externalId, "utf8") > 4096 || externalConversations.has(provenance.externalId)) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid external conversation IDs`); + externalConversations.add(provenance.externalId); + } + const messageIds = new Set(snapshot.messages.map(({ id }) => id)); + const messagesById = new Map(snapshot.messages.map((message) => [message.id, message])); + if (messageIds.size !== snapshot.messages.length) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} repeats message IDs`); + } + for (const message of snapshot.messages) { + if (!conversationIds.has(message.conversationId)) { + throw new CliError("invalid-data", `Message ${message.id} references an unknown conversation`); + } + } + const messageProvenance = new Map(snapshot.messageProvenance.map((value) => [value.messageId, value])); + if (messageProvenance.size !== snapshot.messageProvenance.length || [...messageIds].some((id) => !messageProvenance.has(id))) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid message provenance`); + const externalMessages = new Set; + for (const provenance of snapshot.messageProvenance) { + const message = messagesById.get(provenance.messageId); + if (provenance.externalId.length < 1 || Buffer.byteLength(provenance.externalId, "utf8") > 4096 || externalMessages.has(provenance.externalId) || provenance.attachments.length > 256 || provenance.providerSortKey !== null && (provenance.providerSortKey.length < 1 || Buffer.byteLength(provenance.providerSortKey, "utf8") > 1024 || /[\u0000-\u001f\u007f]/u.test(provenance.providerSortKey)) || (snapshot.source.kind === "bundle" ? message.kind === "reaction" === (provenance.providerSortKey !== null) : provenance.providerSortKey !== null)) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid external message provenance`); + externalMessages.add(provenance.externalId); + } + const auxiliaryIds = new Set; + for (const record of snapshot.auxiliaryRecords ?? []) { + const key = `${record.kind}\x00${record.id}`; + if (!["account", "participant", "reaction", "tombstone", "excluded-message"].includes(record.kind) || record.id.length < 1 || Buffer.byteLength(record.id, "utf8") > 4096 || auxiliaryIds.has(key)) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid auxiliary records`); + const encoded = canonicalJson(record.record); + if (typeof encoded !== "string" || Buffer.byteLength(encoded, "utf8") > 2 * 1024 * 1024) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has an oversized auxiliary record`); + } + auxiliaryIds.add(key); + } + const reactionIds = new Set; + const externalReactionIds = new Set; + for (const reaction of snapshot.reactionFacts ?? []) { + if (reaction.id.length < 1 || reaction.externalId.length < 1 || reaction.targetExternalId.length < 1 || Buffer.byteLength(reaction.id, "utf8") > 4096 || Buffer.byteLength(reaction.externalId, "utf8") > 4096 || Buffer.byteLength(reaction.targetExternalId, "utf8") > 4096 || Buffer.byteLength(reaction.body, "utf8") > 8 * 1024 || reactionIds.has(reaction.id) || externalReactionIds.has(reaction.externalId) || reaction.conversationId !== null && !conversationIds.has(reaction.conversationId) || reaction.direction !== null && reaction.direction !== "incoming" && reaction.direction !== "outgoing" || reaction.state !== "active" && reaction.state !== "removed") + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid reaction facts`); + canonicalTimestampOrNull(reaction.reactedAt, `Corpus source ${snapshot.source.id} reaction time`); + reactionIds.add(reaction.id); + externalReactionIds.add(reaction.externalId); + } + for (const deletion of snapshot.deletions ?? []) { + if (![ + "account", + "participant", + "conversation", + "message", + "reaction", + "reaction-timeline" + ].includes(deletion.entityKind) || deletion.externalId.length < 1 || Buffer.byteLength(deletion.externalId, "utf8") > 4096 || deletion.localEntityId !== null && Buffer.byteLength(deletion.localEntityId, "utf8") > 4096 || deletion.expectedConversationId !== undefined && (deletion.expectedConversationId.length < 1 || Buffer.byteLength(deletion.expectedConversationId, "utf8") > 4096) || deletion.reason !== undefined && ![ + "tombstone", + "explicit-exclusion", + "replacement" + ].includes(deletion.reason)) + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has an invalid deletion`); + canonicalTimestampOrNull(deletion.deletedAt, `Corpus source ${snapshot.source.id} deletion time`); + } +} class LocalStore { #database; @@ -3652,13 +5363,15 @@ class LocalStore { return scalarText(this.#database, "corpus_revision"); } sourceIdentity() { - const encoded = scalarText(this.#database, "source_identity"); - return encoded === null ? null : JSON.parse(encoded); + const encoded = get(this.#database, ` + SELECT identity_json FROM corpus_sources WHERE id=? + `, IMESSAGE_SOURCE_ID)?.identity_json ?? scalarText(this.#database, "source_identity"); + return encoded === null ? null : parsedJson(encoded, "Stored iMessage source identity"); } contactsRevision() { return scalarText(this.#database, "contacts_revision"); } - enrichContacts(snapshot, ingestedAt, hmacKey2) { + enrichContacts(snapshot, ingestedAt, hmacKey3) { if (snapshot.schemaVersion !== 1 || !/^[a-f0-9]{64}$/u.test(snapshot.snapshotSha256)) { throw new CliError("invalid-data", "The Contacts reader returned an invalid snapshot revision"); } @@ -3686,7 +5399,7 @@ class LocalStore { const handles = new Set; for (const handle of contact.handles) { const canonical = normalizeContactHandle(handle.normalizedValue); - if (canonical === null || canonical.kind !== handle.kind || canonical.normalizedValue !== handle.normalizedValue || handle.matchId !== contactHandleMatchId(hmacKey2, canonical) || !/^[a-f0-9]{64}$/u.test(handle.matchId)) + if (canonical === null || canonical.kind !== handle.kind || canonical.normalizedValue !== handle.normalizedValue || handle.matchId !== contactHandleMatchId(hmacKey3, canonical) || !/^[a-f0-9]{64}$/u.test(handle.matchId)) throw new CliError("invalid-data", "The Contacts reader returned a non-canonical handle"); const key = `${handle.kind}\x00${handle.matchId}`; if (handles.has(key)) { @@ -3714,7 +5427,7 @@ class LocalStore { insertHandle.run(contact.id, handle.kind, handle.matchId); } } - const projection = rebuildConversationLabels(this.#database, hmacKey2); + const projection = rebuildConversationLabels(this.#database, hmacKey3); const setMetadata = this.#database.query(` INSERT INTO metadata (key,value) VALUES (?,?) ON CONFLICT (key) DO UPDATE SET value=excluded.value @@ -3758,65 +5471,510 @@ class LocalStore { privateLabel: row.private_label })); } - replaceCorpus(snapshot, ingestedAt, hmacKey2) { - const corpusRevision = snapshot.source.snapshotSha256; - if (!/^[a-f0-9]{64}$/u.test(corpusRevision)) { - throw new CliError("invalid-data", "The iMessage reader returned an invalid corpus revision"); + replaceSources(snapshots, ingestedAt, hmacKey3) { + canonicalTimestampOrNull(ingestedAt, "Source ingest time"); + if (snapshots.length < 1) { + throw new CliError("invalid-data", "A source replacement must contain at least one source"); } - const conversationIds = new Set(snapshot.conversations.map((conversation) => conversation.id)); - if (conversationIds.size !== snapshot.conversations.length) { - throw new CliError("invalid-data", "The iMessage reader returned duplicate conversation IDs"); - } - const messageIds = new Set; - for (const message of snapshot.messages) { - if (!conversationIds.has(message.conversationId)) { - throw new CliError("invalid-data", `Message ${message.id} references an unknown conversation`); + const sourceIds = new Set; + for (const snapshot of snapshots) { + if (sourceIds.has(snapshot.source.id)) { + throw new CliError("invalid-data", `Source replacement repeats ${snapshot.source.id}`); } - if (messageIds.has(message.id)) - throw new CliError("invalid-data", `Duplicate message ID ${message.id}`); - messageIds.add(message.id); + sourceIds.add(snapshot.source.id); + validateSourceSnapshot(snapshot); } - transaction(this.#database, () => { - this.#database.exec("DELETE FROM messages; DELETE FROM conversations;"); - const insertConversation = this.#database.query(` - INSERT INTO conversations ( - id, source_key, private_label, service, participant_count, - participant_ids_json, private_participants_json, is_group - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + return transaction(this.#database, () => { + const upsertSource = this.#database.query(` + INSERT INTO corpus_sources( + id,kind,provider,network,account_id,external_id,input_revision,revision,generated_at, + producer_json,coverage_json,manifest_sha256,identity_json,warnings_json,ingested_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + kind=excluded.kind,provider=excluded.provider,network=excluded.network, + account_id=excluded.account_id,external_id=excluded.external_id, + input_revision=excluded.input_revision,generated_at=excluded.generated_at, + producer_json=excluded.producer_json,coverage_json=excluded.coverage_json, + manifest_sha256=excluded.manifest_sha256,identity_json=excluded.identity_json, + warnings_json=excluded.warnings_json,ingested_at=excluded.ingested_at `); - for (const conversation of snapshot.conversations) { - insertConversation.run(conversation.id, conversation.sourceKey, conversation.privateLabel, conversation.service, conversation.participantCount, canonicalJson(conversation.participantIds), canonicalJson(conversation.privateParticipants), conversation.group ? 1 : 0); - } - const insertMessage = this.#database.query(` - INSERT INTO messages ( - id, source_row_id, source_guid, conversation_id, sent_at, direction, - body, body_source, kind, reply_to_source_guid, edited_at, retracted_at, - service, attachment_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + const relabelSourceConversations = this.#database.query(` + UPDATE conversations SET service=? + WHERE id IN ( + SELECT conversation_id FROM conversation_sources WHERE source_id=? + ) `); - for (const message of snapshot.messages) { - insertMessage.run(message.id, message.sourceRowId, message.sourceGuid, message.conversationId, message.sentAt, message.direction, message.body, message.bodySource, message.kind, message.replyToSourceGuid, message.editedAt, message.retractedAt, message.service, message.attachmentCount); + const relabelSourceMessages = this.#database.query(` + UPDATE messages SET service=? + WHERE id IN ( + SELECT message_id FROM message_provenance WHERE source_id=? + ) + `); + const upsertConversation = this.#database.query(` + INSERT INTO conversations( + id,source_key,private_label,service,participant_count, + participant_ids_json,private_participants_json,is_group + ) VALUES (?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + source_key=excluded.source_key,private_label=excluded.private_label, + service=excluded.service,participant_count=excluded.participant_count, + participant_ids_json=excluded.participant_ids_json, + private_participants_json=excluded.private_participants_json,is_group=excluded.is_group + `); + const upsertConversationSource = this.#database.query(` + INSERT INTO conversation_sources(conversation_id,source_id,external_id,metadata_json) + VALUES (?,?,?,?) + ON CONFLICT(conversation_id) DO UPDATE SET + external_id=excluded.external_id,metadata_json=excluded.metadata_json + `); + const upsertMessage = this.#database.query(` + INSERT INTO messages( + id,source_row_id,source_guid,conversation_id,sent_at,direction, + body,body_source,kind,reply_to_source_guid,edited_at,retracted_at, + service,attachment_count + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + source_guid=excluded.source_guid,conversation_id=excluded.conversation_id, + sent_at=excluded.sent_at,direction=excluded.direction,body=excluded.body, + body_source=excluded.body_source,kind=excluded.kind, + reply_to_source_guid=excluded.reply_to_source_guid,edited_at=excluded.edited_at, + retracted_at=excluded.retracted_at,service=excluded.service, + attachment_count=excluded.attachment_count + `); + const upsertMessageProvenance = this.#database.query(` + INSERT INTO message_provenance( + message_id,source_id,external_id,reply_to_external_id,attachments_json,metadata_json + ) VALUES (?,?,?,?,?,?) + ON CONFLICT(message_id) DO UPDATE SET + external_id=excluded.external_id,reply_to_external_id=excluded.reply_to_external_id, + attachments_json=excluded.attachments_json,metadata_json=excluded.metadata_json + `); + const upsertReactionFact = this.#database.query(` + INSERT INTO corpus_reaction_facts( + id,source_id,external_id,target_external_id,conversation_id, + direction,body,reacted_at,state + ) VALUES (?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + external_id=excluded.external_id,target_external_id=excluded.target_external_id, + conversation_id=excluded.conversation_id,direction=excluded.direction, + body=excluded.body,reacted_at=excluded.reacted_at,state=excluded.state + `); + const upsertSourceRecord = this.#database.query(` + INSERT INTO corpus_source_records(source_id,kind,external_id,record_json) + VALUES (?,?,?,?) + ON CONFLICT(source_id,kind,external_id) DO UPDATE SET record_json=excluded.record_json + `); + const setSuppression = this.#database.query(` + INSERT INTO corpus_source_suppressions( + source_id,kind,local_id,external_id,suppressed_at,reason,suppressed + ) VALUES (?,?,?,?,?,?,?) + ON CONFLICT(source_id,kind,local_id) DO UPDATE SET + external_id=excluded.external_id,suppressed_at=excluded.suppressed_at, + reason=excluded.reason,suppressed=excluded.suppressed + `); + const clearExternalSuppression = this.#database.query(` + UPDATE corpus_source_suppressions + SET suppressed_at=?,reason='reappeared',suppressed=0 + WHERE source_id=? AND kind=? AND external_id=? AND suppressed=1 + `); + const results = []; + let changedAny = false; + for (const snapshot of snapshots) { + const existing = get(this.#database, ` + SELECT kind,network,input_revision,revision,generated_at,manifest_sha256 + FROM corpus_sources WHERE id=? + `, snapshot.source.id); + if (existing !== null && existing.kind !== snapshot.source.kind) { + throw new CliError("conflict", `Source ${snapshot.source.id} changed kind`); + } + if (existing !== null && snapshot.source.kind === "bundle") { + if (existing.generated_at === null || snapshot.source.generatedAt < existing.generated_at) { + throw new CliError("conflict", `Source ${snapshot.source.id} snapshot is older than stored state`); + } + if (snapshot.source.generatedAt === existing.generated_at && (snapshot.source.revision !== existing.input_revision || snapshot.source.manifestSha256 !== existing.manifest_sha256)) + throw new CliError("conflict", `Source ${snapshot.source.id} reuses generatedAt for different input`); + } + const authoritative = snapshot.source.kind === "imessage" || snapshot.source.coverage.history === "complete-current-local"; + if (authoritative) { + for (const row of this.#database.query(` + SELECT conversation_id,external_id FROM conversation_sources WHERE source_id=? + `).iterate(snapshot.source.id)) { + setSuppression.run(snapshot.source.id, "conversation", row.conversation_id, row.external_id, ingestedAt, "authoritative-absence", 1); + } + for (const row of this.#database.query(` + SELECT id,external_id FROM corpus_reaction_facts WHERE source_id=? + `).iterate(snapshot.source.id)) { + setSuppression.run(snapshot.source.id, "reaction", row.id, row.external_id, ingestedAt, "authoritative-absence", 1); + } + for (const row of this.#database.query(` + SELECT provenance.message_id,provenance.external_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + `).iterate(snapshot.source.id)) { + setSuppression.run(snapshot.source.id, row.kind === "reaction" ? "reaction" : "message", row.message_id, row.external_id, ingestedAt, "authoritative-absence", 1); + } + } + upsertSource.run(snapshot.source.id, snapshot.source.kind, snapshot.source.provider, snapshot.source.network, snapshot.source.accountId, snapshot.source.externalId, snapshot.source.revision, existing?.revision ?? snapshot.source.revision, snapshot.source.generatedAt, canonicalJson(snapshot.source.producer), canonicalJson(snapshot.source.coverage), snapshot.source.manifestSha256, canonicalJson(snapshot.source.identity), canonicalJson(snapshot.source.warnings), ingestedAt); + if (existing !== null && existing.network !== snapshot.source.network) { + relabelSourceConversations.run(snapshot.source.network, snapshot.source.id); + relabelSourceMessages.run(snapshot.source.network, snapshot.source.id); + } + const conversationProvenance = new Map(snapshot.conversationProvenance.map((value) => [value.conversationId, value])); + for (const conversation of snapshot.conversations) { + const owner = get(this.#database, ` + SELECT source_id FROM conversation_sources WHERE conversation_id=? + `, conversation.id); + if (owner !== null && owner.source_id !== snapshot.source.id) { + throw new CliError("conflict", `Conversation ${conversation.id} belongs to another source`); + } + upsertConversation.run(conversation.id, conversation.sourceKey, conversation.privateLabel, conversation.service, conversation.participantCount, canonicalJson(conversation.participantIds), canonicalJson(conversation.privateParticipants), conversation.group ? 1 : 0); + const provenance = conversationProvenance.get(conversation.id); + upsertConversationSource.run(conversation.id, snapshot.source.id, provenance.externalId, canonicalJson(provenance.metadata ?? {})); + setSuppression.run(snapshot.source.id, "conversation", conversation.id, provenance.externalId, ingestedAt, "reappeared", 0); + clearExternalSuppression.run(ingestedAt, snapshot.source.id, "conversation", provenance.externalId); + } + const messageProvenance = new Map(snapshot.messageProvenance.map((value) => [value.messageId, value])); + for (const message of snapshot.messages) { + const owner = get(this.#database, ` + SELECT provenance.source_id,message.source_row_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.message_id=? + `, message.id); + if (owner !== null && owner.source_id !== snapshot.source.id) { + throw new CliError("conflict", `Message ${message.id} belongs to another source`); + } + const preferredRowId = authoritative ? message.sourceRowId : null; + const preferredCollision = preferredRowId === null ? null : get(this.#database, "SELECT id FROM messages WHERE conversation_id=? AND source_row_id=?", message.conversationId, preferredRowId); + const sourceRowId = owner?.source_row_id ?? (preferredRowId !== null && preferredCollision === null ? preferredRowId : (get(this.#database, ` + SELECT max(source_row_id) AS value FROM messages WHERE conversation_id=? + `, message.conversationId)?.value ?? 0) + 1); + upsertMessage.run(message.id, sourceRowId, message.sourceGuid, message.conversationId, message.sentAt, message.direction, message.body, message.bodySource, message.kind, message.replyToSourceGuid, message.editedAt, message.retractedAt, message.service, message.attachmentCount); + const provenance = messageProvenance.get(message.id); + upsertMessageProvenance.run(message.id, snapshot.source.id, provenance.externalId, provenance.replyToExternalId, canonicalJson(provenance.attachments), canonicalJson({ + providerSortKey: provenance.providerSortKey, + metadata: provenance.metadata ?? {} + })); + setSuppression.run(snapshot.source.id, message.kind === "reaction" ? "reaction" : "message", message.id, provenance.externalId, ingestedAt, "reappeared", 0); + clearExternalSuppression.run(ingestedAt, snapshot.source.id, message.kind === "reaction" ? "reaction" : "message", provenance.externalId); + if (message.kind === "reaction") { + setSuppression.run(snapshot.source.id, "reaction-timeline", message.id, provenance.externalId, ingestedAt, "reappeared", 0); + clearExternalSuppression.run(ingestedAt, snapshot.source.id, "reaction-timeline", provenance.externalId); + } + } + for (const reaction of snapshot.reactionFacts ?? []) { + const existingReaction = get(this.#database, ` + SELECT source_id,external_id FROM corpus_reaction_facts WHERE id=? + `, reaction.id); + if (existingReaction !== null && (existingReaction.source_id !== snapshot.source.id || existingReaction.external_id !== reaction.externalId)) + throw new CliError("conflict", `Reaction ${reaction.id} belongs to another source coordinate`); + const conversationId = reaction.conversationId ?? get(this.#database, `SELECT message.conversation_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.external_id=?`, snapshot.source.id, reaction.targetExternalId)?.conversation_id ?? null; + upsertReactionFact.run(reaction.id, snapshot.source.id, reaction.externalId, reaction.targetExternalId, conversationId, reaction.direction, reaction.body, reaction.reactedAt, reaction.state); + if (reaction.state === "active") { + setSuppression.run(snapshot.source.id, "reaction", reaction.id, reaction.externalId, ingestedAt, "reappeared", 0); + clearExternalSuppression.run(ingestedAt, snapshot.source.id, "reaction", reaction.externalId); + clearExternalSuppression.run(ingestedAt, snapshot.source.id, "reaction-timeline", reaction.externalId); + } + } + this.#database.query(` + UPDATE corpus_reaction_facts AS reaction + SET conversation_id=( + SELECT message.conversation_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=reaction.source_id + AND provenance.external_id=reaction.target_external_id + ) + WHERE reaction.source_id=? AND reaction.conversation_id IS NULL + AND EXISTS ( + SELECT 1 FROM message_provenance provenance + WHERE provenance.source_id=reaction.source_id + AND provenance.external_id=reaction.target_external_id + ) + `).run(snapshot.source.id); + for (const record of snapshot.auxiliaryRecords ?? []) { + upsertSourceRecord.run(snapshot.source.id, record.kind, record.id, canonicalJson(record.record)); + } + for (const deletion of snapshot.deletions ?? []) { + let localId = deletion.localEntityId; + if (deletion.entityKind === "conversation") { + const specifiedLocal = localId !== null; + const target = localId === null ? get(this.#database, ` + SELECT conversation_id,external_id FROM conversation_sources + WHERE source_id=? AND external_id=? + `, snapshot.source.id, deletion.externalId) : get(this.#database, ` + SELECT conversation_id,external_id FROM conversation_sources + WHERE source_id=? AND conversation_id=? + `, snapshot.source.id, localId); + if (target !== null) { + if (target.external_id !== deletion.externalId) { + throw new CliError("invalid-data", "A conversation deletion has mismatched coordinates"); + } + localId = target.conversation_id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A conversation deletion references an unknown local entity"); + } + } + if (deletion.entityKind === "message") { + const specifiedLocal = localId !== null; + const target = localId === null ? get(this.#database, ` + SELECT provenance.message_id,provenance.external_id, + message.conversation_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.external_id=? + `, snapshot.source.id, deletion.externalId) : get(this.#database, ` + SELECT provenance.message_id,provenance.external_id, + message.conversation_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.message_id=? + `, snapshot.source.id, localId); + if (target !== null) { + if (target.external_id !== deletion.externalId || target.kind === "reaction" || deletion.expectedConversationId !== undefined && deletion.expectedConversationId !== target.conversation_id) + throw new CliError("invalid-data", "A message deletion has mismatched coordinates"); + localId = target.message_id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A message deletion references an unknown local entity"); + } + } + if (deletion.entityKind === "reaction" || deletion.entityKind === "reaction-timeline") { + const specifiedLocal = localId !== null; + const target = localId === null ? get(this.#database, ` + SELECT id,external_id,conversation_id FROM corpus_reaction_facts + WHERE source_id=? AND external_id=? + `, snapshot.source.id, deletion.externalId) : get(this.#database, ` + SELECT id,external_id,conversation_id FROM corpus_reaction_facts + WHERE source_id=? AND id=? + `, snapshot.source.id, localId); + if (target !== null) { + if (target.external_id !== deletion.externalId || deletion.expectedConversationId !== undefined && deletion.expectedConversationId !== target.conversation_id) + throw new CliError("invalid-data", "A reaction deletion has mismatched coordinates"); + localId = target.id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A reaction deletion references an unknown local entity"); + } + } + setSuppression.run(snapshot.source.id, deletion.entityKind, localId ?? `external:${deletion.externalId}`, deletion.externalId, deletion.deletedAt, deletion.reason ?? "tombstone", 1); + } + if (snapshot.source.kind === "bundle") { + rerankBundleMessages(this.#database, snapshot.source.id); + } + const stateRevision = sourceStateRevision(this.#database, snapshot.source.id); + this.#database.query("UPDATE corpus_sources SET revision=? WHERE id=?").run(stateRevision, snapshot.source.id); + const changed = existing?.revision !== stateRevision; + changedAny ||= changed; + const counts = get(this.#database, ` + SELECT count(distinct conversation.id) AS conversations, + count(message.id) AS messages + FROM conversation_sources ownership + JOIN conversations conversation ON conversation.id=ownership.conversation_id + LEFT JOIN messages message ON message.conversation_id=conversation.id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + ) + WHERE ownership.source_id=? + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.local_id=conversation.id + AND suppression.kind='conversation' + AND suppression.suppressed=1 + ) + `, snapshot.source.id) ?? { conversations: 0, messages: 0 }; + results.push(Object.freeze({ id: snapshot.source.id, changed, ...counts })); } + if (changedAny) + rebuildConversationLabels(this.#database, hmacKey3); + const corpusRevision = setCorpusRevision(this.#database); + if (corpusRevision === null) + throw new CliError("internal", "Source replacement produced no corpus revision"); + return Object.freeze({ corpusRevision, sources: Object.freeze(results) }); + }); + } + replaceCorpus(snapshot, ingestedAt, hmacKey3) { + if (!/^[a-f0-9]{64}$/u.test(snapshot.source.snapshotSha256)) { + throw new CliError("invalid-data", "The iMessage reader returned an invalid corpus revision"); + } + const observed = snapshot.messages.map(({ sentAt }) => sentAt).sort(); + const sourceSnapshot = Object.freeze({ + source: Object.freeze({ + id: IMESSAGE_SOURCE_ID, + kind: "imessage", + provider: "apple", + network: null, + accountId: null, + externalId: "local-imessage", + revision: snapshot.source.snapshotSha256, + generatedAt: null, + producer: Object.freeze({ id: "message-like-me", version: "imessage-reader-v1" }), + coverage: Object.freeze({ + history: "complete-current-local", + observedFrom: observed[0] ?? null, + observedTo: observed.at(-1) ?? null + }), + manifestSha256: null, + identity: snapshot.source, + warnings: snapshot.warnings + }), + conversations: snapshot.conversations, + conversationProvenance: Object.freeze(snapshot.conversations.map((conversation) => ({ + conversationId: conversation.id, + externalId: conversation.sourceKey + }))), + messages: snapshot.messages, + messageProvenance: Object.freeze(snapshot.messages.map((message) => ({ + messageId: message.id, + externalId: message.sourceGuid, + providerSortKey: null, + replyToExternalId: message.replyToSourceGuid, + attachments: Object.freeze(Array.from({ length: message.attachmentCount }, (_value, index) => ({ + id: `unavailable-${index + 1}`, + kind: null, + mimeType: null, + fileName: null, + bytes: null + }))) + }))) + }); + const replaced = this.replaceSources([sourceSnapshot], ingestedAt, hmacKey3); + transaction(this.#database, () => { const setMetadata = this.#database.query(` - INSERT INTO metadata (key, value) VALUES (?, ?) - ON CONFLICT (key) DO UPDATE SET value = excluded.value + INSERT INTO metadata(key,value) VALUES (?,?) + ON CONFLICT(key) DO UPDATE SET value=excluded.value `); for (const [key, value] of [ - ["corpus_revision", corpusRevision], ["source_identity", canonicalJson(snapshot.source)], ["ingested_at", ingestedAt], ["warnings", canonicalJson(snapshot.warnings)], ["corpus_schema_version", String(snapshot.schemaVersion)] ]) setMetadata.run(key, value); - rebuildConversationLabels(this.#database, hmacKey2); }); return { - corpusRevision, + corpusRevision: replaced.corpusRevision, conversations: snapshot.conversations.length, messages: snapshot.messages.length }; } + listSources(privateDetails = false) { + const rows = all(this.#database, ` + SELECT source.*, + count(distinct ownership.conversation_id) AS conversations, + count(message.id) AS messages, + CASE source.kind WHEN 'bundle' THEN + (SELECT count(*) FROM corpus_reaction_facts reaction + WHERE reaction.source_id=source.id AND reaction.state='active' + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='reaction' + AND suppression.local_id=reaction.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction.conversation_id + AND suppression.suppressed=1 + )) + ELSE + (SELECT count(*) FROM messages reaction_message + JOIN message_provenance reaction_provenance + ON reaction_provenance.message_id=reaction_message.id + WHERE reaction_provenance.source_id=source.id + AND reaction_message.kind='reaction' AND reaction_message.retracted_at IS NULL + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.local_id=reaction_message.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction_message.conversation_id + AND suppression.suppressed=1 + )) + END AS reactions, + CASE source.kind WHEN 'bundle' THEN + (SELECT count(*) FROM corpus_reaction_facts reaction + WHERE reaction.source_id=source.id AND reaction.state='active' + AND reaction.reacted_at IS NULL + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='reaction' + AND suppression.local_id=reaction.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction.conversation_id + AND suppression.suppressed=1 + )) + ELSE 0 + END AS undated_reactions + FROM corpus_sources source + LEFT JOIN conversation_sources ownership ON ownership.source_id=source.id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind='conversation' + AND suppression.local_id=ownership.conversation_id + AND suppression.suppressed=1 + ) + LEFT JOIN messages message ON message.conversation_id=ownership.conversation_id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.local_id=message.id + AND suppression.suppressed=1 + ) + GROUP BY source.id + ORDER BY source.provider,source.network,source.id + `); + return rows.map((row) => { + const warnings = parsedJson(row.warnings_json, `Source ${row.id} warnings`); + if (!Array.isArray(warnings)) + throw new CliError("invalid-data", `Source ${row.id} warnings are invalid`); + return { + id: row.id, + kind: row.kind, + provider: row.provider, + network: row.network, + revision: row.revision, + generatedAt: row.generated_at, + ingestedAt: row.ingested_at, + coverage: parsedJson(row.coverage_json, `Source ${row.id} coverage`), + warningCount: warnings.length, + conversations: row.conversations, + messages: row.messages, + reactions: row.reactions, + undatedReactions: row.undated_reactions, + ...privateDetails ? { + accountId: row.account_id, + externalId: row.external_id, + manifestSha256: row.manifest_sha256, + inputRevision: row.input_revision, + identity: parsedJson(row.identity_json, `Source ${row.id} identity`), + warnings + } : {} + }; + }); + } + source(sourceId, privateDetails = false) { + if (sourceId.length < 1 || sourceId.length > 256) { + throw new CliError("usage", "Source ID must be bounded non-empty text"); + } + return this.listSources(privateDetails).find(({ id }) => id === sourceId) ?? null; + } listContacts(options) { if (this.corpusRevision() === null) return []; @@ -3828,16 +5986,30 @@ class LocalStore { conversation.id AS conversation_id FROM conversation_contact_scopes association JOIN conversations conversation ON conversation.id=association.conversation_id + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id LEFT JOIN conversation_contact_labels label ON label.conversation_id=association.conversation_id - WHERE conversation.is_group=0 + WHERE conversation.is_group=0 AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) UNION ALL SELECT conversation.id,conversation.private_label,'conversation',conversation.is_group, conversation.participant_count,conversation.id FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id LEFT JOIN conversation_contact_scopes association ON association.conversation_id=conversation.id - WHERE association.conversation_id IS NULL + WHERE association.conversation_id IS NULL AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) ) SELECT scope.id,min(scope.private_label) AS private_label, max(scope.scope_kind) AS scope_kind, @@ -3849,6 +6021,14 @@ class LocalStore { sum(CASE WHEN message.direction = 'outgoing' THEN 1 ELSE 0 END) AS outgoing_count FROM scope_conversations scope JOIN messages message ON message.conversation_id=scope.conversation_id + JOIN message_provenance provenance ON provenance.message_id=message.id + WHERE NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + ) GROUP BY scope.id HAVING outgoing_count >= ? ORDER BY outgoing_count DESC,last_message_at DESC,scope.id @@ -3930,6 +6110,7 @@ class LocalStore { scopeKind: scope.kind, conversationCount: scope.conversationIds.length, service: services.length === 1 ? services[0] : null, + services: Object.freeze(services.sort((left, right) => left < right ? -1 : left > right ? 1 : 0)), participantCount: scope.kind === "person" ? 1 : first.participant_count, participantIds: participants, privateParticipants, @@ -3954,7 +6135,8 @@ class LocalStore { return { corpusRevision, evidenceRevision: scopeEvidenceRevision(this.#database, scope, undefined, window), - messages: messageRowsForScope(this.#database, scope, undefined, window).map(corpusMessage) + messages: messageRowsForScope(this.#database, scope, undefined, window).map(corpusMessage), + reactions: reactionFactsForScope(this.#database, scope, window) }; }); } @@ -4098,6 +6280,7 @@ class LocalStore { foreignKeyViolations: foreignKeys, corpusRevision: this.corpusRevision(), contactsRevision: this.contactsRevision(), + sources: count("corpus_sources"), conversations: count("conversations"), messages: count("messages"), profiles: count("profiles"), @@ -4108,7 +6291,7 @@ class LocalStore { } // src/version.ts -var MESSAGE_LIKE_ME_VERSION = "0.2.0"; +var MESSAGE_LIKE_ME_VERSION = "0.3.0"; // src/commands.ts var HELP = `Message Like Me ${MESSAGE_LIKE_ME_VERSION} @@ -4116,7 +6299,10 @@ var HELP = `Message Like Me ${MESSAGE_LIKE_ME_VERSION} Usage: messagelikeme [--data-dir PATH] init [--json] messagelikeme [--data-dir PATH] ingest imessage [--database PATH] [--json] + messagelikeme [--data-dir PATH] ingest bundle --input ABS_PATH [--json] messagelikeme [--data-dir PATH] ingest contacts [--addressbook PATH] [--json] + messagelikeme [--data-dir PATH] sources list [--private] [--json] + messagelikeme [--data-dir PATH] sources show SOURCE_ID [--private] [--json] messagelikeme [--data-dir PATH] contacts list [--min-outgoing N] [--limit N] [--private] [--json] messagelikeme [--data-dir PATH] contacts show CONTACT_ID [--private] [--json] messagelikeme [--data-dir PATH] contacts resolve QUERY --private [--limit N] [--json] @@ -4137,13 +6323,13 @@ Usage: [--project PATH] [--force] [--json] messagelikeme [--data-dir PATH] doctor [--json] -Message Like Me reads caller-owned macOS Messages and optional Contacts data, -then stores private analysis locally. It has no network, account, AI-provider, -or message-sending surface. +Message Like Me reads caller-owned macOS Messages, optional Contacts data, and +strict private local message bundles, then stores private analysis locally. It +has no network, account, AI-provider, or message-sending surface. `; async function exists2(path) { try { - await lstat3(path); + await lstat4(path); return true; } catch (error) { if (error.code === "ENOENT") @@ -4196,7 +6382,10 @@ function contactEvidence(store, contactId, window) { } function contactMetrics(store, contactId, options = {}) { const evidence = contactEvidence(store, contactId); - return analyzeContact(evidence.messages, evidence.corpusRevision, contactId, options); + return analyzeContact(evidence.messages, evidence.corpusRevision, contactId, { + ...options, + reactionFacts: evidence.reactions + }); } function metricOptions(parsed) { return { @@ -4228,6 +6417,7 @@ function safeContactDetail(store, contactId, privateLabels) { privateParticipants: conversation.privateParticipants } : {}, service: conversation.service, + services: conversation.services, group: conversation.group, participantCount: conversation.participantCount, participantIds: conversation.participantIds, @@ -4261,9 +6451,9 @@ function compactMetrics(metrics) { function absolutePrivatePath(value, label) { if (value === undefined) throw new CliError("usage", `${label} is required`); - if (!isAbsolute4(value)) + if (!isAbsolute5(value)) throw new CliError("unsafe-path", `${label} must be an absolute private path`); - return resolve5(value); + return resolve6(value); } function translateIMessageError(error) { const code = error.code; @@ -4286,6 +6476,18 @@ function translateContactsError(error) { const message = error instanceof Error ? error.message : ""; throw new CliError("invalid-data", message.startsWith("Contacts source ") ? message : "The selected AddressBook source could not be read safely", { cause: error }); } +function translateBundleError(error) { + if (error instanceof CliError) + throw error; + const code = error.code; + if (code === "EACCES" || code === "EPERM") { + throw new CliError("permission", "The selected private bundle is not readable", { cause: error }); + } + if (code === "ENOENT") { + throw new CliError("not-found", "The selected private bundle does not exist", { cause: error }); + } + throw new CliError("invalid-data", "The selected private message bundle could not be read safely", { cause: error }); +} async function runCommand(argv, io) { const parsed = parseArguments(argv); if (parsed.flags.has("version")) { @@ -4303,7 +6505,7 @@ async function runCommand(argv, io) { return; } const json = parsed.flags.has("json"); - const [command, subcommand, identifier, ...extra] = parsed.positionals; + const [command, subcommand, identifier2, ...extra] = parsed.positionals; if (extra.length !== 0) throw new CliError("usage", `Unexpected argument ${extra[0]}`); if (command === "init" && subcommand === undefined) { @@ -4317,7 +6519,7 @@ async function runCommand(argv, io) { } return; } - if (command === "ingest" && subcommand === "imessage" && identifier === undefined) { + if (command === "ingest" && subcommand === "imessage" && identifier2 === undefined) { rejectUnused(parsed, ["data-dir", "database"], ["json"]); const context = await writableStore(parsed); try { @@ -4345,7 +6547,33 @@ async function runCommand(argv, io) { } return; } - if (command === "ingest" && subcommand === "contacts" && identifier === undefined) { + if (command === "ingest" && subcommand === "bundle" && identifier2 === undefined) { + rejectUnused(parsed, ["data-dir", "input"], ["json"]); + const input = absolutePrivatePath(parsed.options.get("input"), "--input"); + const context = await writableStore(parsed); + try { + let bundle; + try { + bundle = await readMessageBundle(input, { hmacKey: context.key }); + } catch (error) { + translateBundleError(error); + } + const stored = context.store.replaceSources(bundle.sources, canonicalNow(io), context.key); + const result = { + schemaVersion: bundle.schemaVersion, + manifestSha256: bundle.manifestSha256, + corpusRevision: stored.corpusRevision, + sources: stored.sources, + conversations: stored.sources.reduce((sum, source) => sum + source.conversations, 0), + messages: stored.sources.reduce((sum, source) => sum + source.messages, 0) + }; + emit(io, json, result, `Ingested ${result.messages} active messages across ${result.conversations} conversations from ${result.sources.length} sources`); + } finally { + context.store.close(); + } + return; + } + if (command === "ingest" && subcommand === "contacts" && identifier2 === undefined) { rejectUnused(parsed, ["data-dir", "addressbook"], ["json"]); const context = await writableStore(parsed); try { @@ -4373,7 +6601,7 @@ async function runCommand(argv, io) { } return; } - if (command === "contacts" && subcommand === "list" && identifier === undefined) { + if (command === "contacts" && subcommand === "list" && identifier2 === undefined) { rejectUnused(parsed, ["data-dir", "min-outgoing", "limit"], ["json", "private"]); const context = await existingStore(parsed); try { @@ -4388,18 +6616,42 @@ async function runCommand(argv, io) { } return; } - if (command === "contacts" && subcommand === "show" && identifier !== undefined) { + if (command === "sources" && subcommand === "list" && identifier2 === undefined) { + rejectUnused(parsed, ["data-dir"], ["json", "private"]); + const context = await existingStore(parsed); + try { + const sources = context.store.listSources(parsed.flags.has("private")); + emit(io, json, { sources }, `${sources.length} message sources`); + } finally { + context.store.close(); + } + return; + } + if (command === "sources" && subcommand === "show" && identifier2 !== undefined) { + rejectUnused(parsed, ["data-dir"], ["json", "private"]); + const context = await existingStore(parsed); + try { + const source = context.store.source(identifier2, parsed.flags.has("private")); + if (source === null) + throw new CliError("not-found", `Unknown source ${identifier2}`); + emit(io, json, source, `Message source ${identifier2}`); + } finally { + context.store.close(); + } + return; + } + if (command === "contacts" && subcommand === "show" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir"], ["json", "private"]); const context = await existingStore(parsed); try { - const detail = safeContactDetail(context.store, identifier, parsed.flags.has("private")); - emit(io, json, detail, `Contact ${identifier}`); + const detail = safeContactDetail(context.store, identifier2, parsed.flags.has("private")); + emit(io, json, detail, `Contact ${identifier2}`); } finally { context.store.close(); } return; } - if (command === "contacts" && subcommand === "resolve" && identifier !== undefined) { + if (command === "contacts" && subcommand === "resolve" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir", "limit"], ["json", "private"]); if (!parsed.flags.has("private")) { throw new CliError("usage", "contacts resolve requires --private"); @@ -4408,7 +6660,7 @@ async function runCommand(argv, io) { try { let matches; try { - matches = context.store.resolvePrivateContacts(identifier, integerOption(parsed, "limit", 10, 1, 50)); + matches = context.store.resolvePrivateContacts(identifier2, integerOption(parsed, "limit", 10, 1, 50)); } catch (error) { if (error instanceof CliError) throw error; @@ -4420,33 +6672,33 @@ async function runCommand(argv, io) { } return; } - if (command === "inspect" && subcommand === "tempo" && identifier !== undefined) { + if (command === "inspect" && subcommand === "tempo" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir", "session-gap", "burst-gap"], ["json"]); const context = await existingStore(parsed); try { - const metrics = contactMetrics(context.store, identifier, metricOptions(parsed)); + const metrics = contactMetrics(context.store, identifier2, metricOptions(parsed)); const result = compactMetrics(metrics); - emit(io, json, result, `Tempo metrics for ${identifier}: ${metrics.tempo.responseEpisodes} response episodes`); + emit(io, json, result, `Tempo metrics for ${identifier2}: ${metrics.tempo.responseEpisodes} response episodes`); } finally { context.store.close(); } return; } - if (command === "inspect" && subcommand === "sessions" && identifier !== undefined) { + if (command === "inspect" && subcommand === "sessions" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir", "limit", "session-gap", "burst-gap"], ["json"]); const context = await existingStore(parsed); try { - const metrics = contactMetrics(context.store, identifier, metricOptions(parsed)); + const metrics = contactMetrics(context.store, identifier2, metricOptions(parsed)); const limit = integerOption(parsed, "limit", 20, 1, 1000); const sessions = metrics.sessions.slice(-limit); - const result = { contactId: identifier, total: metrics.sessions.length, sessions }; - emit(io, json, result, `${sessions.length} of ${metrics.sessions.length} sessions for ${identifier}`); + const result = { contactId: identifier2, total: metrics.sessions.length, sessions }; + emit(io, json, result, `${sessions.length} of ${metrics.sessions.length} sessions for ${identifier2}`); } finally { context.store.close(); } return; } - if (command === "study" && subcommand === "prepare" && identifier !== undefined) { + if (command === "study" && subcommand === "prepare" && identifier2 !== undefined) { rejectUnused(parsed, [ "data-dir", "output", @@ -4463,13 +6715,13 @@ async function runCommand(argv, io) { const before = canonicalTimestampOption(parsed, "before"); let evidence; try { - evidence = contactEvidence(context.store, identifier, { after, before }); + evidence = contactEvidence(context.store, identifier2, { after, before }); } catch (error) { if (error instanceof CliError) throw error; throw new CliError("usage", error instanceof Error ? error.message : String(error), { cause: error }); } - const metrics = analyzeContact(evidence.messages, evidence.corpusRevision, identifier, metricOptions(parsed)); + const metrics = analyzeContact(evidence.messages, evidence.corpusRevision, identifier2, { ...metricOptions(parsed), reactionFacts: evidence.reactions }); const packet = buildStudyPacket(evidence.messages, metrics, { limit: integerOption(parsed, "limit", 24, 1, 50), generatedAt: canonicalNow(io), @@ -4481,7 +6733,7 @@ async function runCommand(argv, io) { await atomicWritePrivate(output, bytes); context.store.recordStudyPacket({ sha256: packetSha256, - contactId: identifier, + contactId: identifier2, corpusRevision: metrics.corpusRevision, evidenceRevision: evidence.evidenceRevision, createdAt: packet.generatedAt, @@ -4500,7 +6752,7 @@ async function runCommand(argv, io) { } }); const result = { - contactId: identifier, + contactId: identifier2, corpusRevision: metrics.corpusRevision, evidenceRevision: evidence.evidenceRevision, packetSha256, @@ -4514,7 +6766,7 @@ async function runCommand(argv, io) { } return; } - if (command === "evaluate" && subcommand === "prepare" && identifier !== undefined) { + if (command === "evaluate" && subcommand === "prepare" && identifier2 !== undefined) { rejectUnused(parsed, [ "data-dir", "after", @@ -4536,13 +6788,13 @@ async function runCommand(argv, io) { try { let evidence; try { - evidence = contactEvidence(context.store, identifier, { after, before }); + evidence = contactEvidence(context.store, identifier2, { after, before }); } catch (error) { if (error instanceof CliError) throw error; throw new CliError("usage", error instanceof Error ? error.message : String(error), { cause: error }); } - const metrics = analyzeContact(evidence.messages, evidence.corpusRevision, identifier, metricOptions(parsed)); + const metrics = analyzeContact(evidence.messages, evidence.corpusRevision, identifier2, { ...metricOptions(parsed), reactionFacts: evidence.reactions }); const packets = buildEvaluationPackets(evidence.messages, metrics, { after, before, @@ -4557,7 +6809,7 @@ async function runCommand(argv, io) { await atomicWritePrivate(referenceOutput, prettyJson(packets.reference)); const result = { evaluationId: packets.prompt.evaluationId, - contactId: identifier, + contactId: identifier2, corpusRevision: evidence.corpusRevision, evidenceRevision: evidence.evidenceRevision, cases: packets.prompt.cases.length, @@ -4572,9 +6824,9 @@ async function runCommand(argv, io) { } return; } - if (command === "profile" && subcommand === "apply" && identifier !== undefined) { + if (command === "profile" && subcommand === "apply" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir"], ["json"]); - const path = absolutePrivatePath(identifier, "Profile path"); + const path = absolutePrivatePath(identifier2, "Profile path"); const profile = await readStyleProfile(path); const context = await existingStore(parsed); try { @@ -4586,38 +6838,38 @@ async function runCommand(argv, io) { } return; } - if (command === "profile" && subcommand === "show" && identifier !== undefined) { + if (command === "profile" && subcommand === "show" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir"], ["json"]); const context = await existingStore(parsed); try { - requireContact(context.store, identifier); - const result = context.store.profile(identifier); + requireContact(context.store, identifier2); + const result = context.store.profile(identifier2); if (result === null) - throw new CliError("not-found", `No profile exists for ${identifier}`); - emit(io, json, result, `${result.state} profile for ${identifier}`); + throw new CliError("not-found", `No profile exists for ${identifier2}`); + emit(io, json, result, `${result.state} profile for ${identifier2}`); } finally { context.store.close(); } return; } - if (command === "profile" && subcommand === "export" && identifier !== undefined) { + if (command === "profile" && subcommand === "export" && identifier2 !== undefined) { rejectUnused(parsed, ["data-dir", "output"], ["json"]); const output = absolutePrivatePath(parsed.options.get("output"), "--output"); const context = await existingStore(parsed); try { - requireContact(context.store, identifier); - const result = context.store.profile(identifier); + requireContact(context.store, identifier2); + const result = context.store.profile(identifier2); if (result === null) - throw new CliError("not-found", `No profile exists for ${identifier}`); + throw new CliError("not-found", `No profile exists for ${identifier2}`); await atomicWritePrivate(output, prettyJson(result.profile)); - const receipt = { contactId: identifier, state: result.state, output }; + const receipt = { contactId: identifier2, state: result.state, output }; emit(io, json, receipt, `Exported ${result.state} profile to ${output}`); } finally { context.store.close(); } return; } - if (command === "context" && subcommand !== undefined && identifier === undefined) { + if (command === "context" && subcommand !== undefined && identifier2 === undefined) { rejectUnused(parsed, ["data-dir"], ["json"]); const contactId = subcommand; const context = await existingStore(parsed); @@ -4633,13 +6885,13 @@ async function runCommand(argv, io) { } return; } - if (command === "skill" && subcommand === "path" && identifier === undefined) { + if (command === "skill" && subcommand === "path" && identifier2 === undefined) { rejectUnused(parsed, ["data-dir"], ["json"]); const path = bundledSkillPath(); emit(io, json, { path }, path); return; } - if (command === "skill" && subcommand === "install" && identifier === undefined) { + if (command === "skill" && subcommand === "install" && identifier2 === undefined) { rejectUnused(parsed, ["data-dir", "target", "scope", "project"], ["force", "json"]); const target = parsed.options.get("target") ?? "codex"; const scope = parsed.options.get("scope") ?? "user"; diff --git a/dist/index.js b/dist/index.js index 7bd5cc0..f674631 100644 --- a/dist/index.js +++ b/dist/index.js @@ -4,18 +4,20 @@ import { CORPUS_SCHEMA_VERSION, EVALUATION_PACKET_SCHEMA_VERSION, LEGACY_PROFILE_SCHEMA_VERSION, + MESSAGE_BUNDLE_SCHEMA_VERSION, METRICS_SCHEMA_VERSION, PROFILE_SCHEMA_VERSION, STUDY_PACKET_SCHEMA_VERSION, canonicalJson, sha256 -} from "./cli-xby0v0et.js"; +} from "./cli-mxxakdqk.js"; export { sha256, canonicalJson, STUDY_PACKET_SCHEMA_VERSION, PROFILE_SCHEMA_VERSION, METRICS_SCHEMA_VERSION, + MESSAGE_BUNDLE_SCHEMA_VERSION, LEGACY_PROFILE_SCHEMA_VERSION, EVALUATION_PACKET_SCHEMA_VERSION, CORPUS_SCHEMA_VERSION, diff --git a/dist/types.d.ts b/dist/types.d.ts index 7e5176e..75ea861 100644 --- a/dist/types.d.ts +++ b/dist/types.d.ts @@ -1,10 +1,11 @@ export declare const CORPUS_SCHEMA_VERSION: 1; -export declare const METRICS_SCHEMA_VERSION: 1; +export declare const METRICS_SCHEMA_VERSION: 2; export declare const PROFILE_SCHEMA_VERSION: 2; export declare const LEGACY_PROFILE_SCHEMA_VERSION: 1; export declare const STUDY_PACKET_SCHEMA_VERSION: 2; export declare const EVALUATION_PACKET_SCHEMA_VERSION: 1; export declare const CONTACTS_SCHEMA_VERSION: 1; +export declare const MESSAGE_BUNDLE_SCHEMA_VERSION: 1; export type Direction = "incoming" | "outgoing"; export type BodySource = "text" | "attributed-body" | "unavailable"; export type MessageKind = "text" | "attachment" | "reaction" | "system" | "unknown"; @@ -50,6 +51,96 @@ export type CorpusSnapshot = Readonly<{ messages: readonly CorpusMessage[]; warnings: readonly string[]; }>; +export type CorpusSourceKind = "imessage" | "bundle"; +export type CorpusSourceCoverage = Readonly<{ + history: "complete-current-local" | "bounded" | "unknown"; + observedFrom: string | null; + observedTo: string | null; + /** Producer-specific completeness classification, when the import format has one. */ + kind?: string; + /** Producer-supplied categorical reason for incomplete coverage. */ + reason?: string | null; +}>; +export type CorpusSourceDescriptor = Readonly<{ + /** Per-install source pseudonym used by the local store and CLI. */ + id: string; + kind: CorpusSourceKind; + provider: string; + network: string | null; + /** Private provider account identifier. Ordinary source views omit it. */ + accountId: string | null; + /** Private producer-local source identifier. Ordinary source views omit it. */ + externalId: string; + revision: string; + generatedAt: string | null; + producer: Readonly<{ + id: string; + version: string; + }>; + coverage: CorpusSourceCoverage; + manifestSha256: string | null; + identity: unknown; + warnings: readonly string[]; +}>; +export type CorpusConversationProvenance = Readonly<{ + conversationId: string; + externalId: string; + metadata?: unknown; +}>; +export type CorpusAttachmentProvenance = Readonly<{ + id: string; + kind: string | null; + mimeType: string | null; + fileName: string | null; + bytes: number | null; +}>; +export type CorpusMessageProvenance = Readonly<{ + messageId: string; + externalId: string; + /** Provider ordering coordinate. Null when this record has no such coordinate. */ + providerSortKey: string | null; + replyToExternalId: string | null; + attachments: readonly CorpusAttachmentProvenance[]; + metadata?: unknown; +}>; +export type CorpusReactionFact = Readonly<{ + id: string; + externalId: string; + targetExternalId: string; + conversationId: string | null; + direction: Direction | null; + body: string; + reactedAt: string | null; + state: "active" | "removed"; +}>; +export type CorpusSourceRecord = Readonly<{ + kind: "account" | "participant" | "reaction" | "tombstone" | "excluded-message"; + id: string; + record: unknown; +}>; +export type CorpusSourceDeletion = Readonly<{ + entityKind: "account" | "participant" | "conversation" | "message" | "reaction" | "reaction-timeline"; + localEntityId: string | null; + externalId: string; + deletedAt: string; + expectedConversationId?: string; + reason?: "tombstone" | "explicit-exclusion" | "replacement"; +}>; +export type SourceCorpusSnapshot = Readonly<{ + source: CorpusSourceDescriptor; + conversations: readonly CorpusConversation[]; + conversationProvenance: readonly CorpusConversationProvenance[]; + messages: readonly CorpusMessage[]; + messageProvenance: readonly CorpusMessageProvenance[]; + reactionFacts?: readonly CorpusReactionFact[]; + auxiliaryRecords?: readonly CorpusSourceRecord[]; + deletions?: readonly CorpusSourceDeletion[]; +}>; +export type MessageBundleSnapshot = Readonly<{ + schemaVersion: typeof MESSAGE_BUNDLE_SCHEMA_VERSION; + manifestSha256: string; + sources: readonly SourceCorpusSnapshot[]; +}>; export type ContactHandle = Readonly<{ kind: "email" | "phone"; normalizedValue: string; @@ -161,6 +252,9 @@ export type ReactionMetrics = Readonly<{ total: number; incoming: number; outgoing: number; + unknownDirection: number; + dated: number; + undated: number; outgoingReactionRatio: number; }>; export type ContactMetrics = Readonly<{ diff --git a/docs/local-message-bundle-v1.md b/docs/local-message-bundle-v1.md new file mode 100644 index 0000000..66a4e14 --- /dev/null +++ b/docs/local-message-bundle-v1.md @@ -0,0 +1,213 @@ +# Local message bundle version 1 + +`message-like-me.local-message-bundle` is a private directory interchange for +moving a bounded local provider observation into Message Like Me. It separates +provider capture from analysis: a producer handles provider access and writes +the bundle, while `messagelikeme ingest bundle` verifies and normalizes it. The +importer never receives provider credentials and never calls the producer. + +The current producer is the local Beeper export in +[Wrench 0.13.0 or newer](https://github.com/hraness/wrench/releases): + +```sh +wrench beeper export-message-like-me \ + --auth \ + --output \ + [--limit-chats ] \ + [--limit-messages ] \ + [--max-participants ] \ + [--json] +``` + +The JSON shape is published as +[`schema/local-message-bundle-v1.schema.json`](../schema/local-message-bundle-v1.schema.json). +Runtime validation also enforces UTF-8 byte bounds, canonical encoding, +filesystem identity, graph joins, and digest laws that JSON Schema cannot +express. + +## Directory inventory + +The input is a normalized absolute path to a current-user-owned physical +directory with mode `0700`. It contains exactly these mode-`0600`, singly +linked physical files: + +```text +manifest.json +accounts.ndjson +participants.ndjson +conversations.ndjson +messages.ndjson +reactions.ndjson +tombstones.ndjson +``` + +The six artifacts always exist, including when they are empty. Every artifact +uses canonical JSON, one object per line, and one final newline per record. +Empty artifacts contain zero bytes. `manifest.json` is canonical JSON followed +by one newline and is written last by the producer. + +The importer rejects symbolic links, extra files, ownership or mode changes, +files that change while read, invalid UTF-8, noncanonical JSON, blank records, +missing final newlines, count or byte mismatches, and digest mismatches. + +## Bounds + +Version one has these hard importer and producer ceilings: + +- 128 connected accounts; +- 500,000 records across all six artifacts; +- 512 MiB across all six artifacts; +- 2 MiB for one encoded NDJSON record, including its final newline; +- 1 MiB of UTF-8 for one message body; +- 1,024 UTF-8 bytes for an identifier, sort key, or provider revision; +- 8 KiB of UTF-8 for a display name, handle, title, reaction body, or + attachment filename; +- 256 UTF-8 bytes for an attachment MIME type; +- 10,000 known participants in one conversation; +- 256 attachment metadata items in one message; and +- 128 unique categorical warning codes. + +Custom producer limits may only lower the total record, byte, and line bounds. +All timestamps are canonical millisecond UTC strings equal to +`Date#toISOString()` output. Identifiers are nonempty and contain no ASCII +control characters. Network and warning values are bounded lowercase tokens. + +## Manifest integrity + +The manifest declares source and provider versions, collection timestamps, +completeness, privacy guarantees, per-kind counts, and each artifact's exact +record count, byte length, and lowercase SHA-256. + +Artifact SHA-256 covers the file's exact bytes, including every final newline. +Artifacts appear in the fixed directory order shown above. The bundle digest +is: + +```text +SHA256(UTF8(canonicalJson(manifest with the entire integrity property omitted))) +``` + +The manifest file's own SHA-256 covers its exact canonical bytes plus final +newline. It is returned by the producer and recorded by Message Like Me, but is +not embedded in the manifest. + +The privacy declaration is fixed to `private-local`, `metadata-only` +attachments, excluded provider URLs, and excluded credentials. This is an +interchange constraint, not anonymization. Bodies, timestamps, handles, +account coordinates, and relationship graphs remain private. + +## Account realms and provenance + +Every line has `schemaVersion`, `kind`, a bundle-local `id`, `accountId`, +`network`, and provenance: + +- `providerId` is the stable provider coordinate for that entity; +- `providerRevision` preserves a provider revision when one exists; +- `observedAt` records when the producer observed this record; and +- `connectedAccountProviderId` is the stable connected-account coordinate. + +An account line has `id === accountId` and +`provenance.providerId === provenance.connectedAccountProviderId`. Every other +record must match one account line on `accountId`, `network`, and connected +account coordinate. Bundle-local IDs exist only for joins inside this one +bundle. Message Like Me derives its stored source and entity IDs from stable +provider, connected-account, and self-participant coordinates with a private +per-install HMAC key. The mutable network label is source metadata and never +part of that identity namespace. + +Provider IDs must be unique within one entity kind and account. Validation +errors name the record kind and ordinal, never the foreign coordinate. Message +and reaction provider IDs are independent domains and may contain the same +value. Message Like Me assigns a separate internal timeline coordinate when a +dated reaction is represented in the normalized messages table; the raw +reaction coordinate remains in the reaction fact and private provenance. + +## Identity and conversation rosters + +Each account names one self participant. Participants carry optional display +names and handles plus `isSelf`. Conversations carry their known participant +IDs and `participantsComplete`: + +- `true` is the only positive assertion that the roster is complete; +- `false` or `null` means the producer cannot assert completeness; and +- a complete direct roster must contain exactly the account's one self + participant and one non-self participant. + +Message senders and reaction actors must agree with direction and any complete +roster. Message Like Me may expose an exact email or E.164 handle from the one +non-self participant of a complete direct conversation to local Contacts +matching. It never uses an incomplete roster for that join. + +## Messages, replies, and attachments + +`sentAt` is the message's actual temporal coordinate. `sortKey` is an opaque +provider ordering key. Within one account and conversation, Message Like Me +orders lexical `sortKey`, then `sentAt` and stable provider ID as deterministic +tie-breakers. + +`bodyTruncated: true` means the body cannot be prose evidence. The record still +becomes a text bubble for tempo, reply, and delivery-shape analysis. A message +with deletion state must have a null body. Attachment entries contain metadata +only; they never contain paths, URLs, or media bytes. + +A reply target has a required provider ID and an optional bundle-local ID. When +the local ID is present, it must resolve to the same provider coordinate in the +same account and conversation. A null local ID preserves a reply to a message +outside the bounded artifact. + +## Edits and deletion + +Edits are discriminated: + +- `in-place` records a terminal mutation under the same provider message ID and + never suppresses that message; and +- `replacement` identifies a different provider message in the same account + and conversation. + +Replacement targets may be outside the bounded artifact. In-bundle targets +must agree on local and provider coordinates. Replacement graphs must be +non-self, single-terminal, and acyclic. A validated replacement suppresses the +older version as evidence. + +Message deletion state is explicit and carries the observation time and +provider revision. Deleted bodies are null. + +## Reactions and tombstones + +A reaction has a required target provider message ID and an optional +bundle-local target. When present, the local target must resolve to the same +provider coordinate. `reactedAt` is nullable because the provider may not +expose a reaction time. Producers never synthesize one. Active undated +reactions contribute to fixed aggregate reaction counts, direction counts, and +timestamp-coverage counts but never enter the message timeline, sessions, +bursts, response episodes, or latency metrics. + +Tombstones identify a conversation, message, or reaction kind, required +provider coordinate, optional bundle-local coordinate, deletion time, scope, +and provider revision. Account and participant tombstones are outside the +version-one contract. A nonnull local coordinate must resolve inside the same +account and agree with +the provider coordinate. A null coordinate preserves deletion knowledge for +an entity outside the bounded artifact. + +## Reimport semantics + +Version-one bundle completeness is `bounded-local`, `truncated`, or `unknown`. +None is authoritative for deletion by absence. Reimport therefore upserts +present records and retains prior records omitted by a later bundle. Explicit +message deletion, removed reaction state, replacement edges, and tombstones +are applied separately. A valid later reappearance clears the matching +suppression. Present and retained messages are reranked together by the provider +ordering coordinates, so a bounded backfill converges with a fresh import of +the same final records. + +The manifest completeness kind and reason apply conservatively to every +account. Stored `observedFrom` and `observedTo` bounds are derived from the +dated message and reaction records for that account, so one account never +inherits another account's time range. An account with no dated timeline +records has null bounds. + +`timestamps.createdAt` is monotonic within one stable connected-account source. +An older bundle is rejected. An equal-time replay is accepted only when its +manifest and input revision match exactly; an equal-time conflict is rejected. +Native iMessage replacement remains scoped to its own source and cannot remove +bundle history. diff --git a/docs/methodology.md b/docs/methodology.md index b23874b..69ded83 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -13,6 +13,10 @@ The CLI makes stable private snapshots and opens only those snapshots through SQLite. It does not modify Messages, Contacts, their databases, or their transactional sidecars. +A caller-owned local message bundle is a separate versioned source +observation. The CLI verifies its complete fixed inventory and digests before +ingest, never obtains its provider credential, and does not call its producer. + The normalized corpus, private installation key, aggregate metrics, profiles, and drafting context stay in the local data root. Study and evaluation files are written only to explicit paths. Ordinary views use keyed pseudonymous IDs @@ -26,11 +30,21 @@ of the user's prose. ## Normalized observations -The corpus preserves message direction, timestamp, body availability and -source, message kind, attachment count, edit or retraction metadata, explicit -reply target, service, and conversation membership where the source supports -them. Unsupported or missing text remains unavailable rather than being -reconstructed. +The corpus preserves source, account, network, message direction, provider +ordering, timestamp, body availability and source, message kind, attachment +metadata, edit or retraction metadata, explicit reply target, service, and +conversation membership where the source supports them. Unsupported, deleted, +or truncated text remains unavailable rather than being reconstructed. A +truncated text record still represents a message bubble for tempo and reply +evidence, but never contributes prose. + +Native iMessage history and each connected bundle account have distinct source +namespaces. A bounded, truncated, or unknown bundle is not an authoritative +statement that omitted history no longer exists. Reimport merges present +records with retained state. Only explicit deletion, removal, replacement, or +tombstone state suppresses evidence, and a later record reappearance clears +that suppression. Bundle creation times are monotonic per source, so an older +snapshot cannot resurrect or overwrite newer state. The analysis uses several operational units: @@ -44,7 +58,11 @@ The analysis uses several operational units: in the same session. - An **explicit reply** is source metadata linking a message to an earlier message. It is distinct from a reaction or an ordinary adjacent response. -- A **reaction** is counted as interaction behavior, not authored prose. +- A **reaction** is counted as interaction behavior, not authored prose. A + reaction without a provider timestamp contributes to counts and direction + but not to temporal order, sessions, bursts, or response episodes. Raw + provider reaction values remain private and are not categorical dimensions + in aggregate metrics or drafting context. Five minutes and eight hours are reproducible segmentation parameters, not claims about natural conversational boundaries. Every metrics artifact records @@ -57,7 +75,11 @@ For each conversation, the CLI reports the evidence window and counts of incoming, outgoing, text, session, burst, and response records. Tempo metrics include response-latency quantiles, outgoing messages per response, the ratio of single-message to multi-message responses, multi-message inbound contexts, -visible multi-question contexts, and explicit reply frequency. +visible multi-question contexts, and explicit reply frequency. Session, burst, +and response construction runs independently for each source conversation +before person-scope results are combined. Adjacent timestamps in two apps or +threads never create one artificial episode. Mixed person scopes expose their +sorted service breakdown. Surface measurements cover characters and words, lowercase starts, terminal punctuation, question and exclamation marks, emoji-bearing messages, and @@ -83,7 +105,7 @@ each direction keeps at most 12 text messages per example, and total emitted body text is capped at 256 KiB. Coverage metadata states what was truncated or omitted. A packet is a sample of response contexts, not a transcript. -Version 0.2 adds temporal bounds to study selection. A profile intended for +Version 0.2 added temporal bounds to study selection. A profile intended for held-out evaluation should use only examples before the chosen cutoff. The cutoff, corpus revision, selection parameters, packet receipt, and evidence window form part of the analysis provenance. Profile validity uses a digest of @@ -171,6 +193,8 @@ sending, reacting, scheduling, or operating a messaging application. Reported behavior can be distorted by: - Messages that are not synchronized to the Mac or are no longer present; +- partial local provider exports whose completeness bounds exclude older or + remote history; - unsupported body encodings, attachments, edits, retractions, or source schema changes; - ambiguous or stale Contacts labels; diff --git a/docs/research.md b/docs/research.md index 189528a..9df965c 100644 --- a/docs/research.md +++ b/docs/research.md @@ -8,7 +8,7 @@ that boundary and the neighboring open-source work that informed it. The cited papers are primary research publications or preprints. Project descriptions link to their official repositories. A paper result is evidence about the task and population it evaluated, not proof that the same result -holds for private iMessage conversations. +holds for private conversations across the messaging sources a user imports. ## Personalization is contextual @@ -26,6 +26,11 @@ context-specific adjustments. Message Like Me therefore treats incoming messages as response context and only the user's outgoing messages as evidence of the user's prose. +[Catch Me If You Can? Not Yet](https://aclanthology.org/2025.findings-emnlp.532/) +evaluates nuanced individual style in informal communication, a task close to +private messaging. Its scope reinforces the same boundary: measured tendencies +can guide a draft without establishing a faithful digital copy of its author. + [LaMP](https://aclanthology.org/2024.acl-long.399/) evaluated personalized classification and generation from user histories and found retrieval-based personalization useful across most of its tasks. Its experiments included @@ -48,6 +53,12 @@ content and writing style separately. Its reported agreement with human judgment improved over the comparison methods in that study. It does not measure message timing, bubble boundaries, or reply-link behavior. +[Can You Make It Sound Like You?](https://aclanthology.org/2026.acl-long.2030/) +studies personalized writing through human review and post-editing. That +workflow supports Message Like Me's product boundary: the output is an unsent +candidate for the user to inspect and revise, not an autonomous act on the +user's behalf. + [Münker, Schwager, and Rettinger](https://arxiv.org/abs/2506.21974) tested LLM-based imitation of social-network communication and argue that a simulation must be validated for empirical realism in the setting where it @@ -110,8 +121,9 @@ conversation partner or make a hosted agent local. A 2026 preprint on [response times in donated WhatsApp and Instagram chats](https://arxiv.org/abs/2605.03687) reported persistent response-speed similarity between chat partners in its sample. This is preliminary evidence from different platforms and cannot set a -norm for iMessage users. It does support comparing tempo within a dyad instead -of treating one global latency distribution as a personal rule. +norm for users of any supported messaging source. It does support comparing +tempo within a dyad instead of treating one global latency distribution as a +personal rule. Historical latency is affected by sleep, work, travel, notifications, device availability, urgency, and missing data. Message Like Me reports it as diff --git a/package.json b/package.json index 9a97e06..fa1d551 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "@hraness/message-like-me", - "version": "0.2.0", - "description": "A local-first CLI and Agent Skill for studying your private iMessage history and drafting messages that sound like you.", + "version": "0.3.0", + "description": "A local-first CLI and Agent Skill for studying private messaging history and drafting messages that sound like you.", "license": "MIT", "type": "module", "packageManager": "bun@1.3.14", @@ -18,10 +18,12 @@ }, "keywords": [ "agent-skill", + "beeper", "bun", "cli", "imessage", "local-first", + "messaging", "messaging-style", "privacy" ], diff --git a/schema/local-message-bundle-v1.schema.json b/schema/local-message-bundle-v1.schema.json new file mode 100644 index 0000000..f46a189 --- /dev/null +++ b/schema/local-message-bundle-v1.schema.json @@ -0,0 +1,446 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://messagelikeme.com/schema/local-message-bundle-v1.schema.json", + "title": "Message Like Me local message bundle v1 JSON objects", + "description": "Validates manifest.json or one NDJSON record. Runtime validation additionally enforces canonical UTF-8 bytes, filesystem safety, byte bounds, digests, account realms, identity joins, and graph laws.", + "oneOf": [ + { "$ref": "#/$defs/manifest" }, + { "$ref": "#/$defs/account" }, + { "$ref": "#/$defs/participant" }, + { "$ref": "#/$defs/conversation" }, + { "$ref": "#/$defs/message" }, + { "$ref": "#/$defs/reaction" }, + { "$ref": "#/$defs/tombstone" } + ], + "$defs": { + "identifier": { + "type": "string", + "minLength": 1, + "maxLength": 1024, + "pattern": "^[^\\u0000-\\u001F\\u007F]+$", + "description": "Runtime bound is 1,024 UTF-8 bytes." + }, + "shortText": { + "type": "string", + "maxLength": 8192, + "pattern": "^[^\\u0000]*$", + "description": "Runtime bound is 8 KiB UTF-8." + }, + "mimeType": { + "type": "string", + "maxLength": 256, + "pattern": "^[^\\u0000]*$", + "description": "Runtime bound is 256 UTF-8 bytes." + }, + "body": { + "type": "string", + "maxLength": 1048576, + "pattern": "^[^\\u0000]*$", + "description": "Runtime bound is 1 MiB UTF-8." + }, + "token": { + "type": "string", + "maxLength": 128, + "pattern": "^[a-z0-9](?:[a-z0-9._+\\-]*[a-z0-9])?$" + }, + "network": { + "type": "string", + "maxLength": 64, + "pattern": "^[a-z0-9](?:[a-z0-9._+\\-]*[a-z0-9])?$" + }, + "version": { + "type": "string", + "maxLength": 128, + "pattern": "^[A-Za-z0-9](?:[A-Za-z0-9._+\\-]*[A-Za-z0-9])?$" + }, + "timestamp": { + "type": "string", + "format": "date-time", + "maxLength": 64, + "description": "Runtime requires exact canonical Date#toISOString() millisecond UTC text." + }, + "digest": { + "type": "string", + "pattern": "^[a-f0-9]{64}$" + }, + "nullableIdentifier": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/identifier" } + ] + }, + "nullableShortText": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/shortText" } + ] + }, + "nullableMimeType": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/mimeType" } + ] + }, + "nullableTimestamp": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/timestamp" } + ] + }, + "provenance": { + "type": "object", + "additionalProperties": false, + "required": [ + "providerId", + "providerRevision", + "observedAt", + "connectedAccountProviderId" + ], + "properties": { + "providerId": { "$ref": "#/$defs/identifier" }, + "providerRevision": { "$ref": "#/$defs/nullableIdentifier" }, + "observedAt": { "$ref": "#/$defs/timestamp" }, + "connectedAccountProviderId": { "$ref": "#/$defs/identifier" } + } + }, + "common": { + "type": "object", + "required": ["schemaVersion", "kind", "id", "accountId", "network", "provenance"], + "properties": { + "schemaVersion": { "const": 1 }, + "kind": { + "enum": ["account", "participant", "conversation", "message", "reaction", "tombstone"] + }, + "id": { "$ref": "#/$defs/identifier" }, + "accountId": { "$ref": "#/$defs/identifier" }, + "network": { "$ref": "#/$defs/network" }, + "provenance": { "$ref": "#/$defs/provenance" } + } + }, + "account": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": ["displayName", "handle", "selfParticipantId"], + "properties": { + "kind": { "const": "account" }, + "displayName": { "$ref": "#/$defs/nullableShortText" }, + "handle": { "$ref": "#/$defs/nullableShortText" }, + "selfParticipantId": { "$ref": "#/$defs/identifier" } + } + }, + "participant": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": ["displayName", "handle", "isSelf"], + "properties": { + "kind": { "const": "participant" }, + "displayName": { "$ref": "#/$defs/nullableShortText" }, + "handle": { "$ref": "#/$defs/nullableShortText" }, + "isSelf": { "type": "boolean" } + } + }, + "conversation": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": [ + "type", + "title", + "participantIds", + "participantsComplete", + "startedAt", + "lastMessageAt" + ], + "properties": { + "kind": { "const": "conversation" }, + "type": { "enum": ["direct", "group", "channel", "unknown"] }, + "title": { "$ref": "#/$defs/nullableShortText" }, + "participantIds": { + "type": "array", + "maxItems": 10000, + "uniqueItems": true, + "items": { "$ref": "#/$defs/identifier" } + }, + "participantsComplete": { "type": ["boolean", "null"] }, + "startedAt": { "$ref": "#/$defs/nullableTimestamp" }, + "lastMessageAt": { "$ref": "#/$defs/nullableTimestamp" } + } + }, + "attachment": { + "type": "object", + "additionalProperties": false, + "required": ["kind", "mimeType", "name", "sizeBytes"], + "properties": { + "kind": { "enum": ["audio", "document", "image", "link", "sticker", "video", "unknown"] }, + "mimeType": { "$ref": "#/$defs/nullableMimeType" }, + "name": { "$ref": "#/$defs/nullableShortText" }, + "sizeBytes": { + "oneOf": [ + { "type": "null" }, + { "type": "integer", "minimum": 0, "maximum": 9007199254740991 } + ] + } + } + }, + "reply": { + "type": "object", + "additionalProperties": false, + "required": ["messageId", "providerId"], + "properties": { + "messageId": { "$ref": "#/$defs/nullableIdentifier" }, + "providerId": { "$ref": "#/$defs/identifier" } + } + }, + "editInPlace": { + "type": "object", + "additionalProperties": false, + "required": ["kind", "editedAt", "providerRevision"], + "properties": { + "kind": { "const": "in-place" }, + "editedAt": { "$ref": "#/$defs/timestamp" }, + "providerRevision": { "$ref": "#/$defs/identifier" } + } + }, + "editReplacement": { + "type": "object", + "additionalProperties": false, + "required": ["kind", "replacesMessageId", "replacesProviderId", "editedAt", "providerRevision"], + "properties": { + "kind": { "const": "replacement" }, + "replacesMessageId": { "$ref": "#/$defs/nullableIdentifier" }, + "replacesProviderId": { "$ref": "#/$defs/identifier" }, + "editedAt": { "$ref": "#/$defs/timestamp" }, + "providerRevision": { "$ref": "#/$defs/identifier" } + } + }, + "deletion": { + "type": "object", + "additionalProperties": false, + "required": ["state", "observedAt", "providerRevision"], + "properties": { + "state": { "enum": ["revoked", "deleted-for-me", "revoked-and-deleted-for-me"] }, + "observedAt": { "$ref": "#/$defs/timestamp" }, + "providerRevision": { "$ref": "#/$defs/nullableIdentifier" } + } + }, + "message": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": [ + "conversationId", + "senderParticipantId", + "direction", + "sentAt", + "sortKey", + "body", + "bodyTruncated", + "replyTo", + "edit", + "deletion", + "attachments" + ], + "properties": { + "kind": { "const": "message" }, + "conversationId": { "$ref": "#/$defs/identifier" }, + "senderParticipantId": { "$ref": "#/$defs/nullableIdentifier" }, + "direction": { "enum": ["incoming", "outgoing", "unknown"] }, + "sentAt": { "$ref": "#/$defs/timestamp" }, + "sortKey": { "$ref": "#/$defs/identifier" }, + "body": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/body" } + ] + }, + "bodyTruncated": { "type": ["boolean", "null"] }, + "replyTo": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/reply" } + ] + }, + "edit": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/editInPlace" }, + { "$ref": "#/$defs/editReplacement" } + ] + }, + "deletion": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/deletion" } + ] + }, + "attachments": { + "type": "array", + "maxItems": 256, + "items": { "$ref": "#/$defs/attachment" } + } + } + }, + "reaction": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": ["messageId", "messageProviderId", "participantId", "body", "reactedAt", "state"], + "properties": { + "kind": { "const": "reaction" }, + "messageId": { "$ref": "#/$defs/nullableIdentifier" }, + "messageProviderId": { "$ref": "#/$defs/identifier" }, + "participantId": { "$ref": "#/$defs/nullableIdentifier" }, + "body": { "$ref": "#/$defs/shortText" }, + "reactedAt": { "$ref": "#/$defs/nullableTimestamp" }, + "state": { "enum": ["active", "removed"] } + } + }, + "tombstone": { + "allOf": [{ "$ref": "#/$defs/common" }], + "type": "object", + "unevaluatedProperties": false, + "required": ["entityKind", "entityId", "entityProviderId", "deletedAt", "scope", "providerRevision"], + "properties": { + "kind": { "const": "tombstone" }, + "entityKind": { "enum": ["conversation", "message", "reaction"] }, + "entityId": { "$ref": "#/$defs/nullableIdentifier" }, + "entityProviderId": { "$ref": "#/$defs/identifier" }, + "deletedAt": { "$ref": "#/$defs/timestamp" }, + "scope": { "enum": ["remote", "local", "unknown"] }, + "providerRevision": { "$ref": "#/$defs/nullableIdentifier" } + } + }, + "artifact": { + "type": "object", + "additionalProperties": false, + "required": ["path", "mediaType", "recordKind", "records", "bytes", "sha256"], + "properties": { + "path": { "enum": ["accounts.ndjson", "participants.ndjson", "conversations.ndjson", "messages.ndjson", "reactions.ndjson", "tombstones.ndjson"] }, + "mediaType": { "const": "application/x-ndjson" }, + "recordKind": { "enum": ["account", "participant", "conversation", "message", "reaction", "tombstone"] }, + "records": { "type": "integer", "minimum": 0, "maximum": 500000 }, + "bytes": { "type": "integer", "minimum": 0, "maximum": 536870912 }, + "sha256": { "$ref": "#/$defs/digest" } + } + }, + "manifest": { + "type": "object", + "additionalProperties": false, + "required": [ + "schemaVersion", + "format", + "source", + "provider", + "timestamps", + "completeness", + "warnings", + "privacy", + "counts", + "artifacts", + "integrity" + ], + "properties": { + "schemaVersion": { "const": 1 }, + "format": { "const": "message-like-me.local-message-bundle" }, + "source": { + "type": "object", + "additionalProperties": false, + "required": ["id", "version"], + "properties": { + "id": { "const": "beeper-local" }, + "version": { "$ref": "#/$defs/version" } + } + }, + "provider": { + "type": "object", + "additionalProperties": false, + "required": ["id", "version"], + "properties": { + "id": { "const": "beeper" }, + "version": { "$ref": "#/$defs/version" } + } + }, + "timestamps": { + "type": "object", + "additionalProperties": false, + "required": ["startedAt", "finishedAt", "createdAt"], + "properties": { + "startedAt": { "$ref": "#/$defs/timestamp" }, + "finishedAt": { "$ref": "#/$defs/timestamp" }, + "createdAt": { "$ref": "#/$defs/timestamp" } + } + }, + "completeness": { + "type": "object", + "additionalProperties": false, + "required": ["kind", "reason", "observedFrom", "observedThrough"], + "properties": { + "kind": { "enum": ["bounded-local", "truncated", "unknown"] }, + "reason": { + "oneOf": [ + { "type": "null" }, + { "$ref": "#/$defs/token" } + ] + }, + "observedFrom": { "$ref": "#/$defs/nullableTimestamp" }, + "observedThrough": { "$ref": "#/$defs/nullableTimestamp" } + } + }, + "warnings": { + "type": "array", + "maxItems": 128, + "uniqueItems": true, + "items": { "$ref": "#/$defs/token" } + }, + "privacy": { + "type": "object", + "additionalProperties": false, + "required": ["classification", "attachments", "providerUrls", "credentials"], + "properties": { + "classification": { "const": "private-local" }, + "attachments": { "const": "metadata-only" }, + "providerUrls": { "const": "excluded" }, + "credentials": { "const": "excluded" } + } + }, + "counts": { + "type": "object", + "additionalProperties": false, + "required": ["account", "participant", "conversation", "message", "reaction", "tombstone"], + "properties": { + "account": { "type": "integer", "minimum": 0, "maximum": 128 }, + "participant": { "type": "integer", "minimum": 0, "maximum": 500000 }, + "conversation": { "type": "integer", "minimum": 0, "maximum": 500000 }, + "message": { "type": "integer", "minimum": 0, "maximum": 500000 }, + "reaction": { "type": "integer", "minimum": 0, "maximum": 500000 }, + "tombstone": { "type": "integer", "minimum": 0, "maximum": 500000 } + } + }, + "artifacts": { + "type": "array", + "minItems": 6, + "maxItems": 6, + "prefixItems": [ + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "accounts.ndjson" }, "recordKind": { "const": "account" } } }] }, + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "participants.ndjson" }, "recordKind": { "const": "participant" } } }] }, + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "conversations.ndjson" }, "recordKind": { "const": "conversation" } } }] }, + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "messages.ndjson" }, "recordKind": { "const": "message" } } }] }, + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "reactions.ndjson" }, "recordKind": { "const": "reaction" } } }] }, + { "allOf": [{ "$ref": "#/$defs/artifact" }, { "type": "object", "properties": { "path": { "const": "tombstones.ndjson" }, "recordKind": { "const": "tombstone" } } }] } + ] + }, + "integrity": { + "type": "object", + "additionalProperties": false, + "required": ["algorithm", "bundleSha256"], + "properties": { + "algorithm": { "const": "sha256" }, + "bundleSha256": { "$ref": "#/$defs/digest" } + } + } + } + } + } +} diff --git a/scripts/check-standalone.ts b/scripts/check-standalone.ts index 4d5bc4b..b3110c6 100644 --- a/scripts/check-standalone.ts +++ b/scripts/check-standalone.ts @@ -15,7 +15,7 @@ import { } from "node:path"; const PACKAGE_ROOT = resolve(fileURLToPath(new URL("../", import.meta.url))); -const PUBLIC_DESCRIPTION = "A local-first CLI and Agent Skill for studying your private iMessage history and drafting messages that sound like you."; +const PUBLIC_DESCRIPTION = "A local-first CLI and Agent Skill for studying private messaging history and drafting messages that sound like you."; const SCANNED_DIRECTORIES = [ ".github", "dist", diff --git a/scripts/local-message-bundle-schema.test.ts b/scripts/local-message-bundle-schema.test.ts new file mode 100644 index 0000000..35bfe15 --- /dev/null +++ b/scripts/local-message-bundle-schema.test.ts @@ -0,0 +1,77 @@ +import { expect, test } from "bun:test"; +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; + +type JsonObject = Record; + +function object(value: unknown, label: string): JsonObject { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + throw new Error(`${label} must be an object`); + } + return value as JsonObject; +} + +function array(value: unknown, label: string): unknown[] { + if (!Array.isArray(value)) throw new Error(`${label} must be an array`); + return value; +} + +test("local message bundle schema publishes the frozen v1 contract", async () => { + const schema = object(JSON.parse(await readFile(join( + import.meta.dir, + "..", + "schema", + "local-message-bundle-v1.schema.json", + ), "utf8")) as unknown, "schema"); + expect(schema.$schema).toBe("https://json-schema.org/draft/2020-12/schema"); + expect(schema.$id).toBe("https://messagelikeme.com/schema/local-message-bundle-v1.schema.json"); + expect(array(schema.oneOf, "schema.oneOf")).toHaveLength(7); + + const definitions = object(schema.$defs, "schema.$defs"); + const manifest = object(definitions.manifest, "manifest"); + const manifestProperties = object(manifest.properties, "manifest.properties"); + expect(object(manifestProperties.format, "manifest.format").const) + .toBe("message-like-me.local-message-bundle"); + const counts = object(object(manifestProperties.counts, "manifest.counts").properties, "counts.properties"); + expect(object(counts.account, "counts.account").maximum).toBe(128); + const artifacts = object(manifestProperties.artifacts, "manifest.artifacts"); + expect(artifacts.minItems).toBe(6); + expect(artifacts.maxItems).toBe(6); + const artifactPrefixes = array(artifacts.prefixItems, "manifest.artifacts.prefixItems"); + expect(artifactPrefixes).toHaveLength(6); + for (const [index, prefixValue] of artifactPrefixes.entries()) { + const prefix = object(prefixValue, `manifest.artifacts.prefixItems[${index}]`); + const refinement = object( + array(prefix.allOf, `manifest.artifacts.prefixItems[${index}].allOf`)[1], + `manifest.artifacts.prefixItems[${index}].allOf[1]`, + ); + expect(refinement.type).toBe("object"); + } + + const mimeType = object(definitions.mimeType, "mimeType"); + expect(mimeType.maxLength).toBe(256); + expect(mimeType.pattern).toBe("^[^\\u0000]*$"); + expect(mimeType.description).toBe("Runtime bound is 256 UTF-8 bytes."); + const attachment = object(definitions.attachment, "attachment"); + const attachmentProperties = object(attachment.properties, "attachment.properties"); + expect(object(attachmentProperties.mimeType, "attachment.mimeType").$ref) + .toBe("#/$defs/nullableMimeType"); + + const message = object(definitions.message, "message"); + const messageProperties = object(message.properties, "message.properties"); + expect(object(messageProperties.sortKey, "message.sortKey").$ref).toBe("#/$defs/identifier"); + expect(array(object(messageProperties.edit, "message.edit").oneOf, "message.edit.oneOf")) + .toHaveLength(3); + + const reaction = object(definitions.reaction, "reaction"); + const reactionProperties = object(reaction.properties, "reaction.properties"); + expect(object(reactionProperties.messageProviderId, "reaction.messageProviderId").$ref) + .toBe("#/$defs/identifier"); + expect(object(reactionProperties.reactedAt, "reaction.reactedAt").$ref) + .toBe("#/$defs/nullableTimestamp"); + + const tombstone = object(definitions.tombstone, "tombstone"); + const tombstoneProperties = object(tombstone.properties, "tombstone.properties"); + expect(object(tombstoneProperties.entityKind, "tombstone.entityKind").enum) + .toEqual(["conversation", "message", "reaction"]); +}); diff --git a/scripts/package-smoke.ts b/scripts/package-smoke.ts index a198bb0..b162155 100644 --- a/scripts/package-smoke.ts +++ b/scripts/package-smoke.ts @@ -269,6 +269,21 @@ export async function packageSmoke(): Promise { if (schema.$schema !== "https://json-schema.org/draft/2020-12/schema") { throw new Error("Packed style profile schema must use JSON Schema draft 2020-12"); } + const bundleSchema = record( + JSON.parse( + await readFile( + join(installedPackage, "schema", "local-message-bundle-v1.schema.json"), + "utf8", + ), + ) as unknown, + "installed local message bundle schema", + ); + if ( + bundleSchema.$schema !== "https://json-schema.org/draft/2020-12/schema" + || bundleSchema.$id !== "https://messagelikeme.com/schema/local-message-bundle-v1.schema.json" + ) { + throw new Error("Packed local message bundle schema has the wrong identity"); + } } finally { await rm(work, { force: true, recursive: true }); } diff --git a/site/AGENTS.md b/site/AGENTS.md index 728d6f8..d6197bf 100644 --- a/site/AGENTS.md +++ b/site/AGENTS.md @@ -9,10 +9,12 @@ - Keep the page informational. It must never accept, upload, transmit, or request message history, contact data, study packets, profiles, or drafts. -- Keep the canonical product description exact and route installation to the - immutable GitHub release. -- Describe the CLI as local-first, bring-your-own-agent, and drafts-only. Never - imply that the site analyzes data or that Message Like Me sends messages. +- Keep the canonical product description exact: “A local-first CLI and Agent + Skill for studying private messaging history and drafting messages that sound + like you.” Route installation to the immutable GitHub release. +- Describe the CLI as local-first, bring-your-own-agent, source-aware, and + drafts-only. Never imply that the site analyzes data or that Message Like Me + sends messages. - Use synthetic examples only. Do not publish real counts, labels, handles, excerpts, identities, private paths, or derived personal profiles. - Use Bun 1.3.14 for installation and scripts. Run Vinext and ESLint through diff --git a/site/app/layout.tsx b/site/app/layout.tsx index da5f136..c36ce2d 100644 --- a/site/app/layout.tsx +++ b/site/app/layout.tsx @@ -3,9 +3,9 @@ import './globals.css'; export const metadata: Metadata = { metadataBase: new URL('https://messagelikeme.com'), - title: 'Message Like Me — Study how you text', + title: 'Message Like Me — Study how you message', description: - 'A local-first CLI and Agent Skill for studying your private iMessage history and drafting messages that sound like you.', + 'A local-first CLI and Agent Skill for studying private messaging history and drafting messages that sound like you.', alternates: { canonical: '/', }, @@ -13,23 +13,23 @@ export const metadata: Metadata = { type: 'website', url: '/', siteName: 'Message Like Me', - title: 'Message Like Me — Study how you text', + title: 'Message Like Me — Study how you message', description: - 'A local-first CLI and Agent Skill for studying your private iMessage history and drafting messages that sound like you.', + 'A local-first CLI and Agent Skill for studying private messaging history and drafting messages that sound like you.', images: [ { url: '/og.png', width: 1200, height: 630, - alt: 'Message Like Me — Study how you text.', + alt: 'Message Like Me — Study how you message.', }, ], }, twitter: { card: 'summary_large_image', - title: 'Message Like Me — Study how you text', + title: 'Message Like Me — Study how you message', description: - 'A local-first CLI and Agent Skill for studying your private iMessage history and drafting messages that sound like you.', + 'A local-first CLI and Agent Skill for studying private messaging history and drafting messages that sound like you.', images: ['/og.png'], }, }; diff --git a/site/app/page.tsx b/site/app/page.tsx index a660433..8f697a3 100644 --- a/site/app/page.tsx +++ b/site/app/page.tsx @@ -1,7 +1,7 @@ import { readmeHtml } from './readme.generated'; const githubUrl = 'https://github.com/hraness/message-like-me'; -const releaseUrl = `${githubUrl}/releases/tag/v0.2.0`; +const releaseUrl = `${githubUrl}/releases/tag/v0.3.0`; export default function Home() { return ( @@ -24,12 +24,13 @@ export default function Home() { Your messages already know how you write.

- Turn your private iMessage history into contact-aware style profiles - an agent can use to draft unsent replies in your voice. + Turn private messaging history from Messages and your connected + accounts into contact-aware style profiles an agent can use to draft + unsent replies in your voice.

- $ messagelikeme ingest imessage + $ wrench beeper export-message-like-me --auth beeper-main --output "$HOME/message-like-me-beeper"

-

✓ corpus stored locally

+

✓ Wrench 0.13.0+ private bundle written

- $ messagelikeme inspect tempo <contact-id> + $ messagelikeme ingest bundle --input "$HOME/message-like-me-beeper"

-

✓ response shape ready

+

✓ source-aware history merged

@@ -85,10 +86,11 @@ export default function Home() {

01 / ingest

Read stable local copies.

- Import Messages and optional Contacts data without opening the - source databases for mutation. + Import Messages and optional Contacts directly, or merge a + private source-aware bundle exported by Wrench 0.13.0+ from + Beeper.

- messagelikeme ingest imessage + messagelikeme ingest bundle --input "$HOME/message-like-me-beeper"

02 / understand

@@ -136,7 +138,8 @@ export default function Home() {

Evidence for a draft.

Message Like Me measures your outgoing prose and delivery shape for - one person, then gives your agent a bounded, inspectable profile. + one person across imported services, then gives your agent a + bounded, inspectable profile.

@@ -167,15 +170,16 @@ export default function Home() {
-

1bun add --global github:hraness/message-like-me#v0.2.0

+

1bun add --global github:hraness/message-like-me#v0.3.0

2messagelikeme skill install

3messagelikeme ingest imessage

+

4messagelikeme ingest bundle --input "$HOME/message-like-me-beeper"

diff --git a/site/app/readme.generated.ts b/site/app/readme.generated.ts index 6a1b34d..e789cba 100644 --- a/site/app/readme.generated.ts +++ b/site/app/readme.generated.ts @@ -1,2 +1,2 @@ // Generated from ../README.md by scripts/sync-readme.ts. -export const readmeHtml = "

Message Like Me

\n

A local-first CLI and Agent Skill for studying your private iMessage history\nand drafting messages that sound like you.

\n

Message Like Me turns a local Messages database into deterministic conversation\nmetrics, bounded study packets, and reusable style profiles. Its Agent Skill\nteaches Codex, Claude, and other coding agents how to interpret those local\nartifacts and draft unsent replies in your voice.

\n

The CLI does not call an AI service, authenticate with a product account, send\nmessages, or operate Messages. The agent already running the skill supplies the\nsemantic analysis and drafting judgment.

\n

This is an evidence layer for relationship-aware drafting, not a digital clone.\nIt does not train a model, represent your identity, infer your beliefs, or claim\nthat a draft is what you would have written. Your current meaning, facts, and\nintent outrank historical style.

\n

Install

\n

Message Like Me requires Bun 1.3.14 or newer. Install the immutable public\nrelease from GitHub, then install the Agent Skill:

\n
bun add --global github:hraness/message-like-me#v0.2.0\nmessagelikeme skill install\n
\n

Start a new agent session after installing the skill. The default target is\nCodex at user scope. Other supported targets and project-local installation are\navailable explicitly:

\n
messagelikeme skill install --target claude\nmessagelikeme skill install --target agents --scope project\nmessagelikeme skill path\n
\n

Message Like Me is distributed directly through GitHub and is not published to\nnpm.

\n

Start with your local history

\n

Initialize the private data store and inspect its location:

\n
messagelikeme init\nmessagelikeme doctor --json\n
\n

On macOS, the default store is:

\n
~/Library/Application Support/Message Like Me/\n
\n

The directory is private to the current user. It contains a local SQLite\ndatabase, stored profiles, and a private installation key used to derive\nstable pseudonymous IDs. Study packets are written only to the explicit path\nyou choose. You can put the store elsewhere by placing\n--data-dir /absolute/private/path before the command.

\n

Import the current user's iMessage database:

\n
messagelikeme ingest imessage --json\n
\n

The default source is the current user's Messages chat.db. Use --database\nonly to name another caller-owned physical database:

\n
messagelikeme ingest imessage --database /absolute/path/to/chat.db --json\n
\n

Ingestion validates the source schema and ownership, makes a stable private\ncopy of the database and its transactional sidecars, and opens only that copy\nwith SQLite. It does not change Messages, chat.db, or its sidecars. macOS may\nrequire permission for the terminal or agent host to read Messages data.

\n

Optionally enrich and join direct conversations with private identities from\nmacOS Contacts:

\n
messagelikeme ingest contacts --json\n
\n

The default source is the current user's AddressBook directory. An explicit\nabsolute AddressBook root, Sources directory, store directory, or\nAddressBook-vN.abcddb file can be selected with --addressbook:

\n
messagelikeme ingest contacts \\\n  --addressbook /absolute/path/to/AddressBook \\\n  --json\n
\n

Contacts ingest may run before or after iMessage ingest. It reads only bounded\nname, email, and phone fields from a stable private copy. Exact normalized\nemail or phone handles can join several one-to-one iMessage, SMS, and email\nthreads for the same AddressBook person into one analysis scope. Existing\nconversation IDs remain aliases for that person scope. Shared handles remain\nambiguous, local phone numbers never gain a guessed country code, unmatched\nthreads stay separate, and groups are never collapsed to one person. Contact\nlabels have their own revision, so a rename does not stale a messaging-style\nprofile. messagelikeme doctor reports local aggregate state without asking\nfor an account or credential.

\n

Inspect behavior without exposing prose

\n

Contact listings and aggregate views omit private labels, handles, and message\nbodies by default:

\n
messagelikeme contacts list --min-outgoing 20 --json\nmessagelikeme contacts show <contact-id> --json\nmessagelikeme inspect tempo <contact-id> --session-gap 28800 --burst-gap 300 --json\nmessagelikeme inspect sessions <contact-id> --limit 20 --json\n
\n

The metrics cover conversation start and end, message counts, incoming and\noutgoing turns, within-session response latency, single-message versus\nmulti-message replies, surface prose features, multi-point response contexts,\nreactions, and explicit reply use. Incoming messages establish what you were\nresponding to; they are never counted as examples of your writing style.\nSession and burst gaps are configurable seconds and are recorded with each\nresult. They are segmentation choices, not universal facts about conversation.

\n

Pass --private to contacts list or contacts show only when you need to\nresolve a pseudonymous contact to its local private label or participants.

\n

When you already know the complete Contacts label, resolve only that exact\nprivate name instead of listing every label:

\n
messagelikeme contacts resolve "Exact Contact Name" --private --json\n
\n

Resolution is normalized for case and Unicode representation, but it does not\nperform prefix, substring, phonetic, or fuzzy matching. It returns only direct\nperson scopes and labels, never handles or message bodies.

\n

Build a style profile

\n

Aggregate metrics cannot explain why a short burst works in one context or why\na longer single message appears in another. For that semantic work, prepare a\nsmall, diverse study packet at an explicit private path:

\n
messagelikeme study prepare <contact-id> \\\n  --output /absolute/private/path/study.json \\\n  --before 2026-08-01T00:00:00.000Z \\\n  --limit 24 \\\n  --json\n
\n

study prepare and evaluate prepare are the only commands that write bounded\nmessage bodies outside the private database. Their outputs are mode 0600.\nA study packet contains incoming context and outgoing responses selected across\ndifferent response shapes; it is not a full transcript export. By default,\neach body is capped at 4 KiB, each example keeps at most 12 text messages per\ndirection, and the entire packet keeps at most 256 KiB of body text. Packet\ncoverage fields report every truncation or omission explicitly.

\n

Keep the JSON receipt with the analysis. Its packetSha256 binds the finished\nprofile to these exact packet bytes; the packet does not contain its own digest.

\n

--after is inclusive and --before is exclusive. Temporal bounds let you\nreserve later conversations for evaluation. Invoke $message-like-me in your\nagent and ask it to analyze that contact. The skill separates measured facts\nfrom inferred patterns, covers prose and tempo, studies how several inbound\npoints are handled, and treats reply links and tapbacks separately from written\ntext.

\n

The agent writes a schema-version-two profile and asks the CLI to validate and\nstore it:

\n
messagelikeme profile apply /absolute/private/path/profile.json --json\nmessagelikeme profile show <contact-id> --json\n
\n

A version-two profile records the global corpus revision for provenance, a\nperson-and-window-specific evidence revision for validity, the exact\nstudy-packet SHA-256, and the packet's non-body evidence manifest. Measured and\ninferred claims cite valid packet example IDs and record counterexamples,\nsupport counts, confidence, and drafting consequences. Messages for someone\nelse or outside the studied time window do not stale it; changes inside its\nactual evidence do.

\n

Export a profile only when you need an explicit private copy:

\n
messagelikeme profile export <contact-id> --output /absolute/private/path/profile.json\n
\n

Version-one profiles remain readable for migration, but new analyses should use\nschema/style-profile-v2.schema.json.

\n

Audit against later conversations

\n

Prepare a separate prompt and reference set from conversations after the study\ncutoff:

\n
messagelikeme evaluate prepare <contact-id> \\\n  --after 2026-08-01T00:00:00.000Z \\\n  --prompt-output /absolute/private/path/evaluation-prompts.json \\\n  --reference-output /absolute/private/path/evaluation-references.json \\\n  --json\n
\n

Give the agent only the prompt file and fix one candidate bubble sequence per\ncase before opening the reference file. Then compare intent coverage, factual\nmeaning, prose, bubble shape, explicit replies, privacy leakage, and\ncalibration. The files support a blind workflow but do not enforce one, and the\nhistorical response is one observation rather than a unique correct answer.\nThe CLI deliberately does not collapse these dimensions into a universal\nfidelity score. See the methodology.

\n

Draft an unsent reply

\n

Ask an agent with the installed $message-like-me skill to draft for a\npseudonymous contact. The compact deterministic context is available through:

\n
messagelikeme context <contact-id> --json\n
\n

The skill preserves your intended meaning, selects the applicable profile,\nand can express the result as one message or a realistic sequence of separate\nbubbles. It uses explicit replies only when your evidence and the current\ncontext support them.

\n

Drafting ends with text in the agent task. Message Like Me has no send, react,\nschedule, or messaging-application command.

\n

Command reference

\n

Run messagelikeme --help for the checked grammar. The public surfaces are:

\n
messagelikeme init [--json]\nmessagelikeme ingest imessage [--database PATH] [--json]\nmessagelikeme ingest contacts [--addressbook PATH] [--json]\nmessagelikeme contacts list [--min-outgoing N] [--limit N] [--private] [--json]\nmessagelikeme contacts show CONTACT_ID [--private] [--json]\nmessagelikeme contacts resolve QUERY --private [--limit N] [--json]\nmessagelikeme inspect tempo CONTACT_ID [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme inspect sessions CONTACT_ID [--limit N] [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme study prepare CONTACT_ID --output FILE [--limit N]\n  [--after ISO_TIMESTAMP] [--before ISO_TIMESTAMP]\n  [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme evaluate prepare CONTACT_ID --after ISO_TIMESTAMP\n  --prompt-output FILE --reference-output FILE [--before ISO_TIMESTAMP]\n  [--limit N] [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme profile apply FILE [--json]\nmessagelikeme profile show CONTACT_ID [--json]\nmessagelikeme profile export CONTACT_ID --output FILE [--json]\nmessagelikeme context CONTACT_ID [--json]\nmessagelikeme skill path [--json]\nmessagelikeme skill install [--target codex|claude|agents]\n  [--scope user|project] [--project PATH] [--force] [--json]\nmessagelikeme doctor [--json]\n
\n

Place global --data-dir PATH before the command.

\n

Privacy model

\n
    \n
  • The original chat.db and AddressBook databases remain authoritative.\nSQLite opens only stable private copies, never the source files or sidecars.
    • \n
  • The normalized corpus, profiles, and installation key stay in a private local\nstore with owner-only permissions.
    • \n
  • Stable contact, conversation, and message IDs are derived with a private\nper-install HMAC key. Pseudonymous IDs are not encryption.
    • \n
  • Aggregate commands omit bodies and private labels. Study and evaluation\npackets are bounded, explicit body-bearing exports.
    • \n
  • Message text never goes to a Message Like Me server. There is no service,\naccount, auth flow, analytics client, or network-backed model call.
    • \n
  • Opening a study packet makes its bounded excerpts visible to the agent\nenvironment already running the skill. Use an agent environment whose data\nhandling you accept; the CLI cannot make a hosted agent local.
    • \n
  • Public fixtures are synthetic. Private corpora, profiles, packets, and drafts\ndo not belong in Git, issues, logs, packages, or examples.
    • \n
  • A draft is never sent.
    • \n
\n

Read SECURITY.md before integrating the library into another\ntool or handling a private packet outside the CLI. The\nmethodology defines every unit and evidence boundary;\nthe research review documents papers, neighboring OSS, and\nthe claims this project does not make.

\n

TypeScript library

\n

The package exports the versioned corpus, metrics, study-packet, and profile\ntypes plus deterministic canonical JSON and SHA-256 helpers:

\n
import type { ContactMetrics, StyleProfileV2 } from "@hraness/message-like-me"\nimport { canonicalJson, sha256 } from "@hraness/message-like-me"\n
\n

The library does not start the CLI, inspect Messages or Contacts, connect to a\nnetwork, or send a draft merely because it is imported.

\n

Development

\n
bun install --frozen-lockfile --ignore-scripts\nbun run check\n
\n

Tests use synthetic Messages and AddressBook databases plus synthetic\nconversations. Never add a real message, handle, group title, attachment,\ncontact record, private path, or derived profile to a fixture.

\n

The canonical repository is\nhraness/message-like-me.\nThe informational project page is\nmessagelikeme.com. The CLI does not connect to\nthe site, and the site never receives message or contact data.

\n

License

\n

MIT.

\n"; +export const readmeHtml = "

Message Like Me

\n

A local-first CLI and Agent Skill for studying private messaging history and\ndrafting messages that sound like you.

\n

Message Like Me turns private local messaging history into deterministic\nconversation metrics, bounded study packets, and reusable style profiles. It\nreads native iMessage history and strict local source bundles, including\nmulti-account Beeper exports produced through Wrench. Its Agent Skill teaches\nCodex, Claude, and other coding agents how to interpret those local artifacts\nand draft unsent replies in your voice.

\n

The CLI does not call an AI service, authenticate with a product account, send\nmessages, or operate Messages. The agent already running the skill supplies the\nsemantic analysis and drafting judgment.

\n

This is an evidence layer for relationship-aware drafting, not a digital clone.\nIt does not train a model, represent your identity, infer your beliefs, or claim\nthat a draft is what you would have written. Your current meaning, facts, and\nintent outrank historical style.

\n

Install

\n

Message Like Me requires Bun 1.3.14 or newer. Install the immutable public\nrelease from GitHub, then install the Agent Skill:

\n
bun add --global github:hraness/message-like-me#v0.3.0\nmessagelikeme skill install\n
\n

Start a new agent session after installing the skill. The default target is\nCodex at user scope. Other supported targets and project-local installation are\navailable explicitly:

\n
messagelikeme skill install --target claude\nmessagelikeme skill install --target agents --scope project\nmessagelikeme skill path\n
\n

Message Like Me is distributed directly through GitHub and is not published to\nnpm.

\n

Start with private local history

\n

Initialize the private data store and inspect its location:

\n
messagelikeme init\nmessagelikeme doctor --json\n
\n

On macOS, the default store is:

\n
~/Library/Application Support/Message Like Me/\n
\n

The directory is private to the current user. It contains a local SQLite\ndatabase, stored profiles, and a private installation key used to derive\nstable pseudonymous IDs. Study packets are written only to the explicit path\nyou choose. You can put the store elsewhere by placing\n--data-dir /absolute/private/path before the command.

\n

Import the current user's iMessage database:

\n
messagelikeme ingest imessage --json\n
\n

The default source is the current user's Messages chat.db. Use --database\nonly to name another caller-owned physical database:

\n
messagelikeme ingest imessage --database /absolute/path/to/chat.db --json\n
\n

Ingestion validates the source schema and ownership, makes a stable private\ncopy of the database and its transactional sidecars, and opens only that copy\nwith SQLite. It does not change Messages, chat.db, or its sidecars. macOS may\nrequire permission for the terminal or agent host to read Messages data.

\n

To study accounts connected through Beeper, install or update to\nWrench 0.13.0 or newer, then ask\nit to create a new private Message Like Me bundle:

\n
wrench beeper export-message-like-me \\\n  --auth <beeper-auth-id> \\\n  --output /absolute/private/path/beeper-bundle \\\n  --json\n
\n

The optional --limit-chats, --limit-messages, and --max-participants\nflags lower the export bounds. The output path must be a normalized absolute\npath to a directory that does not already exist. Wrench calls the pinned\nofficial Beeper CLI directly. It enumerates\nthe connected account realm, invokes export --no-attachments once per\naccount in deterministic order, and reports the account ordinal, elapsed-time\nheartbeats, and cumulative validated chat and message counts on stderr. It\nretains each private raw shard until it can atomically publish the complete\nmode-0700 seven-file bundle with mode-0600 files.

\n

The export does not use the separate\nBeeper Desktop API MCP project.\nThe CLI path supplies the bounded account snapshots and local files needed for\nhash validation, deterministic conversion, crash recovery, and atomic\npublication. Provider URLs and credentials are excluded. Message Like Me does\nnot receive the Beeper credential and does not call Beeper or Wrench itself.

\n

Ingest the finished directory, then inspect its redacted source health:

\n
messagelikeme ingest bundle --input /absolute/private/path/beeper-bundle --json\nmessagelikeme sources list --json\nmessagelikeme sources show <source-id> --json\n
\n

The importer verifies the fixed version-one inventory, canonical UTF-8 NDJSON,\nrecord and byte bounds, owner-only permissions, artifact digests, and manifest\ndigest before changing the store. One bundle may contain several connected\naccounts and networks; each becomes a separate source namespace. Native\niMessage and prior bundle sources remain alongside it.

\n

The complete interchange, integrity, identity, and reimport laws are in the\nversion-one local message bundle contract.

\n

Beeper exports describe bounded local observations. A later bounded export\nthat omits an older record does not delete retained history. Explicit deletion,\nremoval, replacement, and tombstone records suppress their target, and a later\nreappearance restores it. Older snapshots cannot overwrite newer state. Use\nsources show <source-id> --private --json only when you deliberately need the\nprivate provider account and source metadata.

\n

Optionally enrich and join direct conversations with private identities from\nmacOS Contacts:

\n
messagelikeme ingest contacts --json\n
\n

The default source is the current user's AddressBook directory. An explicit\nabsolute AddressBook root, Sources directory, store directory, or\nAddressBook-vN.abcddb file can be selected with --addressbook:

\n
messagelikeme ingest contacts \\\n  --addressbook /absolute/path/to/AddressBook \\\n  --json\n
\n

Contacts ingest may run before or after any message source. It reads only\nbounded name, email, and phone fields from a stable private copy. Exact\nnormalized email or E.164 phone handles can join several one-to-one threads\nfor the same AddressBook person into one analysis scope. A bundle conversation\nis eligible only when the producer positively marks its direct participant\nroster complete. Existing conversation IDs remain aliases for that person\nscope. Shared handles remain ambiguous, local phone numbers never gain a\nguessed country code, unmatched threads stay separate, and groups are never\ncollapsed to one person. Contact labels have their own revision, so a rename\ndoes not stale a messaging-style profile. messagelikeme doctor reports local\naggregate state without asking for an account or credential.

\n

Inspect behavior without exposing prose

\n

Contact listings and aggregate views omit private labels, handles, and message\nbodies by default:

\n
messagelikeme contacts list --min-outgoing 20 --json\nmessagelikeme contacts show <contact-id> --json\nmessagelikeme inspect tempo <contact-id> --session-gap 28800 --burst-gap 300 --json\nmessagelikeme inspect sessions <contact-id> --limit 20 --json\n
\n

The metrics cover conversation start and end, message counts, incoming and\noutgoing turns, within-session response latency, single-message versus\nmulti-message replies, surface prose features, multi-point response contexts,\nreactions, and explicit reply use. Incoming messages establish what you were\nresponding to; they are never counted as examples of your writing style.\nSessions, bursts, and response episodes never cross a source conversation\nboundary. Person scopes spanning several apps expose a sorted services\nbreakdown instead of hiding the mixed-channel evidence behind a null service.\nReactions with no provider timestamp still contribute to reaction counts and\ndirection, but never to temporal metrics. Raw provider reaction values remain\nprivate; ordinary metrics and drafting context expose only fixed-size counts,\ndirection, datedness, and the outgoing reaction ratio. Session and burst gaps\nare configurable seconds and are recorded with each result. They are\nsegmentation choices, not universal facts about conversation.

\n

Pass --private to contacts list or contacts show only when you need to\nresolve a pseudonymous contact to its local private label or participants.

\n

When you already know the complete Contacts label, resolve only that exact\nprivate name instead of listing every label:

\n
messagelikeme contacts resolve "Exact Contact Name" --private --json\n
\n

Resolution is normalized for case and Unicode representation, but it does not\nperform prefix, substring, phonetic, or fuzzy matching. It returns only direct\nperson scopes and labels, never handles or message bodies.

\n

Build a style profile

\n

Aggregate metrics cannot explain why a short burst works in one context or why\na longer single message appears in another. For that semantic work, prepare a\nsmall, diverse study packet at an explicit private path:

\n
messagelikeme study prepare <contact-id> \\\n  --output /absolute/private/path/study.json \\\n  --before 2026-08-01T00:00:00.000Z \\\n  --limit 24 \\\n  --json\n
\n

study prepare and evaluate prepare are the only commands that write bounded\nmessage bodies outside the private database. Their outputs are mode 0600.\nA study packet contains incoming context and outgoing responses selected across\ndifferent response shapes; it is not a full transcript export. By default,\neach body is capped at 4 KiB, each example keeps at most 12 text messages per\ndirection, and the entire packet keeps at most 256 KiB of body text. Packet\ncoverage fields report every truncation or omission explicitly.

\n

Keep the JSON receipt with the analysis. Its packetSha256 binds the finished\nprofile to these exact packet bytes; the packet does not contain its own digest.

\n

--after is inclusive and --before is exclusive. Temporal bounds let you\nreserve later conversations for evaluation. Invoke $message-like-me in your\nagent and ask it to analyze that contact. The skill separates measured facts\nfrom inferred patterns, covers prose and tempo, studies how several inbound\npoints are handled, and treats reply links and tapbacks separately from written\ntext.

\n

The agent writes a schema-version-two profile and asks the CLI to validate and\nstore it:

\n
messagelikeme profile apply /absolute/private/path/profile.json --json\nmessagelikeme profile show <contact-id> --json\n
\n

A version-two profile records the global corpus revision for provenance, a\nperson-and-window-specific evidence revision for validity, the exact\nstudy-packet SHA-256, and the packet's non-body evidence manifest. Measured and\ninferred claims cite valid packet example IDs and record counterexamples,\nsupport counts, confidence, and drafting consequences. Messages for someone\nelse or outside the studied time window do not stale it; changes inside its\nactual evidence do.

\n

Export a profile only when you need an explicit private copy:

\n
messagelikeme profile export <contact-id> --output /absolute/private/path/profile.json\n
\n

Version-one profiles remain readable for migration, but new analyses should use\nschema/style-profile-v2.schema.json.

\n

Audit against later conversations

\n

Prepare a separate prompt and reference set from conversations after the study\ncutoff:

\n
messagelikeme evaluate prepare <contact-id> \\\n  --after 2026-08-01T00:00:00.000Z \\\n  --prompt-output /absolute/private/path/evaluation-prompts.json \\\n  --reference-output /absolute/private/path/evaluation-references.json \\\n  --json\n
\n

Give the agent only the prompt file and fix one candidate bubble sequence per\ncase before opening the reference file. Then compare intent coverage, factual\nmeaning, prose, bubble shape, explicit replies, privacy leakage, and\ncalibration. The files support a blind workflow but do not enforce one, and the\nhistorical response is one observation rather than a unique correct answer.\nThe CLI deliberately does not collapse these dimensions into a universal\nfidelity score. See the methodology.

\n

Draft an unsent reply

\n

Ask an agent with the installed $message-like-me skill to draft for a\npseudonymous contact. The compact deterministic context is available through:

\n
messagelikeme context <contact-id> --json\n
\n

The skill preserves your intended meaning, selects the applicable profile,\nand can express the result as one message or a realistic sequence of separate\nbubbles. It uses explicit replies only when your evidence and the current\ncontext support them.

\n

Drafting ends with text in the agent task. Message Like Me has no send, react,\nschedule, or messaging-application command.

\n

Command reference

\n

Run messagelikeme --help for the checked grammar. The public surfaces are:

\n
messagelikeme init [--json]\nmessagelikeme ingest imessage [--database PATH] [--json]\nmessagelikeme ingest contacts [--addressbook PATH] [--json]\nmessagelikeme ingest bundle --input ABS_PATH [--json]\nmessagelikeme sources list [--private] [--json]\nmessagelikeme sources show SOURCE_ID [--private] [--json]\nmessagelikeme contacts list [--min-outgoing N] [--limit N] [--private] [--json]\nmessagelikeme contacts show CONTACT_ID [--private] [--json]\nmessagelikeme contacts resolve QUERY --private [--limit N] [--json]\nmessagelikeme inspect tempo CONTACT_ID [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme inspect sessions CONTACT_ID [--limit N] [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme study prepare CONTACT_ID --output FILE [--limit N]\n  [--after ISO_TIMESTAMP] [--before ISO_TIMESTAMP]\n  [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme evaluate prepare CONTACT_ID --after ISO_TIMESTAMP\n  --prompt-output FILE --reference-output FILE [--before ISO_TIMESTAMP]\n  [--limit N] [--session-gap N] [--burst-gap N] [--json]\nmessagelikeme profile apply FILE [--json]\nmessagelikeme profile show CONTACT_ID [--json]\nmessagelikeme profile export CONTACT_ID --output FILE [--json]\nmessagelikeme context CONTACT_ID [--json]\nmessagelikeme skill path [--json]\nmessagelikeme skill install [--target codex|claude|agents]\n  [--scope user|project] [--project PATH] [--force] [--json]\nmessagelikeme doctor [--json]\n
\n

Place global --data-dir PATH before the command.

\n

Privacy model

\n
    \n
  • The original chat.db and AddressBook databases remain authoritative.\nSQLite opens only stable private copies, never the source files or sidecars.
    • \n
  • Source bundles remain private caller-owned inputs. Import verifies their\nfixed inventory, canonical bytes, digests, bounds, and owner-only modes.
    • \n
  • The normalized corpus, profiles, and installation key stay in a private local\nstore with owner-only permissions.
    • \n
  • Stable source, contact, participant, conversation, message, and reaction IDs\nare derived with a private per-install HMAC key. Pseudonymous IDs are not\nencryption.
    • \n
  • Aggregate commands omit bodies and private labels. Study and evaluation\npackets are bounded, explicit body-bearing exports.
    • \n
  • Message text never goes to a Message Like Me server. There is no service,\naccount, auth flow, analytics client, or network-backed model call.
    • \n
  • Opening a study packet makes its bounded excerpts visible to the agent\nenvironment already running the skill. Use an agent environment whose data\nhandling you accept; the CLI cannot make a hosted agent local.
    • \n
  • Public fixtures are synthetic. Private corpora, profiles, packets, and drafts\ndo not belong in Git, issues, logs, packages, or examples.
    • \n
  • A draft is never sent.
    • \n
\n

Read SECURITY.md before integrating the library into another\ntool or handling a private packet outside the CLI. The\nmethodology defines every unit and evidence boundary;\nthe research review documents papers, neighboring OSS, and\nthe claims this project does not make.

\n

TypeScript library

\n

The package exports the versioned corpus, metrics, study-packet, and profile\ntypes plus deterministic canonical JSON and SHA-256 helpers:

\n
import type { ContactMetrics, StyleProfileV2 } from "@hraness/message-like-me"\nimport { canonicalJson, sha256 } from "@hraness/message-like-me"\n
\n

The library does not start the CLI, inspect Messages or Contacts, connect to a\nnetwork, or send a draft merely because it is imported.

\n

Development

\n
bun install --frozen-lockfile --ignore-scripts\nbun run check\n
\n

Tests use synthetic Messages and AddressBook databases plus synthetic source\nbundles and conversations. Never add a real message, handle, group title,\nattachment, contact record, private path, or derived profile to a fixture.

\n

The canonical repository is\nhraness/message-like-me.\nThe informational project page is\nmessagelikeme.com. The CLI does not connect to\nthe site, and the site never receives message or contact data.

\n

License

\n

MIT.

\n"; diff --git a/site/package.json b/site/package.json index d4e398b..6e06860 100644 --- a/site/package.json +++ b/site/package.json @@ -1,6 +1,6 @@ { "name": "message-like-me-site", - "version": "0.2.0", + "version": "0.3.0", "private": true, "engines": { "node": ">=22.13.0" diff --git a/site/public/og.png b/site/public/og.png index 40f0997..207d13e 100644 Binary files a/site/public/og.png and b/site/public/og.png differ diff --git a/skills/message-like-me/SKILL.md b/skills/message-like-me/SKILL.md index 0cabd30..dd09689 100644 --- a/skills/message-like-me/SKILL.md +++ b/skills/message-like-me/SKILL.md @@ -63,6 +63,21 @@ database with `messagelikeme ingest imessage --json`; pass `--database` only when the user names a different source. The ingest is read-only. It stores a private normalized corpus and aggregate metrics without changing `chat.db`. +When the user supplies a finished Wrench/Beeper Message Like Me bundle, ingest +only its normalized absolute directory path: + +```sh +messagelikeme ingest bundle --input --json +messagelikeme sources list --json +``` + +Do not request or handle the Beeper credential, call Beeper directly, improvise +a provider parser, or open the bundle's NDJSON files. Wrench owns provider +capture; Message Like Me owns strict verification, normalization, and local +analysis. Use `sources show --json` for redacted completeness and +health. Add `--private` only when the user's task requires provider account +metadata. + When the user wants AddressBook names attached to direct conversations, run `messagelikeme ingest contacts --json`. Pass `--addressbook` only for an explicit alternative AddressBook root, source directory, or database. The @@ -70,10 +85,13 @@ optional enrichment is also read-only, may run before or after Messages ingestion, and keeps ambiguous methods and group conversations unresolved. When Contacts supplies an unambiguous exact handle match, the CLI can combine -that person's direct Messages conversations into one pseudonymous `person_...` -analysis scope. Unmatched conversations and groups remain separate. Treat each -scope as evidence about messaging with that observed person or conversation, -not as a label for the relationship or a complete model of either participant. +that person's complete-roster direct conversations across message sources into +one pseudonymous `person_...` analysis scope. Unmatched conversations, +incomplete rosters, and groups remain separate. Treat each scope as evidence +about messaging with that observed person or conversation, not as a label for +the relationship or a complete model of either participant. Inspect the +`services` breakdown before applying a multi-app profile as though it described +one channel. Use the CLI's aggregate views before requesting message text. Ask for the narrowest bounded study packet that answers the question. Prefer stable local diff --git a/skills/message-like-me/references/analysis.md b/skills/message-like-me/references/analysis.md index 63e412b..0a09a14 100644 --- a/skills/message-like-me/references/analysis.md +++ b/skills/message-like-me/references/analysis.md @@ -12,9 +12,11 @@ segmentation parameters, and exclusions. Check whether the evidence spans enough conversations and contexts to support the requested claim. An AddressBook-matched `person_...` scope can combine several conservatively -matched direct Messages conversations with one person. An unmatched contact ID -or a group remains a conversation scope. Analyze the observed messaging scope -without inferring a relationship category, importance, or status. +matched complete-roster direct conversations with one person across message +sources. An unmatched contact ID, incomplete roster, or group remains a +conversation scope. Inspect the source and `services` breakdown before +generalizing across apps. Analyze the observed messaging scope without +inferring a relationship category, importance, or status. Start with aggregate metrics. Open bounded text samples only for questions the metrics cannot answer, such as how the user acknowledges emotion, resolves @@ -101,6 +103,10 @@ Examine: - explicit reply-link frequency and the situations where replies are used; - tapbacks as lightweight acknowledgements, separate from written replies. +Reactions with no provider timestamp remain valid count and direction evidence. +Do not place them in chronological order, a session, or a response episode, and +do not synthesize a reaction time. + Do not describe within-session response latency as an obligation, promise, availability signal, or general preference. The sample excludes incoming bursts without a later outgoing burst in the same session and may be shaped by diff --git a/skills/message-like-me/references/privacy.md b/skills/message-like-me/references/privacy.md index a003937..3f2a3f4 100644 --- a/skills/message-like-me/references/privacy.md +++ b/skills/message-like-me/references/privacy.md @@ -7,11 +7,15 @@ sensitive local data. ## Data boundary - Use the `messagelikeme` CLI for ingestion and inspection. Do not open, copy, - transform, or query the live Messages or AddressBook databases through an - improvised script. + transform, or query the live Messages or AddressBook databases or a private + message bundle through an improvised script. - Keep the original `chat.db` and AddressBook stores authoritative. Ingestion is read-only and must not change Messages, Contacts, attachments, or database sidecars. +- Treat caller-owned provider bundles as private source observations. Check + their state through `messagelikeme sources list|show`; do not parse their + manifest or NDJSON records in agent context. The bundle must not contain a + provider credential, but it still contains private message and account data. - Do not send message data to a model API, hosted service, analytics system, remote MCP server, or network endpoint. The agent already executing this skill performs the semantic work directly in its current context. diff --git a/src/args.ts b/src/args.ts index 610eb66..a68c0cf 100644 --- a/src/args.ts +++ b/src/args.ts @@ -7,6 +7,7 @@ const VALUE_OPTIONS = new Set([ "burst-gap", "data-dir", "database", + "input", "limit", "min-outgoing", "output", diff --git a/src/bundle.test.ts b/src/bundle.test.ts new file mode 100644 index 0000000..fdfcea9 --- /dev/null +++ b/src/bundle.test.ts @@ -0,0 +1,782 @@ +import { describe, expect, test } from "bun:test"; +import { chmod, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { readMessageBundle } from "./bundle.ts"; +import { canonicalJson, sha256 } from "./canonical-json.ts"; +import { CliError } from "./errors.ts"; +import { LocalStore } from "./store.ts"; +import { + syntheticBundleRecords, + writeSyntheticMessageBundle, +} from "./test-bundle-fixture.ts"; + +const TEST_KEY = "synthetic-bundle-test-key-32-bytes"; +const GOLDEN_MANIFEST_SHA256 = "dcef93293af9af0f3b0ff303992517ce2eece6d4bf0b7477e30c0b9d77a2c7f1"; +const GOLDEN_FILES = Object.freeze([ + "accounts.ndjson", + "participants.ndjson", + "conversations.ndjson", + "messages.ndjson", + "reactions.ndjson", + "tombstones.ndjson", + "manifest.json", +]); + +async function materializeWrenchGoldenBundle(parent: string): Promise { + const source = join(import.meta.dir, "fixtures", "beeper-message-like-me-v1"); + const target = join(parent, "beeper-message-like-me-v1"); + await mkdir(target, { mode: 0o700 }); + await chmod(target, 0o700); + for (const file of GOLDEN_FILES) { + const bytes = await readFile(join(source, file)); + await writeFile(join(target, file), bytes, { mode: 0o600 }); + await chmod(join(target, file), 0o600); + } + return realpath(target); +} + +async function replaceArtifactBytes( + bundlePath: string, + artifactPath: string, + bytes: Uint8Array, +): Promise { + await writeFile(join(bundlePath, artifactPath), bytes, { mode: 0o600 }); + await chmod(join(bundlePath, artifactPath), 0o600); + const manifestPath = join(bundlePath, "manifest.json"); + const manifest = JSON.parse(await readFile(manifestPath, "utf8")) as { + artifacts: Array<{ path: string; records: number; bytes: number; sha256: string }>; + integrity: { algorithm: "sha256"; bundleSha256: string }; + [key: string]: unknown; + }; + const artifact = manifest.artifacts.find(({ path }) => path === artifactPath); + if (artifact === undefined) throw new Error("Synthetic artifact is absent from its manifest"); + artifact.bytes = bytes.byteLength; + artifact.sha256 = sha256(bytes); + const { integrity: _integrity, ...projection } = manifest; + manifest.integrity = { algorithm: "sha256", bundleSha256: sha256(canonicalJson(projection)) }; + await writeFile(manifestPath, `${canonicalJson(manifest)}\n`, { mode: 0o600 }); + await chmod(manifestPath, 0o600); +} + +describe("private local message bundle", () => { + test("imports the exact canonical bundle emitted by Wrench", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-wrench-golden-")); + try { + const vendoredManifest = await readFile(join( + import.meta.dir, + "fixtures", + "beeper-message-like-me-v1", + "manifest.json", + )); + expect(sha256(vendoredManifest)).toBe(GOLDEN_MANIFEST_SHA256); + const path = await materializeWrenchGoldenBundle(root); + const bundle = await readMessageBundle(path, { hmacKey: TEST_KEY }); + expect(bundle.manifestSha256).toBe(GOLDEN_MANIFEST_SHA256); + expect(bundle.sources).toHaveLength(2); + const primary = bundle.sources.find(({ source }) => source.network === "synthetic"); + const secondary = bundle.sources.find(({ source }) => + source.network === "synthetic-secondary"); + expect(primary).toBeDefined(); + expect(secondary).toBeDefined(); + expect(primary!.source).toMatchObject({ + provider: "beeper", + network: "synthetic", + producer: { id: "beeper-local", version: "1.1.0" }, + coverage: { + kind: "bounded-local", + reason: "desktop-local-sequential-export", + }, + }); + expect(primary!.source.warnings).toContain("sequential-account-snapshot"); + expect(secondary!.source).toMatchObject({ + provider: "beeper", + network: "synthetic-secondary", + producer: { id: "beeper-local", version: "1.1.0" }, + coverage: { + kind: "bounded-local", + reason: "desktop-local-sequential-export", + observedFrom: null, + observedTo: null, + }, + }); + expect(secondary!.source.warnings).toContain("sequential-account-snapshot"); + expect(secondary!.conversations).toHaveLength(0); + expect(secondary!.messages).toHaveLength(0); + expect(primary!.messages[0]).toMatchObject({ + body: "edited synthetic reply", + editedAt: "2026-08-21T15:58:30.000Z", + retractedAt: null, + replyToSourceGuid: "beeper-message:synthetic-external-reply-target", + }); + expect(primary!.messages[1]).toMatchObject({ + body: null, + retractedAt: "2026-08-21T15:59:00.000Z", + direction: "incoming", + }); + expect(primary!.reactionFacts).toMatchObject([{ + body: "👍", + reactedAt: null, + direction: "incoming", + }]); + expect(primary!.deletions).toEqual(expect.arrayContaining([ + expect.objectContaining({ + entityKind: "message", + externalId: "beeper-message:synthetic-deleted", + deletedAt: "2026-08-21T15:59:00.000Z", + }), + ])); + expect(primary!.deletions?.some(({ externalId }) => + externalId === "beeper-message:synthetic-edited")).toBeFalse(); + + const store = LocalStore.open(join(root, "golden-store.sqlite3")); + try { + store.replaceSources(bundle.sources, "2026-08-21T16:01:00.000Z", TEST_KEY); + const storedSources = store.listSources(); + expect(storedSources).toHaveLength(2); + expect(storedSources.find(({ network }) => network === "synthetic")).toMatchObject({ + provider: "beeper", + network: "synthetic", + conversations: 1, + messages: 1, + reactions: 1, + undatedReactions: 1, + coverage: { + kind: "bounded-local", + reason: "desktop-local-sequential-export", + }, + }); + expect(storedSources.find(({ network }) => network === "synthetic-secondary")) + .toMatchObject({ + provider: "beeper", + network: "synthetic-secondary", + conversations: 0, + messages: 0, + reactions: 0, + undatedReactions: 0, + coverage: { + kind: "bounded-local", + reason: "desktop-local-sequential-export", + observedFrom: null, + observedTo: null, + }, + }); + } finally { + store.close(); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("verifies and normalizes Wrench's six-artifact contract", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-")); + try { + const path = await writeSyntheticMessageBundle(root); + const bundle = await readMessageBundle(path, { hmacKey: TEST_KEY }); + expect(bundle.schemaVersion).toBe(1); + expect(bundle.manifestSha256).toMatch(/^[a-f0-9]{64}$/u); + expect(bundle.sources).toHaveLength(1); + const source = bundle.sources[0]!; + expect(source.source).toMatchObject({ + kind: "bundle", + provider: "beeper", + network: "whatsapp", + coverage: { + history: "bounded", + kind: "truncated", + reason: "synthetic-limit", + }, + }); + expect(source.source.id).toMatch(/^source_[a-f0-9]{64}$/u); + expect(source.conversations[0]).toMatchObject({ + group: false, + privateParticipants: ["peer@example.test"], + }); + const outgoing = source.messages.find(({ sourceGuid }) => + sourceGuid === "message-provider-outgoing")!; + expect(outgoing).toMatchObject({ + kind: "text", + body: "Synthetic answer.", + replyToSourceGuid: "message-provider-incoming", + attachmentCount: 1, + }); + expect(source.messageProvenance.find(({ messageId }) => messageId === outgoing.id)) + .toMatchObject({ + replyToExternalId: "message-provider-incoming", + attachments: [{ + kind: "image", + mimeType: "image/png", + fileName: "synthetic.png", + bytes: 1234, + }], + }); + expect(source.messages.find(({ sourceGuid }) => sourceGuid === "message-provider-truncated")) + .toMatchObject({ kind: "text", body: null, bodySource: "unavailable" }); + expect(source.messages.find(({ sourceGuid }) => sourceGuid === "message-provider-deleted")) + .toMatchObject({ body: null, retractedAt: "2026-08-20T12:04:00.000Z" }); + expect(source.deletions).toContainEqual(expect.objectContaining({ + entityKind: "message", + externalId: "message-provider-deleted", + deletedAt: "2026-08-20T12:04:00.000Z", + })); + expect(source.messages.filter(({ kind }) => kind === "reaction")).toHaveLength(1); + expect(source.reactionFacts).toMatchObject([ + { body: "heart", reactedAt: "2026-08-20T12:01:30.000Z", direction: "incoming" }, + { body: "thumbs-up", reactedAt: null, direction: "outgoing" }, + ]); + expect(source.auxiliaryRecords?.filter(({ kind }) => kind === "reaction")).toHaveLength(2); + expect(source.source.warnings).toContain("undated-reactions:1"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("uses fatal UTF-8 decoding before canonical byte comparison", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-utf8-")); + try { + const path = await writeSyntheticMessageBundle(root); + const accountsPath = join(path, "accounts.ndjson"); + const bytes = Buffer.from(await readFile(accountsPath)); + bytes[10] = 0xff; + await writeFile(accountsPath, bytes, { mode: 0o600 }); + await expect(readMessageBundle(path, { hmacKey: TEST_KEY })).rejects.toThrow("valid UTF-8"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("requires every nullable provider revision to be an identifier", async () => { + const locations: Array, + value: string, + ) => void; + }>> = [ + { + mutate: (records, value) => { + const provenance = records.account[0]!.provenance as Record; + provenance.providerRevision = value; + }, + }, + { + mutate: (records, value) => { + const deletion = records.message[3]!.deletion as Record; + deletion.providerRevision = value; + }, + }, + { + mutate: (records, value) => { + records.tombstone.push({ + schemaVersion: 1, + kind: "tombstone", + id: "tombstone-local", + accountId: "account-local", + network: "whatsapp", + provenance: { + providerId: "tombstone-provider", + providerRevision: null, + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account", + }, + entityKind: "message", + entityId: "message-incoming", + entityProviderId: "message-provider-incoming", + deletedAt: "2026-08-20T12:05:00.000Z", + scope: "remote", + providerRevision: value, + }); + }, + }, + ]; + const invalidValues = ["", "\t", "\n"]; + for (const [locationIndex, location] of locations.entries()) { + for (const [valueIndex, value] of invalidValues.entries()) { + const root = await mkdtemp(join( + tmpdir(), + `message-like-me-bundle-revision-${locationIndex}-${valueIndex}-`, + )); + try { + const records = syntheticBundleRecords(); + location.mutate(records, value); + const path = await writeSyntheticMessageBundle(root, records); + await expect(readMessageBundle(path, { hmacKey: TEST_KEY })) + .rejects.toThrow("providerRevision"); + } finally { + await rm(root, { recursive: true, force: true }); + } + } + } + }); + + test("enforces the exact UTF-8 byte bound for attachment MIME types", async () => { + const cases = [ + { value: "m".repeat(256), bytes: 256, accepted: true }, + { value: "m".repeat(257), bytes: 257, accepted: false }, + { value: "é".repeat(128), bytes: 256, accepted: true }, + { value: "é".repeat(129), bytes: 258, accepted: false }, + ] as const; + for (const [index, candidate] of cases.entries()) { + const root = await mkdtemp(join(tmpdir(), `message-like-me-bundle-mime-${index}-`)); + try { + expect(Buffer.byteLength(candidate.value, "utf8")).toBe(candidate.bytes); + const records = syntheticBundleRecords(); + const attachments = records.message[1]!.attachments as Array>; + attachments[0]!.mimeType = candidate.value; + const path = await writeSyntheticMessageBundle(root, records); + if (candidate.accepted) { + const bundle = await readMessageBundle(path, { hmacKey: TEST_KEY }); + expect(bundle.sources).toHaveLength(1); + } else { + await expect(readMessageBundle(path, { hmacKey: TEST_KEY })) + .rejects.toThrow("mimeType"); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + } + }); + + test("rejects deeply nested foreign fields before recursive canonicalization", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-depth-")); + const deepValue = `${"[".repeat(20_000)}0${"]".repeat(20_000)}`; + try { + const artifactPath = await writeSyntheticMessageBundle(root, syntheticBundleRecords(), { + directoryName: "deep-artifact", + }); + const messagesPath = join(artifactPath, "messages.ndjson"); + const original = (await readFile(messagesPath, "utf8")).trimEnd(); + const bytes = Buffer.from(`${original.slice(0, -1)},"unexpected":${deepValue}}\n`, "utf8"); + await replaceArtifactBytes(artifactPath, "messages.ndjson", bytes); + try { + await readMessageBundle(artifactPath, { hmacKey: TEST_KEY }); + throw new Error("Expected deep artifact rejection"); + } catch (error) { + expect(error).toBeInstanceOf(CliError); + expect(error).not.toBeInstanceOf(RangeError); + } + + const manifestBundle = await writeSyntheticMessageBundle(root, syntheticBundleRecords(), { + directoryName: "deep-manifest", + }); + const manifestPath = join(manifestBundle, "manifest.json"); + const originalManifest = (await readFile(manifestPath, "utf8")).trimEnd(); + await writeFile( + manifestPath, + `${originalManifest.slice(0, -1)},"unexpected":${deepValue}}\n`, + { mode: 0o600 }, + ); + await chmod(manifestPath, 0o600); + try { + await readMessageBundle(manifestBundle, { hmacKey: TEST_KEY }); + throw new Error("Expected deep manifest rejection"); + } catch (error) { + expect(error).toBeInstanceOf(CliError); + expect(error).not.toBeInstanceOf(RangeError); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("rejects incomplete identity joins before creating local coordinates", async () => { + const cases: Array) => void; + error: string; + }>> = [ + { + name: "unknown roster participant", + mutate: (records) => { + records.conversation[0]!.participantIds = ["participant-self", "participant-missing"]; + }, + error: "unknown participant", + }, + { + name: "complete direct roster without self", + mutate: (records) => { + records.conversation[0]!.participantIds = ["participant-peer"]; + }, + error: "one self and one non-self participant", + }, + { + name: "complete direct roster with only self", + mutate: (records) => { + records.conversation[0]!.participantIds = ["participant-self"]; + }, + error: "one self and one non-self participant", + }, + { + name: "sender direction mismatch", + mutate: (records) => { + records.message[0]!.direction = "outgoing"; + }, + error: "direction conflicts", + }, + { + name: "duplicate provider coordinate", + mutate: (records) => { + records.participant.push({ + ...records.participant[1]!, + id: "participant-duplicate-local", + }); + }, + error: "repeat a provider identity", + }, + { + name: "reply coordinate mismatch", + mutate: (records) => { + records.message[1]!.replyTo = { + messageId: "message-incoming", + providerId: "different-provider-message", + }; + }, + error: "reply has mismatched target coordinates", + }, + { + name: "reaction coordinate mismatch", + mutate: (records) => { + records.reaction[0]!.messageProviderId = "different-provider-message"; + }, + error: "reaction has mismatched target coordinates", + }, + ]; + for (const [index, candidate] of cases.entries()) { + const root = await mkdtemp(join(tmpdir(), `message-like-me-bundle-join-${index}-`)); + try { + const records = syntheticBundleRecords(); + candidate.mutate(records); + const path = await writeSyntheticMessageBundle(root, records); + await expect(readMessageBundle(path, { hmacKey: TEST_KEY })) + .rejects.toThrow(candidate.error); + } finally { + await rm(root, { recursive: true, force: true }); + } + } + }); + + test("rejects a tombstone whose local and provider coordinates disagree", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-tombstone-")); + try { + const records = syntheticBundleRecords(); + records.tombstone.push({ + schemaVersion: 1, + kind: "tombstone", + id: "tombstone-local", + accountId: "account-local", + network: "whatsapp", + provenance: { + providerId: "tombstone-provider", + providerRevision: null, + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account", + }, + entityKind: "message", + entityId: "message-incoming", + entityProviderId: "different-message-provider", + deletedAt: "2026-08-20T12:05:00.000Z", + scope: "remote", + providerRevision: null, + }); + const path = await writeSyntheticMessageBundle(root, records); + await expect(readMessageBundle(path, { hmacKey: TEST_KEY })) + .rejects.toThrow("mismatched message identity"); + const missing = syntheticBundleRecords(); + missing.tombstone.push({ + ...records.tombstone[0]!, + id: "tombstone-missing-local", + provenance: { + providerId: "tombstone-provider-missing", + providerRevision: null, + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account", + }, + entityId: "private-missing-local-id", + entityProviderId: "private-missing-provider-id", + }); + const missingPath = await writeSyntheticMessageBundle(root, missing, { + directoryName: "missing-tombstone-target", + }); + try { + await readMessageBundle(missingPath, { hmacKey: TEST_KEY }); + throw new Error("Expected unresolved tombstone rejection"); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + expect(message).toContain("unknown local message"); + expect(message).not.toContain("private-missing-local-id"); + expect(message).not.toContain("private-missing-provider-id"); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("turns terminal edits and removed reactions into explicit suppressions", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-terminal-")); + try { + const records = syntheticBundleRecords(); + records.message.push({ + ...records.message[1]!, + id: "message-edit-local", + provenance: { + providerId: "message-provider-edit", + providerRevision: "revision-edit", + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account", + }, + sentAt: "2026-08-20T12:01:00.000Z", + sortKey: "0002-edit", + body: "Synthetic edited answer.", + replyTo: null, + edit: { + kind: "replacement", + replacesMessageId: "message-outgoing", + replacesProviderId: "message-provider-outgoing", + editedAt: "2026-08-20T12:02:00.000Z", + providerRevision: "revision-edit", + }, + attachments: [], + }); + records.reaction.push({ + ...records.reaction[0]!, + id: "reaction-removed", + provenance: { + providerId: "reaction-provider-removed", + providerRevision: "reaction-revision", + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account", + }, + reactedAt: null, + state: "removed", + }); + const path = await writeSyntheticMessageBundle(root, records); + const source = (await readMessageBundle(path, { hmacKey: TEST_KEY })).sources[0]!; + expect(source.deletions).toEqual(expect.arrayContaining([ + expect.objectContaining({ + entityKind: "message", + externalId: "message-provider-outgoing", + deletedAt: "2026-08-20T12:02:00.000Z", + }), + expect.objectContaining({ + entityKind: "reaction", + externalId: "reaction-provider-removed", + deletedAt: "2026-08-20T12:05:00.000Z", + }), + ])); + const inPlaceRecords = syntheticBundleRecords(); + inPlaceRecords.message[1]!.edit = { + kind: "in-place", + editedAt: "2026-08-20T12:02:00.000Z", + providerRevision: "revision-in-place", + }; + const inPlacePath = await writeSyntheticMessageBundle(root, inPlaceRecords, { + directoryName: "in-place-edit-bundle", + }); + const inPlace = (await readMessageBundle(inPlacePath, { hmacKey: TEST_KEY })).sources[0]!; + expect(inPlace.messages.find(({ sourceGuid }) => sourceGuid === "message-provider-outgoing")) + .toMatchObject({ editedAt: "2026-08-20T12:02:00.000Z" }); + expect(inPlace.deletions?.some(({ externalId }) => + externalId === "message-provider-outgoing")).toBeFalse(); + + const cyclic = syntheticBundleRecords(); + cyclic.message[0]!.edit = { + kind: "replacement", + replacesMessageId: "message-outgoing", + replacesProviderId: "message-provider-outgoing", + editedAt: "2026-08-20T12:02:00.000Z", + providerRevision: "cycle-a", + }; + cyclic.message[1]!.edit = { + kind: "replacement", + replacesMessageId: "message-incoming", + replacesProviderId: "message-provider-incoming", + editedAt: "2026-08-20T12:02:00.000Z", + providerRevision: "cycle-b", + }; + const cyclicPath = await writeSyntheticMessageBundle(root, cyclic, { + directoryName: "cyclic-edit-bundle", + }); + await expect(readMessageBundle(cyclicPath, { hmacKey: TEST_KEY })) + .rejects.toThrow("contain a cycle"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("validates the private HMAC key before touching bundle input", async () => { + await expect(readMessageBundle("/synthetic/does-not-exist", { hmacKey: "short" })) + .rejects.toThrow("HMAC key"); + }); + + test("orders opaque provider sort keys by code units", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-order-")); + try { + const records = syntheticBundleRecords(); + records.message[0]!.sortKey = "Z"; + records.message[1]!.sortKey = "a"; + records.message[2]!.sortKey = "b"; + records.message[3]!.sortKey = "c"; + records.message.reverse(); + const path = await writeSyntheticMessageBundle(root, records); + const source = (await readMessageBundle(path, { hmacKey: TEST_KEY })).sources[0]!; + expect(source.messages.filter(({ kind }) => kind !== "reaction").map(({ sourceGuid }) => sourceGuid)) + .toEqual([ + "message-provider-incoming", + "message-provider-outgoing", + "message-provider-truncated", + "message-provider-deleted", + ]); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("keeps one stable source and entity namespace when a network label changes", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-network-")); + try { + const firstPath = await writeSyntheticMessageBundle(root, syntheticBundleRecords(), { + directoryName: "first-network", + }); + const changedRecords = syntheticBundleRecords(); + for (const values of Object.values(changedRecords)) { + for (const record of values) record.network = "whatsapp-business"; + } + changedRecords.participant = [changedRecords.participant[0]!]; + changedRecords.conversation = []; + changedRecords.message = []; + changedRecords.reaction = []; + changedRecords.tombstone = []; + const changedPath = await writeSyntheticMessageBundle(root, changedRecords, { + directoryName: "renamed-network", + createdAt: "2026-08-20T12:06:00.000Z", + }); + const first = (await readMessageBundle(firstPath, { hmacKey: TEST_KEY })).sources[0]!; + const changed = (await readMessageBundle(changedPath, { hmacKey: TEST_KEY })).sources[0]!; + expect(changed.source.id).toBe(first.source.id); + expect(changed.conversations).toEqual([]); + expect(changed.messages).toEqual([]); + + const store = LocalStore.open(join(root, "network-store.sqlite3")); + try { + store.replaceSources([first], "2026-08-20T12:05:30.000Z", TEST_KEY); + store.replaceSources([changed], "2026-08-20T12:06:30.000Z", TEST_KEY); + expect(store.listSources()).toMatchObject([{ + id: first.source.id, + network: "whatsapp-business", + conversations: 1, + messages: 4, + }]); + const contactId = first.conversations[0]!.id; + expect(store.conversation(contactId, true)).toMatchObject({ + service: "whatsapp-business", + services: ["whatsapp-business"], + }); + const retainedMessages = store.contactCorpus(contactId)?.messages; + expect(retainedMessages).toHaveLength(4); + expect(retainedMessages?.every(({ service }) => service === "whatsapp-business")) + .toBeTrue(); + } finally { + store.close(); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("derives completeness time bounds independently for each account", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-account-bounds-")); + try { + const records = syntheticBundleRecords(); + records.account.push({ + schemaVersion: 1, + kind: "account", + id: "account-empty", + accountId: "account-empty", + network: "signal", + provenance: { + providerId: "synthetic-connected-account-empty", + providerRevision: null, + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account-empty", + }, + displayName: "Synthetic Empty Account", + handle: null, + selfParticipantId: "participant-empty-self", + }); + records.participant.push({ + schemaVersion: 1, + kind: "participant", + id: "participant-empty-self", + accountId: "account-empty", + network: "signal", + provenance: { + providerId: "participant-provider-empty-self", + providerRevision: null, + observedAt: "2026-08-20T12:05:00.000Z", + connectedAccountProviderId: "synthetic-connected-account-empty", + }, + displayName: "Synthetic Empty Self", + handle: null, + isSelf: true, + }); + const path = await writeSyntheticMessageBundle(root, records); + const bundle = await readMessageBundle(path, { hmacKey: TEST_KEY }); + expect(bundle.sources).toHaveLength(2); + expect(bundle.sources.find(({ source }) => source.network === "whatsapp")?.source.coverage) + .toMatchObject({ + observedFrom: "2026-08-20T12:00:00.000Z", + observedTo: "2026-08-20T12:03:00.000Z", + }); + expect(bundle.sources.find(({ source }) => source.network === "signal")?.source.coverage) + .toMatchObject({ observedFrom: null, observedTo: null }); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + test("domain-separates a reaction timeline coordinate from an equal message provider ID", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-bundle-cross-kind-id-")); + try { + const records = syntheticBundleRecords(); + const sharedProviderId = records.message[0]!.provenance as Record; + (records.reaction[0]!.provenance as Record).providerId = sharedProviderId.providerId; + const path = await writeSyntheticMessageBundle(root, records); + const source = (await readMessageBundle(path, { hmacKey: TEST_KEY })).sources[0]!; + const rawProviderId = String(sharedProviderId.providerId); + const message = source.messages.find(({ kind, sourceGuid }) => + kind !== "reaction" && sourceGuid === rawProviderId); + const timelineReaction = source.messages.find(({ kind }) => kind === "reaction"); + expect(message).toBeDefined(); + expect(timelineReaction).toBeDefined(); + expect(timelineReaction!.sourceGuid).not.toBe(rawProviderId); + expect(timelineReaction!.sourceGuid.startsWith("\u001freaction-timeline:")).toBeTrue(); + expect(source.reactionFacts?.find(({ id }) => id === timelineReaction!.id)?.externalId) + .toBe(rawProviderId); + expect(source.messageProvenance.find(({ messageId }) => messageId === timelineReaction!.id)) + .toMatchObject({ + externalId: timelineReaction!.sourceGuid, + metadata: { provenance: { providerId: rawProviderId } }, + }); + + const store = LocalStore.open(join(root, "cross-kind-store.sqlite3")); + try { + expect(() => store.replaceSources( + [source], + "2026-08-20T12:06:00.000Z", + TEST_KEY, + )).not.toThrow(); + expect(store.listSources()).toMatchObject([{ + conversations: 1, + messages: 4, + reactions: 2, + undatedReactions: 1, + }]); + } finally { + store.close(); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); diff --git a/src/bundle.ts b/src/bundle.ts new file mode 100644 index 0000000..83a4058 --- /dev/null +++ b/src/bundle.ts @@ -0,0 +1,1448 @@ +import { createHash, createHmac } from "node:crypto"; +import { constants as fsConstants, createReadStream, type BigIntStats } from "node:fs"; +import { lstat, open, readdir, realpath, type FileHandle } from "node:fs/promises"; +import { isAbsolute, join, resolve } from "node:path"; + +import { canonicalJson, sha256 } from "./canonical-json.ts"; +import { normalizeContactHandle } from "./contacts.ts"; +import { CliError } from "./errors.ts"; +import { + MESSAGE_BUNDLE_SCHEMA_VERSION, + type CorpusAttachmentProvenance, + type CorpusConversation, + type CorpusMessage, + type CorpusMessageProvenance, + type CorpusReactionFact, + type CorpusSourceDeletion, + type CorpusSourceRecord, + type MessageBundleSnapshot, + type SourceCorpusSnapshot, +} from "./types.ts"; + +type JsonObject = Record; +type RecordKind = "account" | "participant" | "conversation" | "message" | "reaction" | "tombstone"; + +const MAX_MANIFEST_BYTES = 1024 * 1024; +const MAX_RECORDS = 500_000; +const MAX_RECORD_BYTES = 2 * 1024 * 1024; +const MAX_TOTAL_BYTES = 512 * 1024 * 1024; +const MAX_ACCOUNTS = 128; +const MAX_IDENTIFIER_BYTES = 1_024; +const MAX_SHORT_TEXT_BYTES = 8 * 1024; +const MAX_BODY_BYTES = 1024 * 1024; +const MAX_PARTICIPANTS = 10_000; +const MAX_ATTACHMENTS = 256; +const MAX_WARNINGS = 128; + +const ARTIFACTS = Object.freeze([ + Object.freeze({ path: "accounts.ndjson", kind: "account" as const }), + Object.freeze({ path: "participants.ndjson", kind: "participant" as const }), + Object.freeze({ path: "conversations.ndjson", kind: "conversation" as const }), + Object.freeze({ path: "messages.ndjson", kind: "message" as const }), + Object.freeze({ path: "reactions.ndjson", kind: "reaction" as const }), + Object.freeze({ path: "tombstones.ndjson", kind: "tombstone" as const }), +]); + +type Provenance = Readonly<{ + providerId: string; + providerRevision: string | null; + observedAt: string; + connectedAccountProviderId: string; +}>; + +type CommonRecord = Readonly<{ + schemaVersion: 1; + kind: RecordKind; + id: string; + accountId: string; + network: string; + provenance: Provenance; +}>; + +type AccountRecord = CommonRecord & Readonly<{ + kind: "account"; + displayName: string | null; + handle: string | null; + selfParticipantId: string; +}>; + +type ParticipantRecord = CommonRecord & Readonly<{ + kind: "participant"; + displayName: string | null; + handle: string | null; + isSelf: boolean; +}>; + +type ConversationRecord = CommonRecord & Readonly<{ + kind: "conversation"; + type: "direct" | "group" | "channel" | "unknown"; + title: string | null; + participantIds: readonly string[]; + participantsComplete: boolean | null; + startedAt: string | null; + lastMessageAt: string | null; +}>; + +type AttachmentRecord = Readonly<{ + kind: "audio" | "document" | "image" | "link" | "sticker" | "video" | "unknown"; + mimeType: string | null; + name: string | null; + sizeBytes: number | null; +}>; + +type MessageRecord = CommonRecord & Readonly<{ + kind: "message"; + conversationId: string; + senderParticipantId: string | null; + direction: "incoming" | "outgoing" | "unknown"; + sentAt: string; + sortKey: string; + body: string | null; + bodyTruncated: boolean | null; + replyTo: Readonly<{ messageId: string | null; providerId: string }> | null; + edit: Readonly<{ + kind: "in-place"; + editedAt: string; + providerRevision: string; + }> | Readonly<{ + kind: "replacement"; + replacesMessageId: string | null; + replacesProviderId: string; + editedAt: string; + providerRevision: string; + }> | null; + deletion: Readonly<{ + state: "revoked" | "deleted-for-me" | "revoked-and-deleted-for-me"; + observedAt: string; + providerRevision: string | null; + }> | null; + attachments: readonly AttachmentRecord[]; +}>; + +type ReactionRecord = CommonRecord & Readonly<{ + kind: "reaction"; + messageId: string | null; + messageProviderId: string; + participantId: string | null; + body: string; + reactedAt: string | null; + state: "active" | "removed"; +}>; + +type TombstoneRecord = CommonRecord & Readonly<{ + kind: "tombstone"; + entityKind: "conversation" | "message" | "reaction"; + entityId: string | null; + entityProviderId: string; + deletedAt: string; + scope: "remote" | "local" | "unknown"; + providerRevision: string | null; +}>; + +type BundleRecord = AccountRecord | ParticipantRecord | ConversationRecord | MessageRecord | ReactionRecord | TombstoneRecord; + +type Artifact = Readonly<{ + path: string; + mediaType: "application/x-ndjson"; + recordKind: RecordKind; + records: number; + bytes: number; + sha256: string; +}>; + +type Manifest = Readonly<{ + schemaVersion: 1; + format: "message-like-me.local-message-bundle"; + source: Readonly<{ id: "beeper-local"; version: string }>; + provider: Readonly<{ id: "beeper"; version: string }>; + timestamps: Readonly<{ startedAt: string; finishedAt: string; createdAt: string }>; + completeness: Readonly<{ + kind: "bounded-local" | "truncated" | "unknown"; + reason: string | null; + observedFrom: string | null; + observedThrough: string | null; + }>; + warnings: readonly string[]; + privacy: Readonly<{ + classification: "private-local"; + attachments: "metadata-only"; + providerUrls: "excluded"; + credentials: "excluded"; + }>; + counts: Readonly>; + artifacts: readonly Artifact[]; + integrity: Readonly<{ algorithm: "sha256"; bundleSha256: string }>; +}>; + +function object(value: unknown, label: string): JsonObject { + if ( + value === null + || typeof value !== "object" + || Array.isArray(value) + || (Object.getPrototypeOf(value) !== Object.prototype && Object.getPrototypeOf(value) !== null) + ) throw new CliError("invalid-data", `${label} must be a plain object`); + return value as JsonObject; +} + +function exactKeys(value: JsonObject, keys: readonly string[], label: string): void { + const expected = [...keys].sort(); + const observed = Object.keys(value).sort(); + if ( + expected.length !== observed.length + || observed.some((key, index) => key !== expected[index]) + ) throw new CliError("invalid-data", `${label} must contain exactly: ${keys.join(", ")}`); +} + +function boundedText(value: unknown, label: string, maximum: number): string { + if (typeof value !== "string" || Buffer.byteLength(value, "utf8") > maximum || value.includes("\u0000")) { + throw new CliError("invalid-data", `${label} must be NUL-free text within ${maximum} UTF-8 bytes`); + } + return value; +} + +function nullableText(value: unknown, label: string, maximum: number): string | null { + return value === null ? null : boundedText(value, label, maximum); +} + +function identifier(value: unknown, label: string): string { + const result = boundedText(value, label, MAX_IDENTIFIER_BYTES); + if (result.length === 0 || /[\u0000-\u001f\u007f]/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a non-empty identifier without ASCII controls`); + } + return result; +} + +function nullableIdentifier(value: unknown, label: string): string | null { + return value === null ? null : identifier(value, label); +} + +function token(value: unknown, label: string, maximum = 128): string { + const result = boundedText(value, label, maximum); + if (!/^[a-z0-9](?:[a-z0-9._+-]*[a-z0-9])?$/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a lowercase categorical token`); + } + return result; +} + +function version(value: unknown, label: string): string { + const result = boundedText(value, label, 128); + if (!/^[A-Za-z0-9](?:[A-Za-z0-9._+-]*[A-Za-z0-9])?$/u.test(result)) { + throw new CliError("invalid-data", `${label} must be a bounded version token`); + } + return result; +} + +function oneOf(value: unknown, values: T, label: string): T[number] { + if (typeof value !== "string" || !values.includes(value)) { + throw new CliError("invalid-data", `${label} must be one of: ${values.join(", ")}`); + } + return value as T[number]; +} + +function integer(value: unknown, label: string, maximum = Number.MAX_SAFE_INTEGER): number { + if (!Number.isSafeInteger(value) || (value as number) < 0 || (value as number) > maximum) { + throw new CliError("invalid-data", `${label} must be a non-negative safe integer`); + } + return value as number; +} + +function nullableInteger(value: unknown, label: string): number | null { + return value === null ? null : integer(value, label); +} + +function boolean(value: unknown, label: string): boolean { + if (typeof value !== "boolean") throw new CliError("invalid-data", `${label} must be boolean`); + return value; +} + +function nullableBoolean(value: unknown, label: string): boolean | null { + return value === null ? null : boolean(value, label); +} + +function timestamp(value: unknown, label: string): string { + const result = boundedText(value, label, 64); + const date = new Date(result); + if (!Number.isFinite(date.getTime()) || date.toISOString() !== result) { + throw new CliError("invalid-data", `${label} must be a canonical UTC timestamp`); + } + return result; +} + +function nullableTimestamp(value: unknown, label: string): string | null { + return value === null ? null : timestamp(value, label); +} + +function digest(value: unknown, label: string): string { + const result = boundedText(value, label, 64); + if (!/^[a-f0-9]{64}$/u.test(result)) throw new CliError("invalid-data", `${label} must be lowercase SHA-256`); + return result; +} + +function array(value: unknown, label: string, maximum: number): readonly unknown[] { + if (!Array.isArray(value) || value.length > maximum) { + throw new CliError("invalid-data", `${label} must contain at most ${maximum} items`); + } + return value; +} + +function identifiers(value: unknown, label: string, maximum: number): readonly string[] { + const result = array(value, label, maximum).map((item, index) => identifier(item, `${label}[${index}]`)); + if (new Set(result).size !== result.length) throw new CliError("invalid-data", `${label} repeats an ID`); + return Object.freeze(result); +} + +function parseProvenance(value: unknown, label: string): Provenance { + const record = object(value, label); + exactKeys(record, ["providerId", "providerRevision", "observedAt", "connectedAccountProviderId"], label); + return Object.freeze({ + providerId: identifier(record.providerId, `${label}.providerId`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`), + observedAt: timestamp(record.observedAt, `${label}.observedAt`), + connectedAccountProviderId: identifier( + record.connectedAccountProviderId, + `${label}.connectedAccountProviderId`, + ), + }); +} + +function parseCommon( + record: JsonObject, + kind: RecordKind, + extraKeys: readonly string[], + label: string, +): Omit & Readonly<{ kind: RecordKind }> { + exactKeys(record, ["schemaVersion", "kind", "id", "accountId", "network", "provenance", ...extraKeys], label); + if (record.schemaVersion !== 1 || record.kind !== kind) { + throw new CliError("invalid-data", `${label} has the wrong schemaVersion or kind`); + } + return Object.freeze({ + schemaVersion: 1, + kind, + id: identifier(record.id, `${label}.id`), + accountId: identifier(record.accountId, `${label}.accountId`), + network: token(record.network, `${label}.network`, 64), + provenance: parseProvenance(record.provenance, `${label}.provenance`), + }); +} + +function parseAccount(record: JsonObject, label: string): AccountRecord { + const common = parseCommon(record, "account", ["displayName", "handle", "selfParticipantId"], label); + if (common.id !== common.accountId || common.provenance.providerId !== common.provenance.connectedAccountProviderId) { + throw new CliError("invalid-data", `${label} does not establish one connected account realm`); + } + return Object.freeze({ + ...common, + kind: "account", + displayName: nullableText(record.displayName, `${label}.displayName`, MAX_SHORT_TEXT_BYTES), + handle: nullableText(record.handle, `${label}.handle`, MAX_SHORT_TEXT_BYTES), + selfParticipantId: identifier(record.selfParticipantId, `${label}.selfParticipantId`), + }); +} + +function parseParticipant(record: JsonObject, label: string): ParticipantRecord { + const common = parseCommon(record, "participant", ["displayName", "handle", "isSelf"], label); + return Object.freeze({ + ...common, + kind: "participant", + displayName: nullableText(record.displayName, `${label}.displayName`, MAX_SHORT_TEXT_BYTES), + handle: nullableText(record.handle, `${label}.handle`, MAX_SHORT_TEXT_BYTES), + isSelf: boolean(record.isSelf, `${label}.isSelf`), + }); +} + +function parseConversation(record: JsonObject, label: string): ConversationRecord { + const common = parseCommon(record, "conversation", [ + "type", "title", "participantIds", "participantsComplete", "startedAt", "lastMessageAt", + ], label); + const startedAt = nullableTimestamp(record.startedAt, `${label}.startedAt`); + const lastMessageAt = nullableTimestamp(record.lastMessageAt, `${label}.lastMessageAt`); + if (startedAt !== null && lastMessageAt !== null && startedAt > lastMessageAt) { + throw new CliError("invalid-data", `${label}.startedAt must not follow lastMessageAt`); + } + return Object.freeze({ + ...common, + kind: "conversation", + type: oneOf(record.type, ["direct", "group", "channel", "unknown"] as const, `${label}.type`), + title: nullableText(record.title, `${label}.title`, MAX_SHORT_TEXT_BYTES), + participantIds: identifiers(record.participantIds, `${label}.participantIds`, MAX_PARTICIPANTS), + participantsComplete: nullableBoolean(record.participantsComplete, `${label}.participantsComplete`), + startedAt, + lastMessageAt, + }); +} + +function parseReply(value: unknown, label: string): MessageRecord["replyTo"] { + if (value === null) return null; + const record = object(value, label); + exactKeys(record, ["messageId", "providerId"], label); + return Object.freeze({ + messageId: record.messageId === null ? null : identifier(record.messageId, `${label}.messageId`), + providerId: identifier(record.providerId, `${label}.providerId`), + }); +} + +function parseEdit(value: unknown, sentAt: string, label: string): MessageRecord["edit"] { + if (value === null) return null; + const record = object(value, label); + if (record.kind === "in-place") { + exactKeys(record, ["kind", "editedAt", "providerRevision"], label); + const editedAt = timestamp(record.editedAt, `${label}.editedAt`); + if (editedAt < sentAt) throw new CliError("invalid-data", `${label} precedes the message`); + return Object.freeze({ + kind: "in-place", + editedAt, + providerRevision: identifier(record.providerRevision, `${label}.providerRevision`), + }); + } + if (record.kind !== "replacement") { + throw new CliError("invalid-data", `${label}.kind must be in-place or replacement`); + } + exactKeys(record, [ + "kind", "replacesMessageId", "replacesProviderId", "editedAt", "providerRevision", + ], label); + const replacesMessageId = record.replacesMessageId === null + ? null + : identifier(record.replacesMessageId, `${label}.replacesMessageId`); + const replacesProviderId = identifier(record.replacesProviderId, `${label}.replacesProviderId`); + const editedAt = timestamp(record.editedAt, `${label}.editedAt`); + if (editedAt < sentAt) throw new CliError("invalid-data", `${label} precedes the message`); + return Object.freeze({ + kind: "replacement", + replacesMessageId, + replacesProviderId, + editedAt, + providerRevision: identifier(record.providerRevision, `${label}.providerRevision`), + }); +} + +function parseDeletion(value: unknown, label: string): MessageRecord["deletion"] { + if (value === null) return null; + const record = object(value, label); + exactKeys(record, ["state", "observedAt", "providerRevision"], label); + return Object.freeze({ + state: oneOf(record.state, ["revoked", "deleted-for-me", "revoked-and-deleted-for-me"] as const, `${label}.state`), + observedAt: timestamp(record.observedAt, `${label}.observedAt`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`), + }); +} + +function parseAttachments(value: unknown, label: string): readonly AttachmentRecord[] { + return Object.freeze(array(value, label, MAX_ATTACHMENTS).map((item, index) => { + const itemLabel = `${label}[${index}]`; + const record = object(item, itemLabel); + exactKeys(record, ["kind", "mimeType", "name", "sizeBytes"], itemLabel); + const name = nullableText(record.name, `${itemLabel}.name`, MAX_SHORT_TEXT_BYTES); + if (name !== null && (name === "." || name === ".." || name.includes("/") || name.includes("\\"))) { + throw new CliError("invalid-data", `${itemLabel}.name must not be a path`); + } + return Object.freeze({ + kind: oneOf(record.kind, ["audio", "document", "image", "link", "sticker", "video", "unknown"] as const, `${itemLabel}.kind`), + mimeType: nullableText(record.mimeType, `${itemLabel}.mimeType`, 256), + name, + sizeBytes: nullableInteger(record.sizeBytes, `${itemLabel}.sizeBytes`), + }); + })); +} + +function parseMessage(record: JsonObject, label: string): MessageRecord { + const common = parseCommon(record, "message", [ + "conversationId", "senderParticipantId", "direction", "sentAt", "sortKey", "body", + "bodyTruncated", "replyTo", "edit", "deletion", "attachments", + ], label); + const sentAt = timestamp(record.sentAt, `${label}.sentAt`); + const deletion = parseDeletion(record.deletion, `${label}.deletion`); + const body = nullableText(record.body, `${label}.body`, MAX_BODY_BYTES); + if (deletion !== null && body !== null) { + throw new CliError("invalid-data", `${label}.body must be null for a deleted message`); + } + return Object.freeze({ + ...common, + kind: "message", + conversationId: identifier(record.conversationId, `${label}.conversationId`), + senderParticipantId: record.senderParticipantId === null + ? null + : identifier(record.senderParticipantId, `${label}.senderParticipantId`), + direction: oneOf(record.direction, ["incoming", "outgoing", "unknown"] as const, `${label}.direction`), + sentAt, + sortKey: identifier(record.sortKey, `${label}.sortKey`), + body, + bodyTruncated: nullableBoolean(record.bodyTruncated, `${label}.bodyTruncated`), + replyTo: parseReply(record.replyTo, `${label}.replyTo`), + edit: parseEdit(record.edit, sentAt, `${label}.edit`), + deletion, + attachments: parseAttachments(record.attachments, `${label}.attachments`), + }); +} + +function parseReaction(record: JsonObject, label: string): ReactionRecord { + const common = parseCommon(record, "reaction", [ + "messageId", "messageProviderId", "participantId", "body", "reactedAt", "state", + ], label); + return Object.freeze({ + ...common, + kind: "reaction", + messageId: record.messageId === null ? null : identifier(record.messageId, `${label}.messageId`), + messageProviderId: identifier(record.messageProviderId, `${label}.messageProviderId`), + participantId: record.participantId === null + ? null + : identifier(record.participantId, `${label}.participantId`), + body: boundedText(record.body, `${label}.body`, MAX_SHORT_TEXT_BYTES), + reactedAt: nullableTimestamp(record.reactedAt, `${label}.reactedAt`), + state: oneOf(record.state, ["active", "removed"] as const, `${label}.state`), + }); +} + +function parseTombstone(record: JsonObject, label: string): TombstoneRecord { + const common = parseCommon(record, "tombstone", [ + "entityKind", "entityId", "entityProviderId", "deletedAt", "scope", "providerRevision", + ], label); + return Object.freeze({ + ...common, + kind: "tombstone", + entityKind: oneOf(record.entityKind, ["conversation", "message", "reaction"] as const, `${label}.entityKind`), + entityId: record.entityId === null ? null : identifier(record.entityId, `${label}.entityId`), + entityProviderId: identifier(record.entityProviderId, `${label}.entityProviderId`), + deletedAt: timestamp(record.deletedAt, `${label}.deletedAt`), + scope: oneOf(record.scope, ["remote", "local", "unknown"] as const, `${label}.scope`), + providerRevision: nullableIdentifier(record.providerRevision, `${label}.providerRevision`), + }); +} + +function parseRecord(value: unknown, kind: RecordKind, label: string): BundleRecord { + const record = object(value, label); + switch (kind) { + case "account": return parseAccount(record, label); + case "participant": return parseParticipant(record, label); + case "conversation": return parseConversation(record, label); + case "message": return parseMessage(record, label); + case "reaction": return parseReaction(record, label); + case "tombstone": return parseTombstone(record, label); + } +} + +function parseArtifact(value: unknown, index: number): Artifact { + const expected = ARTIFACTS[index]!; + const label = `manifest.artifacts[${index}]`; + const record = object(value, label); + exactKeys(record, ["path", "mediaType", "recordKind", "records", "bytes", "sha256"], label); + if ( + record.path !== expected.path + || record.mediaType !== "application/x-ndjson" + || record.recordKind !== expected.kind + ) throw new CliError("invalid-data", `${label} does not match the fixed artifact inventory`); + return Object.freeze({ + path: expected.path, + mediaType: "application/x-ndjson", + recordKind: expected.kind, + records: integer(record.records, `${label}.records`, MAX_RECORDS), + bytes: integer(record.bytes, `${label}.bytes`, MAX_TOTAL_BYTES), + sha256: digest(record.sha256, `${label}.sha256`), + }); +} + +function parseManifest(value: unknown): Manifest { + const record = object(value, "manifest"); + exactKeys(record, [ + "schemaVersion", "format", "source", "provider", "timestamps", "completeness", + "warnings", "privacy", "counts", "artifacts", "integrity", + ], "manifest"); + if (record.schemaVersion !== MESSAGE_BUNDLE_SCHEMA_VERSION || record.format !== "message-like-me.local-message-bundle") { + throw new CliError("invalid-data", "Manifest has an unsupported schemaVersion or format"); + } + const source = object(record.source, "manifest.source"); + exactKeys(source, ["id", "version"], "manifest.source"); + if (source.id !== "beeper-local") throw new CliError("invalid-data", "manifest.source.id must be beeper-local"); + const provider = object(record.provider, "manifest.provider"); + exactKeys(provider, ["id", "version"], "manifest.provider"); + if (provider.id !== "beeper") throw new CliError("invalid-data", "manifest.provider.id must be beeper"); + const timestamps = object(record.timestamps, "manifest.timestamps"); + exactKeys(timestamps, ["startedAt", "finishedAt", "createdAt"], "manifest.timestamps"); + const startedAt = timestamp(timestamps.startedAt, "manifest.timestamps.startedAt"); + const finishedAt = timestamp(timestamps.finishedAt, "manifest.timestamps.finishedAt"); + const createdAt = timestamp(timestamps.createdAt, "manifest.timestamps.createdAt"); + if (startedAt > finishedAt || finishedAt > createdAt) { + throw new CliError("invalid-data", "Manifest timestamps are not monotonic"); + } + const completeness = object(record.completeness, "manifest.completeness"); + exactKeys(completeness, ["kind", "reason", "observedFrom", "observedThrough"], "manifest.completeness"); + const observedFrom = nullableTimestamp(completeness.observedFrom, "manifest.completeness.observedFrom"); + const observedThrough = nullableTimestamp(completeness.observedThrough, "manifest.completeness.observedThrough"); + if (observedFrom !== null && observedThrough !== null && observedFrom > observedThrough) { + throw new CliError("invalid-data", "Manifest completeness bounds are reversed"); + } + const warnings = array(record.warnings, "manifest.warnings", MAX_WARNINGS) + .map((value, index) => token(value, `manifest.warnings[${index}]`)); + if (new Set(warnings).size !== warnings.length) throw new CliError("invalid-data", "Manifest warnings repeat"); + const privacy = object(record.privacy, "manifest.privacy"); + exactKeys(privacy, ["classification", "attachments", "providerUrls", "credentials"], "manifest.privacy"); + if ( + privacy.classification !== "private-local" + || privacy.attachments !== "metadata-only" + || privacy.providerUrls !== "excluded" + || privacy.credentials !== "excluded" + ) throw new CliError("invalid-data", "Manifest privacy guarantees are unsupported"); + const counts = object(record.counts, "manifest.counts"); + exactKeys(counts, ARTIFACTS.map(({ kind }) => kind), "manifest.counts"); + const parsedCounts = Object.fromEntries(ARTIFACTS.map(({ kind }) => [ + kind, + integer(counts[kind], `manifest.counts.${kind}`, MAX_RECORDS), + ])) as Record; + if (parsedCounts.account > MAX_ACCOUNTS) { + throw new CliError("invalid-data", `Manifest exceeds the ${MAX_ACCOUNTS}-account safety bound`); + } + if (!Array.isArray(record.artifacts) || record.artifacts.length !== ARTIFACTS.length) { + throw new CliError("invalid-data", "Manifest must list the fixed six artifacts"); + } + const artifacts = Object.freeze(record.artifacts.map(parseArtifact)); + let totalRecords = 0; + let totalBytes = 0; + for (const artifact of artifacts) { + if (artifact.records !== parsedCounts[artifact.recordKind]) { + throw new CliError("invalid-data", `${artifact.path} count disagrees with manifest.counts`); + } + totalRecords += artifact.records; + totalBytes += artifact.bytes; + } + if (totalRecords > MAX_RECORDS || totalBytes > MAX_TOTAL_BYTES) { + throw new CliError("invalid-data", "Manifest exceeds the bundle record or byte bound"); + } + const integrity = object(record.integrity, "manifest.integrity"); + exactKeys(integrity, ["algorithm", "bundleSha256"], "manifest.integrity"); + if (integrity.algorithm !== "sha256") throw new CliError("invalid-data", "Manifest integrity algorithm is unsupported"); + const result: Manifest = Object.freeze({ + schemaVersion: 1, + format: "message-like-me.local-message-bundle", + source: Object.freeze({ id: "beeper-local", version: version(source.version, "manifest.source.version") }), + provider: Object.freeze({ id: "beeper", version: version(provider.version, "manifest.provider.version") }), + timestamps: Object.freeze({ startedAt, finishedAt, createdAt }), + completeness: Object.freeze({ + kind: oneOf(completeness.kind, ["bounded-local", "truncated", "unknown"] as const, "manifest.completeness.kind"), + reason: completeness.reason === null ? null : token(completeness.reason, "manifest.completeness.reason"), + observedFrom, + observedThrough, + }), + warnings: Object.freeze(warnings), + privacy: Object.freeze({ + classification: "private-local", + attachments: "metadata-only", + providerUrls: "excluded", + credentials: "excluded", + }), + counts: Object.freeze(parsedCounts), + artifacts, + integrity: Object.freeze({ algorithm: "sha256", bundleSha256: digest(integrity.bundleSha256, "manifest.integrity.bundleSha256") }), + }); + const { integrity: _integrity, ...projection } = result; + if (sha256(canonicalJson(projection)) !== result.integrity.bundleSha256) { + throw new CliError("invalid-data", "Manifest bundle SHA-256 does not match its canonical projection"); + } + return result; +} + +function sameFile(left: Awaited>, right: Awaited>): boolean { + return left.dev === right.dev && left.ino === right.ino; +} + +async function bundleDirectory(path: string): Promise { + if (!isAbsolute(path) || resolve(path) !== path) { + throw new CliError("unsafe-path", "Bundle input must be a normalized absolute path"); + } + const before = await lstat(path); + if ( + !before.isDirectory() + || before.isSymbolicLink() + || (before.mode & 0o777) !== 0o700 + || (typeof process.getuid === "function" && before.uid !== process.getuid()) + ) throw new CliError("unsafe-path", "Bundle input must be a current-user-owned mode-0700 physical directory"); + const physical = await realpath(path); + if (physical !== path) throw new CliError("unsafe-path", "Bundle input path must not traverse a symbolic link"); + const after = await lstat(physical); + if (!sameFile(before, after)) throw new CliError("unsafe-path", "Bundle directory changed while resolving"); + const expected = ["manifest.json", ...ARTIFACTS.map(({ path: artifactPath }) => artifactPath)].sort(); + const entries = (await readdir(physical)).sort(); + if (entries.length !== expected.length || entries.some((entry, index) => entry !== expected[index])) { + throw new CliError("invalid-data", "Bundle directory does not contain exactly the version-one inventory"); + } + return physical; +} + +async function openPrivateFile(path: string, maximumBytes: number, allowEmpty: boolean): Promise<{ + handle: FileHandle; + before: BigIntStats; +}> { + const handle = await open(path, fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW); + try { + const before = await handle.stat({ bigint: true }); + if ( + !before.isFile() + || before.nlink !== 1n + || before.size > BigInt(maximumBytes) + || (!allowEmpty && before.size < 1n) + || (before.mode & 0o777n) !== 0o600n + || (typeof process.getuid === "function" && before.uid !== BigInt(process.getuid())) + ) throw new CliError("unsafe-path", `${path} must be a private physical file within its bound`); + return { handle, before }; + } catch (error) { + await handle.close(); + throw error; + } +} + +async function assertFileUnchanged( + path: string, + handle: FileHandle, + before: BigIntStats, +): Promise { + const after = await handle.stat({ bigint: true }); + if ( + before.dev !== after.dev + || before.ino !== after.ino + || before.size !== after.size + || before.mtimeNs !== after.mtimeNs + || before.ctimeNs !== after.ctimeNs + ) throw new CliError("unsafe-path", `${path} changed while it was read`); +} + +async function closeReadHandle(handle: FileHandle): Promise { + try { + await handle.close(); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EBADF") throw error; + } +} + +function decodeUtf8(bytes: Uint8Array, label: string): string { + try { + return new TextDecoder("utf-8", { fatal: true }).decode(bytes); + } catch (error) { + throw new CliError("invalid-data", `${label} is not valid UTF-8`, { cause: error }); + } +} + +async function readManifest(path: string): Promise> { + const opened = await openPrivateFile(path, MAX_MANIFEST_BYTES, false); + try { + const bytes = Uint8Array.from(await opened.handle.readFile()); + await assertFileUnchanged(path, opened.handle, opened.before); + let value: unknown; + try { + value = JSON.parse(decodeUtf8(bytes, "manifest.json")) as unknown; + } catch (error) { + throw new CliError("invalid-data", "manifest.json is not valid UTF-8 JSON", { cause: error }); + } + const manifest = parseManifest(value); + if (!Buffer.from(`${canonicalJson(manifest)}\n`, "utf8").equals(Buffer.from(bytes))) { + throw new CliError("invalid-data", "manifest.json must use canonical JSON with one final newline"); + } + return Object.freeze({ bytes, manifest }); + } finally { + await closeReadHandle(opened.handle); + } +} + +async function readArtifact(root: string, artifact: Artifact): Promise { + const path = join(root, artifact.path); + const opened = await openPrivateFile(path, artifact.bytes, true); + const hash = createHash("sha256"); + const records: BundleRecord[] = []; + let totalBytes = 0; + let pending: Buffer = Buffer.alloc(0); + let endedWithNewline = false; + try { + const stream = createReadStream(path, { + fd: opened.handle.fd, + autoClose: false, + start: 0, + highWaterMark: 64 * 1024, + }); + for await (const value of stream) { + const chunk: Buffer = Buffer.from(value as Uint8Array); + hash.update(chunk); + totalBytes += chunk.byteLength; + if (totalBytes > artifact.bytes) throw new CliError("invalid-data", `${artifact.path} exceeds manifest bytes`); + pending = pending.length === 0 ? chunk : Buffer.concat([pending, chunk]); + let newline = pending.indexOf(0x0a); + while (newline >= 0) { + const line = pending.subarray(0, newline); + pending = pending.subarray(newline + 1); + endedWithNewline = true; + if (line.byteLength < 1 || line.byteLength + 1 > MAX_RECORD_BYTES) { + throw new CliError("invalid-data", `${artifact.path} contains a blank or oversized record`); + } + let parsed: unknown; + try { + parsed = JSON.parse(decodeUtf8(line, `${artifact.path} record`)) as unknown; + } catch (error) { + throw new CliError("invalid-data", `${artifact.path} contains invalid UTF-8 JSON`, { cause: error }); + } + const normalized = parseRecord( + parsed, + artifact.recordKind, + `${artifact.path}:${records.length + 1}`, + ); + if (!Buffer.from(canonicalJson(normalized), "utf8").equals(line)) { + throw new CliError("invalid-data", `${artifact.path} records must use canonical JSON`); + } + records.push(normalized); + if (records.length > artifact.records) { + throw new CliError("invalid-data", `${artifact.path} exceeds its manifest record count`); + } + newline = pending.indexOf(0x0a); + } + if (pending.byteLength + 1 > MAX_RECORD_BYTES) { + throw new CliError("invalid-data", `${artifact.path} contains an oversized record`); + } + if (pending.length > 0) endedWithNewline = false; + } + await assertFileUnchanged(path, opened.handle, opened.before); + } finally { + await closeReadHandle(opened.handle); + } + if (pending.byteLength !== 0 || (artifact.records > 0 && !endedWithNewline)) { + throw new CliError("invalid-data", `${artifact.path} must end every record with a newline`); + } + if ( + totalBytes !== artifact.bytes + || records.length !== artifact.records + || hash.digest("hex") !== artifact.sha256 + ) throw new CliError("invalid-data", `${artifact.path} does not match its manifest integrity`); + return Object.freeze(records); +} + +function hmacKey(value: string | Uint8Array): Uint8Array { + const key = typeof value === "string" ? new TextEncoder().encode(value) : value; + if (!(key instanceof Uint8Array) || key.byteLength < 16 || key.byteLength > 1_024) { + throw new CliError("invalid-data", "Bundle HMAC key must contain 16 through 1024 bytes"); + } + return Uint8Array.from(key); +} + +function hmac(key: Uint8Array, namespace: string, value: string): string { + return createHmac("sha256", key) + .update(`message-like-me\0bundle-${namespace}\0`, "utf8") + .update(value, "utf8") + .digest("hex"); +} + +function compareCodeUnits(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0; +} + +function recordMap(records: readonly T[], label: string): Map { + const result = new Map(); + for (const record of records) { + if (result.has(record.id)) throw new CliError("invalid-data", `${label} repeats a bundle-local ID`); + result.set(record.id, record); + } + return result; +} + +function groupByAccount(records: readonly T[]): Map { + const grouped = new Map(); + for (const record of records) { + const values = grouped.get(record.accountId) ?? []; + values.push(record); + grouped.set(record.accountId, values); + } + return grouped; +} + +function attachmentProvenance( + messageId: string, + attachments: readonly AttachmentRecord[], +): readonly CorpusAttachmentProvenance[] { + return Object.freeze(attachments.map((attachment, index) => ({ + id: `${messageId}:attachment:${index + 1}`, + kind: attachment.kind, + mimeType: attachment.mimeType, + fileName: attachment.name, + bytes: attachment.sizeBytes, + }))); +} + +/** + * Timeline reactions share the messages table, but provider message and + * reaction IDs occupy independent foreign domains. The control-prefixed local + * coordinate cannot equal a provider identifier because the bundle parser + * rejects ASCII controls in every provider ID. + */ +function reactionTimelineCoordinate(localReactionId: string): string { + return `\u001freaction-timeline:${localReactionId}`; +} + +function normalizeBundle( + manifest: Manifest, + manifestSha256: string, + records: Readonly>, + key: Uint8Array, +): readonly SourceCorpusSnapshot[] { + const accounts = records.account as readonly AccountRecord[]; + const participants = records.participant as readonly ParticipantRecord[]; + const conversations = records.conversation as readonly ConversationRecord[]; + const messages = records.message as readonly MessageRecord[]; + const reactions = records.reaction as readonly ReactionRecord[]; + const tombstones = records.tombstone as readonly TombstoneRecord[]; + if (accounts.length > MAX_ACCOUNTS) { + throw new CliError("invalid-data", `Bundle exceeds the ${MAX_ACCOUNTS}-account safety bound`); + } + const accountById = recordMap(accounts, "accounts"); + recordMap(participants, "participants"); + recordMap(conversations, "conversations"); + const messageRecordById = recordMap(messages, "messages"); + const reactionById = recordMap(reactions, "reactions"); + recordMap(tombstones, "tombstones"); + for (const [kind, values] of [ + ["account", accounts], + ["participant", participants], + ["conversation", conversations], + ["message", messages], + ["reaction", reactions], + ["tombstone", tombstones], + ] as const) { + const providerCoordinates = new Set(); + for (const record of values) { + const coordinate = `${record.accountId}\0${record.provenance.providerId}`; + if (providerCoordinates.has(coordinate)) { + throw new CliError( + "invalid-data", + `${kind} records repeat a provider identity within one account`, + ); + } + providerCoordinates.add(coordinate); + } + } + for (const record of [...participants, ...conversations, ...messages, ...reactions, ...tombstones]) { + const account = accountById.get(record.accountId); + if ( + account === undefined + || account.network !== record.network + || account.provenance.connectedAccountProviderId !== record.provenance.connectedAccountProviderId + ) throw new CliError("invalid-data", "A record does not match its connected account realm"); + } + const participantsByAccount = groupByAccount(participants); + const conversationsByAccount = groupByAccount(conversations); + const messagesByAccount = groupByAccount(messages); + const reactionsByAccount = groupByAccount(reactions); + const tombstonesByAccount = groupByAccount(tombstones); + + const result: SourceCorpusSnapshot[] = []; + const sourceIds = new Set(); + for (const account of accounts) { + const accountParticipants = participantsByAccount.get(account.id) ?? []; + const participantById = new Map(accountParticipants.map((participant) => [participant.id, participant])); + const self = participantById.get(account.selfParticipantId); + if ( + self === undefined + || !self.isSelf + || accountParticipants.filter(({ isSelf }) => isSelf).length !== 1 + ) { + throw new CliError("invalid-data", "An account must have exactly one matching self participant"); + } + const accountConversations = conversationsByAccount.get(account.id) ?? []; + const conversationParticipantIds = new Map(accountConversations.map((conversation) => [ + conversation.id, + new Set(conversation.participantIds), + ])); + for (const conversation of accountConversations) { + for (const participantId of conversation.participantIds) { + if (!participantById.has(participantId)) { + throw new CliError( + "invalid-data", + "A conversation references an unknown participant", + ); + } + } + if ( + conversation.type === "direct" + && conversation.participantsComplete === true + && ( + conversation.participantIds.length !== 2 + || !conversation.participantIds.includes(account.selfParticipantId) + || conversation.participantIds.filter((participantId) => + participantById.get(participantId)?.isSelf === false).length !== 1 + ) + ) { + throw new CliError( + "invalid-data", + "A complete direct conversation must contain one self and one non-self participant", + ); + } + } + const conversationById = new Map(accountConversations.map((conversation) => [conversation.id, conversation])); + const namespace = [ + manifest.provider.id, + account.provenance.connectedAccountProviderId, + self.provenance.providerId, + ].join("\0"); + const sourceId = `source_${hmac(key, "source", namespace)}`; + if (sourceIds.has(sourceId)) { + throw new CliError( + "invalid-data", + "Connected accounts repeat a stable source realm", + ); + } + sourceIds.add(sourceId); + const conversationLocalIds = new Map(accountConversations.map((conversation) => [ + conversation.id, + `conversation_${hmac(key, "conversation", `${namespace}\0${conversation.provenance.providerId}`)}`, + ])); + const participantLocalIds = new Map(accountParticipants.map((participant) => [ + participant.id, + `participant_${hmac(key, "participant", `${namespace}\0${participant.provenance.providerId}`)}`, + ])); + const normalizedConversations: CorpusConversation[] = accountConversations.map((conversation) => { + const known = conversation.participantIds.flatMap((id) => { + const participant = participantById.get(id); + return participant === undefined ? [] : [participant]; + }); + const peers = known.filter(({ isSelf }) => !isSelf); + const completeDirectPeer = conversation.type === "direct" + && conversation.participantsComplete === true + && peers.length === 1 + ? peers[0]! + : null; + const canonicalHandle = completeDirectPeer?.handle === null || completeDirectPeer === null + ? null + : normalizeContactHandle(completeDirectPeer.handle); + return Object.freeze({ + id: conversationLocalIds.get(conversation.id)!, + sourceKey: conversation.provenance.providerId, + privateLabel: conversation.title, + service: account.network, + participantCount: conversation.type === "direct" ? 1 : peers.length, + participantIds: Object.freeze(peers.map((participant) => participantLocalIds.get(participant.id)!)), + privateParticipants: canonicalHandle === null + ? Object.freeze([]) + : Object.freeze([canonicalHandle.normalizedValue]), + group: conversation.type !== "direct", + }); + }); + + const accountMessages = messagesByAccount.get(account.id) ?? []; + const messageById = new Map(accountMessages.map((message) => [message.id, message])); + const messageByProviderId = new Map(accountMessages.map((message) => [ + message.provenance.providerId, + message, + ])); + const replacementTargets = new Map>(); + const replacerByTarget = new Map(); + for (const message of accountMessages) { + if (!conversationById.has(message.conversationId)) { + throw new CliError("invalid-data", "A message references an unknown conversation"); + } + if (message.senderParticipantId !== null && !participantById.has(message.senderParticipantId)) { + throw new CliError("invalid-data", "A message references an unknown sender participant"); + } + const sender = message.senderParticipantId === null + ? null + : participantById.get(message.senderParticipantId)!; + const conversation = conversationById.get(message.conversationId)!; + if ( + sender !== null + && ( + (message.direction === "outgoing" && !sender.isSelf) + || (message.direction === "incoming" && sender.isSelf) + ) + ) throw new CliError("invalid-data", "A message direction conflicts with its sender identity"); + if ( + sender !== null + && conversation.participantsComplete === true + && !conversationParticipantIds.get(conversation.id)!.has(sender.id) + ) throw new CliError("invalid-data", "A message sender is outside its complete conversation roster"); + if (message.replyTo !== null) { + const localTarget = message.replyTo.messageId === null + ? undefined + : messageRecordById.get(message.replyTo.messageId); + if ( + message.replyTo.messageId !== null + && ( + localTarget === undefined + || localTarget.accountId !== account.id + || localTarget.provenance.providerId !== message.replyTo.providerId + ) + ) throw new CliError("invalid-data", "A message reply has mismatched target coordinates"); + const providerTarget = messageByProviderId.get(message.replyTo.providerId); + const target = localTarget ?? providerTarget; + if ( + message.replyTo.providerId === message.provenance.providerId + || (target !== undefined && target.conversationId !== message.conversationId) + ) throw new CliError("invalid-data", "A message reply has an invalid conversation target"); + } + if (message.edit?.kind === "replacement") { + const localTarget = message.edit.replacesMessageId === null + ? undefined + : messageRecordById.get(message.edit.replacesMessageId); + if ( + message.edit.replacesMessageId !== null + && ( + localTarget === undefined + || localTarget.accountId !== account.id + || localTarget.provenance.providerId !== message.edit.replacesProviderId + ) + ) throw new CliError("invalid-data", "A message edit has mismatched replacement coordinates"); + const providerTarget = messageByProviderId.get(message.edit.replacesProviderId); + const target = localTarget ?? providerTarget; + if ( + message.edit.replacesProviderId === message.provenance.providerId + || (target !== undefined && target.conversationId !== message.conversationId) + ) throw new CliError("invalid-data", "A message edit has an invalid replacement target"); + if (replacerByTarget.has(message.edit.replacesProviderId)) { + throw new CliError("invalid-data", "A message version has multiple replacements"); + } + replacerByTarget.set(message.edit.replacesProviderId, message.provenance.providerId); + replacementTargets.set(message.id, Object.freeze({ + target, + externalId: message.edit.replacesProviderId, + })); + } + } + const editEdges = new Map([...replacementTargets.entries()].map(([messageId, target]) => [ + messageById.get(messageId)!.provenance.providerId, + target.externalId, + ])); + const completedEditNodes = new Set(); + for (const start of editEdges.keys()) { + if (completedEditNodes.has(start)) continue; + const seen = new Set(); + const chain: string[] = []; + let current: string | undefined = start; + while (current !== undefined && !completedEditNodes.has(current)) { + if (seen.has(current)) throw new CliError("invalid-data", "Message replacement edits contain a cycle"); + seen.add(current); + chain.push(current); + current = editEdges.get(current); + } + for (const node of chain) completedEditNodes.add(node); + } + const analyzableMessages = accountMessages.filter(({ direction }) => direction !== "unknown") + .sort((left, right) => + compareCodeUnits(left.conversationId, right.conversationId) + || compareCodeUnits(left.sortKey, right.sortKey) + || compareCodeUnits(left.sentAt, right.sentAt) + || compareCodeUnits(left.provenance.providerId, right.provenance.providerId)); + const normalizedMessages: CorpusMessage[] = []; + const messageProvenance: CorpusMessageProvenance[] = []; + const localMessageIds = new Map(); + const localReactionIds = new Map(); + const timelineReactionIds = new Set(); + for (const [index, message] of analyzableMessages.entries()) { + const localId = `message_${hmac(key, "message", `${namespace}\0${message.provenance.providerId}`)}`; + localMessageIds.set(message.id, localId); + const body = message.bodyTruncated === true || message.deletion !== null ? null : message.body; + normalizedMessages.push(Object.freeze({ + id: localId, + sourceRowId: index + 1, + sourceGuid: message.provenance.providerId, + conversationId: conversationLocalIds.get(message.conversationId)!, + sentAt: message.sentAt, + direction: message.direction as "incoming" | "outgoing", + body, + bodySource: body === null ? "unavailable" : "text", + kind: body !== null || message.bodyTruncated === true + ? "text" + : message.attachments.length > 0 ? "attachment" : "unknown", + replyToSourceGuid: message.replyTo?.providerId ?? null, + editedAt: message.edit?.editedAt ?? null, + retractedAt: message.deletion?.observedAt ?? null, + service: account.network, + attachmentCount: message.attachments.length, + })); + messageProvenance.push(Object.freeze({ + messageId: localId, + externalId: message.provenance.providerId, + providerSortKey: message.sortKey, + replyToExternalId: message.replyTo?.providerId ?? null, + attachments: attachmentProvenance(localId, message.attachments), + metadata: message, + })); + } + + const accountReactions = reactionsByAccount.get(account.id) ?? []; + for (const reaction of accountReactions) { + localReactionIds.set( + reaction.id, + `message_${hmac(key, "reaction", `${namespace}\0${reaction.provenance.providerId}`)}`, + ); + } + const reactionFacts: CorpusReactionFact[] = []; + for (const reaction of accountReactions) { + const localTarget = reaction.messageId === null + ? undefined + : messageRecordById.get(reaction.messageId); + if ( + reaction.messageId !== null + && ( + localTarget === undefined + || localTarget.accountId !== account.id + || localTarget.provenance.providerId !== reaction.messageProviderId + ) + ) throw new CliError("invalid-data", "A reaction has mismatched target coordinates"); + if (reaction.participantId !== null && !participantById.has(reaction.participantId)) { + throw new CliError("invalid-data", "A reaction references an unknown participant"); + } + const target = localTarget ?? messageByProviderId.get(reaction.messageProviderId); + const participant = reaction.participantId === null + ? null + : participantById.get(reaction.participantId)!; + const targetConversationId = target === undefined + ? null + : conversationLocalIds.get(target.conversationId) ?? null; + if (target !== undefined && participant !== null) { + const targetConversation = conversationById.get(target.conversationId)!; + if ( + targetConversation.participantsComplete === true + && !conversationParticipantIds.get(targetConversation.id)!.has(participant.id) + ) throw new CliError("invalid-data", "A reaction participant is outside its complete conversation roster"); + } + const localId = localReactionIds.get(reaction.id)!; + reactionFacts.push(Object.freeze({ + id: localId, + externalId: reaction.provenance.providerId, + targetExternalId: reaction.messageProviderId, + conversationId: targetConversationId, + direction: participant === null ? null : participant.isSelf ? "outgoing" : "incoming", + body: reaction.body, + reactedAt: reaction.reactedAt, + state: reaction.state, + })); + if (reaction.state !== "active" || reaction.reactedAt === null || reaction.participantId === null) continue; + if (participant === null || target === undefined || targetConversationId === null) continue; + timelineReactionIds.add(reaction.id); + const timelineCoordinate = reactionTimelineCoordinate(localId); + normalizedMessages.push(Object.freeze({ + id: localId, + sourceRowId: normalizedMessages.length + 1, + sourceGuid: timelineCoordinate, + conversationId: targetConversationId, + sentAt: reaction.reactedAt, + direction: participant.isSelf ? "outgoing" : "incoming", + body: null, + bodySource: "unavailable", + kind: "reaction", + replyToSourceGuid: reaction.messageProviderId, + editedAt: null, + retractedAt: null, + service: account.network, + attachmentCount: 0, + })); + messageProvenance.push(Object.freeze({ + messageId: localId, + externalId: timelineCoordinate, + providerSortKey: null, + replyToExternalId: reaction.messageProviderId, + attachments: Object.freeze([]), + metadata: reaction, + })); + } + const reactionFactByExternal = new Map(reactionFacts.map((fact) => [fact.externalId, fact])); + + const auxiliaryRecords: CorpusSourceRecord[] = [ + { kind: "account", id: account.provenance.providerId, record: account }, + ...accountParticipants.map((participant) => ({ + kind: "participant" as const, + id: participant.provenance.providerId, + record: participant, + })), + ...accountReactions.map((reaction) => ({ + kind: "reaction" as const, + id: reaction.provenance.providerId, + record: reaction, + })), + ...(tombstonesByAccount.get(account.id) ?? []).map((tombstone) => ({ + kind: "tombstone" as const, + id: tombstone.provenance.providerId, + record: tombstone, + })), + ...accountMessages.filter(({ direction }) => direction === "unknown").map((message) => ({ + kind: "excluded-message" as const, + id: message.provenance.providerId, + record: message, + })), + ]; + const accountTombstones = tombstonesByAccount.get(account.id) ?? []; + const deletions: CorpusSourceDeletion[] = accountTombstones.map((tombstone) => { + const entityId = tombstone.entityId; + let localEntityId: string | null = null; + if (entityId !== null) { + if (tombstone.entityKind === "conversation") { + const target = conversationById.get(entityId); + if (target === undefined) { + throw new CliError("invalid-data", "A tombstone references an unknown local conversation"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched conversation identity"); + } + localEntityId = conversationLocalIds.get(entityId) ?? null; + } else if (tombstone.entityKind === "message") { + const target = messageRecordById.get(entityId); + if (target === undefined || target.accountId !== account.id) { + throw new CliError("invalid-data", "A tombstone references an unknown local message"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched message identity"); + } + localEntityId = localMessageIds.get(entityId) ?? null; + } else if (tombstone.entityKind === "reaction") { + const target = reactionById.get(entityId); + if (target === undefined || target.accountId !== account.id) { + throw new CliError("invalid-data", "A tombstone references an unknown local reaction"); + } + if (target.provenance.providerId !== tombstone.entityProviderId) { + throw new CliError("invalid-data", "A tombstone has mismatched reaction identity"); + } + localEntityId = localReactionIds.get(entityId) ?? null; + } + } + return Object.freeze({ + entityKind: tombstone.entityKind, + localEntityId, + externalId: tombstone.entityProviderId, + deletedAt: tombstone.deletedAt, + reason: "tombstone" as const, + }); + }); + for (const [messageId, replacement] of replacementTargets) { + const message = messageById.get(messageId)!; + deletions.push(Object.freeze({ + entityKind: "message" as const, + localEntityId: replacement.target === undefined + ? null + : localMessageIds.get(replacement.target.id) ?? null, + externalId: replacement.externalId, + deletedAt: message.edit!.editedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId)!, + reason: "replacement" as const, + })); + } + for (const message of accountMessages) { + if (message.deletion === null) continue; + deletions.push(Object.freeze({ + entityKind: "message" as const, + localEntityId: localMessageIds.get(message.id) ?? null, + externalId: message.provenance.providerId, + deletedAt: message.deletion.observedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId)!, + reason: "tombstone" as const, + })); + } + for (const message of accountMessages) { + if (message.direction !== "unknown") continue; + deletions.push(Object.freeze({ + entityKind: "message" as const, + localEntityId: null, + externalId: message.provenance.providerId, + deletedAt: message.provenance.observedAt, + expectedConversationId: conversationLocalIds.get(message.conversationId)!, + reason: "explicit-exclusion" as const, + })); + } + for (const reaction of accountReactions) { + if (timelineReactionIds.has(reaction.id)) continue; + const fact = reactionFactByExternal.get(reaction.provenance.providerId)!; + deletions.push(Object.freeze({ + entityKind: reaction.state === "removed" ? "reaction" as const : "reaction-timeline" as const, + localEntityId: localReactionIds.get(reaction.id)!, + externalId: reaction.provenance.providerId, + deletedAt: reaction.provenance.observedAt, + ...(fact.conversationId === null ? {} : { expectedConversationId: fact.conversationId }), + reason: reaction.state === "removed" ? "tombstone" as const : "explicit-exclusion" as const, + })); + } + const sourceWarnings = [...manifest.warnings]; + const unknownDirections = accountMessages.filter(({ direction }) => direction === "unknown").length; + const undatedReactions = accountReactions.filter(({ reactedAt }) => reactedAt === null).length; + if (unknownDirections > 0) sourceWarnings.push(`unknown-direction-messages:${unknownDirections}`); + if (undatedReactions > 0) sourceWarnings.push(`undated-reactions:${undatedReactions}`); + const accountTimelineBounds = [ + ...accountMessages.map(({ sentAt }) => sentAt), + ...accountReactions.flatMap(({ reactedAt }) => reactedAt === null ? [] : [reactedAt]), + ].sort(compareCodeUnits); + const accountObservedFrom = accountTimelineBounds[0] ?? null; + const accountObservedThrough = accountTimelineBounds.at(-1) ?? null; + const revisionHash = createHash("sha256"); + const revisionHeader = canonicalJson({ + schemaVersion: 1, + source: manifest.source, + provider: manifest.provider, + completeness: manifest.completeness, + warnings: manifest.warnings, + }); + revisionHash.update(`${revisionHeader.length}:`, "utf8").update(revisionHeader, "utf8"); + for (const [kind, values] of [ + ["account", [account]], + ["participant", accountParticipants], + ["conversation", accountConversations], + ["message", accountMessages], + ["reaction", accountReactions], + ["tombstone", accountTombstones], + ] as const) { + revisionHash.update(`${kind.length}:${kind}`, "utf8"); + for (const record of values) { + const encoded = canonicalJson(record); + revisionHash.update(`${Buffer.byteLength(encoded, "utf8")}:`, "utf8").update(encoded, "utf8"); + } + } + const revision = revisionHash.digest("hex"); + result.push(Object.freeze({ + source: Object.freeze({ + id: sourceId, + kind: "bundle", + provider: manifest.provider.id, + network: account.network, + accountId: account.provenance.connectedAccountProviderId, + externalId: account.provenance.connectedAccountProviderId, + revision, + generatedAt: manifest.timestamps.createdAt, + producer: manifest.source, + coverage: Object.freeze({ + history: manifest.completeness.kind === "unknown" ? "unknown" : "bounded", + observedFrom: accountObservedFrom, + observedTo: accountObservedThrough, + kind: manifest.completeness.kind, + reason: manifest.completeness.reason, + }), + manifestSha256, + identity: Object.freeze({ account, selfParticipantProviderId: self.provenance.providerId }), + warnings: Object.freeze(sourceWarnings), + }), + conversations: Object.freeze(normalizedConversations), + conversationProvenance: Object.freeze(accountConversations.map((conversation) => ({ + conversationId: conversationLocalIds.get(conversation.id)!, + externalId: conversation.provenance.providerId, + metadata: conversation, + }))), + messages: Object.freeze(normalizedMessages), + messageProvenance: Object.freeze(messageProvenance), + reactionFacts: Object.freeze(reactionFacts), + auxiliaryRecords: Object.freeze(auxiliaryRecords), + deletions: Object.freeze(deletions), + })); + } + return Object.freeze(result); +} + +/** Read one complete private Wrench/Beeper replacement-snapshot bundle. */ +export async function readMessageBundle( + path: string, + options: Readonly<{ hmacKey: string | Uint8Array }>, +): Promise { + const key = hmacKey(options.hmacKey); + const root = await bundleDirectory(path); + const manifestResult = await readManifest(join(root, "manifest.json")); + const manifest = manifestResult.manifest; + const manifestSha256 = sha256(manifestResult.bytes); + const parsedRecords: Array = []; + for (const artifact of manifest.artifacts) parsedRecords.push(await readArtifact(root, artifact)); + const records = Object.fromEntries(manifest.artifacts.map((artifact, index) => [ + artifact.recordKind, + parsedRecords[index]!, + ])) as Record; + return Object.freeze({ + schemaVersion: MESSAGE_BUNDLE_SCHEMA_VERSION, + manifestSha256, + sources: normalizeBundle(manifest, manifestSha256, records, key), + }); +} diff --git a/src/commands.test.ts b/src/commands.test.ts index 587b61b..d9c64e2 100644 --- a/src/commands.test.ts +++ b/src/commands.test.ts @@ -8,6 +8,7 @@ import type { CommandIo } from "./io.ts"; import { dataPaths, initializeDataPaths, loadOrCreateInstallKey } from "./paths.ts"; import { LocalStore } from "./store.ts"; import { syntheticProfileV2 } from "./test-fixtures.ts"; +import { writeSyntheticMessageBundle } from "./test-bundle-fixture.ts"; import type { CorpusSnapshot, StudyPacket } from "./types.ts"; function corpus(): CorpusSnapshot { @@ -108,6 +109,91 @@ async function createContactsFixture(root: string): Promise { } describe("messagelikeme CLI", () => { + test("ingests a strict local bundle and exposes redacted source health", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-cli-bundle-")); + const capture = ioCapture(); + try { + const bundlePath = await writeSyntheticMessageBundle(root); + const state = join(root, "state"); + expect(await main([ + "--data-dir", state, "ingest", "bundle", "--input", bundlePath, "--json", + ], capture.io)).toBe(0); + expect(capture.stderr()).toBe(""); + const receipt = JSON.parse(capture.stdout()) as { + sources: Array<{ id: string }>; + messages: number; + }; + expect(receipt.messages).toBe(4); + expect(receipt.sources[0]!.id).toMatch(/^source_[a-f0-9]{64}$/u); + expect(capture.stdout()).not.toContain("Synthetic Peer"); + expect(capture.stdout()).not.toContain("peer@example.test"); + expect(capture.stdout()).not.toContain("Synthetic answer"); + + capture.clear(); + expect(await main([ + "--data-dir", state, "sources", "list", "--json", + ], capture.io)).toBe(0); + const listed = JSON.parse(capture.stdout()) as { + sources: Array>; + }; + expect(listed.sources[0]).toMatchObject({ + id: receipt.sources[0]!.id, + provider: "beeper", + network: "whatsapp", + conversations: 1, + messages: 4, + reactions: 2, + undatedReactions: 1, + }); + expect(listed.sources[0]).not.toHaveProperty("accountId"); + expect(capture.stdout()).not.toContain("synthetic-connected-account"); + + capture.clear(); + expect(await main([ + "--data-dir", state, "contacts", "list", "--json", + ], capture.io)).toBe(0); + const contacts = JSON.parse(capture.stdout()) as { contacts: Array<{ id: string }> }; + expect(contacts.contacts).toHaveLength(1); + capture.clear(); + expect(await main([ + "--data-dir", state, "inspect", "tempo", contacts.contacts[0]!.id, "--json", + ], capture.io)).toBe(0); + const tempoOutput = capture.stdout(); + expect(JSON.parse(tempoOutput)).toMatchObject({ + reactions: { + total: 2, + incoming: 1, + outgoing: 1, + undated: 1, + }, + }); + expect(tempoOutput).not.toContain("heart"); + expect(tempoOutput).not.toContain("thumbs-up"); + expect(Object.hasOwn( + (JSON.parse(tempoOutput) as { reactions: Record }).reactions, + "byBody", + )).toBeFalse(); + + capture.clear(); + expect(await main([ + "--data-dir", state, "context", contacts.contacts[0]!.id, "--json", + ], capture.io)).toBe(0); + const contextOutput = capture.stdout(); + expect(contextOutput).not.toContain("heart"); + expect(contextOutput).not.toContain("thumbs-up"); + expect(contextOutput).not.toContain("byBody"); + + capture.clear(); + expect(await main([ + "--data-dir", state, "sources", "show", receipt.sources[0]!.id, + "--private", "--json", + ], capture.io)).toBe(0); + expect(capture.stdout()).toContain("synthetic-connected-account"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + test("keeps aggregate views redacted and completes packet-to-profile flow", async () => { const root = await mkdtemp(join(tmpdir(), "message-like-me-cli-")); const paths = await initializeDataPaths(dataPaths(join(root, "state"))); diff --git a/src/commands.ts b/src/commands.ts index 123a137..870905e 100644 --- a/src/commands.ts +++ b/src/commands.ts @@ -2,6 +2,7 @@ import { lstat } from "node:fs/promises"; import { isAbsolute, resolve } from "node:path"; import { integerOption, parseArguments, rejectUnused, type ParsedArguments } from "./args.ts"; import { prettyJson, sha256 } from "./canonical-json.ts"; +import { readMessageBundle } from "./bundle.ts"; import { DEFAULT_CONTACTS_DIRECTORY, readMacOSContacts } from "./contacts.ts"; import { CliError } from "./errors.ts"; import { DEFAULT_IMESSAGE_DATABASE, readIMessageDatabase } from "./imessage.ts"; @@ -29,7 +30,10 @@ export const HELP = `Message Like Me ${MESSAGE_LIKE_ME_VERSION} Usage: messagelikeme [--data-dir PATH] init [--json] messagelikeme [--data-dir PATH] ingest imessage [--database PATH] [--json] + messagelikeme [--data-dir PATH] ingest bundle --input ABS_PATH [--json] messagelikeme [--data-dir PATH] ingest contacts [--addressbook PATH] [--json] + messagelikeme [--data-dir PATH] sources list [--private] [--json] + messagelikeme [--data-dir PATH] sources show SOURCE_ID [--private] [--json] messagelikeme [--data-dir PATH] contacts list [--min-outgoing N] [--limit N] [--private] [--json] messagelikeme [--data-dir PATH] contacts show CONTACT_ID [--private] [--json] messagelikeme [--data-dir PATH] contacts resolve QUERY --private [--limit N] [--json] @@ -50,9 +54,9 @@ Usage: [--project PATH] [--force] [--json] messagelikeme [--data-dir PATH] doctor [--json] -Message Like Me reads caller-owned macOS Messages and optional Contacts data, -then stores private analysis locally. It has no network, account, AI-provider, -or message-sending surface. +Message Like Me reads caller-owned macOS Messages, optional Contacts data, and +strict private local message bundles, then stores private analysis locally. It +has no network, account, AI-provider, or message-sending surface. `; async function exists(path: string): Promise { @@ -124,7 +128,10 @@ function contactMetrics( options: Parameters[3] = {}, ): ContactMetrics { const evidence = contactEvidence(store, contactId); - return analyzeContact(evidence.messages, evidence.corpusRevision, contactId, options); + return analyzeContact(evidence.messages, evidence.corpusRevision, contactId, { + ...options, + reactionFacts: evidence.reactions, + }); } function metricOptions(parsed: ParsedArguments): Readonly<{ @@ -165,6 +172,7 @@ function safeContactDetail(store: LocalStore, contactId: string, privateLabels: privateParticipants: conversation.privateParticipants, } : {}), service: conversation.service, + services: conversation.services, group: conversation.group, participantCount: conversation.participantCount, participantIds: conversation.participantIds, @@ -240,6 +248,22 @@ function translateContactsError(error: unknown): never { ); } +function translateBundleError(error: unknown): never { + if (error instanceof CliError) throw error; + const code = (error as NodeJS.ErrnoException).code; + if (code === "EACCES" || code === "EPERM") { + throw new CliError("permission", "The selected private bundle is not readable", { cause: error }); + } + if (code === "ENOENT") { + throw new CliError("not-found", "The selected private bundle does not exist", { cause: error }); + } + throw new CliError( + "invalid-data", + "The selected private message bundle could not be read safely", + { cause: error }, + ); +} + export async function runCommand(argv: readonly string[], io: CommandIo): Promise { const parsed = parseArguments(argv); if (parsed.flags.has("version")) { @@ -299,6 +323,38 @@ export async function runCommand(argv: readonly string[], io: CommandIo): Promis return; } + if (command === "ingest" && subcommand === "bundle" && identifier === undefined) { + rejectUnused(parsed, ["data-dir", "input"], ["json"]); + const input = absolutePrivatePath(parsed.options.get("input"), "--input"); + const context = await writableStore(parsed); + try { + let bundle; + try { + bundle = await readMessageBundle(input, { hmacKey: context.key }); + } catch (error) { + translateBundleError(error); + } + const stored = context.store.replaceSources(bundle.sources, canonicalNow(io), context.key); + const result = { + schemaVersion: bundle.schemaVersion, + manifestSha256: bundle.manifestSha256, + corpusRevision: stored.corpusRevision, + sources: stored.sources, + conversations: stored.sources.reduce((sum, source) => sum + source.conversations, 0), + messages: stored.sources.reduce((sum, source) => sum + source.messages, 0), + }; + emit( + io, + json, + result, + `Ingested ${result.messages} active messages across ${result.conversations} conversations from ${result.sources.length} sources`, + ); + } finally { + context.store.close(); + } + return; + } + if (command === "ingest" && subcommand === "contacts" && identifier === undefined) { rejectUnused(parsed, ["data-dir", "addressbook"], ["json"]); const context = await writableStore(parsed); @@ -351,6 +407,31 @@ export async function runCommand(argv: readonly string[], io: CommandIo): Promis return; } + if (command === "sources" && subcommand === "list" && identifier === undefined) { + rejectUnused(parsed, ["data-dir"], ["json", "private"]); + const context = await existingStore(parsed); + try { + const sources = context.store.listSources(parsed.flags.has("private")); + emit(io, json, { sources }, `${sources.length} message sources`); + } finally { + context.store.close(); + } + return; + } + + if (command === "sources" && subcommand === "show" && identifier !== undefined) { + rejectUnused(parsed, ["data-dir"], ["json", "private"]); + const context = await existingStore(parsed); + try { + const source = context.store.source(identifier, parsed.flags.has("private")); + if (source === null) throw new CliError("not-found", `Unknown source ${identifier}`); + emit(io, json, source, `Message source ${identifier}`); + } finally { + context.store.close(); + } + return; + } + if (command === "contacts" && subcommand === "show" && identifier !== undefined) { rejectUnused(parsed, ["data-dir"], ["json", "private"]); const context = await existingStore(parsed); @@ -435,7 +516,7 @@ export async function runCommand(argv: readonly string[], io: CommandIo): Promis evidence.messages, evidence.corpusRevision, identifier, - metricOptions(parsed), + { ...metricOptions(parsed), reactionFacts: evidence.reactions }, ); const packet = buildStudyPacket(evidence.messages, metrics, { limit: integerOption(parsed, "limit", 24, 1, 50), @@ -510,7 +591,7 @@ export async function runCommand(argv: readonly string[], io: CommandIo): Promis evidence.messages, evidence.corpusRevision, identifier, - metricOptions(parsed), + { ...metricOptions(parsed), reactionFacts: evidence.reactions }, ); const packets = buildEvaluationPackets(evidence.messages, metrics, { after, diff --git a/src/fixtures/beeper-message-like-me-v1/accounts.ndjson b/src/fixtures/beeper-message-like-me-v1/accounts.ndjson new file mode 100644 index 0000000..2c91aff --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/accounts.ndjson @@ -0,0 +1,2 @@ +{"accountId":"account:synthetic:primary","displayName":"Synthetic Primary","handle":"+15555550100","id":"account:synthetic:primary","kind":"account","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-account:synthetic-primary","providerRevision":null},"schemaVersion":1,"selfParticipantId":"participant:synthetic:self"} +{"accountId":"account:synthetic:secondary","displayName":"Synthetic Secondary","handle":"synthetic-secondary@example.invalid","id":"account:synthetic:secondary","kind":"account","network":"synthetic-secondary","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-secondary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-account:synthetic-secondary","providerRevision":null},"schemaVersion":1,"selfParticipantId":"participant:synthetic:secondary-self"} diff --git a/src/fixtures/beeper-message-like-me-v1/conversations.ndjson b/src/fixtures/beeper-message-like-me-v1/conversations.ndjson new file mode 100644 index 0000000..7f4f218 --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/conversations.ndjson @@ -0,0 +1 @@ +{"accountId":"account:synthetic:primary","id":"conversation:synthetic:friend","kind":"conversation","lastMessageAt":"2026-08-21T15:58:45.000Z","network":"synthetic","participantIds":["participant:synthetic:self","participant:synthetic:peer"],"participantsComplete":true,"provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-conversation:synthetic-friend","providerRevision":"chat-r1"},"schemaVersion":1,"startedAt":"2026-08-21T15:50:00.000Z","title":"Synthetic Friend","type":"direct"} diff --git a/src/fixtures/beeper-message-like-me-v1/manifest.json b/src/fixtures/beeper-message-like-me-v1/manifest.json new file mode 100644 index 0000000..0d9b70e --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/manifest.json @@ -0,0 +1 @@ +{"artifacts":[{"bytes":913,"mediaType":"application/x-ndjson","path":"accounts.ndjson","recordKind":"account","records":2,"sha256":"822cb760142762378a27aa32d35aa2f8422c353cf139700bdd754454c9516db6"},{"bytes":1262,"mediaType":"application/x-ndjson","path":"participants.ndjson","recordKind":"participant","records":3,"sha256":"1287cb91bcce7e5e6299115a86e4c18f57bb6801b07424e95652974a1ad9d87f"},{"bytes":571,"mediaType":"application/x-ndjson","path":"conversations.ndjson","recordKind":"conversation","records":1,"sha256":"87957914b3f21ece69815719db37ea6413efdf3201b68bd60a0cf7abbbacb529"},{"bytes":1492,"mediaType":"application/x-ndjson","path":"messages.ndjson","recordKind":"message","records":2,"sha256":"dbe97f5b4a5c46c6c2f3e02294511ceb7868e45a8738f8736c0ec93d6707c394"},{"bytes":521,"mediaType":"application/x-ndjson","path":"reactions.ndjson","recordKind":"reaction","records":1,"sha256":"683c263ce93e82bca0d905bcf6ef4cec6b32d8028aaa52595d0f295f068a6b01"},{"bytes":539,"mediaType":"application/x-ndjson","path":"tombstones.ndjson","recordKind":"tombstone","records":1,"sha256":"34f5a61aa9756c10979372b6147337053a563a6a7c63cd44eab602e2594e05d5"}],"completeness":{"kind":"bounded-local","observedFrom":"2026-08-21T15:50:00.000Z","observedThrough":"2026-08-21T15:59:00.000Z","reason":"desktop-local-sequential-export"},"counts":{"account":2,"conversation":1,"message":2,"participant":3,"reaction":1,"tombstone":1},"format":"message-like-me.local-message-bundle","integrity":{"algorithm":"sha256","bundleSha256":"f4c1c8f99e9ff74a4e8ba300a7e0e417f6e2cadb21a90dc3e712e3f50b518c1d"},"privacy":{"attachments":"metadata-only","classification":"private-local","credentials":"excluded","providerUrls":"excluded"},"provider":{"id":"beeper","version":"0.6.2"},"schemaVersion":1,"source":{"id":"beeper-local","version":"1.1.0"},"timestamps":{"createdAt":"2026-08-21T16:00:01.000Z","finishedAt":"2026-08-21T16:00:01.000Z","startedAt":"2026-08-21T16:00:00.000Z"},"warnings":["attachments-metadata-only","connected-account-backfill-coverage-unknown","remote-history-not-claimed","sequential-account-snapshot","synthetic-golden-fixture"]} diff --git a/src/fixtures/beeper-message-like-me-v1/messages.ndjson b/src/fixtures/beeper-message-like-me-v1/messages.ndjson new file mode 100644 index 0000000..86c0af4 --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/messages.ndjson @@ -0,0 +1,2 @@ +{"accountId":"account:synthetic:primary","attachments":[],"body":"edited synthetic reply","bodyTruncated":false,"conversationId":"conversation:synthetic:friend","deletion":null,"direction":"outgoing","edit":{"editedAt":"2026-08-21T15:58:30.000Z","kind":"in-place","providerRevision":"edit-r2"},"id":"message:synthetic:edited","kind":"message","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-message:synthetic-edited","providerRevision":"edit-r2"},"replyTo":{"messageId":null,"providerId":"beeper-message:synthetic-external-reply-target"},"schemaVersion":1,"senderParticipantId":"participant:synthetic:self","sentAt":"2026-08-21T15:58:00.000Z","sortKey":"00000000000000000001"} +{"accountId":"account:synthetic:primary","attachments":[],"body":null,"bodyTruncated":false,"conversationId":"conversation:synthetic:friend","deletion":{"observedAt":"2026-08-21T15:59:00.000Z","providerRevision":"delete-r3","state":"revoked"},"direction":"incoming","edit":null,"id":"message:synthetic:deleted","kind":"message","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-message:synthetic-deleted","providerRevision":"delete-r3"},"replyTo":null,"schemaVersion":1,"senderParticipantId":"participant:synthetic:peer","sentAt":"2026-08-21T15:58:45.000Z","sortKey":"00000000000000000002"} diff --git a/src/fixtures/beeper-message-like-me-v1/participants.ndjson b/src/fixtures/beeper-message-like-me-v1/participants.ndjson new file mode 100644 index 0000000..9bf8961 --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/participants.ndjson @@ -0,0 +1,3 @@ +{"accountId":"account:synthetic:primary","displayName":"Synthetic Self","handle":"+15555550100","id":"participant:synthetic:self","isSelf":true,"kind":"participant","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-participant:synthetic-self","providerRevision":null},"schemaVersion":1} +{"accountId":"account:synthetic:primary","displayName":"Synthetic Peer","handle":"+15555550101","id":"participant:synthetic:peer","isSelf":false,"kind":"participant","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-participant:synthetic-peer","providerRevision":null},"schemaVersion":1} +{"accountId":"account:synthetic:secondary","displayName":"Synthetic Secondary Self","handle":"synthetic-secondary@example.invalid","id":"participant:synthetic:secondary-self","isSelf":true,"kind":"participant","network":"synthetic-secondary","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-secondary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-participant:synthetic-secondary-self","providerRevision":null},"schemaVersion":1} diff --git a/src/fixtures/beeper-message-like-me-v1/reactions.ndjson b/src/fixtures/beeper-message-like-me-v1/reactions.ndjson new file mode 100644 index 0000000..37e789a --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/reactions.ndjson @@ -0,0 +1 @@ +{"accountId":"account:synthetic:primary","body":"👍","id":"reaction:synthetic:undated","kind":"reaction","messageId":"message:synthetic:edited","messageProviderId":"beeper-message:synthetic-edited","network":"synthetic","participantId":"participant:synthetic:peer","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-reaction:synthetic-undated","providerRevision":"reaction-r1"},"reactedAt":null,"schemaVersion":1,"state":"active"} diff --git a/src/fixtures/beeper-message-like-me-v1/tombstones.ndjson b/src/fixtures/beeper-message-like-me-v1/tombstones.ndjson new file mode 100644 index 0000000..e3dcebe --- /dev/null +++ b/src/fixtures/beeper-message-like-me-v1/tombstones.ndjson @@ -0,0 +1 @@ +{"accountId":"account:synthetic:primary","deletedAt":"2026-08-21T15:59:00.000Z","entityId":"message:synthetic:deleted","entityKind":"message","entityProviderId":"beeper-message:synthetic-deleted","id":"tombstone:synthetic:message","kind":"tombstone","network":"synthetic","provenance":{"connectedAccountProviderId":"beeper-account:synthetic-primary","observedAt":"2026-08-21T15:59:00.000Z","providerId":"beeper-tombstone:synthetic-message","providerRevision":"delete-r3"},"providerRevision":"delete-r3","schemaVersion":1,"scope":"remote"} diff --git a/src/metrics.test.ts b/src/metrics.test.ts index bdd2c68..4733086 100644 --- a/src/metrics.test.ts +++ b/src/metrics.test.ts @@ -22,13 +22,14 @@ function message( replyTo?: string | null; attachments?: number; retractedAt?: string | null; + conversationId?: string; }> = {}, ): CorpusMessage { return Object.freeze({ id, sourceRowId, sourceGuid: `source-${id}`, - conversationId: CONTACT_ID, + conversationId: options.conversationId ?? CONTACT_ID, sentAt, direction, body, @@ -184,8 +185,13 @@ describe("analyzeContact", () => { total: 2, incoming: 1, outgoing: 1, + unknownDirection: 0, + dated: 2, + undated: 0, outgoingReactionRatio: 0.125, }); + expect(analyzeContact(messages, CORPUS_REVISION, CONTACT_ID, { reactionFacts: [] }).reactions) + .toEqual(metrics.reactions); expect(metrics.surface).toMatchObject({ outgoingTextMessages: 6, lowercaseStartsRatio: 0.833333, @@ -209,6 +215,40 @@ describe("analyzeContact", () => { .toThrow("repeat ID"); }); + test("never joins sessions, bursts, or responses across conversation boundaries", () => { + const messages = [ + message("thread-a-in", 1, "2024-06-01T00:00:00.000Z", "incoming", "Question in A?", { + conversationId: "thread-a", + }), + message("thread-b-out", 2, "2024-06-01T00:00:10.000Z", "outgoing", "Unrelated answer in B", { + conversationId: "thread-b", + }), + message("thread-a-out", 3, "2024-06-01T00:01:00.000Z", "outgoing", "Answer in A", { + conversationId: "thread-a", + }), + message("thread-b-in", 4, "2024-06-01T00:02:00.000Z", "incoming", "Question in B?", { + conversationId: "thread-b", + }), + ] as const; + + const metrics = analyzeContact(messages, CORPUS_REVISION, CONTACT_ID); + + expect(metrics.sessions).toHaveLength(2); + expect(metrics.bursts).toHaveLength(4); + expect(metrics.responses).toHaveLength(1); + expect(metrics.responses[0]).toMatchObject({ + incomingMessageIds: ["thread-a-in"], + outgoingMessageIds: ["thread-a-out"], + latencySeconds: 60, + }); + for (const response of metrics.responses) { + const ids = [...response.incomingMessageIds, ...response.outgoingMessageIds]; + const conversations = new Set(ids.map((id) => + messages.find((candidate) => candidate.id === id)!.conversationId)); + expect(conversations.size).toBe(1); + } + }); + test("returns well-defined empty distributions and rejects invalid bounds", () => { const metrics = analyzeContact([], CORPUS_REVISION, CONTACT_ID); expect(metrics.sessions).toEqual([]); @@ -220,6 +260,9 @@ describe("analyzeContact", () => { total: 0, incoming: 0, outgoing: 0, + unknownDirection: 0, + dated: 0, + undated: 0, outgoingReactionRatio: 0, }); expect(() => analyzeContact([], CORPUS_REVISION, CONTACT_ID, { @@ -228,6 +271,83 @@ describe("analyzeContact", () => { })).toThrow("cannot exceed"); }); + test("counts undated reaction facts without inventing timeline timestamps", () => { + const metrics = analyzeContact([ + message("outgoing-text", 1, "2026-08-21T12:00:00.000Z", "outgoing", "ok"), + ], CORPUS_REVISION, CONTACT_ID, { + reactionFacts: [ + { + id: "reaction-in", + externalId: "external-in", + targetExternalId: "target-1", + conversationId: "conversation_1", + direction: "incoming", + body: "heart", + reactedAt: null, + state: "active", + }, + { + id: "reaction-out", + externalId: "external-out", + targetExternalId: "target-1", + conversationId: "conversation_1", + direction: "outgoing", + body: "heart", + reactedAt: null, + state: "active", + }, + { + id: "reaction-unknown", + externalId: "external-unknown", + targetExternalId: "target-1", + conversationId: "conversation_1", + direction: null, + body: "question", + reactedAt: null, + state: "active", + }, + ], + }); + expect(metrics.reactions).toEqual({ + total: 3, + incoming: 1, + outgoing: 1, + unknownDirection: 1, + dated: 0, + undated: 3, + outgoingReactionRatio: 0.5, + }); + }); + + test("keeps aggregate output fixed-size for many unique maximum-size reaction values", () => { + const reactionFacts = Object.freeze(Array.from({ length: 2_048 }, (_, index) => { + const prefix = `private-reaction-value-${index}:`; + return Object.freeze({ + id: `reaction-${index}`, + externalId: `external-${index}`, + targetExternalId: "target-1", + conversationId: "conversation_1", + direction: index % 3 === 0 ? "incoming" as const + : index % 3 === 1 ? "outgoing" as const + : null, + body: `${prefix}${"x".repeat(8_192 - prefix.length)}`, + reactedAt: null, + state: "active" as const, + }); + })); + expect(Buffer.byteLength(reactionFacts[0]!.body, "utf8")).toBe(8_192); + expect(Buffer.byteLength(reactionFacts.at(-1)!.body, "utf8")).toBe(8_192); + + const metrics = analyzeContact([], CORPUS_REVISION, CONTACT_ID, { reactionFacts }); + expect(metrics.reactions.total).toBe(reactionFacts.length); + expect(metrics.reactions.incoming + metrics.reactions.outgoing + + metrics.reactions.unknownDirection).toBe(reactionFacts.length); + expect(Object.hasOwn(metrics.reactions, "byBody")).toBeFalse(); + const encoded = JSON.stringify(metrics.reactions); + expect(Buffer.byteLength(encoded, "utf8")).toBeLessThan(256); + expect(encoded).not.toContain("private-reaction-value"); + }); + test("excludes retracted and system records from style and tempo evidence", () => { const messages = [ message("eligible-in", 1, "2024-05-01T00:00:00.000Z", "incoming", "Are you coming?"), @@ -278,6 +398,9 @@ describe("analyzeContact", () => { total: 0, incoming: 0, outgoing: 0, + unknownDirection: 0, + dated: 0, + undated: 0, outgoingReactionRatio: 0, }); }); @@ -400,6 +523,8 @@ describe("buildStudyPacket", () => { expect(Object.hasOwn(packet.metrics, "responses")).toBeFalse(); expect(Object.hasOwn(packet.metrics, "corpusRevision")).toBeFalse(); expect(Object.hasOwn(packet.metrics, "contactId")).toBeFalse(); + expect(Object.hasOwn(packet.metrics.reactions, "byBody")).toBeFalse(); + expect(JSON.stringify(packet.metrics.reactions)).not.toContain(':"unknown"'); expect(packet.examples).toHaveLength(2); expect(packet.examples[0]!.messages.map(({ id }) => id)).toEqual(["m01", "m02", "m03", "m04"]); expect(packet.examples[0]!.messages.map(({ offsetSeconds }) => offsetSeconds)).toEqual([0, 10, 70, 90]); diff --git a/src/metrics.ts b/src/metrics.ts index f3eee57..09adc19 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -7,6 +7,7 @@ import { type BurstMetric, type ContactMetrics, type CorpusMessage, + type CorpusReactionFact, type Direction, type EvaluationPromptPacket, type EvaluationReferencePacket, @@ -79,6 +80,7 @@ type StudySelection = Readonly<{ export type AnalyzeContactOptions = Readonly<{ sessionGapSeconds?: number; burstGapSeconds?: number; + reactionFacts?: readonly CorpusReactionFact[]; }>; export type BuildStudyPacketOptions = Readonly<{ @@ -508,16 +510,52 @@ function tempoMetrics(messages: readonly OrderedMessage[], responses: readonly R }); } -function reactionMetrics(messages: readonly OrderedMessage[]): ReactionMetrics { - const reactions = messages.filter(({ message }) => - message.kind === "reaction" && message.retractedAt === null); - const outgoing = reactions.filter(({ message }) => message.direction === "outgoing").length; +function reactionMetrics( + messages: readonly OrderedMessage[], + facts: readonly CorpusReactionFact[] | undefined, +): ReactionMetrics { + const legacy = messages.filter(({ message }) => + message.kind === "reaction" && message.retractedAt === null).map(({ message }) => ({ + id: message.id, + externalId: message.sourceGuid, + targetExternalId: message.replyToSourceGuid ?? message.sourceGuid, + conversationId: message.conversationId, + direction: message.direction, + body: "unknown", + reactedAt: message.sentAt, + state: "active" as const, + })); + const merged = new Map(legacy.map((fact) => [fact.id, fact as CorpusReactionFact])); + for (const fact of facts ?? []) merged.set(fact.id, fact); + const source = [...merged.values()]; + const ids = new Set(); + const reactions = source.filter((fact, index) => { + if ( + typeof fact.id !== "string" + || fact.id.length === 0 + || ids.has(fact.id) + || (fact.direction !== null && fact.direction !== "incoming" && fact.direction !== "outgoing") + || typeof fact.body !== "string" + || (fact.state !== "active" && fact.state !== "removed") + ) throw new Error(`reactionFacts[${index}] is invalid`); + if (fact.reactedAt !== null) canonicalTimestamp(fact.reactedAt, `reactionFacts[${index}].reactedAt`); + ids.add(fact.id); + return fact.state === "active"; + }); + const outgoing = reactions.filter(({ direction }) => direction === "outgoing").length; + const incoming = reactions.filter(({ direction }) => direction === "incoming").length; + const unknownDirection = reactions.length - outgoing - incoming; const outgoingActions = messages.filter(({ message }) => - message.direction === "outgoing" && timelineEligible(message)).length; + message.kind !== "reaction" + && message.direction === "outgoing" + && timelineEligible(message)).length + outgoing; return Object.freeze({ total: reactions.length, - incoming: reactions.length - outgoing, + incoming, outgoing, + unknownDirection, + dated: reactions.filter(({ reactedAt }) => reactedAt !== null).length, + undated: reactions.filter(({ reactedAt }) => reactedAt === null).length, outgoingReactionRatio: ratio(outgoing, outgoingActions), }); } @@ -549,19 +587,44 @@ export function analyzeContact( throw new Error("burstGapSeconds cannot exceed sessionGapSeconds"); } const ordered = orderedMessages(messages); - const sessions = sessionsFor(ordered, corpusRevision, contactId, sessionGapSeconds); - const burstRecords = burstsFor( - ordered, - sessions, - corpusRevision, - contactId, - burstGapSeconds, - ); - const responses = responsesFor( - burstRecords, - corpusRevision, - contactId, - ); + const byConversation = new Map(); + for (const row of ordered) { + const rows = byConversation.get(row.message.conversationId) ?? []; + rows.push(row); + byConversation.set(row.message.conversationId, rows); + } + const sessions: SessionMetric[] = []; + const burstRecords: BurstRecord[] = []; + const responses: ResponseEpisode[] = []; + for (const conversationId of [...byConversation.keys()].sort((left, right) => + left.localeCompare(right, "en-US"))) { + const rows = Object.freeze(byConversation.get(conversationId)!); + const conversationSessions = sessionsFor( + rows, + corpusRevision, + contactId, + sessionGapSeconds, + ); + const conversationBursts = burstsFor( + rows, + conversationSessions, + corpusRevision, + contactId, + burstGapSeconds, + ); + sessions.push(...conversationSessions); + burstRecords.push(...conversationBursts); + responses.push(...responsesFor(conversationBursts, corpusRevision, contactId)); + } + sessions.sort((left, right) => + left.startedAt.localeCompare(right.startedAt, "en-US") + || left.id.localeCompare(right.id, "en-US")); + burstRecords.sort((left, right) => + left.metric.startedAt.localeCompare(right.metric.startedAt, "en-US") + || left.metric.id.localeCompare(right.metric.id, "en-US")); + responses.sort((left, right) => + left.startedAt.localeCompare(right.startedAt, "en-US") + || left.id.localeCompare(right.id, "en-US")); return Object.freeze({ schemaVersion: METRICS_SCHEMA_VERSION, corpusRevision, @@ -575,11 +638,11 @@ export function analyzeContact( message.retractedAt === null && message.kind === "text" && message.body !== null).length, sessionGapSeconds, burstGapSeconds, - sessions, + sessions: Object.freeze(sessions), bursts: Object.freeze(burstRecords.map(({ metric }) => metric)), - responses, + responses: Object.freeze(responses), tempo: tempoMetrics(ordered, responses), - reactions: reactionMetrics(ordered), + reactions: reactionMetrics(ordered, options.reactionFacts), surface: surfaceMetrics(ordered), }); } @@ -870,6 +933,9 @@ function aggregateStudyMetrics(metrics: ContactMetrics): StudyAggregateMetrics { total: metrics.reactions.total, incoming: metrics.reactions.incoming, outgoing: metrics.reactions.outgoing, + unknownDirection: metrics.reactions.unknownDirection, + dated: metrics.reactions.dated, + undated: metrics.reactions.undated, outgoingReactionRatio: metrics.reactions.outgoingReactionRatio, }), surface: Object.freeze({ diff --git a/src/store.test.ts b/src/store.test.ts index 3f8cdb7..71b746c 100644 --- a/src/store.test.ts +++ b/src/store.test.ts @@ -5,9 +5,16 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { parseStyleProfile } from "./profile.ts"; import { contactHandleMatchId, normalizeContactHandle } from "./contacts.ts"; -import { LocalStore } from "./store.ts"; +import { analyzeContact } from "./metrics.ts"; +import { IMESSAGE_SOURCE_ID, LocalStore } from "./store.ts"; import { syntheticProfile, syntheticProfileV2 } from "./test-fixtures.ts"; -import type { ContactsSnapshot, CorpusSnapshot } from "./types.ts"; +import type { + ContactsSnapshot, + CorpusMessage, + CorpusReactionFact, + CorpusSnapshot, + SourceCorpusSnapshot, +} from "./types.ts"; const CONTACTS_TEST_KEY = "synthetic-contacts-store-key-32"; @@ -90,6 +97,110 @@ function snapshotWithLaterMessage(revision: string): CorpusSnapshot { }; } +const BUNDLE_SOURCE_ID = `source_${"7".repeat(64)}`; +const BUNDLE_CONVERSATION_ID = "conversation_bundle_synthetic"; + +function bundleMessage( + id: string, + sourceRowId: number, + sentAt: string, + kind: CorpusMessage["kind"] = "text", +): CorpusMessage { + return { + id: `bundle-message-${id}`, + sourceRowId, + sourceGuid: `provider-message-${id}`, + conversationId: BUNDLE_CONVERSATION_ID, + sentAt, + direction: id === "a" ? "incoming" : "outgoing", + body: kind === "text" ? `Synthetic bundle ${id}.` : null, + bodySource: kind === "text" ? "text" : "unavailable", + kind, + replyToSourceGuid: null, + editedAt: null, + retractedAt: null, + service: "whatsapp", + attachmentCount: 0, + }; +} + +function bundleReaction(state: "active" | "removed" = "active"): CorpusReactionFact { + return { + id: "bundle-reaction-r", + externalId: "provider-reaction-r", + targetExternalId: "provider-message-a", + conversationId: BUNDLE_CONVERSATION_ID, + direction: "outgoing", + body: "heart", + reactedAt: null, + state, + }; +} + +function bundleSnapshot(options: Readonly<{ + revision: string; + generatedAt: string; + messages: readonly CorpusMessage[]; + reactions?: readonly CorpusReactionFact[]; + history?: "bounded" | "complete-current-local"; + deletions?: SourceCorpusSnapshot["deletions"]; + includeConversation?: boolean; + providerSortKeys?: Readonly>; +}>): SourceCorpusSnapshot { + const conversations: SourceCorpusSnapshot["conversations"] = options.includeConversation === false + ? [] + : [{ + id: BUNDLE_CONVERSATION_ID, + sourceKey: "provider-conversation-bundle", + privateLabel: "Synthetic Bundle Contact", + service: "whatsapp", + participantCount: 1, + participantIds: ["participant-bundle-peer"], + privateParticipants: [], + group: false, + }]; + return { + source: { + id: BUNDLE_SOURCE_ID, + kind: "bundle", + provider: "beeper", + network: "whatsapp", + accountId: "synthetic-connected-account", + externalId: "synthetic-connected-account", + revision: options.revision, + generatedAt: options.generatedAt, + producer: { id: "beeper-local", version: "test" }, + coverage: { + history: options.history ?? "bounded", + observedFrom: "2026-08-20T10:00:00.000Z", + observedTo: "2026-08-20T11:00:00.000Z", + kind: options.history === "complete-current-local" ? "complete-current-local" : "bounded-local", + reason: null, + }, + manifestSha256: options.revision, + identity: { synthetic: true }, + warnings: [], + }, + conversations, + conversationProvenance: conversations.map((conversation) => ({ + conversationId: conversation.id, + externalId: "provider-conversation-bundle", + })), + messages: options.messages, + messageProvenance: options.messages.map((message) => ({ + messageId: message.id, + externalId: message.sourceGuid, + providerSortKey: message.kind === "reaction" + ? null + : options.providerSortKeys?.[message.id] ?? message.sourceGuid, + replyToExternalId: message.replyToSourceGuid, + attachments: [], + })), + reactionFacts: options.reactions ?? [], + deletions: options.deletions ?? [], + }; +} + function enrichmentCorpus(revision: string): CorpusSnapshot { const conversation = ( id: string, @@ -270,6 +381,31 @@ function createLegacyV1Store(path: string): void { } } +function createLegacyV2Store(path: string): void { + createLegacyV1Store(path); + const database = new Database(path, { strict: true }); + try { + database.exec(` + ALTER TABLE study_packets ADD COLUMN scope_id TEXT; + ALTER TABLE study_packets ADD COLUMN evidence_revision TEXT; + ALTER TABLE study_packets ADD COLUMN example_ids_json TEXT; + ALTER TABLE study_packets ADD COLUMN evidence_json TEXT; + ALTER TABLE profiles ADD COLUMN scope_id TEXT; + ALTER TABLE profiles ADD COLUMN evidence_revision TEXT; + CREATE TABLE conversation_contact_scopes( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + contact_id TEXT NOT NULL REFERENCES addressbook_contacts(id) ON DELETE CASCADE, + contacts_revision TEXT NOT NULL + ) STRICT; + CREATE INDEX conversation_contact_scopes_lookup + ON conversation_contact_scopes(contact_id,conversation_id); + PRAGMA user_version=2; + `); + } finally { + database.close(); + } +} + function contactsSnapshot(revision: string, label = "Synthetic Friend"): ContactsSnapshot { const handle = (value: string) => { const normalized = normalizeContactHandle(value)!; @@ -358,7 +494,7 @@ describe("local corpus store", () => { store.replaceCorpus(snapshot("d".repeat(64)), "2026-08-21T13:00:00.000Z"); expect(store.profile(profile.contactId)?.state).toBe("current"); expect(store.doctor()).toMatchObject({ - storeSchemaVersion: 2, + storeSchemaVersion: 3, quickCheck: "ok", foreignKeyViolations: 0, }); @@ -536,6 +672,8 @@ describe("local corpus store", () => { messageCount: 4, incomingCount: 2, outgoingCount: 2, + service: null, + services: ["SMS", "iMessage"], }); expect(store.conversation("email-conversation", true)?.id).toBe(personId); expect(store.contactCorpus(personId)?.messages.map(({ conversationId }) => conversationId)) @@ -800,6 +938,384 @@ describe("local corpus store", () => { } }); + test("merges bounded sources, applies explicit suppression, and rejects stale snapshots", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-source-merge-")); + const store = LocalStore.open(join(root, "store.sqlite3")); + const a = bundleMessage("a", 1, "2026-08-20T10:00:00.000Z"); + const b = bundleMessage("b", 2, "2026-08-20T10:01:00.000Z"); + try { + store.replaceCorpus(snapshot("a".repeat(64)), "2026-08-21T12:00:00.000Z"); + const first = bundleSnapshot({ + revision: "1".repeat(64), + generatedAt: "2026-08-21T12:01:00.000Z", + messages: [a, b], + reactions: [bundleReaction()], + }); + store.replaceSources([first], "2026-08-21T12:01:01.000Z"); + expect(store.listSources()).toHaveLength(2); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toMatchObject({ + messages: [{ id: a.id }, { id: b.id }], + reactions: [{ id: "bundle-reaction-r", reactedAt: null, state: "active" }], + }); + + const retained = store.replaceSources([bundleSnapshot({ + revision: "2".repeat(64), + generatedAt: "2026-08-21T12:02:00.000Z", + messages: [a], + reactions: [bundleReaction()], + })], "2026-08-21T12:02:01.000Z"); + expect(retained.sources[0]?.changed).toBeFalse(); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)?.messages.map(({ id }) => id)) + .toEqual([a.id, b.id]); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ conversations: 1, messages: 2 }); + + store.replaceSources([bundleSnapshot({ + revision: "3".repeat(64), + generatedAt: "2026-08-21T12:03:00.000Z", + messages: [a], + reactions: [bundleReaction("removed")], + deletions: [ + { + entityKind: "message", + localEntityId: null, + externalId: b.sourceGuid, + deletedAt: "2026-08-21T12:03:00.000Z", + expectedConversationId: BUNDLE_CONVERSATION_ID, + reason: "tombstone", + }, + { + entityKind: "reaction", + localEntityId: "bundle-reaction-r", + externalId: "provider-reaction-r", + deletedAt: "2026-08-21T12:03:00.000Z", + expectedConversationId: BUNDLE_CONVERSATION_ID, + reason: "tombstone", + }, + ], + })], "2026-08-21T12:03:01.000Z"); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toMatchObject({ + messages: [{ id: a.id }], + reactions: [], + }); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ conversations: 1, messages: 1 }); + + const reappeared = bundleSnapshot({ + revision: "4".repeat(64), + generatedAt: "2026-08-21T12:04:00.000Z", + messages: [a, b], + reactions: [bundleReaction()], + }); + store.replaceSources([reappeared], "2026-08-21T12:04:01.000Z"); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toMatchObject({ + messages: [{ id: a.id }, { id: b.id }], + reactions: [{ id: "bundle-reaction-r", state: "active" }], + }); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ conversations: 1, messages: 2 }); + expect(() => store.replaceSources([first], "2026-08-21T12:05:00.000Z")) + .toThrow("older than stored state"); + expect(() => store.replaceSources([bundleSnapshot({ + ...reappeared, + revision: "5".repeat(64), + generatedAt: "2026-08-21T12:04:00.000Z", + messages: [a, b], + })], "2026-08-21T12:05:00.000Z")).toThrow("reuses generatedAt"); + expect(store.replaceSources([reappeared], "2026-08-21T12:05:00.000Z").sources[0]) + .toMatchObject({ changed: false, conversations: 1, messages: 2 }); + } finally { + store.close(); + await rm(root, { recursive: true, force: true }); + } + }); + + test("converges provider order across bounded backfill, omission, and replay", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-provider-order-")); + const incremental = LocalStore.open(join(root, "incremental.sqlite3")); + const fresh = LocalStore.open(join(root, "fresh.sqlite3")); + const omitted = LocalStore.open(join(root, "omitted.sqlite3")); + const sentAt = "2026-08-20T10:00:00.000Z"; + const incoming = bundleMessage("a", 1, sentAt); + const outgoing = bundleMessage("b", 2, sentAt); + const first = bundleSnapshot({ + revision: "1".repeat(64), + generatedAt: "2026-08-21T12:01:00.000Z", + messages: [outgoing], + providerSortKeys: { [outgoing.id]: "b" }, + }); + const complete = bundleSnapshot({ + revision: "2".repeat(64), + generatedAt: "2026-08-21T12:02:00.000Z", + messages: [incoming, outgoing], + providerSortKeys: { [incoming.id]: "a", [outgoing.id]: "b" }, + }); + try { + incremental.replaceSources([first], "2026-08-21T12:01:01.000Z"); + incremental.replaceSources([complete], "2026-08-21T12:02:01.000Z"); + fresh.replaceSources([complete], "2026-08-21T12:02:01.000Z"); + + const incrementalCorpus = incremental.contactCorpus(BUNDLE_CONVERSATION_ID)!; + const freshCorpus = fresh.contactCorpus(BUNDLE_CONVERSATION_ID)!; + expect(incrementalCorpus.messages.map(({ id }) => id)).toEqual([incoming.id, outgoing.id]); + expect(incrementalCorpus.messages).toEqual(freshCorpus.messages); + expect(incrementalCorpus.evidenceRevision).toBe(freshCorpus.evidenceRevision); + expect(incremental.source(BUNDLE_SOURCE_ID)?.revision) + .toBe(fresh.source(BUNDLE_SOURCE_ID)?.revision); + expect(incremental.corpusRevision()).toBe(fresh.corpusRevision()); + expect(analyzeContact( + incrementalCorpus.messages, + incrementalCorpus.corpusRevision, + BUNDLE_CONVERSATION_ID, + { reactionFacts: incrementalCorpus.reactions }, + ).tempo.responseEpisodes).toBe(1); + + const stableRevision = incremental.corpusRevision(); + expect(incremental.replaceSources([complete], "2026-08-21T12:03:00.000Z").sources[0]) + .toMatchObject({ changed: false, conversations: 1, messages: 2 }); + expect(incremental.corpusRevision()).toBe(stableRevision); + + omitted.replaceSources([first], "2026-08-21T12:01:01.000Z"); + omitted.replaceSources([bundleSnapshot({ + revision: "3".repeat(64), + generatedAt: "2026-08-21T12:03:00.000Z", + messages: [incoming], + providerSortKeys: { [incoming.id]: "a" }, + })], "2026-08-21T12:03:01.000Z"); + const omittedCorpus = omitted.contactCorpus(BUNDLE_CONVERSATION_ID)!; + expect(omittedCorpus.messages.map(({ id }) => id)).toEqual([incoming.id, outgoing.id]); + expect(omittedCorpus.messages).toEqual(freshCorpus.messages); + expect(omittedCorpus.evidenceRevision).toBe(freshCorpus.evidenceRevision); + expect(omitted.source(BUNDLE_SOURCE_ID)?.revision) + .toBe(fresh.source(BUNDLE_SOURCE_ID)?.revision); + expect(omitted.corpusRevision()).toBe(fresh.corpusRevision()); + } finally { + omitted.close(); + fresh.close(); + incremental.close(); + await rm(root, { recursive: true, force: true }); + } + }); + + test("clears external-only conversation, message, and reaction tombstones on reappearance", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-external-reappearance-")); + const incremental = LocalStore.open(join(root, "incremental.sqlite3")); + const fresh = LocalStore.open(join(root, "fresh.sqlite3")); + const message = bundleMessage("a", 1, "2026-08-20T10:00:00.000Z"); + const tombstones = bundleSnapshot({ + revision: "1".repeat(64), + generatedAt: "2026-08-21T12:01:00.000Z", + includeConversation: false, + messages: [], + deletions: [ + { + entityKind: "conversation", + localEntityId: null, + externalId: "provider-conversation-bundle", + deletedAt: "2026-08-21T12:01:00.000Z", + reason: "tombstone", + }, + { + entityKind: "message", + localEntityId: null, + externalId: message.sourceGuid, + deletedAt: "2026-08-21T12:01:00.000Z", + reason: "tombstone", + }, + { + entityKind: "reaction", + localEntityId: null, + externalId: "provider-reaction-r", + deletedAt: "2026-08-21T12:01:00.000Z", + reason: "tombstone", + }, + ], + }); + const active = bundleSnapshot({ + revision: "2".repeat(64), + generatedAt: "2026-08-21T12:02:00.000Z", + messages: [message], + reactions: [bundleReaction()], + }); + try { + incremental.replaceSources([tombstones], "2026-08-21T12:01:01.000Z"); + expect(incremental.source(BUNDLE_SOURCE_ID)).toMatchObject({ + conversations: 0, + messages: 0, + reactions: 0, + }); + incremental.replaceSources([active], "2026-08-21T12:02:01.000Z"); + fresh.replaceSources([active], "2026-08-21T12:02:01.000Z"); + + const incrementalCorpus = incremental.contactCorpus(BUNDLE_CONVERSATION_ID)!; + const freshCorpus = fresh.contactCorpus(BUNDLE_CONVERSATION_ID)!; + expect(incrementalCorpus).toMatchObject({ + messages: [{ id: message.id }], + reactions: [{ id: "bundle-reaction-r" }], + }); + expect(incrementalCorpus.messages).toEqual(freshCorpus.messages); + expect(incrementalCorpus.reactions).toEqual(freshCorpus.reactions); + expect(incrementalCorpus.evidenceRevision).toBe(freshCorpus.evidenceRevision); + expect(incremental.source(BUNDLE_SOURCE_ID)?.revision) + .toBe(fresh.source(BUNDLE_SOURCE_ID)?.revision); + expect(incremental.corpusRevision()).toBe(fresh.corpusRevision()); + expect(incremental.replaceSources([active], "2026-08-21T12:03:00.000Z").sources[0]?.changed) + .toBeFalse(); + } finally { + fresh.close(); + incremental.close(); + await rm(root, { recursive: true, force: true }); + } + }); + + test("reports native iMessage reactions consistently with contact metrics", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-imessage-reaction-health-")); + const store = LocalStore.open(join(root, "store.sqlite3")); + const base = snapshot("a".repeat(64)); + const reaction: CorpusMessage = { + ...base.messages[1]!, + id: "message_reaction", + sourceRowId: 3, + sourceGuid: "synthetic-reaction-guid", + sentAt: "2026-08-21T11:02:00.000Z", + body: null, + bodySource: "unavailable", + kind: "reaction", + replyToSourceGuid: base.messages[0]!.sourceGuid, + }; + try { + store.replaceCorpus({ ...base, messages: [...base.messages, reaction] }, "2026-08-21T12:00:00.000Z"); + expect(store.source(IMESSAGE_SOURCE_ID)).toMatchObject({ + conversations: 1, + messages: 3, + reactions: 1, + undatedReactions: 0, + }); + const corpus = store.contactCorpus(base.conversations[0]!.id)!; + const metrics = analyzeContact( + corpus.messages, + corpus.corpusRevision, + base.conversations[0]!.id, + { reactionFacts: corpus.reactions }, + ); + expect(metrics.reactions).toMatchObject({ + total: 1, + incoming: 0, + outgoing: 1, + dated: 1, + undated: 0, + }); + } finally { + store.close(); + await rm(root, { recursive: true, force: true }); + } + }); + + test("hides source reaction health with a conversation tombstone and restores it on reappearance", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-source-conversation-suppression-")); + const store = LocalStore.open(join(root, "store.sqlite3")); + const message = bundleMessage("a", 1, "2026-08-20T10:00:00.000Z"); + try { + store.replaceSources([bundleSnapshot({ + revision: "1".repeat(64), + generatedAt: "2026-08-21T12:01:00.000Z", + messages: [message], + reactions: [bundleReaction()], + })], "2026-08-21T12:01:01.000Z"); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ + conversations: 1, + messages: 1, + reactions: 1, + undatedReactions: 1, + }); + + store.replaceSources([bundleSnapshot({ + revision: "2".repeat(64), + generatedAt: "2026-08-21T12:02:00.000Z", + messages: [message], + reactions: [bundleReaction()], + deletions: [{ + entityKind: "conversation", + localEntityId: BUNDLE_CONVERSATION_ID, + externalId: "provider-conversation-bundle", + deletedAt: "2026-08-21T12:02:00.000Z", + reason: "tombstone", + }], + })], "2026-08-21T12:02:01.000Z"); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ + conversations: 0, + messages: 0, + reactions: 0, + undatedReactions: 0, + }); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toBeNull(); + + store.replaceSources([bundleSnapshot({ + revision: "3".repeat(64), + generatedAt: "2026-08-21T12:03:00.000Z", + messages: [message], + reactions: [bundleReaction()], + })], "2026-08-21T12:03:01.000Z"); + expect(store.source(BUNDLE_SOURCE_ID)).toMatchObject({ + conversations: 1, + messages: 1, + reactions: 1, + undatedReactions: 1, + }); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toMatchObject({ + messages: [{ id: message.id }], + reactions: [{ id: "bundle-reaction-r" }], + }); + } finally { + store.close(); + await rm(root, { recursive: true, force: true }); + } + }); + + test("authoritative reaction absence and reappearance affect only that source", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-reaction-reappearance-")); + const store = LocalStore.open(join(root, "store.sqlite3")); + const a = bundleMessage("a", 1, "2026-08-20T10:00:00.000Z"); + const reactionMessage = { + ...bundleMessage("reaction-r", 2, "2026-08-20T10:00:30.000Z", "reaction"), + sourceGuid: "provider-reaction-r", + }; + const reaction = { ...bundleReaction(), id: reactionMessage.id }; + try { + store.replaceSources([bundleSnapshot({ + revision: "1".repeat(64), + generatedAt: "2026-08-21T12:01:00.000Z", + messages: [a, reactionMessage], + reactions: [reaction], + history: "complete-current-local", + })], "2026-08-21T12:01:01.000Z"); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)?.reactions).toHaveLength(1); + + store.replaceSources([bundleSnapshot({ + revision: "2".repeat(64), + generatedAt: "2026-08-21T12:02:00.000Z", + messages: [a], + reactions: [], + history: "complete-current-local", + })], "2026-08-21T12:02:01.000Z"); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)).toMatchObject({ + messages: [{ id: a.id }], + reactions: [], + }); + + store.replaceSources([bundleSnapshot({ + revision: "3".repeat(64), + generatedAt: "2026-08-21T12:03:00.000Z", + messages: [a, reactionMessage], + reactions: [reaction], + history: "complete-current-local", + })], "2026-08-21T12:03:01.000Z"); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)?.reactions).toHaveLength(1); + expect(store.contactCorpus(BUNDLE_CONVERSATION_ID)?.messages.map(({ id }) => id)) + .toEqual([a.id, reactionMessage.id]); + } finally { + store.close(); + await rm(root, { recursive: true, force: true }); + } + }); + test("keeps a person profile current across unrelated changes and stales relevant evidence", async () => { const root = await mkdtemp(join(tmpdir(), "message-like-me-scope-revision-")); const store = LocalStore.open(join(root, "store.sqlite3")); @@ -934,7 +1450,7 @@ describe("local corpus store", () => { createLegacyV1Store(path); const store = LocalStore.open(path); try { - expect(store.doctor()).toMatchObject({ storeSchemaVersion: 2, profiles: 1 }); + expect(store.doctor()).toMatchObject({ storeSchemaVersion: 3, profiles: 1 }); expect(store.profile("contact_0123456789abcdef")).toMatchObject({ state: "current", profile: { schemaVersion: 1, corpusRevision: "a".repeat(64) }, @@ -945,7 +1461,7 @@ describe("local corpus store", () => { const migrated = new Database(path, { strict: true }); try { - expect(migrated.query("PRAGMA user_version").get()).toEqual({ user_version: 2 }); + expect(migrated.query("PRAGMA user_version").get()).toEqual({ user_version: 3 }); const profileColumns = migrated.query("PRAGMA table_info(profiles)").all() as Array<{ name: string }>; expect(profileColumns.map(({ name }) => name)).toContain("scope_id"); expect(profileColumns.map(({ name }) => name)).toContain("evidence_revision"); @@ -961,4 +1477,50 @@ describe("local corpus store", () => { await rm(root, { recursive: true, force: true }); } }); + + test("upgrades a populated v0.2 store in place without rebuilding evidence rows", async () => { + const root = await mkdtemp(join(tmpdir(), "message-like-me-v2-upgrade-")); + const path = join(root, "store.sqlite3"); + createLegacyV2Store(path); + const store = LocalStore.open(path); + try { + expect(store.doctor()).toMatchObject({ + storeSchemaVersion: 3, + conversations: 1, + messages: 1, + profiles: 1, + sources: 1, + }); + expect(store.corpusRevision()).toBe("a".repeat(64)); + expect(store.profile("contact_0123456789abcdef")).toMatchObject({ + state: "current", + profile: { schemaVersion: 1, corpusRevision: "a".repeat(64) }, + }); + } finally { + store.close(); + } + const migrated = new Database(path, { readonly: true, strict: true }); + try { + expect(migrated.query("PRAGMA user_version").get()).toEqual({ user_version: 3 }); + expect(migrated.query("SELECT count(*) AS value FROM conversations").get()) + .toEqual({ value: 1 }); + expect(migrated.query("SELECT count(*) AS value FROM messages").get()) + .toEqual({ value: 1 }); + expect(migrated.query("SELECT count(*) AS value FROM profiles").get()) + .toEqual({ value: 1 }); + expect(migrated.query("SELECT count(*) AS value FROM study_packets").get()) + .toEqual({ value: 1 }); + expect(migrated.query(`SELECT source_id,external_id FROM conversation_sources`).get()) + .toEqual({ source_id: "source_imessage_local", external_id: "legacy-conversation" }); + expect(migrated.query(`SELECT source_id,external_id FROM message_provenance`).get()) + .toEqual({ source_id: "source_imessage_local", external_id: "legacy-guid" }); + expect(migrated.query(`SELECT evidence_revision IS NOT NULL AS value FROM profiles`).get()) + .toEqual({ value: 1 }); + expect(migrated.query(`SELECT evidence_revision IS NOT NULL AS value FROM study_packets`).get()) + .toEqual({ value: 1 }); + } finally { + migrated.close(); + await rm(root, { recursive: true, force: true }); + } + }); }); diff --git a/src/store.ts b/src/store.ts index f0f3d58..b988bdc 100644 --- a/src/store.ts +++ b/src/store.ts @@ -1,4 +1,5 @@ import { Database } from "bun:sqlite"; +import { createHash } from "node:crypto"; import { closeSync, constants as fsConstants, @@ -20,8 +21,11 @@ import type { ContactSummary, CorpusConversation, CorpusMessage, + CorpusReactionFact, CorpusSnapshot, + CorpusSourceDescriptor, ProfileEvidenceV2, + SourceCorpusSnapshot, StyleProfile, StyleProfileV2, } from "./types.ts"; @@ -29,8 +33,9 @@ import type { type Binding = string | number | bigint | Uint8Array | null; type Row = Record; -const STORE_SCHEMA_VERSION = 2; +const STORE_SCHEMA_VERSION = 3; const PERSON_SCOPE_PREFIX = "person_"; +export const IMESSAGE_SOURCE_ID = "source_imessage_local"; const SCHEMA = ` PRAGMA foreign_keys = ON; @@ -38,6 +43,23 @@ const SCHEMA = ` key TEXT PRIMARY KEY, value TEXT NOT NULL ) STRICT; + CREATE TABLE IF NOT EXISTS corpus_sources ( + id TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('imessage', 'bundle')), + provider TEXT NOT NULL, + network TEXT, + account_id TEXT, + external_id TEXT NOT NULL, + input_revision TEXT NOT NULL, + revision TEXT NOT NULL, + generated_at TEXT, + producer_json TEXT NOT NULL, + coverage_json TEXT NOT NULL, + manifest_sha256 TEXT, + identity_json TEXT NOT NULL, + warnings_json TEXT NOT NULL, + ingested_at TEXT NOT NULL + ) STRICT; CREATE TABLE IF NOT EXISTS conversations ( id TEXT PRIMARY KEY, source_key TEXT NOT NULL, @@ -48,6 +70,15 @@ const SCHEMA = ` private_participants_json TEXT NOT NULL, is_group INTEGER NOT NULL CHECK (is_group IN (0, 1)) ) STRICT; + CREATE TABLE IF NOT EXISTS conversation_sources ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS conversation_sources_lookup + ON conversation_sources(source_id, conversation_id); CREATE TABLE IF NOT EXISTS messages ( id TEXT PRIMARY KEY, source_row_id INTEGER NOT NULL, @@ -68,6 +99,54 @@ const SCHEMA = ` CREATE INDEX IF NOT EXISTS messages_conversation_time ON messages(conversation_id, sent_at, source_row_id, id); CREATE INDEX IF NOT EXISTS messages_source_guid ON messages(source_guid); + CREATE TABLE IF NOT EXISTS message_provenance ( + message_id TEXT PRIMARY KEY REFERENCES messages(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + reply_to_external_id TEXT, + attachments_json TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS message_provenance_source + ON message_provenance(source_id, message_id); + CREATE TABLE IF NOT EXISTS corpus_reaction_facts ( + id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + external_id TEXT NOT NULL, + target_external_id TEXT NOT NULL, + conversation_id TEXT REFERENCES conversations(id) ON DELETE SET NULL, + direction TEXT CHECK (direction IN ('incoming','outgoing')), + body TEXT NOT NULL, + reacted_at TEXT, + state TEXT NOT NULL CHECK (state IN ('active','removed')), + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS corpus_reaction_facts_source + ON corpus_reaction_facts(source_id,conversation_id,id); + CREATE TABLE IF NOT EXISTS corpus_source_records ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('account','participant','reaction','tombstone','excluded-message') + ), + external_id TEXT NOT NULL, + record_json TEXT NOT NULL, + PRIMARY KEY (source_id, kind, external_id) + ) WITHOUT ROWID, STRICT; + CREATE TABLE IF NOT EXISTS corpus_source_suppressions ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('conversation','message','reaction','reaction-timeline','participant','account') + ), + local_id TEXT NOT NULL, + external_id TEXT NOT NULL, + suppressed_at TEXT NOT NULL, + reason TEXT NOT NULL CHECK ( + reason IN ('authoritative-absence','tombstone','explicit-exclusion','replacement','reappeared') + ), + suppressed INTEGER NOT NULL CHECK (suppressed IN (0,1)), + PRIMARY KEY (source_id, kind, local_id) + ) WITHOUT ROWID, STRICT; CREATE TABLE IF NOT EXISTS study_packets ( sha256 TEXT PRIMARY KEY, contact_id TEXT NOT NULL, @@ -125,6 +204,83 @@ const SCHEMA = ` ON conversation_contact_labels(normalized_label, conversation_id); `; +const SOURCE_SCHEMA = ` + CREATE TABLE IF NOT EXISTS corpus_sources ( + id TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('imessage', 'bundle')), + provider TEXT NOT NULL, + network TEXT, + account_id TEXT, + external_id TEXT NOT NULL, + input_revision TEXT NOT NULL, + revision TEXT NOT NULL, + generated_at TEXT, + producer_json TEXT NOT NULL, + coverage_json TEXT NOT NULL, + manifest_sha256 TEXT, + identity_json TEXT NOT NULL, + warnings_json TEXT NOT NULL, + ingested_at TEXT NOT NULL + ) STRICT; + CREATE TABLE IF NOT EXISTS conversation_sources ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS conversation_sources_lookup + ON conversation_sources(source_id, conversation_id); + CREATE TABLE IF NOT EXISTS message_provenance ( + message_id TEXT PRIMARY KEY REFERENCES messages(id) ON DELETE CASCADE, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE RESTRICT, + external_id TEXT NOT NULL, + reply_to_external_id TEXT, + attachments_json TEXT NOT NULL, + metadata_json TEXT NOT NULL, + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS message_provenance_source + ON message_provenance(source_id, message_id); + CREATE TABLE IF NOT EXISTS corpus_reaction_facts ( + id TEXT PRIMARY KEY, + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + external_id TEXT NOT NULL, + target_external_id TEXT NOT NULL, + conversation_id TEXT REFERENCES conversations(id) ON DELETE SET NULL, + direction TEXT CHECK (direction IN ('incoming','outgoing')), + body TEXT NOT NULL, + reacted_at TEXT, + state TEXT NOT NULL CHECK (state IN ('active','removed')), + UNIQUE (source_id, external_id) + ) STRICT; + CREATE INDEX IF NOT EXISTS corpus_reaction_facts_source + ON corpus_reaction_facts(source_id,conversation_id,id); + CREATE TABLE IF NOT EXISTS corpus_source_records ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('account','participant','reaction','tombstone','excluded-message') + ), + external_id TEXT NOT NULL, + record_json TEXT NOT NULL, + PRIMARY KEY (source_id, kind, external_id) + ) WITHOUT ROWID, STRICT; + CREATE TABLE IF NOT EXISTS corpus_source_suppressions ( + source_id TEXT NOT NULL REFERENCES corpus_sources(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK ( + kind IN ('conversation','message','reaction','reaction-timeline','participant','account') + ), + local_id TEXT NOT NULL, + external_id TEXT NOT NULL, + suppressed_at TEXT NOT NULL, + reason TEXT NOT NULL CHECK ( + reason IN ('authoritative-absence','tombstone','explicit-exclusion','replacement','reappeared') + ), + suppressed INTEGER NOT NULL CHECK (suppressed IN (0,1)), + PRIMARY KEY (source_id, kind, local_id) + ) WITHOUT ROWID, STRICT; +`; + const CONTACT_SCOPE_SCHEMA = ` CREATE TABLE IF NOT EXISTS conversation_contact_scopes ( conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, @@ -413,7 +569,15 @@ function personScope( SELECT association.conversation_id FROM conversation_contact_scopes association JOIN conversations conversation ON conversation.id=association.conversation_id + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id WHERE association.contact_id=? AND conversation.is_group=0 + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) ORDER BY association.conversation_id `, addressBookContactId); if (rows.length === 0) return null; @@ -434,7 +598,15 @@ function analysisScope(database: Database, contactId: string): AnalysisScope | n } const conversation = get<{ id: string }>( database, - "SELECT id FROM conversations WHERE id=?", + `SELECT conversation.id FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id + WHERE conversation.id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + )`, contactId, ); if (conversation === null) return null; @@ -458,20 +630,37 @@ function messageRowsForScope( ): StoredMessageRow[] { if (exactConversationId !== undefined) { return all(database, ` - SELECT * FROM messages WHERE conversation_id=? - AND (? IS NULL OR sent_at>=?) AND (? IS NULL OR sent_at=?) AND (? IS NULL OR message.sent_at(database, ` SELECT message.* FROM messages message + JOIN message_provenance provenance ON provenance.message_id=message.id JOIN conversation_contact_scopes association ON association.conversation_id=message.conversation_id WHERE association.contact_id=? AND (? IS NULL OR message.sent_at>=?) AND (? IS NULL OR message.sent_at(database, ` - SELECT * FROM messages WHERE conversation_id=? - AND (? IS NULL OR sent_at>=?) AND (? IS NULL OR sent_at=?) AND (? IS NULL OR message.sent_at; + +function reactionFactsForScope( + database: Database, + scope: AnalysisScope, + window: EvidenceWindow = UNBOUNDED_EVIDENCE_WINDOW, +): CorpusReactionFact[] { + const select = `SELECT reaction.id,reaction.external_id,reaction.target_external_id, + reaction.conversation_id,reaction.direction,reaction.body,reaction.reacted_at,reaction.state + FROM corpus_reaction_facts reaction`; + const suppression = `NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=reaction.source_id + AND suppression.kind='reaction' + AND suppression.local_id=reaction.id + AND suppression.suppressed=1 + )`; + const rows = scope.kind === "person" + ? all(database, `${select} + JOIN conversation_contact_scopes association + ON association.conversation_id=reaction.conversation_id + WHERE association.contact_id=? AND reaction.state='active' AND ${suppression} + ORDER BY reaction.reacted_at IS NULL,reaction.reacted_at,reaction.id`, scope.addressBookContactId) + : all(database, `${select} + WHERE reaction.conversation_id=? AND reaction.state='active' AND ${suppression} + ORDER BY reaction.reacted_at IS NULL,reaction.reacted_at,reaction.id`, scope.conversationIds[0]!); + return rows.filter((row) => row.reacted_at === null + ? window.after === null && window.before === null + : (window.after === null || row.reacted_at >= window.after) + && (window.before === null || row.reacted_at < window.before)).map((row) => ({ + id: row.id, + externalId: row.external_id, + targetExternalId: row.target_external_id, + conversationId: row.conversation_id, + direction: row.direction, + body: row.body, + reactedAt: row.reacted_at, + state: row.state, + })); +} + function scopeEvidenceRevision( database: Database, scope: AnalysisScope, @@ -521,8 +769,18 @@ function scopeEvidenceRevision( ? scope.conversationIds : Object.freeze([exactConversationId]); const messages = messageRowsForScope(database, scope, exactConversationId, window).map(corpusMessage); + const reactions = reactionFactsForScope(database, scope, window); return sha256(canonicalJson( - window.after === null && window.before === null + reactions.length > 0 + ? { + schemaVersion: 3, + scopeId: scope.id, + conversationIds, + evidenceWindow: window, + messages, + reactions, + } + : window.after === null && window.before === null ? { schemaVersion: 1, scopeId: scope.id, @@ -577,16 +835,32 @@ function scopeMessageCounts(database: Database, scope: AnalysisScope): Readonly< outgoing_count: number; }>(database, `${select} FROM messages message + JOIN message_provenance provenance ON provenance.message_id=message.id JOIN conversation_contact_scopes association ON association.conversation_id=message.conversation_id - WHERE association.contact_id=?`, scope.addressBookContactId) + WHERE association.contact_id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + )`, scope.addressBookContactId) : get<{ first_message_at: string | null; last_message_at: string | null; message_count: number; incoming_count: number; outgoing_count: number; - }>(database, `${select} FROM messages message WHERE message.conversation_id=?`, scope.conversationIds[0]!); + }>(database, `${select} + FROM messages message + JOIN message_provenance provenance ON provenance.message_id=message.id + WHERE message.conversation_id=? AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + )`, scope.conversationIds[0]!); return { firstMessageAt: row?.first_message_at ?? null, lastMessageAt: row?.last_message_at ?? null, @@ -637,6 +911,78 @@ function backfillLegacyEvidence(database: Database): void { } } +function backfillLegacySource(database: Database): void { + const conversations = get<{ value: number }>( + database, + "SELECT count(*) AS value FROM conversations", + )?.value ?? 0; + const assigned = get<{ value: number }>( + database, + "SELECT count(*) AS value FROM conversation_sources", + )?.value ?? 0; + if (assigned !== 0 && assigned !== conversations) { + throw new CliError("invalid-data", "Local store has partially assigned corpus source ownership"); + } + if (conversations === 0 || assigned === conversations) return; + const revision = scalarText(database, "corpus_revision"); + if (revision === null || !/^[a-f0-9]{64}$/u.test(revision)) { + throw new CliError("invalid-data", "Legacy local store has no valid corpus revision"); + } + const identity = scalarText(database, "source_identity") ?? canonicalJson({ migrated: true }); + const warnings = scalarText(database, "warnings") ?? canonicalJson([]); + const ingestedAt = scalarText(database, "ingested_at") ?? "1970-01-01T00:00:00.000Z"; + database.query(` + INSERT INTO corpus_sources( + id,kind,provider,network,account_id,external_id,input_revision,revision,generated_at, + producer_json,coverage_json,manifest_sha256,identity_json,warnings_json,ingested_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + `).run( + IMESSAGE_SOURCE_ID, + "imessage", + "apple", + null, + null, + "local-imessage", + revision, + revision, + null, + canonicalJson({ id: "message-like-me", version: "legacy" }), + canonicalJson({ history: "complete-current-local", observedFrom: null, observedTo: null }), + null, + identity, + warnings, + ingestedAt, + ); + database.exec(` + INSERT INTO conversation_sources(conversation_id,source_id,external_id,metadata_json) + SELECT id,'${IMESSAGE_SOURCE_ID}',source_key,'{}' FROM conversations; + `); + const rows = all<{ + id: string; + source_guid: string; + reply_to_source_guid: string | null; + attachment_count: number; + }>(database, ` + SELECT id,source_guid,reply_to_source_guid,attachment_count + FROM messages ORDER BY id + `); + const insert = database.query(` + INSERT INTO message_provenance( + message_id,source_id,external_id,reply_to_external_id,attachments_json,metadata_json + ) VALUES (?,?,?,?,?,?) + `); + for (const row of rows) { + insert.run( + row.id, + IMESSAGE_SOURCE_ID, + row.source_guid, + row.reply_to_source_guid, + canonicalJson({ count: row.attachment_count, detailsAvailable: false }), + canonicalJson({ migrated: true }), + ); + } +} + function initializeStoreSchema(database: Database): void { const existingStore = tableExists(database, "metadata"); const version = userVersion(database); @@ -656,6 +1002,7 @@ function initializeStoreSchema(database: Database): void { throw new CliError("invalid-data", `Local store is missing required table ${table}`); } } + database.exec(SOURCE_SCHEMA); transaction(database, () => { database.exec(CONTACT_SCOPE_SCHEMA); database.exec(` @@ -671,6 +1018,7 @@ function initializeStoreSchema(database: Database): void { addColumn(database, "study_packets", "evidence_json TEXT"); addColumn(database, "profiles", "scope_id TEXT"); addColumn(database, "profiles", "evidence_revision TEXT"); + backfillLegacySource(database); backfillLegacyEvidence(database); database.exec(`PRAGMA user_version=${STORE_SCHEMA_VERSION}`); }); @@ -713,7 +1061,17 @@ function rebuildConversationLabels( owners.set(key, values); } const conversations = all<{ id: string; private_participants_json: string }>(database, ` - SELECT id,private_participants_json FROM conversations WHERE is_group=0 ORDER BY id + SELECT conversation.id,conversation.private_participants_json + FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id + WHERE conversation.is_group=0 AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) + ORDER BY conversation.id `); const insertScope = database.query(`INSERT INTO conversation_contact_scopes( conversation_id,contact_id,contacts_revision @@ -849,6 +1207,329 @@ function hardenDatabaseFiles(path: string): void { } } +function globalCorpusRevision(database: Database): string | null { + const sources = all<{ + id: string; + kind: "imessage" | "bundle"; + input_revision: string; + revision: string; + }>(database, "SELECT id,kind,input_revision,revision FROM corpus_sources ORDER BY id"); + if (sources.length === 0) return null; + if ( + sources.length === 1 + && sources[0]!.id === IMESSAGE_SOURCE_ID + && sources[0]!.kind === "imessage" + ) return sources[0]!.input_revision; + return sha256(canonicalJson({ + schemaVersion: 1, + sources: sources.map(({ id, kind, revision }) => ({ id, kind, revision })), + })); +} + +function sourceStateRevision(database: Database, sourceId: string): string { + const hash = createHash("sha256"); + hash.update("message-like-me\0stored-source-state-v1\0", "utf8"); + const append = (kind: string, row: Row): void => { + const encoded = canonicalJson(row); + hash.update(`${kind.length}:${kind}${encoded.length}:`, "utf8").update(encoded, "utf8"); + }; + const source = get(database, ` + SELECT kind,provider,network,account_id,external_id,producer_json, + coverage_json,warnings_json + FROM corpus_sources WHERE id=? + `, sourceId); + if (source === null) throw new CliError("internal", `Missing corpus source ${sourceId}`); + append("source", source); + for (const row of database.query(` + SELECT conversation.id,conversation.source_key,conversation.private_label, + conversation.service,conversation.participant_count, + conversation.participant_ids_json,conversation.private_participants_json, + conversation.is_group + FROM conversation_sources ownership + JOIN conversations conversation ON conversation.id=ownership.conversation_id + WHERE ownership.source_id=? + ORDER BY ownership.external_id,conversation.id + `).iterate(sourceId) as Iterable) append("conversation", row); + for (const row of database.query(` + SELECT message.id,message.source_row_id,message.source_guid,message.conversation_id, + message.sent_at,message.direction,message.body,message.body_source,message.kind, + message.reply_to_source_guid,message.edited_at,message.retracted_at,message.service, + message.attachment_count,provenance.external_id, + provenance.reply_to_external_id,provenance.attachments_json + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + ORDER BY provenance.external_id,message.id + `).iterate(sourceId) as Iterable) append("message", row); + for (const row of database.query(` + SELECT id,external_id,target_external_id,conversation_id,direction,body,reacted_at,state + FROM corpus_reaction_facts WHERE source_id=? ORDER BY external_id,id + `).iterate(sourceId) as Iterable) append("reaction-fact", row); + for (const row of database.query(` + SELECT kind,local_id,external_id,reason FROM corpus_source_suppressions + WHERE source_id=? AND suppressed=1 ORDER BY kind,local_id + `).iterate(sourceId) as Iterable) append("suppression", row); + return hash.digest("hex"); +} + +type BundleMessageOrderRow = Readonly<{ + id: string; + conversation_id: string; + sent_at: string; + kind: CorpusMessage["kind"]; + external_id: string; + metadata_json: string; +}>; + +type RankedBundleMessageOrderRow = BundleMessageOrderRow & Readonly<{ + provider_sort_key: string | null; +}>; + +function compareCodeUnits(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0; +} + +function storedProviderSortKey(row: BundleMessageOrderRow): string | null { + const parsed = parsedJson(row.metadata_json, `Message ${row.id} provenance`); + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return null; + const record = parsed as Record; + const value = "providerSortKey" in record ? record.providerSortKey : record.sortKey; + return typeof value === "string" ? value : null; +} + +function rerankBundleMessages(database: Database, sourceId: string): void { + const rows = all(database, ` + SELECT message.id,message.conversation_id,message.sent_at,message.kind, + provenance.external_id,provenance.metadata_json + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + ORDER BY message.conversation_id,message.id + `, sourceId); + const byConversation = new Map(); + for (const value of rows) { + const row = Object.freeze({ ...value, provider_sort_key: storedProviderSortKey(value) }); + const values = byConversation.get(row.conversation_id) ?? []; + values.push(row); + byConversation.set(row.conversation_id, values); + } + const update = database.query("UPDATE messages SET source_row_id=? WHERE id=?"); + for (const values of byConversation.values()) { + for (const [index, row] of values.entries()) update.run(-(index + 1), row.id); + values.sort((left, right) => { + const leftReaction = left.kind === "reaction"; + const rightReaction = right.kind === "reaction"; + if (leftReaction !== rightReaction) return leftReaction ? 1 : -1; + if (!leftReaction) { + const sort = compareCodeUnits( + left.provider_sort_key ?? left.external_id, + right.provider_sort_key ?? right.external_id, + ); + if (sort !== 0) return sort; + } + return compareCodeUnits(left.sent_at, right.sent_at) + || compareCodeUnits(left.external_id, right.external_id) + || compareCodeUnits(left.id, right.id); + }); + for (const [index, row] of values.entries()) update.run(index + 1, row.id); + } +} + +function setCorpusRevision(database: Database): string | null { + const revision = globalCorpusRevision(database); + if (revision === null) { + database.query("DELETE FROM metadata WHERE key='corpus_revision'").run(); + return null; + } + database.query(` + INSERT INTO metadata(key,value) VALUES ('corpus_revision',?) + ON CONFLICT(key) DO UPDATE SET value=excluded.value + `).run(revision); + return revision; +} + +function validSourceDescriptor(source: CorpusSourceDescriptor): void { + if ( + (source.id !== IMESSAGE_SOURCE_ID && !/^source_[a-f0-9]{64}$/u.test(source.id)) + || (source.kind !== "imessage" && source.kind !== "bundle") + || source.provider.length < 1 + || Buffer.byteLength(source.provider, "utf8") > 256 + || !/^[a-f0-9]{64}$/u.test(source.revision) + || source.externalId.length < 1 + || Buffer.byteLength(source.externalId, "utf8") > 4_096 + || source.warnings.length > 130 + ) throw new CliError("invalid-data", `Corpus source ${source.id} is invalid`); + canonicalTimestampOrNull(source.generatedAt, `Corpus source ${source.id} generatedAt`); + if (source.kind === "bundle" && source.generatedAt === null) { + throw new CliError("invalid-data", `Bundle source ${source.id} requires generatedAt`); + } + canonicalTimestampOrNull(source.coverage.observedFrom, `Corpus source ${source.id} observedFrom`); + canonicalTimestampOrNull(source.coverage.observedTo, `Corpus source ${source.id} observedTo`); + if ( + ( + source.coverage.observedFrom !== null + && source.coverage.observedTo !== null + && source.coverage.observedFrom > source.coverage.observedTo + ) + ) throw new CliError("invalid-data", `Corpus source ${source.id} has invalid coverage bounds`); + if ( + source.coverage.history !== "complete-current-local" + && source.coverage.history !== "bounded" + && source.coverage.history !== "unknown" + ) throw new CliError("invalid-data", `Corpus source ${source.id} has invalid history coverage`); + if ( + (source.coverage.kind !== undefined && ( + source.coverage.kind.length < 1 + || Buffer.byteLength(source.coverage.kind, "utf8") > 128 + || /\p{Cc}/u.test(source.coverage.kind) + )) + || (source.coverage.reason !== undefined && source.coverage.reason !== null && ( + source.coverage.reason.length < 1 + || Buffer.byteLength(source.coverage.reason, "utf8") > 128 + || /\p{Cc}/u.test(source.coverage.reason) + )) + ) throw new CliError("invalid-data", `Corpus source ${source.id} has invalid coverage metadata`); + if ( + source.manifestSha256 !== null + && !/^[a-f0-9]{64}$/u.test(source.manifestSha256) + ) throw new CliError("invalid-data", `Corpus source ${source.id} has an invalid manifest digest`); + if ( + source.producer.id.length < 1 + || source.producer.version.length < 1 + || Buffer.byteLength(source.producer.id, "utf8") > 256 + || Buffer.byteLength(source.producer.version, "utf8") > 256 + ) throw new CliError("invalid-data", `Corpus source ${source.id} has invalid producer identity`); + for (const warning of source.warnings) { + if (Buffer.byteLength(warning, "utf8") > 1_024 || warning.includes("\u0000")) { + throw new CliError("invalid-data", `Corpus source ${source.id} has an invalid warning`); + } + } +} + +function validateSourceSnapshot(snapshot: SourceCorpusSnapshot): void { + validSourceDescriptor(snapshot.source); + if ( + snapshot.conversations.length > 2_000_000 + || snapshot.messages.length > 2_000_000 + || (snapshot.reactionFacts?.length ?? 0) > 2_000_000 + || snapshot.conversationProvenance.length !== snapshot.conversations.length + || snapshot.messageProvenance.length !== snapshot.messages.length + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} exceeds its result bounds`); + const conversationIds = new Set(snapshot.conversations.map(({ id }) => id)); + if (conversationIds.size !== snapshot.conversations.length) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} repeats conversation IDs`); + } + const conversationProvenance = new Map( + snapshot.conversationProvenance.map((value) => [value.conversationId, value]), + ); + if ( + conversationProvenance.size !== snapshot.conversationProvenance.length + || [...conversationIds].some((id) => !conversationProvenance.has(id)) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid conversation provenance`); + const externalConversations = new Set(); + for (const provenance of snapshot.conversationProvenance) { + if ( + provenance.externalId.length < 1 + || Buffer.byteLength(provenance.externalId, "utf8") > 4_096 + || externalConversations.has(provenance.externalId) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid external conversation IDs`); + externalConversations.add(provenance.externalId); + } + const messageIds = new Set(snapshot.messages.map(({ id }) => id)); + const messagesById = new Map(snapshot.messages.map((message) => [message.id, message])); + if (messageIds.size !== snapshot.messages.length) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} repeats message IDs`); + } + for (const message of snapshot.messages) { + if (!conversationIds.has(message.conversationId)) { + throw new CliError("invalid-data", `Message ${message.id} references an unknown conversation`); + } + } + const messageProvenance = new Map(snapshot.messageProvenance.map((value) => [value.messageId, value])); + if ( + messageProvenance.size !== snapshot.messageProvenance.length + || [...messageIds].some((id) => !messageProvenance.has(id)) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid message provenance`); + const externalMessages = new Set(); + for (const provenance of snapshot.messageProvenance) { + const message = messagesById.get(provenance.messageId)!; + if ( + provenance.externalId.length < 1 + || Buffer.byteLength(provenance.externalId, "utf8") > 4_096 + || externalMessages.has(provenance.externalId) + || provenance.attachments.length > 256 + || ( + provenance.providerSortKey !== null + && ( + provenance.providerSortKey.length < 1 + || Buffer.byteLength(provenance.providerSortKey, "utf8") > 1_024 + || /[\u0000-\u001f\u007f]/u.test(provenance.providerSortKey) + ) + ) + || ( + snapshot.source.kind === "bundle" + ? (message.kind === "reaction") === (provenance.providerSortKey !== null) + : provenance.providerSortKey !== null + ) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid external message provenance`); + externalMessages.add(provenance.externalId); + } + const auxiliaryIds = new Set(); + for (const record of snapshot.auxiliaryRecords ?? []) { + const key = `${record.kind}\0${record.id}`; + if ( + !["account", "participant", "reaction", "tombstone", "excluded-message"].includes(record.kind) + || record.id.length < 1 + || Buffer.byteLength(record.id, "utf8") > 4_096 + || auxiliaryIds.has(key) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid auxiliary records`); + const encoded = canonicalJson(record.record); + if (typeof encoded !== "string" || Buffer.byteLength(encoded, "utf8") > 2 * 1024 * 1024) { + throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has an oversized auxiliary record`); + } + auxiliaryIds.add(key); + } + const reactionIds = new Set(); + const externalReactionIds = new Set(); + for (const reaction of snapshot.reactionFacts ?? []) { + if ( + reaction.id.length < 1 + || reaction.externalId.length < 1 + || reaction.targetExternalId.length < 1 + || Buffer.byteLength(reaction.id, "utf8") > 4_096 + || Buffer.byteLength(reaction.externalId, "utf8") > 4_096 + || Buffer.byteLength(reaction.targetExternalId, "utf8") > 4_096 + || Buffer.byteLength(reaction.body, "utf8") > 8 * 1_024 + || reactionIds.has(reaction.id) + || externalReactionIds.has(reaction.externalId) + || (reaction.conversationId !== null && !conversationIds.has(reaction.conversationId)) + || (reaction.direction !== null && reaction.direction !== "incoming" && reaction.direction !== "outgoing") + || (reaction.state !== "active" && reaction.state !== "removed") + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has invalid reaction facts`); + canonicalTimestampOrNull(reaction.reactedAt, `Corpus source ${snapshot.source.id} reaction time`); + reactionIds.add(reaction.id); + externalReactionIds.add(reaction.externalId); + } + for (const deletion of snapshot.deletions ?? []) { + if ( + ![ + "account", "participant", "conversation", "message", "reaction", "reaction-timeline", + ].includes(deletion.entityKind) + || deletion.externalId.length < 1 + || Buffer.byteLength(deletion.externalId, "utf8") > 4_096 + || (deletion.localEntityId !== null && Buffer.byteLength(deletion.localEntityId, "utf8") > 4_096) + || (deletion.expectedConversationId !== undefined && ( + deletion.expectedConversationId.length < 1 + || Buffer.byteLength(deletion.expectedConversationId, "utf8") > 4_096 + )) + || (deletion.reason !== undefined && ![ + "tombstone", "explicit-exclusion", "replacement", + ].includes(deletion.reason)) + ) throw new CliError("invalid-data", `Corpus source ${snapshot.source.id} has an invalid deletion`); + canonicalTimestampOrNull(deletion.deletedAt, `Corpus source ${snapshot.source.id} deletion time`); + } +} + export class LocalStore { readonly #database: Database; @@ -886,8 +1567,10 @@ export class LocalStore { } sourceIdentity(): unknown | null { - const encoded = scalarText(this.#database, "source_identity"); - return encoded === null ? null : JSON.parse(encoded) as unknown; + const encoded = get<{ identity_json: string }>(this.#database, ` + SELECT identity_json FROM corpus_sources WHERE id=? + `, IMESSAGE_SOURCE_ID)?.identity_json ?? scalarText(this.#database, "source_identity"); + return encoded === null ? null : parsedJson(encoded, "Stored iMessage source identity"); } contactsRevision(): string | null { @@ -1027,98 +1710,813 @@ export class LocalStore { })); } - replaceCorpus( - snapshot: CorpusSnapshot, + replaceSources( + snapshots: readonly SourceCorpusSnapshot[], ingestedAt: string, hmacKey?: string | Uint8Array, ): Readonly<{ corpusRevision: string; - conversations: number; - messages: number; + sources: readonly Readonly<{ + id: string; + changed: boolean; + conversations: number; + messages: number; + }>[]; }> { - const corpusRevision = snapshot.source.snapshotSha256; - if (!/^[a-f0-9]{64}$/u.test(corpusRevision)) { - throw new CliError("invalid-data", "The iMessage reader returned an invalid corpus revision"); + canonicalTimestampOrNull(ingestedAt, "Source ingest time"); + if (snapshots.length < 1) { + throw new CliError("invalid-data", "A source replacement must contain at least one source"); } - const conversationIds = new Set(snapshot.conversations.map((conversation) => conversation.id)); - if (conversationIds.size !== snapshot.conversations.length) { - throw new CliError("invalid-data", "The iMessage reader returned duplicate conversation IDs"); - } - const messageIds = new Set(); - for (const message of snapshot.messages) { - if (!conversationIds.has(message.conversationId)) { - throw new CliError("invalid-data", `Message ${message.id} references an unknown conversation`); + const sourceIds = new Set(); + for (const snapshot of snapshots) { + if (sourceIds.has(snapshot.source.id)) { + throw new CliError("invalid-data", `Source replacement repeats ${snapshot.source.id}`); } - if (messageIds.has(message.id)) throw new CliError("invalid-data", `Duplicate message ID ${message.id}`); - messageIds.add(message.id); + sourceIds.add(snapshot.source.id); + validateSourceSnapshot(snapshot); } - transaction(this.#database, () => { - this.#database.exec("DELETE FROM messages; DELETE FROM conversations;"); - const insertConversation = this.#database.query(` - INSERT INTO conversations ( - id, source_key, private_label, service, participant_count, - participant_ids_json, private_participants_json, is_group - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + return transaction(this.#database, () => { + const upsertSource = this.#database.query(` + INSERT INTO corpus_sources( + id,kind,provider,network,account_id,external_id,input_revision,revision,generated_at, + producer_json,coverage_json,manifest_sha256,identity_json,warnings_json,ingested_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + kind=excluded.kind,provider=excluded.provider,network=excluded.network, + account_id=excluded.account_id,external_id=excluded.external_id, + input_revision=excluded.input_revision,generated_at=excluded.generated_at, + producer_json=excluded.producer_json,coverage_json=excluded.coverage_json, + manifest_sha256=excluded.manifest_sha256,identity_json=excluded.identity_json, + warnings_json=excluded.warnings_json,ingested_at=excluded.ingested_at `); - for (const conversation of snapshot.conversations) { - insertConversation.run( - conversation.id, - conversation.sourceKey, - conversation.privateLabel, - conversation.service, - conversation.participantCount, - canonicalJson(conversation.participantIds), - canonicalJson(conversation.privateParticipants), - conversation.group ? 1 : 0, - ); - } - const insertMessage = this.#database.query(` - INSERT INTO messages ( - id, source_row_id, source_guid, conversation_id, sent_at, direction, - body, body_source, kind, reply_to_source_guid, edited_at, retracted_at, - service, attachment_count - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + const relabelSourceConversations = this.#database.query(` + UPDATE conversations SET service=? + WHERE id IN ( + SELECT conversation_id FROM conversation_sources WHERE source_id=? + ) + `); + const relabelSourceMessages = this.#database.query(` + UPDATE messages SET service=? + WHERE id IN ( + SELECT message_id FROM message_provenance WHERE source_id=? + ) + `); + const upsertConversation = this.#database.query(` + INSERT INTO conversations( + id,source_key,private_label,service,participant_count, + participant_ids_json,private_participants_json,is_group + ) VALUES (?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + source_key=excluded.source_key,private_label=excluded.private_label, + service=excluded.service,participant_count=excluded.participant_count, + participant_ids_json=excluded.participant_ids_json, + private_participants_json=excluded.private_participants_json,is_group=excluded.is_group + `); + const upsertConversationSource = this.#database.query(` + INSERT INTO conversation_sources(conversation_id,source_id,external_id,metadata_json) + VALUES (?,?,?,?) + ON CONFLICT(conversation_id) DO UPDATE SET + external_id=excluded.external_id,metadata_json=excluded.metadata_json + `); + const upsertMessage = this.#database.query(` + INSERT INTO messages( + id,source_row_id,source_guid,conversation_id,sent_at,direction, + body,body_source,kind,reply_to_source_guid,edited_at,retracted_at, + service,attachment_count + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + source_guid=excluded.source_guid,conversation_id=excluded.conversation_id, + sent_at=excluded.sent_at,direction=excluded.direction,body=excluded.body, + body_source=excluded.body_source,kind=excluded.kind, + reply_to_source_guid=excluded.reply_to_source_guid,edited_at=excluded.edited_at, + retracted_at=excluded.retracted_at,service=excluded.service, + attachment_count=excluded.attachment_count + `); + const upsertMessageProvenance = this.#database.query(` + INSERT INTO message_provenance( + message_id,source_id,external_id,reply_to_external_id,attachments_json,metadata_json + ) VALUES (?,?,?,?,?,?) + ON CONFLICT(message_id) DO UPDATE SET + external_id=excluded.external_id,reply_to_external_id=excluded.reply_to_external_id, + attachments_json=excluded.attachments_json,metadata_json=excluded.metadata_json + `); + const upsertReactionFact = this.#database.query(` + INSERT INTO corpus_reaction_facts( + id,source_id,external_id,target_external_id,conversation_id, + direction,body,reacted_at,state + ) VALUES (?,?,?,?,?,?,?,?,?) + ON CONFLICT(id) DO UPDATE SET + external_id=excluded.external_id,target_external_id=excluded.target_external_id, + conversation_id=excluded.conversation_id,direction=excluded.direction, + body=excluded.body,reacted_at=excluded.reacted_at,state=excluded.state + `); + const upsertSourceRecord = this.#database.query(` + INSERT INTO corpus_source_records(source_id,kind,external_id,record_json) + VALUES (?,?,?,?) + ON CONFLICT(source_id,kind,external_id) DO UPDATE SET record_json=excluded.record_json + `); + const setSuppression = this.#database.query(` + INSERT INTO corpus_source_suppressions( + source_id,kind,local_id,external_id,suppressed_at,reason,suppressed + ) VALUES (?,?,?,?,?,?,?) + ON CONFLICT(source_id,kind,local_id) DO UPDATE SET + external_id=excluded.external_id,suppressed_at=excluded.suppressed_at, + reason=excluded.reason,suppressed=excluded.suppressed `); - for (const message of snapshot.messages) { - insertMessage.run( - message.id, - message.sourceRowId, - message.sourceGuid, - message.conversationId, - message.sentAt, - message.direction, - message.body, - message.bodySource, - message.kind, - message.replyToSourceGuid, - message.editedAt, - message.retractedAt, - message.service, - message.attachmentCount, + const clearExternalSuppression = this.#database.query(` + UPDATE corpus_source_suppressions + SET suppressed_at=?,reason='reappeared',suppressed=0 + WHERE source_id=? AND kind=? AND external_id=? AND suppressed=1 + `); + const results: Array> = []; + let changedAny = false; + for (const snapshot of snapshots) { + const existing = get<{ + kind: string; + network: string | null; + input_revision: string; + revision: string; + generated_at: string | null; + manifest_sha256: string | null; + }>(this.#database, ` + SELECT kind,network,input_revision,revision,generated_at,manifest_sha256 + FROM corpus_sources WHERE id=? + `, snapshot.source.id); + if (existing !== null && existing.kind !== snapshot.source.kind) { + throw new CliError("conflict", `Source ${snapshot.source.id} changed kind`); + } + if (existing !== null && snapshot.source.kind === "bundle") { + if (existing.generated_at === null || snapshot.source.generatedAt! < existing.generated_at) { + throw new CliError("conflict", `Source ${snapshot.source.id} snapshot is older than stored state`); + } + if ( + snapshot.source.generatedAt === existing.generated_at + && ( + snapshot.source.revision !== existing.input_revision + || snapshot.source.manifestSha256 !== existing.manifest_sha256 + ) + ) throw new CliError("conflict", `Source ${snapshot.source.id} reuses generatedAt for different input`); + } + const authoritative = snapshot.source.kind === "imessage" + || snapshot.source.coverage.history === "complete-current-local"; + if (authoritative) { + for (const row of this.#database.query(` + SELECT conversation_id,external_id FROM conversation_sources WHERE source_id=? + `).iterate(snapshot.source.id) as Iterable<{ + conversation_id: string; + external_id: string; + }>) { + setSuppression.run( + snapshot.source.id, + "conversation", + row.conversation_id, + row.external_id, + ingestedAt, + "authoritative-absence", + 1, + ); + } + for (const row of this.#database.query(` + SELECT id,external_id FROM corpus_reaction_facts WHERE source_id=? + `).iterate(snapshot.source.id) as Iterable<{ id: string; external_id: string }>) { + setSuppression.run( + snapshot.source.id, + "reaction", + row.id, + row.external_id, + ingestedAt, + "authoritative-absence", + 1, + ); + } + for (const row of this.#database.query(` + SELECT provenance.message_id,provenance.external_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? + `).iterate(snapshot.source.id) as Iterable<{ + message_id: string; + external_id: string; + kind: string; + }>) { + setSuppression.run( + snapshot.source.id, + row.kind === "reaction" ? "reaction" : "message", + row.message_id, + row.external_id, + ingestedAt, + "authoritative-absence", + 1, + ); + } + } + upsertSource.run( + snapshot.source.id, + snapshot.source.kind, + snapshot.source.provider, + snapshot.source.network, + snapshot.source.accountId, + snapshot.source.externalId, + snapshot.source.revision, + existing?.revision ?? snapshot.source.revision, + snapshot.source.generatedAt, + canonicalJson(snapshot.source.producer), + canonicalJson(snapshot.source.coverage), + snapshot.source.manifestSha256, + canonicalJson(snapshot.source.identity), + canonicalJson(snapshot.source.warnings), + ingestedAt, + ); + if (existing !== null && existing.network !== snapshot.source.network) { + relabelSourceConversations.run(snapshot.source.network, snapshot.source.id); + relabelSourceMessages.run(snapshot.source.network, snapshot.source.id); + } + const conversationProvenance = new Map( + snapshot.conversationProvenance.map((value) => [value.conversationId, value]), + ); + for (const conversation of snapshot.conversations) { + const owner = get<{ source_id: string }>(this.#database, ` + SELECT source_id FROM conversation_sources WHERE conversation_id=? + `, conversation.id); + if (owner !== null && owner.source_id !== snapshot.source.id) { + throw new CliError("conflict", `Conversation ${conversation.id} belongs to another source`); + } + upsertConversation.run( + conversation.id, + conversation.sourceKey, + conversation.privateLabel, + conversation.service, + conversation.participantCount, + canonicalJson(conversation.participantIds), + canonicalJson(conversation.privateParticipants), + conversation.group ? 1 : 0, + ); + const provenance = conversationProvenance.get(conversation.id)!; + upsertConversationSource.run( + conversation.id, + snapshot.source.id, + provenance.externalId, + canonicalJson(provenance.metadata ?? {}), + ); + setSuppression.run( + snapshot.source.id, + "conversation", + conversation.id, + provenance.externalId, + ingestedAt, + "reappeared", + 0, + ); + clearExternalSuppression.run( + ingestedAt, + snapshot.source.id, + "conversation", + provenance.externalId, + ); + } + const messageProvenance = new Map( + snapshot.messageProvenance.map((value) => [value.messageId, value]), ); + for (const message of snapshot.messages) { + const owner = get<{ source_id: string; source_row_id: number }>(this.#database, ` + SELECT provenance.source_id,message.source_row_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.message_id=? + `, message.id); + if (owner !== null && owner.source_id !== snapshot.source.id) { + throw new CliError("conflict", `Message ${message.id} belongs to another source`); + } + const preferredRowId = authoritative ? message.sourceRowId : null; + const preferredCollision = preferredRowId === null ? null : get<{ id: string }>( + this.#database, + "SELECT id FROM messages WHERE conversation_id=? AND source_row_id=?", + message.conversationId, + preferredRowId, + ); + const sourceRowId = owner?.source_row_id ?? ( + preferredRowId !== null && preferredCollision === null + ? preferredRowId + : (get<{ value: number | null }>(this.#database, ` + SELECT max(source_row_id) AS value FROM messages WHERE conversation_id=? + `, message.conversationId)?.value ?? 0) + 1 + ); + upsertMessage.run( + message.id, + sourceRowId, + message.sourceGuid, + message.conversationId, + message.sentAt, + message.direction, + message.body, + message.bodySource, + message.kind, + message.replyToSourceGuid, + message.editedAt, + message.retractedAt, + message.service, + message.attachmentCount, + ); + const provenance = messageProvenance.get(message.id)!; + upsertMessageProvenance.run( + message.id, + snapshot.source.id, + provenance.externalId, + provenance.replyToExternalId, + canonicalJson(provenance.attachments), + canonicalJson({ + providerSortKey: provenance.providerSortKey, + metadata: provenance.metadata ?? {}, + }), + ); + setSuppression.run( + snapshot.source.id, + message.kind === "reaction" ? "reaction" : "message", + message.id, + provenance.externalId, + ingestedAt, + "reappeared", + 0, + ); + clearExternalSuppression.run( + ingestedAt, + snapshot.source.id, + message.kind === "reaction" ? "reaction" : "message", + provenance.externalId, + ); + if (message.kind === "reaction") { + setSuppression.run( + snapshot.source.id, + "reaction-timeline", + message.id, + provenance.externalId, + ingestedAt, + "reappeared", + 0, + ); + clearExternalSuppression.run( + ingestedAt, + snapshot.source.id, + "reaction-timeline", + provenance.externalId, + ); + } + } + for (const reaction of snapshot.reactionFacts ?? []) { + const existingReaction = get<{ source_id: string; external_id: string }>(this.#database, ` + SELECT source_id,external_id FROM corpus_reaction_facts WHERE id=? + `, reaction.id); + if ( + existingReaction !== null + && ( + existingReaction.source_id !== snapshot.source.id + || existingReaction.external_id !== reaction.externalId + ) + ) throw new CliError("conflict", `Reaction ${reaction.id} belongs to another source coordinate`); + const conversationId = reaction.conversationId ?? get<{ conversation_id: string }>( + this.#database, + `SELECT message.conversation_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.external_id=?`, + snapshot.source.id, + reaction.targetExternalId, + )?.conversation_id ?? null; + upsertReactionFact.run( + reaction.id, + snapshot.source.id, + reaction.externalId, + reaction.targetExternalId, + conversationId, + reaction.direction, + reaction.body, + reaction.reactedAt, + reaction.state, + ); + if (reaction.state === "active") { + setSuppression.run( + snapshot.source.id, + "reaction", + reaction.id, + reaction.externalId, + ingestedAt, + "reappeared", + 0, + ); + clearExternalSuppression.run( + ingestedAt, + snapshot.source.id, + "reaction", + reaction.externalId, + ); + clearExternalSuppression.run( + ingestedAt, + snapshot.source.id, + "reaction-timeline", + reaction.externalId, + ); + } + } + this.#database.query(` + UPDATE corpus_reaction_facts AS reaction + SET conversation_id=( + SELECT message.conversation_id + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=reaction.source_id + AND provenance.external_id=reaction.target_external_id + ) + WHERE reaction.source_id=? AND reaction.conversation_id IS NULL + AND EXISTS ( + SELECT 1 FROM message_provenance provenance + WHERE provenance.source_id=reaction.source_id + AND provenance.external_id=reaction.target_external_id + ) + `).run(snapshot.source.id); + for (const record of snapshot.auxiliaryRecords ?? []) { + upsertSourceRecord.run( + snapshot.source.id, + record.kind, + record.id, + canonicalJson(record.record), + ); + } + for (const deletion of snapshot.deletions ?? []) { + let localId = deletion.localEntityId; + if (deletion.entityKind === "conversation") { + const specifiedLocal = localId !== null; + const target = localId === null + ? get<{ conversation_id: string; external_id: string }>(this.#database, ` + SELECT conversation_id,external_id FROM conversation_sources + WHERE source_id=? AND external_id=? + `, snapshot.source.id, deletion.externalId) + : get<{ conversation_id: string; external_id: string }>(this.#database, ` + SELECT conversation_id,external_id FROM conversation_sources + WHERE source_id=? AND conversation_id=? + `, snapshot.source.id, localId); + if (target !== null) { + if (target.external_id !== deletion.externalId) { + throw new CliError("invalid-data", "A conversation deletion has mismatched coordinates"); + } + localId = target.conversation_id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A conversation deletion references an unknown local entity"); + } + } + if (deletion.entityKind === "message") { + const specifiedLocal = localId !== null; + const target = localId === null + ? get<{ + message_id: string; + external_id: string; + conversation_id: string; + kind: string; + }>(this.#database, ` + SELECT provenance.message_id,provenance.external_id, + message.conversation_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.external_id=? + `, snapshot.source.id, deletion.externalId) + : get<{ + message_id: string; + external_id: string; + conversation_id: string; + kind: string; + }>(this.#database, ` + SELECT provenance.message_id,provenance.external_id, + message.conversation_id,message.kind + FROM message_provenance provenance + JOIN messages message ON message.id=provenance.message_id + WHERE provenance.source_id=? AND provenance.message_id=? + `, snapshot.source.id, localId); + if (target !== null) { + if ( + target.external_id !== deletion.externalId + || target.kind === "reaction" + || ( + deletion.expectedConversationId !== undefined + && deletion.expectedConversationId !== target.conversation_id + ) + ) throw new CliError("invalid-data", "A message deletion has mismatched coordinates"); + localId = target.message_id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A message deletion references an unknown local entity"); + } + } + if ( + deletion.entityKind === "reaction" + || deletion.entityKind === "reaction-timeline" + ) { + const specifiedLocal = localId !== null; + const target = localId === null + ? get<{ + id: string; + external_id: string; + conversation_id: string | null; + }>(this.#database, ` + SELECT id,external_id,conversation_id FROM corpus_reaction_facts + WHERE source_id=? AND external_id=? + `, snapshot.source.id, deletion.externalId) + : get<{ + id: string; + external_id: string; + conversation_id: string | null; + }>(this.#database, ` + SELECT id,external_id,conversation_id FROM corpus_reaction_facts + WHERE source_id=? AND id=? + `, snapshot.source.id, localId); + if (target !== null) { + if ( + target.external_id !== deletion.externalId + || ( + deletion.expectedConversationId !== undefined + && deletion.expectedConversationId !== target.conversation_id + ) + ) throw new CliError("invalid-data", "A reaction deletion has mismatched coordinates"); + localId = target.id; + } else if (specifiedLocal) { + throw new CliError("invalid-data", "A reaction deletion references an unknown local entity"); + } + } + setSuppression.run( + snapshot.source.id, + deletion.entityKind, + localId ?? `external:${deletion.externalId}`, + deletion.externalId, + deletion.deletedAt, + deletion.reason ?? "tombstone", + 1, + ); + } + if (snapshot.source.kind === "bundle") { + rerankBundleMessages(this.#database, snapshot.source.id); + } + const stateRevision = sourceStateRevision(this.#database, snapshot.source.id); + this.#database.query("UPDATE corpus_sources SET revision=? WHERE id=?") + .run(stateRevision, snapshot.source.id); + const changed = existing?.revision !== stateRevision; + changedAny ||= changed; + const counts = get<{ conversations: number; messages: number }>(this.#database, ` + SELECT count(distinct conversation.id) AS conversations, + count(message.id) AS messages + FROM conversation_sources ownership + JOIN conversations conversation ON conversation.id=ownership.conversation_id + LEFT JOIN messages message ON message.conversation_id=conversation.id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + ) + WHERE ownership.source_id=? + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.local_id=conversation.id + AND suppression.kind='conversation' + AND suppression.suppressed=1 + ) + `, snapshot.source.id) ?? { conversations: 0, messages: 0 }; + results.push(Object.freeze({ id: snapshot.source.id, changed, ...counts })); } + if (changedAny) rebuildConversationLabels(this.#database, hmacKey); + const corpusRevision = setCorpusRevision(this.#database); + if (corpusRevision === null) throw new CliError("internal", "Source replacement produced no corpus revision"); + return Object.freeze({ corpusRevision, sources: Object.freeze(results) }); + }); + } + + replaceCorpus( + snapshot: CorpusSnapshot, + ingestedAt: string, + hmacKey?: string | Uint8Array, + ): Readonly<{ + corpusRevision: string; + conversations: number; + messages: number; + }> { + if (!/^[a-f0-9]{64}$/u.test(snapshot.source.snapshotSha256)) { + throw new CliError("invalid-data", "The iMessage reader returned an invalid corpus revision"); + } + const observed = snapshot.messages.map(({ sentAt }) => sentAt).sort(); + const sourceSnapshot: SourceCorpusSnapshot = Object.freeze({ + source: Object.freeze({ + id: IMESSAGE_SOURCE_ID, + kind: "imessage", + provider: "apple", + network: null, + accountId: null, + externalId: "local-imessage", + revision: snapshot.source.snapshotSha256, + generatedAt: null, + producer: Object.freeze({ id: "message-like-me", version: "imessage-reader-v1" }), + coverage: Object.freeze({ + history: "complete-current-local", + observedFrom: observed[0] ?? null, + observedTo: observed.at(-1) ?? null, + }), + manifestSha256: null, + identity: snapshot.source, + warnings: snapshot.warnings, + }), + conversations: snapshot.conversations, + conversationProvenance: Object.freeze(snapshot.conversations.map((conversation) => ({ + conversationId: conversation.id, + externalId: conversation.sourceKey, + }))), + messages: snapshot.messages, + messageProvenance: Object.freeze(snapshot.messages.map((message) => ({ + messageId: message.id, + externalId: message.sourceGuid, + providerSortKey: null, + replyToExternalId: message.replyToSourceGuid, + attachments: Object.freeze(Array.from({ length: message.attachmentCount }, (_value, index) => ({ + id: `unavailable-${index + 1}`, + kind: null, + mimeType: null, + fileName: null, + bytes: null, + }))), + }))), + }); + const replaced = this.replaceSources([sourceSnapshot], ingestedAt, hmacKey); + transaction(this.#database, () => { const setMetadata = this.#database.query(` - INSERT INTO metadata (key, value) VALUES (?, ?) - ON CONFLICT (key) DO UPDATE SET value = excluded.value + INSERT INTO metadata(key,value) VALUES (?,?) + ON CONFLICT(key) DO UPDATE SET value=excluded.value `); for (const [key, value] of [ - ["corpus_revision", corpusRevision], ["source_identity", canonicalJson(snapshot.source)], ["ingested_at", ingestedAt], ["warnings", canonicalJson(snapshot.warnings)], ["corpus_schema_version", String(snapshot.schemaVersion)], ] as const) setMetadata.run(key, value); - rebuildConversationLabels(this.#database, hmacKey); }); - return { - corpusRevision, + corpusRevision: replaced.corpusRevision, conversations: snapshot.conversations.length, messages: snapshot.messages.length, }; } + listSources(privateDetails = false): ReadonlyArray> { + const rows = all<{ + id: string; + kind: "imessage" | "bundle"; + provider: string; + network: string | null; + account_id: string | null; + external_id: string; + input_revision: string; + revision: string; + generated_at: string | null; + coverage_json: string; + manifest_sha256: string | null; + identity_json: string; + warnings_json: string; + ingested_at: string; + conversations: number; + messages: number; + reactions: number; + undated_reactions: number; + }>(this.#database, ` + SELECT source.*, + count(distinct ownership.conversation_id) AS conversations, + count(message.id) AS messages, + CASE source.kind WHEN 'bundle' THEN + (SELECT count(*) FROM corpus_reaction_facts reaction + WHERE reaction.source_id=source.id AND reaction.state='active' + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='reaction' + AND suppression.local_id=reaction.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction.conversation_id + AND suppression.suppressed=1 + )) + ELSE + (SELECT count(*) FROM messages reaction_message + JOIN message_provenance reaction_provenance + ON reaction_provenance.message_id=reaction_message.id + WHERE reaction_provenance.source_id=source.id + AND reaction_message.kind='reaction' AND reaction_message.retracted_at IS NULL + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.local_id=reaction_message.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction_message.conversation_id + AND suppression.suppressed=1 + )) + END AS reactions, + CASE source.kind WHEN 'bundle' THEN + (SELECT count(*) FROM corpus_reaction_facts reaction + WHERE reaction.source_id=source.id AND reaction.state='active' + AND reaction.reacted_at IS NULL + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='reaction' + AND suppression.local_id=reaction.id AND suppression.suppressed=1 + ) + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id AND suppression.kind='conversation' + AND suppression.local_id=reaction.conversation_id + AND suppression.suppressed=1 + )) + ELSE 0 + END AS undated_reactions + FROM corpus_sources source + LEFT JOIN conversation_sources ownership ON ownership.source_id=source.id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind='conversation' + AND suppression.local_id=ownership.conversation_id + AND suppression.suppressed=1 + ) + LEFT JOIN messages message ON message.conversation_id=ownership.conversation_id + AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=source.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.local_id=message.id + AND suppression.suppressed=1 + ) + GROUP BY source.id + ORDER BY source.provider,source.network,source.id + `); + return rows.map((row) => { + const warnings = parsedJson(row.warnings_json, `Source ${row.id} warnings`); + if (!Array.isArray(warnings)) throw new CliError("invalid-data", `Source ${row.id} warnings are invalid`); + return { + id: row.id, + kind: row.kind, + provider: row.provider, + network: row.network, + revision: row.revision, + generatedAt: row.generated_at, + ingestedAt: row.ingested_at, + coverage: parsedJson(row.coverage_json, `Source ${row.id} coverage`), + warningCount: warnings.length, + conversations: row.conversations, + messages: row.messages, + reactions: row.reactions, + undatedReactions: row.undated_reactions, + ...(privateDetails ? { + accountId: row.account_id, + externalId: row.external_id, + manifestSha256: row.manifest_sha256, + inputRevision: row.input_revision, + identity: parsedJson(row.identity_json, `Source ${row.id} identity`), + warnings, + } : {}), + }; + }); + } + + source(sourceId: string, privateDetails = false): ReturnType[number] | null { + if (sourceId.length < 1 || sourceId.length > 256) { + throw new CliError("usage", "Source ID must be bounded non-empty text"); + } + return this.listSources(privateDetails).find(({ id }) => id === sourceId) ?? null; + } + listContacts(options: Readonly<{ privateLabels: boolean; minimumOutgoing: number; @@ -1145,16 +2543,30 @@ export class LocalStore { conversation.id AS conversation_id FROM conversation_contact_scopes association JOIN conversations conversation ON conversation.id=association.conversation_id + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id LEFT JOIN conversation_contact_labels label ON label.conversation_id=association.conversation_id - WHERE conversation.is_group=0 + WHERE conversation.is_group=0 AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) UNION ALL SELECT conversation.id,conversation.private_label,'conversation',conversation.is_group, conversation.participant_count,conversation.id FROM conversations conversation + JOIN conversation_sources ownership ON ownership.conversation_id=conversation.id LEFT JOIN conversation_contact_scopes association ON association.conversation_id=conversation.id - WHERE association.conversation_id IS NULL + WHERE association.conversation_id IS NULL AND NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=ownership.source_id + AND suppression.kind='conversation' + AND suppression.local_id=conversation.id + AND suppression.suppressed=1 + ) ) SELECT scope.id,min(scope.private_label) AS private_label, max(scope.scope_kind) AS scope_kind, @@ -1166,6 +2578,14 @@ export class LocalStore { sum(CASE WHEN message.direction = 'outgoing' THEN 1 ELSE 0 END) AS outgoing_count FROM scope_conversations scope JOIN messages message ON message.conversation_id=scope.conversation_id + JOIN message_provenance provenance ON provenance.message_id=message.id + WHERE NOT EXISTS ( + SELECT 1 FROM corpus_source_suppressions suppression + WHERE suppression.source_id=provenance.source_id + AND suppression.local_id=message.id + AND suppression.kind IN ('message','reaction','reaction-timeline') + AND suppression.suppressed=1 + ) GROUP BY scope.id HAVING outgoing_count >= ? ORDER BY outgoing_count DESC,last_message_at DESC,scope.id @@ -1224,6 +2644,7 @@ export class LocalStore { messageCount: number; incomingCount: number; outgoingCount: number; + services: readonly string[]; }) | null { const scope = analysisScope(this.#database, contactId); if (scope === null) return null; @@ -1286,6 +2707,8 @@ export class LocalStore { scopeKind: scope.kind, conversationCount: scope.conversationIds.length, service: services.length === 1 ? services[0]! : null, + services: Object.freeze(services.sort((left, right) => + left < right ? -1 : left > right ? 1 : 0)), participantCount: scope.kind === "person" ? 1 : first.participant_count, participantIds: participants, privateParticipants, @@ -1306,6 +2729,7 @@ export class LocalStore { corpusRevision: string; evidenceRevision: string; messages: CorpusMessage[]; + reactions: CorpusReactionFact[]; }> | null { const window = evidenceWindow(options, "Evidence window"); return readTransaction(this.#database, () => { @@ -1321,6 +2745,7 @@ export class LocalStore { corpusRevision, evidenceRevision: scopeEvidenceRevision(this.#database, scope, undefined, window), messages: messageRowsForScope(this.#database, scope, undefined, window).map(corpusMessage), + reactions: reactionFactsForScope(this.#database, scope, window), }; }); } @@ -1556,6 +2981,7 @@ export class LocalStore { foreignKeyViolations: number; corpusRevision: string | null; contactsRevision: string | null; + sources: number; conversations: number; messages: number; profiles: number; @@ -1564,7 +2990,7 @@ export class LocalStore { }> { const quick = get<{ quick_check: string }>(this.#database, "PRAGMA quick_check")?.quick_check ?? "unknown"; const foreignKeys = all(this.#database, "PRAGMA foreign_key_check").length; - const count = (table: "conversations" | "messages" | "profiles" | "addressbook_contacts" | "conversation_contact_labels") => + const count = (table: "corpus_sources" | "conversations" | "messages" | "profiles" | "addressbook_contacts" | "conversation_contact_labels") => get<{ value: number }>(this.#database, `SELECT count(*) AS value FROM ${table}`)?.value ?? 0; return { storeSchemaVersion: userVersion(this.#database), @@ -1572,6 +2998,7 @@ export class LocalStore { foreignKeyViolations: foreignKeys, corpusRevision: this.corpusRevision(), contactsRevision: this.contactsRevision(), + sources: count("corpus_sources"), conversations: count("conversations"), messages: count("messages"), profiles: count("profiles"), diff --git a/src/test-bundle-fixture.ts b/src/test-bundle-fixture.ts new file mode 100644 index 0000000..679bcb0 --- /dev/null +++ b/src/test-bundle-fixture.ts @@ -0,0 +1,228 @@ +import { chmod, mkdir, realpath, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { canonicalJson, sha256 } from "./canonical-json.ts"; + +export const BUNDLE_ARTIFACTS = Object.freeze([ + Object.freeze({ path: "accounts.ndjson", kind: "account" as const }), + Object.freeze({ path: "participants.ndjson", kind: "participant" as const }), + Object.freeze({ path: "conversations.ndjson", kind: "conversation" as const }), + Object.freeze({ path: "messages.ndjson", kind: "message" as const }), + Object.freeze({ path: "reactions.ndjson", kind: "reaction" as const }), + Object.freeze({ path: "tombstones.ndjson", kind: "tombstone" as const }), +]); + +export type BundleArtifactKind = typeof BUNDLE_ARTIFACTS[number]["kind"]; +export type SyntheticBundleRecords = Record>>; + +const CONNECTED_ACCOUNT = "synthetic-connected-account"; +const OBSERVED_AT = "2026-08-20T12:05:00.000Z"; + +function provenance(providerId: string): Record { + return { + providerId, + providerRevision: null, + observedAt: OBSERVED_AT, + connectedAccountProviderId: CONNECTED_ACCOUNT, + }; +} + +function common(kind: BundleArtifactKind, id: string, providerId: string): Record { + return { + schemaVersion: 1, + kind, + id, + accountId: "account-local", + network: "whatsapp", + provenance: provenance(providerId), + }; +} + +export function syntheticBundleRecords(): SyntheticBundleRecords { + return { + account: [{ + ...common("account", "account-local", CONNECTED_ACCOUNT), + displayName: "Synthetic Account", + handle: null, + selfParticipantId: "participant-self", + }], + participant: [ + { + ...common("participant", "participant-self", "participant-provider-self"), + displayName: "Synthetic Self", + handle: "+15555550100", + isSelf: true, + }, + { + ...common("participant", "participant-peer", "participant-provider-peer"), + displayName: "Synthetic Peer", + handle: "peer@example.test", + isSelf: false, + }, + ], + conversation: [{ + ...common("conversation", "conversation-local", "conversation-provider-1"), + type: "direct", + title: "Synthetic Direct", + participantIds: ["participant-self", "participant-peer"], + participantsComplete: true, + startedAt: "2026-08-20T12:00:00.000Z", + lastMessageAt: "2026-08-20T12:04:00.000Z", + }], + message: [ + { + ...common("message", "message-incoming", "message-provider-incoming"), + conversationId: "conversation-local", + senderParticipantId: "participant-peer", + direction: "incoming", + sentAt: "2026-08-20T12:00:00.000Z", + sortKey: "0001", + body: "Synthetic question?", + bodyTruncated: false, + replyTo: null, + edit: null, + deletion: null, + attachments: [], + }, + { + ...common("message", "message-outgoing", "message-provider-outgoing"), + conversationId: "conversation-local", + senderParticipantId: "participant-self", + direction: "outgoing", + sentAt: "2026-08-20T12:01:00.000Z", + sortKey: "0002", + body: "Synthetic answer.", + bodyTruncated: false, + replyTo: { messageId: "message-incoming", providerId: "message-provider-incoming" }, + edit: null, + deletion: null, + attachments: [{ + kind: "image", + mimeType: "image/png", + name: "synthetic.png", + sizeBytes: 1234, + }], + }, + { + ...common("message", "message-truncated", "message-provider-truncated"), + conversationId: "conversation-local", + senderParticipantId: "participant-self", + direction: "outgoing", + sentAt: "2026-08-20T12:02:00.000Z", + sortKey: "0003", + body: "Synthetic partial body", + bodyTruncated: true, + replyTo: null, + edit: null, + deletion: null, + attachments: [], + }, + { + ...common("message", "message-deleted", "message-provider-deleted"), + conversationId: "conversation-local", + senderParticipantId: "participant-peer", + direction: "incoming", + sentAt: "2026-08-20T12:03:00.000Z", + sortKey: "0004", + body: null, + bodyTruncated: null, + replyTo: null, + edit: null, + deletion: { + state: "revoked", + observedAt: "2026-08-20T12:04:00.000Z", + providerRevision: null, + }, + attachments: [], + }, + ], + reaction: [ + { + ...common("reaction", "reaction-dated", "reaction-provider-dated"), + messageId: "message-outgoing", + messageProviderId: "message-provider-outgoing", + participantId: "participant-peer", + body: "heart", + reactedAt: "2026-08-20T12:01:30.000Z", + state: "active", + }, + { + ...common("reaction", "reaction-undated", "reaction-provider-undated"), + messageId: "message-incoming", + messageProviderId: "message-provider-incoming", + participantId: "participant-self", + body: "thumbs-up", + reactedAt: null, + state: "active", + }, + ], + tombstone: [], + }; +} + +export async function writeSyntheticMessageBundle( + parent: string, + records: SyntheticBundleRecords = syntheticBundleRecords(), + options: Readonly<{ + directoryName?: string; + completenessKind?: "bounded-local" | "truncated" | "unknown"; + completenessReason?: string | null; + createdAt?: string; + }> = {}, +): Promise { + const directory = join(parent, options.directoryName ?? "synthetic-message-bundle"); + await mkdir(directory, { mode: 0o700 }); + await chmod(directory, 0o700); + const artifacts: Array> = []; + const counts: Record = {}; + for (const artifact of BUNDLE_ARTIFACTS) { + const values = records[artifact.kind]; + const bytes = Buffer.from(values.map((value) => `${canonicalJson(value)}\n`).join(""), "utf8"); + const path = join(directory, artifact.path); + await writeFile(path, bytes, { mode: 0o600 }); + await chmod(path, 0o600); + counts[artifact.kind] = values.length; + artifacts.push({ + path: artifact.path, + mediaType: "application/x-ndjson", + recordKind: artifact.kind, + records: values.length, + bytes: bytes.byteLength, + sha256: sha256(bytes), + }); + } + const projection = { + schemaVersion: 1, + format: "message-like-me.local-message-bundle", + source: { id: "beeper-local", version: "0.1.0-test" }, + provider: { id: "beeper", version: "1.2.3-test" }, + timestamps: { + startedAt: "2026-08-20T12:00:00.000Z", + finishedAt: "2026-08-20T12:05:00.000Z", + createdAt: options.createdAt ?? "2026-08-20T12:05:01.000Z", + }, + completeness: { + kind: options.completenessKind ?? "truncated", + reason: options.completenessReason === undefined ? "synthetic-limit" : options.completenessReason, + observedFrom: "2026-08-20T12:00:00.000Z", + observedThrough: "2026-08-20T12:04:00.000Z", + }, + warnings: ["synthetic-fixture"], + privacy: { + classification: "private-local", + attachments: "metadata-only", + providerUrls: "excluded", + credentials: "excluded", + }, + counts, + artifacts, + }; + const manifest = { + ...projection, + integrity: { algorithm: "sha256", bundleSha256: sha256(canonicalJson(projection)) }, + }; + const path = join(directory, "manifest.json"); + await writeFile(path, `${canonicalJson(manifest)}\n`, { mode: 0o600 }); + await chmod(path, 0o600); + return realpath(directory); +} diff --git a/src/types.ts b/src/types.ts index e667f98..f099486 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1,10 +1,11 @@ export const CORPUS_SCHEMA_VERSION = 1 as const; -export const METRICS_SCHEMA_VERSION = 1 as const; +export const METRICS_SCHEMA_VERSION = 2 as const; export const PROFILE_SCHEMA_VERSION = 2 as const; export const LEGACY_PROFILE_SCHEMA_VERSION = 1 as const; export const STUDY_PACKET_SCHEMA_VERSION = 2 as const; export const EVALUATION_PACKET_SCHEMA_VERSION = 1 as const; export const CONTACTS_SCHEMA_VERSION = 1 as const; +export const MESSAGE_BUNDLE_SCHEMA_VERSION = 1 as const; export type Direction = "incoming" | "outgoing"; export type BodySource = "text" | "attributed-body" | "unavailable"; @@ -61,6 +62,104 @@ export type CorpusSnapshot = Readonly<{ warnings: readonly string[]; }>; +export type CorpusSourceKind = "imessage" | "bundle"; + +export type CorpusSourceCoverage = Readonly<{ + history: "complete-current-local" | "bounded" | "unknown"; + observedFrom: string | null; + observedTo: string | null; + /** Producer-specific completeness classification, when the import format has one. */ + kind?: string; + /** Producer-supplied categorical reason for incomplete coverage. */ + reason?: string | null; +}>; + +export type CorpusSourceDescriptor = Readonly<{ + /** Per-install source pseudonym used by the local store and CLI. */ + id: string; + kind: CorpusSourceKind; + provider: string; + network: string | null; + /** Private provider account identifier. Ordinary source views omit it. */ + accountId: string | null; + /** Private producer-local source identifier. Ordinary source views omit it. */ + externalId: string; + revision: string; + generatedAt: string | null; + producer: Readonly<{ id: string; version: string }>; + coverage: CorpusSourceCoverage; + manifestSha256: string | null; + identity: unknown; + warnings: readonly string[]; +}>; + +export type CorpusConversationProvenance = Readonly<{ + conversationId: string; + externalId: string; + metadata?: unknown; +}>; + +export type CorpusAttachmentProvenance = Readonly<{ + id: string; + kind: string | null; + mimeType: string | null; + fileName: string | null; + bytes: number | null; +}>; + +export type CorpusMessageProvenance = Readonly<{ + messageId: string; + externalId: string; + /** Provider ordering coordinate. Null when this record has no such coordinate. */ + providerSortKey: string | null; + replyToExternalId: string | null; + attachments: readonly CorpusAttachmentProvenance[]; + metadata?: unknown; +}>; + +export type CorpusReactionFact = Readonly<{ + id: string; + externalId: string; + targetExternalId: string; + conversationId: string | null; + direction: Direction | null; + body: string; + reactedAt: string | null; + state: "active" | "removed"; +}>; + +export type CorpusSourceRecord = Readonly<{ + kind: "account" | "participant" | "reaction" | "tombstone" | "excluded-message"; + id: string; + record: unknown; +}>; + +export type CorpusSourceDeletion = Readonly<{ + entityKind: "account" | "participant" | "conversation" | "message" | "reaction" | "reaction-timeline"; + localEntityId: string | null; + externalId: string; + deletedAt: string; + expectedConversationId?: string; + reason?: "tombstone" | "explicit-exclusion" | "replacement"; +}>; + +export type SourceCorpusSnapshot = Readonly<{ + source: CorpusSourceDescriptor; + conversations: readonly CorpusConversation[]; + conversationProvenance: readonly CorpusConversationProvenance[]; + messages: readonly CorpusMessage[]; + messageProvenance: readonly CorpusMessageProvenance[]; + reactionFacts?: readonly CorpusReactionFact[]; + auxiliaryRecords?: readonly CorpusSourceRecord[]; + deletions?: readonly CorpusSourceDeletion[]; +}>; + +export type MessageBundleSnapshot = Readonly<{ + schemaVersion: typeof MESSAGE_BUNDLE_SCHEMA_VERSION; + manifestSha256: string; + sources: readonly SourceCorpusSnapshot[]; +}>; + export type ContactHandle = Readonly<{ kind: "email" | "phone"; normalizedValue: string; @@ -160,6 +259,9 @@ export type ReactionMetrics = Readonly<{ total: number; incoming: number; outgoing: number; + unknownDirection: number; + dated: number; + undated: number; outgoingReactionRatio: number; }>; diff --git a/src/version.ts b/src/version.ts index 61356c6..cef521b 100644 --- a/src/version.ts +++ b/src/version.ts @@ -1 +1 @@ -export const MESSAGE_LIKE_ME_VERSION = "0.2.0" as const; +export const MESSAGE_LIKE_ME_VERSION = "0.3.0" as const;