A sync engine that keeps two systems in agreement — and shows its working. Incremental reads with a resumable cursor, idempotent writes, a dead-letter queue, reconciliation against the destination, and rate limits shaped per service.
git clone https://github.com/vinkurov/converge.git
cd converge
docker compose upThen open http://127.0.0.1:3000 — and break it on purpose. There are buttons for it.
The Notion, Google Sheets and Telegram connectors are not written yet. What runs today is an in-memory source and destination with a generated world of orders, and everything above the connector line — cursors, fingerprints, batching, dead letters, reconciliation, rate limits — is real and covered by tests. The API constraints those three connectors will have to satisfy were read from vendor documentation and are already encoded in DOCUMENTED_LIMITS; the connectors themselves are the next piece of work.
Saying so here rather than further down, because a README that implies working integrations would be the one dishonest thing in the repository.
Every sync tool's screenshot shows a sync working. That proves nothing: the interesting behaviour is what happens when it doesn't.
So the dashboard has switches. Press Destination returns 429 and watch the rate-limited records stay unresolved while the ones that got through are kept. Press Fail halfway through the batch and watch the watermark stop at the failure instead of the last success. Press Reject the newest record and watch it move to the dead-letter queue after the destination refuses it — and watch the sync stop asking about it, rather than re-filing it once a run forever. Click any row at the destination to edit or delete it behind the sync's back, then press Check for drift: the source is untouched, so nothing in the incremental loop can possibly notice, and only a full read of the destination finds it.
It is a real sync through the real engine. Only the records are generated.
Keeping two systems in agreement is not message passing, and the difference is where the work is. A relay hands a message on and is finished. A sync has to answer questions that persist between runs:
| Where did I get to? | A cursor, resumable after a crash, that never claims to be past a record it did not handle. |
| Has this actually changed? | Otherwise every run rewrites everything and burns the destination's quota. |
| What if half the batch failed? | Rolling back a write that succeeded is impossible at most destinations and pointless at the rest. |
| When do I give up on a record? | Retrying forever blocks the queue; giving up silently loses data. |
| What if someone edits the destination? | The source's copy is untouched, so the incremental loop is blind to it by construction. |
| How fast am I allowed to go? | Every service says it differently, and one global "requests per second" is wrong for all of them. |
Six decisions, each of which could have gone the other way. If you are reading the code, these are the parts worth understanding first — the modules are small and the reasoning is written where the code is.
A position is (timestamp, record id), not a timestamp. Sources order changes by modification time, and timestamps collide — a bulk edit writes twenty rows in the same millisecond, and "time T" cannot express "I handled three of those twenty".
If the third of ten records is unresolved, the watermark stops at the second even though four through ten succeeded. Advancing past it would make that record invisible forever, and the run would report success having silently dropped it. The cost is re-reading records four to ten, which is a read and nothing more.
Each run starts reading before the watermark. A source's ordering is only as trustworthy as its clock, and clocks lie in two ways here: a record can be committed with a time slightly behind ones already returned, and a multi-node source has skew between nodes. Re-reading a window is a cheaper correctness argument than an exact boundary, which is only correct while an assumption about someone else's clock holds.
It is affordable because a re-read record that has not changed never reaches the destination — which brings us to the next one.
Source records carry fields that move whenever anyone touches the row: a last-edited timestamp, a revision counter, who saved it. Hash the whole record and every touch reads as a change, so the sync rewrites the destination for edits that altered nothing it copies. Hash the mapped result — the fields actually being written — and "changed" means what it says.
JSON.stringify cannot be the encoder, for four reasons that are defects rather than preferences: key order is insertion order; {"a": "1"} and {"a": 1} collapse together without a type tag; concatenation without lengths lets ["ab","c"] and ["a","bc"] produce identical bytes; and NaN becomes null, so a changed record would hash the same as a null one and leave the destination stale forever.
A transient failure leaves the record unresolved, which is what holds the watermark. A permanent one — a value the destination refuses — goes to the dead-letter queue on the first attempt, because retrying a rejection never starts working.
The attempt budget belongs to the fingerprint, not the record: a row that failed four times because it was malformed gets a clean start the moment someone fixes it. And a record that was given up on is remembered as such, so the next run does not plan the same write, collect the same refusal and file a second dead letter — once a run, forever. Edit the record to anything else and it is picked up immediately, with no flag for anyone to clear.
| Service | Documented limit | Shape |
|---|---|---|
| Notion | ~3 requests/second average, bursts allowed | average with burst |
| Google Sheets | 60 writes per minute per user | count in a rolling window |
| Telegram | ~1 message/second into one chat, ~30/second overall | minimum gap, plus a ceiling |
One counter for the whole process would be simultaneously too strict and too loose. The primitive is reserve, not "may I go?" — asking is unusable once more than one call is being scheduled, because five callers ask at the same instant, all hear yes, and all go.
Two findings from the tests are written on the code: an accumulator in floating point drifts until an idle limiter refuses to say "now" (it keeps integer microseconds instead), and the burst algorithm assumes reservations arrive in time order, which the per-chat gap breaks — so Telegram's overall ceiling is a rolling window, which gives the same answer whatever order it is asked in.
The watermark, the per-record state and the dead letters are one fact told three ways. A crash between any two of them leaves the store lying: a cursor past a record whose state was never saved means the next run skips work it never did. They are written in a single transaction, and there is a test that makes the middle of a commit throw and asserts that nothing moved — not even the run's own status.
An incremental sync only looks at records the source says changed. A row deleted or hand-edited at the destination is invisible to that loop forever: the source's copy is untouched and its modification time has not moved. The only way to see it is to read the destination back in full and compare — which is exactly the work the incremental loop exists to avoid, so it is scheduled rarely and on purpose. Where a destination cannot be read back at all, the dashboard says so rather than reporting "no drift found".
Every setting is an environment variable prefixed CONVERGE_, validated once at startup. Two of them can be made to contradict each other, and the resulting failure is silent — a sync that reports success and makes no progress — so the service refuses to start instead.
| Variable | Default | What it does |
|---|---|---|
CONVERGE_PORT |
3000 |
Port to listen on. |
CONVERGE_HOST |
0.0.0.0 |
Interface to bind. A container needs 0.0.0.0 for its port mapping to reach anything; on a shared host set 127.0.0.1 and put a proxy in front. |
CONVERGE_DB_PATH |
./data/converge.sqlite |
SQLite file. |
CONVERGE_INTERVAL_MS |
15000 |
How often a sync runs. |
CONVERGE_OVERLAP_MS |
300000 |
How far before the watermark each run starts reading. Sized from clock skew, not data volume. |
CONVERGE_PAGE_SIZE |
100 |
Records requested per read. |
CONVERGE_WRITE_BATCH_SIZE |
50 |
Records handed to the destination per write. |
CONVERGE_MAX_NEW_RECORDS_PER_RUN |
1000 |
How much new work a run takes on. Counts records past the watermark; counting re-read ones would let a wide overlap window stall the sync forever. |
CONVERGE_MAX_RECORDS_PER_RUN |
5000 |
Hard ceiling on records pulled, protecting memory. Hitting it without reaching a new record is reported as a misconfiguration. |
CONVERGE_MAX_ATTEMPTS |
5 |
Attempts one version of a record gets before it is dead-lettered. |
CONVERGE_RECONCILE_EVERY_MS |
300000 |
How often the destination is read back in full. 0 disables it. |
CONVERGE_MAX_EVENT_STREAMS |
32 |
Dashboard streams allowed open at once. |
CONVERGE_DEMO |
true |
The generated world. There are no real connectors yet, so false currently refuses to start. |
CONVERGE_LOG_LEVEL |
info |
fatal … trace, or silent. |
The engine knows nothing about any particular service. A source hands back records in modification order; a destination writes them idempotently and reports each record's fate separately. Everything about cursors, retries, dead letters and rate limiting then applies to it for free.
interface Source {
name: string;
limits: readonly Limit[];
pull(input: { since: number | null; page: string | null; limit: number }): Promise<PullResult>;
}
interface Destination {
name: string;
limits: readonly Limit[];
write(records: readonly WriteInput[]): Promise<Map<string, WriteResult>>;
snapshot?(): Promise<Map<string, unknown>>; // only needed for drift checking
remove?(recordIds: readonly string[]): Promise<Map<string, WriteResult>>;
}Two requirements that are not negotiable. write must be idempotent — the engine re-sends after a crash and after a retry, and has no way to know whether a request that timed out was applied. And the source must order by modification time; one that cannot promise that cannot be synced incrementally at all.
Stated plainly, because they matter more than the feature list:
- No real connectors yet. See the top of this file.
- No authentication. Anyone who can reach the port sees every synced record and can press the buttons that break the sync. This is why
docker-compose.ymlbinds to127.0.0.1. - One-way. The source is the truth; there is no conflict resolution, because two-way sync is a different and much larger problem.
- One machine. Storage is SQLite. That is a deliberate ceiling, not an oversight.
- The demo resets on restart. Its destination is a Map that does not survive the process, so the record states describing it are wiped at startup — state that outlives the system it describes is worse than no state. A deployment with real connectors would not do this.
npm install
npm run dev
npm test # 164 tests
npm run test:coverage
npm run lint
npm run typecheck
npm run buildCoverage is 98% of lines. The more useful number is that running the thing found four defects the tests had not, each now covered: a permanently refused record re-filed once per run forever, a dead letter closed by the very commit that filed it, a fault switch that silently did nothing because the record it targeted was skipped by fingerprint, and record states that outlived the destination they described. They are in the git history with the reasoning.
Layout
src/domain/ watermark, fingerprint, rate limit shapes — pure, no I/O
src/engine/ batch planning and settlement, the run, reconciliation
src/store/ SQLite schema and the atomic run commit
src/connectors/ the two interfaces, and in-memory implementations with fault injection
src/http/ API, event stream, dashboard assets
public/ the dashboard: three files, no build step
Hono for HTTP, better-sqlite3 for storage, zod for configuration. The image is 281 MB on arm64; docker compose stop with a dashboard connected takes 0.15s and exits 0, which CI asserts.
MIT — see LICENSE.