Skip to content

feat: add ordered gatekeeper sync#1213

Open
Bushstar wants to merge 8 commits into
mainfrom
bush/ordered-sync
Open

feat: add ordered gatekeeper sync#1213
Bushstar wants to merge 8 commits into
mainfrom
bush/ordered-sync

Conversation

@Bushstar
Copy link
Copy Markdown
Contributor

@Bushstar Bushstar commented Jun 4, 2026

Overview

This PR updates the Hyperswarm mediator to use Gatekeeper’s index export endpoint for local bootstrap and ongoing local sync-store hydration.

The intent is to make Hyperswarm sync faster and more predictable by preserving Gatekeeper acceptance order where it is available. Negentropy still reconciles by stable operation identity, but nodes that are empty or far behind can now catch up from an ordered peer before falling back to normal Negentropy repair.

What Changed

Hyperswarm Bootstrap

  • Replaced the old bootstrap path that used getDIDs() / exportBatch().
  • Hyperswarm now hydrates its sync store from Gatekeeper with:
exportIndex({ mode: 'snapshot', ... })
exportIndex({ mode: 'changes', includeOperations: true, ... })
  • Added mediator sync state for:
    • snapshot completion
    • snapshot cursor
    • snapshot checkpoint cursor
    • changes cursor
  • Sync cursors are saved atomically with operation persistence.
  • The mediator no longer resets its sync store on every startup.

Ordered Operation Storage

  • Added syncOrder to Hyperswarm sync-store records.
  • Added ordered iteration and ordered counts to sync-store adapters.
  • Added sync-state storage to memory, SQLite, and Postgres sync stores.
  • Existing rows can be backfilled with syncOrder when Gatekeeper later exports ordered operation metadata.

Ordered Catch-Up

  • Added ordered catch-up capabilities to Hyperswarm peer negotiation.
  • Peers advertise whether ordered catch-up is available and ready.
  • A node can use ordered catch-up when it is empty or sufficiently far behind a peer with a fully ordered sync store.
  • Ordered catch-up pages operations by (syncOrder, id).
  • After ordered catch-up completes, the mediator starts normal Negentropy reconciliation.

Negentropy Transfer Ordering

  • Requested operation pushes are now sorted by:
    1. syncOrder when available
    2. signed timestamp
    3. operation hash

Gatekeeper Export Additions

  • Added optional includeOperations support for changes export.

Gatekeeper DB Cleanup

  • Updated Gatekeeper DB adapters so operation-level change rows are only written for newly accepted operation events.
  • setEvents() now supports operationEvents to avoid treating retained history as newly accepted operations.
  • Flattened unreleased Gatekeeper DB migration layers:
    • Postgres now starts directly with the current schema.
    • SQLite creates the final index_changes shape directly.
    • Redis keeps its real release-to-current DID index bootstrap.
    • Mongo keeps its real deployed-index repair and transaction requirement.

Config

  • KC_HYPR_ORDERED_CATCHUP_ENABLE

Resolves

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants