Skip to content

host: atomic apply batch verb for local-store, per-call durability model #11

Description

@mfw78

From the local-store durability study (three-lens fable analysis, 2026-07-25). Decision recorded in ADR-0014 (#610).

Decision

The local store is per-call committed durability, not per-event atomic: each set/delete is an individually fsync-durable redb transaction, program-order, read-your-writes, and a trap (fault, panic, OutOfFuel, deadline, crash) freezes state at the last completed call and never rewinds it. Per-event atomicity is rejected: it would roll back the keeper Journal's RESERVED marker on exactly the trap it exists to survive, breaking the reserve/commit/reconcile design (#583-#587, nullislabs/shepherd#574), and it defends against nothing (dispatch is single-actor serialised, redb single-process). The sanctioned atomicity scope is a single opt-in batch verb.

API surface: no new host verbs beyond apply

The host primitive is complete: get/set/delete/list-keys/count/contains/len plus apply(set|delete). set is already an upsert (insert or overwrite). No compare-and-swap, atomic increment, or prefix-delete host verb is warranted, because every module is key-isolated by a keccak prefix (ADR-0003) and dispatch is single-actor serialised, so no two writers ever touch the same key: get-then-apply is effectively atomic, and prefix-delete composes as list-keys + apply(deletes). WIT variants stay additively extensible, so a conditional write-op can be added later if a concrete need appears; not now. Enrich the SDK, not the seam: every WIT addition is a versioned cross-repo change after the cleave, so a minimal host seam is itself the low-effort path.

Build

Host (Wave 1, done)

  • apply WIT verb + ModuleStore::apply (#611): additive apply(list<write-op>) with write-op = set(key,value) | delete(key), one begin_write/commit, whole-batch net quota projection, op-count and byte caps, all-or-nothing. Landed.

SDK ergonomics (Wave 2, after nullislabs/shepherd#611 merges, needs apply in the bindings)

The pain is hand-serialising every list<u8> and rewriting get-modify-set and batch patterns; all pure SDK sugar over the primitives, zero host surface. A module should write typed, intention-revealing code and never touch raw bytes or hand-roll a batch.

  • TypedCell<T>: a borsh-typed key (cell.get/cell.set). The single biggest win; kills manual serialisation.
  • Typed keyed collection: list-by-prefix plus typed values, for sets of things (watches, orders), so modules stop hand-rolling indexes.
  • Counter / accumulator helper (get-inc-set, safe under single-actor dispatch).
  • WriteBatch builder over apply (stage in guest memory, flush in one host call; a pre-flush trap discards for free) and a clear_prefix helper (list-keys + apply-deletes, atomic).
  • Journal guard combinator: reserve, run the submit closure, then commit or park, enforcing reserve-before-effect ordering by API shape. The Journal's single-key markers stay on plain set (they span an await, so they structurally cannot ride one apply).

Test kit (Wave 1, done)

  • Trap-injection harness (#612): a mock local store that traps after the Nth write, sweeping every torn prefix of the keeper reserve/commit flow and asserting reconcile converges exactly-once. Test-only. Plus the review rule: no in-store invariant may span two set calls unless the intermediate state is self-healing or the writes use apply.

Constraints

apply is an additive nexum:host@0.1.0 verb landed pre-cleave (cheap in-monorepo fold, expensive cross-repo ripple after the carve). The Journal and the reserve/commit/reconcile code stay untouched; the SDK helpers add no host surface.

Acceptance

apply lands additive, quota-projected, capped, all-or-nothing (done, nullislabs/shepherd#611); the trap-injection harness proves crash convergence from every torn prefix (done, nullislabs/shepherd#612); ADR-0014 and the docs state the per-call contract with no rollback claim (#610, nullislabs/shepherd#606); the Wave 2 SDK helpers give module authors typed cells, collections, counters, atomic batches and a Journal guard, so no module reimplements store atomicity or serialisation by hand.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions