docs(protocol): real-time capsule-holder discovery (find_providers + holdings-announce)

docs/protocol/peer-network.md

@@ -1,7 +1,7 @@
 ---
 sidebar_position: 13
 title: "L7 · DIG Node peer network"
-description: "The normative node↔node protocol: mTLS peer identity (peer_id = SHA-256(TLS SPKI DER)), the two RPC tiers (mTLS-authenticated PEER/CONTROL vs anonymous PUBLIC-READ so browsers can retrieve content), the dual-mode public gateway (rpc.dig.net's mTLS front for node-class clients — the dig-store CLI, the SDK, any DIG-identity-key holder — across the full dig.local/localhost/rpc.dig.net ladder, plus its plain-HTTPS+CORS front for browsers, with an ephemeral self-signed client certificate for channel-bound anonymous reads), the ordered NAT-traversal ladder (direct → UPnP → NAT-PMP → PCP → relay-coordinated hole-punch (signalling only) → relayed/TURN transport), the relay's four roles (STUN, introducer, hole-punch signalling, relayed transport), STUN reflexive-address discovery, introducer + gossip peer discovery, PEX peer-exchange (node↔node stream + the RLY-008 relay introducer binding), the Kademlia DHT with provider records that locate which peers hold content (find_node/find_providers/add_provider/ping over a framed dig-nat mTLS stream; content-key = SHA-256(domain-tag ‖ store_id[‖root[‖retrieval_key]])), the relay RelayMessage wire (RLY-001..RLY-008), the peer RPC methods (dig.getPeers/dig.announce/dig.getNetworkInfo/dig.getAvailability/dig.listInventory/dig.fetchRange), and the relay-last-fallback invariant (prefer hole-punch signalling over full relaying)."
+description: "The normative node↔node protocol: mTLS peer identity (peer_id = SHA-256(TLS SPKI DER)), the two RPC tiers (mTLS-authenticated PEER/CONTROL vs anonymous PUBLIC-READ so browsers can retrieve content), the dual-mode public gateway (rpc.dig.net's mTLS front for node-class clients — the dig-store CLI, the SDK, any DIG-identity-key holder — across the full dig.local/localhost/rpc.dig.net ladder, plus its plain-HTTPS+CORS front for browsers, with an ephemeral self-signed client certificate for channel-bound anonymous reads), the ordered NAT-traversal ladder (direct → UPnP → NAT-PMP → PCP → relay-coordinated hole-punch (signalling only) → relayed/TURN transport), the relay's four roles (STUN, introducer, hole-punch signalling, relayed transport), STUN reflexive-address discovery, introducer + gossip peer discovery, PEX peer-exchange (node↔node stream + the RLY-008 relay introducer binding), the Kademlia DHT with provider records that locate which peers hold content (find_node/find_providers/add_provider/ping over a framed dig-nat mTLS stream; content-key = SHA-256(domain-tag ‖ store_id[‖root[‖retrieval_key]])), provider records announced promptly on an inventory gain (whichever path put a capsule on disk — so a reader that caches a capsule becomes a discoverable holder of it) and withdrawn on removal, the holdings-announce wire (gossip-pool broadcast opcode 222 — the signed, batched add/remove statement of a peer's holdings, TLS-leaf-key ECDSA with the `dig:holdings:v1` preimage and fail-closed acceptance), the gossip/peer-RPC port relationship (gossip 9445 = peer 9444 + 1 — translate before dialing), the relay RelayMessage wire (RLY-001..RLY-008), the peer RPC methods (dig.getPeers/dig.announce/dig.getNetworkInfo/dig.getAvailability/dig.listInventory/dig.fetchRange), and the relay-last-fallback invariant (prefer hole-punch signalling over full relaying)."
 keywords:
   - peer network
   - peer_id
@@ -24,6 +24,10 @@ keywords:
   - Kademlia
   - provider record
   - content discovery
+  - holdings announce
+  - opcode 222
+  - capsule holder discovery
+  - find_providers
 tags:
   - dig-node
   - relay
@@ -289,13 +293,70 @@ A provider record is **soft state**, not a permanent entry, so an offline holder
 - **Absolute expiry.** `expires_at` is set to `now + TTL` in absolute Unix seconds. A record at or after its `expires_at` is treated as **absent**.
 - **Republish before expiry.** The holder re-announces (a fresh record with a new `expires_at`) on an interval strictly shorter than the TTL, so its records never expire while it is online.
 - **Withdraw on removal.** A node that no longer holds content stops announcing it; the record then ages out on its TTL (no explicit delete is required).
+- **Announce the gain promptly, don't wait for the republish tick.** A holder diffs its inventory against the content-id set it last announced and PUTs a record for each newly-held key **as soon as it gains it**, so freshness is not bounded by the republish interval. This is what makes the network self-scaling: whichever path put a capsule on disk — a cache fill after a remote read, a hosted pin, a store sync — the node announces it and a reader that caches a capsule becomes a discoverable holder of it, so content replicates toward wherever it is read.
 - **GC drops the expired.** A responder discards expired records on read and does not return them.
 - **Inbound RPC populates the routing table bidirectionally.** On **every** inbound DHT RPC, the responder folds the **mTLS-verified caller** (its `Contact`) into its own routing table — every request is evidence the caller is alive, so a node that queries you teaches you about itself. The caller identity MUST come from the authenticated transport, never from a field the caller sets.
 
+:::warning Two ports, one node — translate before you dial
+A node's **gossip** listener (`DIG_GOSSIP_PORT`, default **9445**) and its **peer-RPC / DHT** listener (`DIG_PEER_PORT`, default **9444**) are different sockets on the same identity — `gossip_port = peer_port + 1`, a fixed offset. Every address a peer offers as a dial target for a DHT stream or for [`dig.getAvailability`](#availability) / [`dig.fetchRange`](#range) is a **peer-RPC** address; the addresses a node observes on its own gossip pool are **gossip** addresses. A path that seeds the DHT routing table or a download-candidate list from a gossip-pool address MUST subtract the offset first. Dialing `9445` for a peer-RPC stream reaches the gossip protocol instead, and the failure surfaces as a content miss rather than as a connection error.
+:::
+
+#### The holdings-announce wire — opcode 222 {#holdings-announce}
+
+Beside the pulled provider records, the peer protocol defines a **signed holdings announce**: one message in which a peer states, under its own key, which content keys it serves and which it no longer serves. It is a gossip-pool broadcast message type — **`HoldingsAnnounce`, opcode 222** of the peer message set, classed as a Plumtree eager/lazy-push broadcast at **bulk** priority — and not one of the four [DHT RPC](#dht-wire) methods.
+
+The shapes below are a **frozen, byte-exact contract**: what a conforming implementation emits, and what it requires before it accepts a statement about another peer's holdings. Every field is covered by the holder's signature.
+
+**The message.** All integers big-endian; `⟨lp⟩` marks a `u16`-length-prefixed byte string.
+
+| Field | Shape | Meaning |
+|---|---|---|
+| `provider_peer_id` | 64 lowercase hex | the holder's `peer_id`; equals `SHA-256(provider_spki)`, and an acceptor **derives** it from the SPKI rather than trusting the carried value |
+| `provider_spki` | TLS leaf `SubjectPublicKeyInfo` DER (P-256, ~91 B) | both what the `peer_id` hashes **and** the public key that verifies `signature` — no certificate and no chain are on the wire |
+| `seq` | `u64` | monotonic per holder; a higher `seq` states a later position than a lower one |
+| `announced_at` | `u64` | Unix seconds the announce was produced |
+| `changes` | 1..**256** deltas | the batch (`MAX_CHANGES` = 256; a larger batch is rejected, not truncated) |
+| `signature` | ECDSA-P256, ASN.1 DER (~70-72 B) | the holder's signature over the signing message below |
+
+Each delta is one of two kinds:
+
+```text
+Add    = 0x01 ‖ content_key[32] ‖ addr_count(u16)
+                ‖ ( host_len(u16) ‖ host ‖ port(u16) )*
+                ‖ expires_at(u64)      — now serving content_key at these addresses
+Remove = 0x02 ‖ content_key[32]        — no longer serving content_key
+```
+
+`content_key` is the same 64-hex [content key](#dht) the DHT is keyed by (store `0x01` / capsule `0x02` / resource `0x03`), so an announce and a `find_providers` name content identically; `addresses` and `expires_at` carry the same meaning they do on a [`ProviderRecord`](#dht-wire). The delta encoding above is the **canonical encoding** — deterministic, no maps or sets — and both the wire body and the signed bytes use it, so signer and verifier never diverge:
+
+```text
+wire     = peer_id⟨lp⟩ ‖ spki⟨lp⟩ ‖ seq(u64) ‖ announced_at(u64)
+           ‖ change_count(u16) ‖ canonical_encode(changes) ‖ signature⟨lp⟩
+
+signed   = "dig:holdings:v1" ‖ peer_id[32] ‖ seq(u64) ‖ announced_at(u64)
+           ‖ canonical_encode(changes)
+```
+
+The `dig:holdings:v1` domain tag is part of the signed preimage, so a holdings signature can never be replayed as a signature over any other DIG message; the version suffix makes a future preimage change an explicit, distinguishable domain. The signature is computed over the **full preimage** (ECDSA-P256 with SHA-256 internally), not over a pre-hashed digest. A truncated, over-long, or trailing-byte frame is rejected outright.
+
+**Acceptance is fail-closed — five checks, in order.** An announce is acceptable only when every one of them passes; the first failure rejects the whole message, and with it every delta the message carried:
+
+1. `changes` holds at most 256 deltas.
+2. `provider_peer_id` decodes as exactly 64 hex characters.
+3. `SHA-256(provider_spki)` equals that `peer_id` — the identity is derived, never taken on the announcer's word.
+4. `provider_spki` parses as a P-256 (`prime256v1`) EC public key.
+5. the signature verifies over the signing message under that key.
+
+Because the signing key is the **TLS leaf key the `peer_id` already commits to**, the message is unforgeable standalone — no handshake, proof of possession, or certificate chain is needed, and there is no separate identity binding a third party could graft onto a copied public key. This is what the signature is for: only the holder can state holdings under the holder's `peer_id`, or name the addresses a reader would dial for them.
+
+A holdings announce is **public discovery data addressed to every peer**, so it is signed and mTLS-carried rather than sealed to a recipient — the same treatment public consensus broadcasts get, and unlike the recipient-sealed directed peer messages.
+
+**What a valid signature does and does not establish.** It establishes *who* made the statement. It does not establish that the content is there: an announced address is authenticated only as far as the announcer's `peer_id`, so addresses are untrusted dial hints that MUST NOT be amplified, and a false claim costs a reader one failed fetch attempt — content is merkle-verified against the [chain-anchored root](#range-integrity) before it is trusted or re-served.
+
 #### How a node uses the DHT
 
 - **On content-want** (a user asks for `store_id`, `store_id:root`, or a specific resource): derive the matching content key, run `find_providers`, then reach each returned provider over the [NAT ladder](#nat-traversal) and fetch via [`dig.getAvailability`](#availability) + [`dig.fetchRange`](#range). The DHT is **step 1 of the [multi-source download](#multi-source)** — it finds the candidate holders the download then fans out across.
-- **On inventory-change** (the node gains or loses content it serves): `add_provider` for each new content key, and stop announcing what it no longer holds. Run republish on the configured interval.
+- **On inventory-change** (the node gains or loses content it serves): diff the current inventory's content ids against the set last announced, `add_provider` each newly-held key promptly, and withdraw each key it no longer holds. Run republish on the configured interval. A node announces its inventory at **store** and **capsule** granularity, so a peer can be found by store id or by exact `store_id:root`.
 - **Bootstrap** the routing table from existing discovery — the gossip peer pool ([§4b](#discovery)) or the relay introducer ([§4a](#discovery)) — then a self-lookup (`find_node` on the node's own id) fills the table. The DHT never hard-depends on a live relay.
 
 ### 4d · PEX — peer exchange {#pex}
@@ -740,6 +801,7 @@ The peer network is implemented by several crates that must interoperate byte-fo
 | **NAT ladder** | the ordered strategies DIRECT → UPnP → NAT-PMP → PCP → hole-punch (relay signalling only) → RELAYED/TURN (relay carries data), relay-data-last | that every `connect(peer)` implementation prefers direct, prefers hole-punch signalling over full relaying, and proxies the stream only as a last resort |
 | **DHT content key** | `SHA-256(tag ‖ canonical bytes)` with tags `0x01` store (`store_id`), `0x02` root/capsule (`store_id ‖ root`), `0x03` resource (`store_id ‖ root ‖ retrieval_key`); node id = `peer_id`; distance = XOR; bucket = `255 − leading_zeros` ([§4c](#dht)) | that every node derives the identical content key for the same content, and places nodes + content in one 256-bit keyspace, so a provider record announced by one implementation is found by another |
 | **DHT RPC wire** | the four `type`-tagged methods `find_node` / `find_providers` / `add_provider` / `ping` (+ the `error` envelope), `u32`-BE length-prefixed JSON over an authenticated dig-nat mTLS stream (same framing as the relay control messages), `find_providers` always returning `closer` ([§4c](#dht-wire)) | that any node's DHT speaks the same locate-the-holders wire; `dig-nat`/`dig-dht`/`dig-node` conform |
+| **Holdings-announce wire** | gossip-pool broadcast opcode **222**; wire `peer_id⟨lp⟩ ‖ spki⟨lp⟩ ‖ seq(u64) ‖ announced_at(u64) ‖ change_count(u16) ‖ canonical_encode(changes) ‖ signature⟨lp⟩` (BE, `u16` length prefixes), deltas `0x01` add (`content_key ‖ addresses ‖ expires_at`) / `0x02` remove (`content_key`), at most 256 per batch; signed `"dig:holdings:v1" ‖ peer_id[32] ‖ seq ‖ announced_at ‖ canonical_encode(changes)` with the TLS leaf key (ECDSA-P256-SHA256, ASN.1); acceptance fail-closed on the five checks, `peer_id` derived as `SHA-256(spki)` ([§4c](#holdings-announce)) | that a signed statement of holdings is byte-identical and independently verifiable across implementations, and that no peer can state holdings under another peer's identity |
 | **DHT shapes** | `Contact { peer_id:<64hex>, addresses:[{host,port,kind}] }` and `ProviderRecord { content_key, provider_peer_id, addresses, expires_at }`, `addresses[]` byte-compatible with `dig.getPeers`; provider records are TTL'd (absolute `expires_at`), republished before expiry, GC'd when stale, and every inbound RPC folds the mTLS-verified caller into the routing table | that returned contacts/records drop straight into a dial target and that provider state is soft state that ages out |
 
 A reimplementation of any peer crate conforms iff it reproduces these — the same discipline that keeps the [read path parity-locked](./conformance-and-parity.md).

package.json

@@ -1,6 +1,6 @@
 {
   "name": "docs-dig-net",
-  "version": "0.5.3",
+  "version": "0.6.0",
   "private": true,
   "scripts": {
     "docusaurus": "docusaurus",