diff --git a/src/css/specs.css b/src/css/specs.css index 4caa2fb12..9a51c8f94 100644 --- a/src/css/specs.css +++ b/src/css/specs.css @@ -30,6 +30,49 @@ body { background: url('/img/watermark-ratified.svg'); } +.ipip-historic .watermark { + display: block; + background: url('/img/watermark-historic.svg'); + background-color: #faf8f5; +} + +.ipip-historic { + --highlight-colour: #b08050; + --link-colour: #8b6914; + --note-colour: #b08050; + --warning-colour: #c07030; + --standard-gradient: linear-gradient(to right, #5c4a3a, #b08050); +} + +.ipip-historic header { + border-bottom-color: #b08050; +} + +.ipip-historic h1, +.ipip-historic h2, +.ipip-historic h3, +.ipip-historic h4, +.ipip-historic h5, +.ipip-historic h6 { + color: #5c4a3a; +} + +.ipip-historic :is(h2, h3, h4, h5, h6) bdi.secno { + color: #9a8a7a; +} + +.ipip-historic pre { + background: #faf6f0; + border-color: #b08050; + box-shadow: -1px 0 0 0 #b08050, 0 0 0 1px #e5d8c8; +} + +.ipip-historic div.note, +.ipip-historic div.warning, +.ipip-historic div.example { + background: #faf6f0; +} + #ipseity-back-to-root { margin-bottom: 2rem; background: var(--standard-gradient); diff --git a/src/img/watermark-historic.svg b/src/img/watermark-historic.svg new file mode 100644 index 000000000..ab34bee09 --- /dev/null +++ b/src/img/watermark-historic.svg @@ -0,0 +1,8 @@ + + + + Historic + + + + diff --git a/src/ipips/ipip-0526.md b/src/ipips/ipip-0526.md new file mode 100644 index 000000000..770b5c9c1 --- /dev/null +++ b/src/ipips/ipip-0526.md @@ -0,0 +1,333 @@ +--- +title: 'IPIP-0526: Historic Bitswap Provider Publishing HTTP API' +date: 2026-08-08 +ipip: historic +editors: + - name: Marcin Rataj + github: lidel + affiliation: + name: Shipyard + url: https://ipshipyard.com +thanks: + - name: Adin Schmahmann + github: aschmahmann + affiliation: + name: Shipyard + url: https://ipshipyard.com + - name: Andrew Gillis + github: gammazero + affiliation: + name: Shipyard + url: https://ipshipyard.com + - name: Henrique Dias + url: https://hacdias.com/ + github: hacdias + - name: Rinse + github: Rinse12 + affiliation: + name: Bitsocial + url: https://bitsocial.net +relatedIssues: + - https://github.com/ipni/index-provider/issues/403 + - https://discuss.ipfs.tech/t/19545/9 +order: 526 +tags: ['ipips'] +xref: + - http-routing-v1 +--- + +## Summary + +This IPIP documents the `PUT /routing/v1/providers/` HTTP endpoint for publishing signed Bitswap provider records, introduced in 2022 for [index-provider](https://github.com/ipni/index-provider) integration with IPNI. + +The API is deprecated because it was never formally specified, lacks conformance tests, has only partial implementation (see notes below), and cannot announce content available via HTTP ([Trustless Gateways](https://specs.ipfs.tech/http-gateways/trustless-gateway/)) due to being naively hardcoded to the `bitswap` schema. + +This document describes the wire format and the observed behavior of the reference implementation, including reproducible test vectors, so that existing deployments keep an interoperability reference while a replacement is designed. + +:::warning + +This API was never formally specified, and community projects have been reverse engineering it. This historic IPIP discourages further adoption and is not intended to be merged into :cite[http-routing-v1]. + +::: + +## Motivation + +The Delegated Routing V1 HTTP API (:cite[http-routing-v1]) was designed primarily for content discovery via `GET` endpoints. However, the ecosystem also needed a mechanism for peers to publish their provider records to routing servers without relying solely on DHT announcements. + +The [index-provider](https://github.com/ipni/index-provider) project (created in 2021) required a way for IPFS nodes (such as Kubo) to advertise their content to indexers (like cid.contact) alongside DHT. This created a parallel discovery mechanism where providers could reach indexing networks without relying exclusively on the distributed hash table. + +The provider publishing API was first added to the Reframe protocol in August 2022 ([ipfs/specs#285](https://github.com/ipfs/specs/pull/285), [go-delegated-routing#37](https://github.com/ipfs/go-delegated-routing/pull/37)). In November 2022, the code was refactored to use `/routing/v1` paths, and the `WriteBitswapRecord` schema was introduced. The code was later migrated to [go-libipfs](https://github.com/ipfs/go-libipfs) and then to [boxo](https://github.com/ipfs/boxo). + +When [IPIP-337](https://github.com/ipfs/specs/pull/337) introduced the Delegated Routing V1 HTTP API specification in February 2023, write operations were [explicitly excluded](https://github.com/ipfs/specs/pull/370) to reduce scope. [IPIP-378](https://github.com/ipfs/specs/pull/378) was opened to design a modernized, protocol-agnostic replacement that would support announcing multiple CIDs in a single request, but work on it was never completed before Protocol Labs restructuring. As a result, the Bitswap-specific `PUT /routing/v1/providers/` endpoint remained in boxo but was never formally documented. + +The wire format has not changed since January 2023, when the HTTP method was changed from POST to PUT. The implementation remains in [boxo v0.35.2](https://github.com/ipfs/boxo/tree/v0.35.2/routing/http) but is marked as deprecated, pending replacement by a protocol-agnostic alternative. + +Despite being undocumented, community projects have reverse engineered this API to implement provider publishing. [Bitsocial](https://bitsocial.net) (previously named [Plebbit](https://plebbit.com)), a peer-to-peer social media protocol, uses it so that nodes running apps such as [Seedit](https://seedit.app) can announce content to volunteer-run trackers instead of the public DHT (see Known users below). Creating this historic IPIP discourages further adoption of this deprecated API, while providing a reference for its existing users and for discussions about replacements. + +## Detailed design + +This IPIP documents the `PUT /routing/v1/providers/` endpoint as a historical artifact. This specification is self-contained and not intended to be merged into :cite[http-routing-v1]. + +### `PUT /routing/v1/providers/` + +This endpoint allows peers to publish signed provider records announcing which CIDs they can provide. + +In this document, "reference client" and "reference server" refer to the implementations in [boxo v0.35.2 `routing/http`](https://github.com/ipfs/boxo/tree/v0.35.2/routing/http): the client is what Kubo uses to publish, and the server is the handler that index-provider mounts. + +#### Request Headers + +The reference client sets no `Content-Type` header on the request (only `User-Agent`), and the reference server never reads it: the body is decoded as JSON regardless of headers. Clients MAY send `Content-Type: application/json`; servers MUST NOT require it. + +#### Request Body + +```json +{ + "Providers": [ + { + "Schema": "bitswap", + "Protocol": "transport-bitswap", + "Signature": "", + "Payload": { + "Keys": ["", ...], + "Timestamp": , + "AdvisoryTTL": , + "ID": "", + "Addrs": ["", ...] + } + } + ] +} +``` + +Example with concrete values (a validly signed record; see [Test fixtures](#test-fixtures) for the key material): + +```json +{ + "Providers": [ + { + "Schema": "bitswap", + "Protocol": "transport-bitswap", + "Signature": "mlGWqELZSoVjY22w6NxT7TuNUj5BmQRlrv/x27jQHo5pM3CYZRlJ834bi0UazFuXlH2SuMBxdwfELdXXwVkjZBQ", + "Payload": {"Keys":["bafkreigur6gzxm3ykiol7ywou3iy3obruzs2q7boizj7oznznid34dzc3e"],"Timestamp":1725833163372,"AdvisoryTTL":86400000000000,"ID":"12D3KooWKcTtoUYcQVfURdwUZtPsanm97SdEf7DkesyxTz7kkD2z","Addrs":["/ip4/198.51.100.1/tcp/4001","/ip4/198.51.100.1/udp/4001/quic-v1"]} + } + ] +} +``` + +:::note + +The `Signature` above is a real signature that verifies against this exact `Payload`. Because the signature covers the `Payload` bytes exactly as sent, the single-line `Payload` value must stay byte-identical: reformatting it (for example, pretty-printing) invalidates the signature. + +::: + +Where: + +- `Schema`: MUST be `"bitswap"` for this record type. +- `Protocol`: SHOULD be `"transport-bitswap"`. The reference server does not validate this field. +- `Signature`: Multibase base64 (prefix `m`, no padding) signature over the SHA-256 hash of the `Payload` bytes exactly as they appear in the request body (see Signature Creation below). +- `Payload`: The signed provider information: + - `Keys`: Array of string-encoded CIDs that the peer provides. The reference server accepts both CIDv0 and CIDv1; new records SHOULD use CIDv1. + - `Timestamp`: Unix timestamp in milliseconds when the record was created. + - `AdvisoryTTL`: Suggested time-to-live in nanoseconds (as JSON number). + - `ID`: The provider's Peer ID (string-encoded as Multihash in Base58btc or CIDv1 with libp2p-key codec). + - `Addrs`: Array of multiaddrs where the provider can be reached. + +`ID`, `Timestamp`, and `AdvisoryTTL` are required in practice: a record without `ID` fails signature verification (`403`), and the reference server assumes `Timestamp` and `AdvisoryTTL` are present and crashes on records without them (the client sees a dropped connection instead of an HTTP error). + +The `Providers` array can carry multiple records, but the reference client always sends exactly one record per request. Multi-record requests are accepted by the reference server, yet remain unexercised by real clients (see the warning about non-atomic processing under Response Status Codes). + +:::note + +Implementation detail: At the time of writing this IPIP (2025Q4), [boxo v0.35.2](https://github.com/ipfs/boxo/releases/tag/v0.35.2) enforces `Schema` (must be `"bitswap"`, returns 400 otherwise) and signature verification (returns 403 on failure). However, `Protocol` is not validated: it is echoed in the response and never passed to the backing service. In [index-provider v0.16.0](https://github.com/ipni/index-provider/releases/tag/v0.16.0), `Timestamp` and `AdvisoryTTL` from the request are ignored: the server uses `time.Now()` for record timestamps and returns its own configured `cidTtl` in the response. Only `Keys`, `ID`, and `Addrs` are used for IPNI announcements, making parts of this specification effectively unused in practice. index-provider also accepts records from only one provider identity per instance (the configured `ProviderID`, or the first peer that publishes when none is configured, tracked in memory until restart); correctly signed records from any other peer are rejected with `500` and a `delegate error: provider ... isn't allowed` message. + +::: + +#### Signature Creation + +The signature covers the `Payload` bytes exactly as they appear in the request body: clients serialize the payload first, sign those bytes, and embed the identical bytes in the request. + +1. Serialize `Payload` to JSON. The reference client emits compact JSON with HTML escaping disabled (Go's `encoding/json`); any serialization works, as long as the transmitted `Payload` bytes are byte-identical to the signed ones. +2. Compute the SHA-256 hash of the payload bytes. +3. Sign the 32-byte digest with the libp2p private key corresponding to `Payload.ID` (for Ed25519 keys this is a plain Ed25519 signature over the digest). +4. Encode the result as multibase base64: prefix `m` followed by standard-alphabet base64 without padding. + +#### Signature Verification + +The reference server verifies every record and responds with `403 Forbidden` when any step fails: + +1. Reject records that have no `Signature` or no `Payload.ID`. +2. Extract the public key from the `Payload.ID` Peer ID. This only succeeds for Peer IDs that embed the public key as an identity multihash, which is the case for Ed25519 and Secp256k1 keys. RSA and ECDSA Peer IDs contain only a hash of the key, so their records always fail verification, even when correctly signed. +3. Decode the `Signature` from multibase. +4. Compute the SHA-256 hash of the received `Payload` bytes and verify the signature against it using the extracted public key. + +Verification is server policy, not a wire-format guarantee: + +- The reference server always verifies: [boxo v0.35.2](https://github.com/ipfs/boxo/tree/v0.35.2/routing/http) has no option to disable verification, and its client cannot produce unsigned records. index-provider inherits this behavior and performs no signature checks of its own. +- Bitsocial trackers do not verify signatures and treat the `Signature` field as optional; they bound abuse by requiring announced addresses to match the request IP instead. +- Skipping verification is reasonable only when the server does not accept announcements from untrusted parties, for example when client and server run on the same machine or in a private network. A publicly reachable server that skips verification allows anyone to publish records for any Peer ID and any CID. + +Unsigned records (no `Signature` field) interoperate only with servers that skip verification; the reference server rejects them with `403`. + +#### Response Status Codes + +Observed behavior of the reference server implementation: + +- `200` (OK): All provider records were accepted. +- `400` (Bad Request): The request body is not valid JSON or does not decode into the structure above, or a record's `Schema` is not `"bitswap"`. +- `403` (Forbidden): Signature verification failed. This includes a missing or invalid `Signature`, a missing `Payload.ID`, and Peer IDs from which a public key cannot be extracted (see Signature Verification above). +- `500` (Internal Server Error): The backing service rejected the records (`delegate error: ...`), for example due to index-provider's provider identity restriction, or another server-side failure occurred. + +:::warning + +Acceptance is not atomic. The reference server processes `Providers` sequentially and stops at the first failing record: records before it were already accepted and passed to the backing service, and there is no rollback. Clients MUST NOT treat an error response as proof that no records were accepted. + +A signed record without `Timestamp` or `AdvisoryTTL` crashes the reference server's request handler: the client sees a dropped connection with no HTTP status code at all. + +::: + +#### Response Body + +```json +{ + "ProvideResults": [ + { + "Schema": "bitswap", + "Protocol": "transport-bitswap", + "AdvisoryTTL": 86400000000000 + } + ] +} +``` + +Where: + +- `Schema`: Echoes `"bitswap"`. +- `Protocol`: Echoes `"transport-bitswap"`. +- `AdvisoryTTL`: TTL in nanoseconds chosen by the server (not the value from the request), indicating how long the record will be cached. Clients SHOULD republish before this TTL expires. + +#### Unspecified Behavior + +The API leaves the following undefined. Server implementations differ, and a client has no way to discover a given server's policy. + +- **Request limits**: There is no limit on records per request, keys per record, or body size, and no mechanism for a server to advertise one (no `413` or `429` semantics). The reference server buffers and decodes the whole body in memory; index-provider bounds requests only with read timeouts. In practice, request size is shaped by the client: Kubo sends at most `MaxProvideBatchSize` keys per request (default 100). +- **Address policy**: Nothing defines what a server may do with `Addrs`. The reference server passes them to the backing service verbatim, index-provider forwards them to IPNI unchanged (or ignores them wholesale when `ProviderID` and `Addrs` are configured), and Bitsocial trackers drop announced addresses that do not match the request IP. Filtering, rewriting, and reachability checks are all server-defined. +- **Refresh cadence**: Nothing consumes the response `AdvisoryTTL`: the reference client discards the returned value and re-announces on its own schedule (Kubo defaults to `Provide.DHT.Interval`, 22 hours, which stays under index-provider's default 24-hour expiry only by the choice of defaults). On the server side, index-provider treats a repeated announcement as a refresh: it resets the record's clock to the server's own time and publishes no duplicate IPNI advertisement. +- **Deletion**: There is no way to withdraw an announcement. The only removal path is to stop announcing and let the record expire server-side (index-provider: configured `CidTtl`, default 24 hours, evaluated only while processing later announcements). +- **HTTP authentication**: The record signature is the only authentication mechanism. The API defines no bearer tokens, API keys, or other HTTP-level credentials; operators who need a closed endpoint rely on network-level controls or on checks in the backing service. + +## Design rationale + +### Bitswap-Specific Schema + +The `bitswap` schema was designed specifically for the Bitswap transfer protocol. The schema name and `Protocol` field make it explicit that these records are for peers providing content over Bitswap. + +Historically, this made sense when Bitswap was the only means of retrieving data from IPFS peers. In modern times, many providers use HTTP [Trustless Gateways](https://specs.ipfs.tech/http-gateways/trustless-gateway/) in addition to Bitswap, and sometimes instead of it. + +This tight coupling to Bitswap is one of the reasons this API is considered deprecated. Future iterations should support protocol-agnostic provider announcements. + +### User benefit + +This API provided a convenient way for [index-provider](https://github.com/ipni/index-provider) to enable Kubo users to announce content to IPNI indexers (like cid.contact) without relying solely on DHT. + +Although built for IPNI, the API was later picked up outside that ecosystem: Bitsocial reverse engineered it to let its apps announce content to community-run trackers (see Known users below). Documenting the endpoint, together with test vectors, gives these existing deployments an interoperability reference. The API's utility remains limited by its Bitswap-only scope and the lack of conformance tests. + +### Known users + +Two kinds of deployments are known to use this API: + +- **index-provider (IPNI)**: the deployment the API was created for. [index-provider](https://github.com/ipni/index-provider) mounts boxo's reference server handler and runs as a sidecar on the same machine as a Kubo node, which is configured with `Routing.Type` set to `custom` and an HTTP router pointing at the sidecar (usually over localhost) for the `provide` method. Signatures are generated by Kubo and verified by the sidecar. Each sidecar instance accepts announcements from a single provider identity only: the configured `ProviderID`, or the first peer that publishes when none is configured. +- **Bitsocial trackers**: [Bitsocial](https://bitsocial.net) (previously named [Plebbit](https://plebbit.com)) is a peer-to-peer social media protocol whose apps, such as [Seedit](https://seedit.app), bundle a Kubo node. These nodes run with the public DHT disabled and announce content over this API to volunteer-run trackers whose default endpoints are hardcoded in the [pkc-js client SDK](https://github.com/pkcprotocol/pkc-js) (PKC, "Public Key Communities", is the project's name for its protocol layer). The [pkc-http-router tracker server](https://github.com/pkcprotocol/pkc-http-router) stores only `Keys`, `ID`, and `Addrs`, does not verify signatures, and instead requires announced addresses to match the IP the request came from. On this path the `Signature` is usually not even valid: a local proxy on the client machine [rewrites `Payload.Addrs`](https://github.com/pkcprotocol/pkc-js/blob/49f6b9c5028a870dcaaae08a5a709e9b5b506600/src/runtime/node/addresses-rewriter-proxy-server.ts) after Kubo signs the record, without re-signing it. + +This deployment is described first-hand, under the Plebbit name, in the [discussion that led to this IPIP](https://discuss.ipfs.tech/t/only-peers-found-from-dht-seem-to-be-getting-used-as-relays-so-cant-use-http-routers/19545). + +### Compatibility + +This is a legacy API. The existing client/server implementation in [boxo v0.35.2](https://github.com/ipfs/boxo/tree/v0.35.2/routing/http) serves as the historical reference. + +The related types (`WriteBitswapRecord`, `WriteBitswapRecordResponse`, `SchemaBitswap`) were marked as deprecated in boxo in August 2023, when they were revived specifically to maintain backwards compatibility with index-provider. + +Kubo remains capable of publishing via this API, but only when explicitly configured with `Routing.Type` set to `custom` and an HTTP router that handles the `provide` method ([delegated routing docs](https://github.com/ipfs/kubo/blob/master/docs/delegated-routing.md)); the default configuration does not publish provider records over HTTP. + +New implementations SHOULD NOT implement this endpoint. Existing deployments MAY continue using it but should plan migration to future protocol-agnostic alternatives. + +### Security + +- **Authentication**: A valid signature proves the record was created by the holder of the private key for the announced Peer ID, so third parties cannot publish records under someone else's identity. It proves nothing about content possession: any peer can announce any CID under its own Peer ID, so signatures alone do not prevent index spam or announcements of content the peer cannot serve. +- **Replay Protection**: The `Timestamp` field was intended to allow servers to reject stale records. +- **TTL Enforcement**: The `AdvisoryTTL` was intended to limit how long records persist. + +:::note + +As noted earlier, [index-provider v0.16.0](https://github.com/ipni/index-provider/releases/tag/v0.16.0) ignores `Timestamp` and `AdvisoryTTL`, providing no replay protection, and Bitsocial trackers additionally skip signature verification. These relaxations are acceptable only where the server does not accept records from untrusted parties, for example when both client and server are operated by the same party on one machine or in a private network, or where abuse is bounded by other checks (see Signature Verification above). + +::: + +### Alternatives + +[IPIP-378](https://github.com/ipfs/specs/pull/378) was an attempt to design a modernized, protocol-agnostic replacement that would support announcing multiple CIDs in a single request using signed DAG-CBOR and/or JSON payloads. [ipni/index-provider#403](https://github.com/ipni/index-provider/issues/403) was opened to migrate index-provider to IPIP-378 once ratified, but IPIP-378 was never completed. + +At the time of writing this IPIP (2025Q4), there is no vendor-agnostic alternative to this HTTP API. A future IPIP could address this gap by extending :cite[http-routing-v1] with a modern specification for delegated publishing of provider information over HTTP. Such work should incorporate lessons from this historic API and support both Bitswap and HTTP ([Trustless Gateway](https://specs.ipfs.tech/http-gateways/trustless-gateway/)) providers natively. + +## Test fixtures + +The signed example in the Detailed design section is reproducible from the following key material (a throwaway Ed25519 key generated for this document): + +- Ed25519 seed (hex), derived as the SHA-256 of the ASCII string `ipip-0526-test-vector`: + + ``` + 829ecb5f7f2b24c936a194361c919249ae9d6b88701e33939daec2fe6f08d747 + ``` + +- Private key (libp2p protobuf encoding, base64): + + ``` + CAESQIKey19/KyTJNqGUNhyRkkmunWuIcB4zk52uwv5vCNdHkYhfe8CtcuHgLiYw9NjGH5MWhlZ5lPeWEFq1aEG7kos= + ``` + +- Peer ID: `12D3KooWKcTtoUYcQVfURdwUZtPsanm97SdEf7DkesyxTz7kkD2z` + +### Signed record + +The exact `Payload` bytes to sign (a single line, compact JSON): + +```json +{"Keys":["bafkreigur6gzxm3ykiol7ywou3iy3obruzs2q7boizj7oznznid34dzc3e"],"Timestamp":1725833163372,"AdvisoryTTL":86400000000000,"ID":"12D3KooWKcTtoUYcQVfURdwUZtPsanm97SdEf7DkesyxTz7kkD2z","Addrs":["/ip4/198.51.100.1/tcp/4001","/ip4/198.51.100.1/udp/4001/quic-v1"]} +``` + +- SHA-256 of the payload bytes (hex), which is the message passed to Ed25519: + + ``` + 198b43ea6aba9071845059c5e91eba3926df35926da35c61c1b7429fd7ffb3b8 + ``` + +- Signature (multibase base64): + + ``` + mlGWqELZSoVjY22w6NxT7TuNUj5BmQRlrv/x27jQHo5pM3CYZRlJ834bi0UazFuXlH2SuMBxdwfELdXXwVkjZBQ + ``` + +A conforming verifier accepts this record: extract the Ed25519 public key from the Peer ID, decode the signature from multibase, and verify it against the SHA-256 digest of the payload bytes. This vector was generated and round-trip verified with the signing code in [boxo v0.35.2](https://github.com/ipfs/boxo/blob/v0.35.2/routing/http/types/record_bitswap.go). + +### Unsigned record + +For servers that skip signature verification (see Signature Verification above), the same announcement without the `Signature` field: + +```json +{ + "Providers": [ + { + "Schema": "bitswap", + "Protocol": "transport-bitswap", + "Payload": {"Keys":["bafkreigur6gzxm3ykiol7ywou3iy3obruzs2q7boizj7oznznid34dzc3e"],"Timestamp":1725833163372,"AdvisoryTTL":86400000000000,"ID":"12D3KooWKcTtoUYcQVfURdwUZtPsanm97SdEf7DkesyxTz7kkD2z","Addrs":["/ip4/198.51.100.1/tcp/4001","/ip4/198.51.100.1/udp/4001/quic-v1"]} + } + ] +} +``` + +The reference server rejects this request with `403 Forbidden` (`signature verification failed`); non-verifying servers such as Bitsocial trackers accept it. + +This API never had an official conformance test suite; the historical absence of fixtures is one of the reasons it never became part of the official `/routing/v1` specification. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).