Fleet trust, stage 3: a directory the relay cannot read, and a browser that pairs once

[karngyan]

Stage 3 of spec/fleet-trust.md, end to end: the relay
grows a store it cannot read, the daemons keep themselves converged with it,
and the browser stops needing a ceremony per machine. Pair a device once and
the fleet is the unit of trust.

Three parts, in the order they were built and have to be read.

A — the directory Durable Object (486b8d0)

One more Durable Object, and the only one that is not per machine
(idFromName("directory"), because one relay is one fleet). It holds signed
blobs — machine certs and revocations — and stores and serves and never
verifies: the fleet key never touches the Worker, so it cannot tell a cert
from a revocation from 200 bytes of noise, and must not try.

• PUT /directory (daemon secret), GET /directory (credential-less,
  metered), DELETE /directory (daemon secret, empties the set),
  WS /directory (daemon secret, push on write).
• Content-addressed. The key is the SHA-256 of the exact bytes and there is
  no other name for one. A caller-supplied name would put the relay in charge
  of a namespace it cannot check, and one hostile secret-holder could then PUT
  a machine cert over a revocation with the relay's help. A PUT can only add;
  duplicates are idempotent; a blob comes back byte for byte.
• Bounds, and no eviction. 4 KiB a blob, 512 entries, 507 over the cap
  rather than making room — every eviction policy can drop a revocation, and a
  directory that forgets a revocation re-admits the device it revoked. The way
  out of a full one is emptying the whole set, which needs no opinion about any
  blob: DELETE /directory, flue relay reset. See the review fixes below.
• spec/relay-protocol.md gains the leg, the routes table and the ranking
  rule that is explicitly not the relay's.

B — the Go daemon's leg (d382196)

internal/transport/relay/directory.go: socket first, snapshot second (a write
that lands between the two arrives by one path or both, and ingest is
idempotent all the way down), then everything this machine holds, re-offered on
every connect and every half hour.

• Nothing the relay says is trusted. Every blob, by GET or by push, is
  verified under the fleet public key and dropped if it fails; the relay's own
  bounds are re-enforced on this side.
• Ingest is revocations only. A verified device cert read from the
  directory is deliberately not written to the local registry: possession of a
  public blob proves nothing, and a registry filled from a credential-less GET
  would be honoured by rule 1, which never looks at a cert — quietly turning
  "the fleet vouches for this key today" into "this machine pairs with it
  forever", surviving the fleet-key rotation meant to withdraw it.
• The deploy learned to send migration history, so an existing relay at
  migration v1 gets FleetDirectory at v2; flue relay status grew a fleet:
  line that reads the directory and verifies it locally.

C — the browser (this part)

• The fleet public key is pinned from the pairing link. internal/daemon
  writes &f=<fleet public key> beside k=; the pairing page reads it on the
  first render, pins it in one record per origin, and the scrub takes it with
  t and k. The QR is the one leg of the ceremony no intermediary can sit
  in — a fleet key learned from the answer to the POST would be
  trust-on-first-use one level up, and whoever supplied it could mint a machine
  cert for every machine the browser will ever dial. The public half only; the
  seed never reaches a browser.
• web/src/crypto/cert.ts decodes and verifies internal/fleet's
  length-prefixed canonical encoding, and encodes it too — not because anything
  in the product signs, but because a decoder tested against its own output
  agrees with itself about a bug. Trailing bytes are refused rather than
  ignored; every fault is one answer, null.
• Noise IK message A stops sealing an empty payload. It carries the device
  cert, which is what rule 2 of the daemon's acceptance order reads.
• GET /directory builds the machine list. web/src/relay/directory.ts
  verifies every blob under the pinned fleet key, bounds the answer locally
  (4 KiB, 512 entries, a ceiling on the document — the relay's word for its own
  limits is not evidence), collects everything before ranking anything, and
  applies the reader's rule: a revocation outranks a device cert for the same
  key whatever the iat — the certificate it outranks being the one this
  browser now holds locally rather than one read off the relay. machines.ts folds the fleet's list into the stored
  records (mergeMachines, still holding no key); fleetSources builds a
  source per row — the ceremony's pinned key where there is one, the
  certificate's noise key otherwise.
• The Remote screen carries part B's directory counts under the address the
  machine is reachable at, because a machine can be perfectly reachable and
  deaf to its fleet, and nothing said so.

The ordering constraint from the stage-2 review, and how it is satisfied

Stage 2 shipped rule 2 — a daemon admits an unknown key that presents a
fleet-signed device cert — and the review's condition on it was that fleet-wide
admission must not be switched on while the kill switch is still local. It was
dormant, and safely so: no browser ever sent a cert, so nothing was ever
admitted by it.

Part C is what switches it on, and it lands in the same PR as part B. The
first release in which a browser presents a device cert is the first release in
which a revoke performed on any machine is published to the directory, pushed
to every connected daemon, and applied there — registry row dropped by key,
live channels closed with the existing revoked{reason} flow. There is no
version of flue in which a device can walk into a machine it never paired and
a revoke reaches only the machine it was typed on.

Two properties keep that true rather than merely arranged:

• The reader's rule is enforced on both sides. A daemon checks the revocation
  list inside the registry write (AddFromFleetCert), so a concurrent revoke
  cannot land between a check and a write; the browser refuses to present a
  cert whose key it has seen revoked, however much fresher the cert is.
• Revocations only ever subtract authority, which is why honouring one from an
  untrusted channel is safe, and device certs are never ingested from that same
  channel, which is why the reverse is not.

Privacy delta

The directory makes visible to whoever runs the Worker — and to anyone who
reads the credential-less route — machine ids and display names, and who
revoked what and when. All of it signed rather than secret. The relay already
routed by machine id; the delta is machine names and the revocation history.

Device public keys and device names are not in it. They were in the first
three commits; the restructure took them out, because that route needs no
credential and a device certificate names a key and the label its owner typed.
A device now gets its own certificate from the machine that minted it — in the
pairing answer and in every relayed welcome — which is both more private and
the end of the directory's only unbounded growth term.

What that does and does not buy, precisely, because it is easy to overclaim.
It does not hide a device certificate from the relay's operator. A relayed
pairing transits the Worker in cleartext — the token, the device key, the label
you typed, and the certificate in the answer — and no directory arrangement
changes that, because the request is the ceremony. What it removes is the
anonymous reader: GET /directory takes no credential, so a published
certificate was a device key and its owner's label readable by anyone who knew
the relay's address, permanently, plus one of 512 entries per ceremony. "Not
published to strangers", not "not seen by the operator". docs/RELAY.md,
spec/fleet-trust.md and spec/relay-protocol.md's "What the relay sees" all
now draw that line rather than blurring it.

Flag day

An existing relay must run flue relay update once. FleetDirectory is a
second Durable Object class, and a Worker deployed before it has no such
binding: /directory answers 503 {"error":"directory unavailable"} and
flue relay status names the fix in as many words —

fleet:    unreachable (this relay has no directory; run `flue relay update` to redeploy it)

Until then no revocation crosses machines and no browser learns of a machine it
did not pair with by hand. Device certificates are unaffected either way: they
never travel through the directory, so a device gets its own from the machine
that paired it on a relay of any vintage. Nothing else is affected: sessions,
pairing and every already-paired machine keep working, which is what makes this
an upgrade rather than an outage. The deploy reads the migration tag the
account's copy of the script already carries and sends only the steps behind
it, so a fresh account runs v1 then v2 and an existing relay runs v2 alone.

Review fixes

An adversarial review of the three commits above returned SHIP WITH FIXES, no
critical findings. What follows is what changed, one commit each.

A full directory permanently disabled the fleet-wide kill switch (HIGH).
put answered 507 before the store and before the fan-out, so a revocation
past the cap was neither kept nor pushed: it reached only the machine it was
typed on, and re-offering it was answered 507 on every reconnect forever.
Nothing evicts, the object is named by a constant, and Durable Object storage
outlives every redeploy — so the documented escape hatch ("flue relay setup
mints a fresh secret and fleet key, and the new relay starts empty") was simply
false. DELETE /directory empties the whole set and the count, gated by the
same daemon secret a PUT presents, and closes every push socket with 1012 so
each daemon reconnects and re-publishes in seconds rather than at its next
half-hourly republish. All-or-nothing is what makes it a thing the relay may
do: a wipe needs no opinion about any blob, where a prune would need the fleet
key. flue relay reset is the operator's side, behind a typed "yes". The
residual — a blob whose only remaining holder never reconnects — is stated in
the class comment, in the spec, in the command's warning and in the docs.
Setup deliberately does not reset: it would be racing the workers.dev
subdomain and a freshly bound secret, on the one command a user runs first.

The 507 story the docs told was wrong (MEDIUM). "reaches only the machines
that were connected to hear the push" — there is no push. The docs, the spec's
Bounds paragraph and the daemon's log line now say what happens: not stored,
not pushed, reaching nobody, and a revoke made past the cap takes effect on the
machine it was typed on and on no other.

The migration tag read failed open (MEDIUM). deployedMigrationTag
answered "" for every failure, and "" meant "fresh account". So an existing
v1 relay whose pre-flight GET .../scripts timed out sent the whole history
untagged → 10079 → the blanket already-applied recovery dropped migrations →
re-PUT bound FleetDirectory to a class no migration created → 10061, and
flue relay update died. The read now answers (tag, known); an unknown tag
still sends the full history so a first deploy works, but the guess is
corrected rather than abandoned — 10079 names the tag the script really
carries, so the retry parses it out and recomputes. A precondition it cannot
read fails closed, the way wrangler does, and one naming the tag already sent
is not retried at all.

Nothing warned before the cap (MEDIUM). flue relay status and the Remote
screen now warn from 90% and name flue relay reset, from single constants on
each side rather than a second copy of 512.

"Availability is the relay's only power" was overstated (MEDIUM). True of
the routing legs and of certificates; for revocations, omission adds — a
machine that never hears one keeps admitting a device that was cut off — and a
GET is shape-identical complete or filtered. spec/relay-protocol.md gains
"What withholding costs": the asymmetry, what bounds it (every holder
republishes every revocation on connect and every 30 minutes; a machine that
heard one never unhears it; the relay still cannot mint the certificate it was
about), and the real fix as future work — a manifest or epoch signed under the
fleet key. Not built here.

Four low-severity items. The migration history, DO bindings,
run_worker_first and rate rule are now compared against relay/wrangler.jsonc
by a test rather than by prose — a wrangler-only v3 would have made every
later flue relay update skip migrations silently. An unknown deployed tag says
so instead of being a silent no-op that surfaces as a baffling 10061.
cert.ts's encoder explains why its iat ceiling is 2^53 where Go's is 2^62
(a number cannot carry more exactly, and signing a rounded value is the bug
the refusal prevents), and a test pins that the decoders agree to the unit.
MAX_DAEMON_SOCKETS has a test; GET /directory has an ETag and answers a
matching If-None-Match with 304, which the daemon's leg now uses — caching
the tag only for a snapshot it fully applied, so an unapplied revocation is
retried rather than skipped; and the ratelimits comment lists /directory.

The CSP finding: real. The daemon serves its UI under connect-src 'self' ws://127.0.0.1:* ws://localhost:*. A loopback tab reaches every machine that
is not this one over wss://<relay>/client/<id> — pre-existing, adoptRemotes
on main has the same shape — and this PR adds fetch('https://<relay>/directory').
Neither is same-origin, so the browser blocked both, and the fetch failed
silently because readDirectory answers "no machines" for every fault: a
loopback tab showed a fleet of one. daemon.LocalCSPFor adds the one relay
origin in both schemes, fed from relay.json rather than from the transport's
status, because a document's policy is fixed when it is served. The
relay-served copy needed nothing — there the relay is the origin.

Two restructure changes, one taken and one refused

Taken: device certificates leave the directory. A device now gets its own
certificate from the machine that minted it — in the pairing answer, verified
against the fleet key the browser pinned from the QR seconds earlier, and again
in the welcome of every connection it opens anywhere in the fleet, inside
Noise, to a device that has just proved it holds the key the certificate names.
That second channel is the re-supply path: a browser that never stored one, or
lost it, or was paired before its machine had a fleet key, picks one up from
any machine it can still reach.

Publishing them as well bought one thing — a browser being able to find its own
— and cost two: a credential-less roster of the operator's device keys and
labels, and one permanent entry per pairing ceremony ever performed, which was
the growth that made a full directory reachable at all. No daemon ever read one
from there; rule 2 reads the certificate the device presents.

The UX constraint holds and is the point: pair once and every machine works,
later-joined ones included, because the certificate travels with the device;
revoke anywhere and the device dies everywhere, because revocations still ride
the directory and its push socket, and the browser still reads them to stop
presenting a certificate the fleet has cut off.

Refused: pruning superseded entries. Because there is next to nothing left
to prune, not because a delete would be dangerous. A revocation is never
superseded; a device certificate is not in the store; what remains is a machine
certificate re-minted by a re-join, single digits out of 512 over a fleet's
life. A mechanism and its tests to reclaim a fraction of a percent of a store
that no longer grows with use is not a trade worth making, flue relay reset
covers the failure mode, and the 90% warning makes the wall a decision.

Two arguments are recorded as not the reason, so nobody re-derives them: that
a targeted delete on the daemon secret would let one compromised machine delete
any entry — true, and DELETE /directory on that same secret already deletes
every entry including every revocation, self-healing for the same reason a
targeted delete would be; and that a signed delete needs the relay to verify —
true, but an argument about which mechanism, not about whether to have one. The
verify-only gate the review raised (the fleet public key bound to the Worker,
authorising a digest delete) is recorded in spec/fleet-trust.md as future
work and rejected for the reasons that are about it: it makes the Worker a
policy engine, and it creates a fleet-key rotation problem nothing else here
has.

Delta-review fixes on the restructure

The restructure above was reviewed on its own and came back SHIP WITH FIXES:
the trust plumbing was sound — both delivery paths verify under the pinned
fleet key and check the certificate's subject against the recipient's own
static key — but it was half-applied. One commit each.

1. The ceremony still published the certificate. Directory.mine() (the
   bulk republish) had stopped; pairing.go had not, so every ceremony still
   spent a permanent entry and still put a device key and its owner's label on
   the credential-less GET /directory — both costs the restructure exists to
   remove. Removed, and PublishFleetBlob now parses the kind and drops a
   device certificate outright so it cannot come back. The branch's own
   TestPairingPublishesTheDeviceCertToTheFleet asserted the wrong behaviour
   and is replaced by TestPairingPublishesNothingToTheFleet, which also
   asserts the certificate is minted into the registry so the new test cannot
   pass for the wrong reason. It fails against the un-fixed code with "a
   pairing published [device] to the fleet directory, want nothing". This is
   what makes the four documentation claims in this description true.
2. The "older flue" diagnostic can never be true. No released flue has a
   fleet directory at all, so nothing released can have published to one. Both
   surfaces now say what is true — "nothing in this fleet should be publishing
   one" — and stop there; a build from this branch's middle commits or a
   hand-written PUT is not a diagnosis a count can make.
3. Re-supply took effect only on the next page load. A browser that booted
   without a certificate built sources for its pinned machines, picked the
   certificate up from the first welcome, and did not gain the machines it
   unlocks until a reload. It happened to work because IndexedDB beats a
   network fetch — timing, not construction, and untested. adoptFleetCert now
   reports when it stored a certificate the browser did not have, and
   FleetClient re-runs the expansion once. Three tests, including the race
   that used to decide it.
4. A loopback tab has no re-supply path, and four comments said otherwise:
   the loopback welcome carries no certificate (the daemon cannot know a
   cookie-authenticated tab's device key) and a loopback pairing writes no
   per-machine pin, so such a tab has nobody to ask and re-pairs. Comments
   corrected rather than a mechanism invented — the only path that would work
   is a new request in which a tab names its own device key, which is protocol
   surface this review was not the place to add.
5. No browser test for the pairing-answer delivery, which is the primary
   channel. Seven cases through the real router and the real signer: kept when
   the link's fleet key signed it and it names this browser's key; dropped for
   another signer, another subject, a machine certificate in that slot, and
   garbage; absent is not an error. The seventh pins the load-bearing ordering
   — the verifying key comes from the QR and is never re-read from storage —
   and all four refusals were checked by mutation.
6. fleetCertFor matched on the 48-bit DeviceID, where FindByKey's own
   comment explains at length why an id-only match is unsafe (Add permits
   colliding ids). ConnMeta now carries the static key the handshake proved
   and the lookup resolves on those bytes.
7. The pruning refusal is rewritten around the reason that is true; see
   above.
8. Stale comments across publishAll, FleetDirectory, the browser's
   directory reader and docs/RELAY.md's flag-day paragraph, plus the privacy
   delta stated precisely in the three documents that owe it.

Test evidence

make web relay clean; go vet ./... clean.

suite	result
go test ./...	all packages ok (incl. internal/fleet, internal/daemon, internal/transport/relay, internal/cloudflare, internal/relaydeploy)
go test -race ./internal/crypto/... ./internal/transport/... ./internal/fleet/...	ok
cd relay && pnpm test	161 passed (6 files)
cd web && pnpm vitest run	1118 passed (58 files)
cd web && pnpm run lint, cd relay && pnpm lint	clean

Watched failing against the un-fixed code, for the two the review asked for:

• relay/test/directory.test.ts → "lets a revocation that was refused at the
  cap be published after a reset" — 404 from the router, which had no DELETE.
• internal/cloudflare → TestDeployRecoversTheUpgradeWhenTheTagReadFailed
  — dies with the real cloudflare: 10061 … binding DIRECTORY refers to class FleetDirectory which is not exported by the script, which is the chain that
  killed flue relay update. Its two siblings —
  TestDeployFailsClosedOnAnUnreadablePrecondition and
  TestDeployDoesNotRetryATagItAlreadyClaimed — both see the blind second PUT.
• And the migration drift guard: adding {"tag":"v3",…} to relay/wrangler.jsonc
  alone fails TestMigrationHistoryMatchesTheWranglerCopy.

Negative tests are the point on a trust boundary, so here is what each of them
catches, and which ones were watched failing against the un-fixed code:

• web/src/crypto/noise.test.ts → "reproduces msg1 with the certificate in
  the payload" — verified failing against the previous noise.ts, which
  sealed an empty payload: expected '7aa6099ce8…' to be '7aa6099ce8…', the two
  differing from the payload ciphertext onward. testdata/noise/ik-payload.json
  is the Go responder's own vector and msg1Payload is the signed device cert
  from testdata/fleet/certs.json byte for byte, so this single case ties the
  TS cert encoder and the TS initiator to Go at once.
• The wrong-key cases — verified failing with the signature check
  stubbed out of verifyCert: five tests fall, including "never builds a source
  from a machine certificate signed by another key" and "never makes a machine
  out of a certificate signed by another key". A machine cert that does not
  verify under the pinned fleet key never becomes a record.
• A revoked device is refused even with a fresher cert beside the
  revocation, in either order in the answer, and gets no fleet-derived
  sources; it also stops presenting the cert to a machine it did pair with.
• A blob cut short or with a byte appended is refused, per kind, both in
  cert.test.ts against the committed vectors and in directory.test.ts.
• A directory the relay truncated costs visibility only — fewer machines,
  no throw, paired machines untouched — as does a 503, an unreachable relay, a
  body that is not a directory, and a document over the size ceiling.
• The encoder round-trips the Go fixtures: all four cases' canonicalHex
  and signedHex, including the astral-character name that catches a length
  prefix counting UTF-16 units.

🤖 Generated with Claude Code