manifest: keep every pubkey — rotation destroyed the old key's records forever

[Nik7A]

The manifest held ONE pubkey_pem and LocalFileSink.__init__ assigned straight over it (sinks/local_file.py:132-138). Starting a recorder with a different key deleted the previous public key from the only place it was stored, and every record it had signed became permanently unverifiable. Not harder to verify — there is no key left to check the signature against, and nothing recovers it.

It already fired, on real evidence

330 records on bosun's evento chains are in exactly that state: evento-2026-06-30 (76), evento-2026-07-01 (238), and the first 16 of evento-2026-07-02, all signed with b0ee6d6c582ec87b — a key searched for exhaustively that exists nowhere.

KEY_RESOLUTION at offset 1: unknown_key_id: no public key for key_id=b0ee6d6c582ec87b

The destruction is timestamped 2026-07-02T08:38:01Z, matching the new key's mtime exactly. The host's contribution was minting a new key when the file was missing — its own defect, fixed separately — but a host mistake must not be able to destroy the verifiability of records already written. A single mutable field is what turned a recoverable host bug into permanent evidence loss.

A public key is not secret. Single-copy storage in a mutable field bought nothing and cost everything.

What changes

• manifest.pubkeys: key_id -> PEM, append-only. declare_pubkey adds and never replaces. Every record's envelope already carries its key_id, so the verifier only ever needed somewhere to look the id up — nothing about the signature form changes.
• pubkey_id / pubkey_pem stay, tracking the most recently declared key, for verifiers that predate the map. They are no longer the only copy, so overwriting them is safe.
• verify_tree resolves from the map, deriving each id from its PEM rather than trusting the map key — the same principle by which pubkey_id is never trusted.

Back-compat

schema_version is not bumped. from_dict rejects any version but its own, so a bump would make every existing manifest unreadable. pubkeys is additive, and a manifest written before it migrates on load by deriving the key_id from the PEM it stores — not from pubkey_id, which was set-once and may name a different key entirely. Only keys still present can be recovered; a PEM already overwritten by a rotation is gone, and nothing here brings it back.

The comment that named the fix

sinks/local_file.py:287-290 said pubkey_id was "overwritten if subsequent records use a different key — that's a v0.2 multi-key story". Both halves were false: the code is if ... is None (set-once), and the multi-key story never landed in v0.2. We are past v0.2. This is that story, and the comment is corrected rather than deleted.

Four existing tests changed — this part is worth reading

test_key_rotation_midchain_both_keys_available already covered this exact scenario and pinned the loss as correct:

# verify_tree derives B from the manifest PEM; pass A explicitly for rotation.
r = verify_tree(d, {kid_a: sk_a.public_key})
# pubkey_id is stale (set to A on first write, pem overwritten to B) → mismatch
assert r.outcome == ChainCheckOutcome.MANIFEST_PUBKEY_MISMATCH

It knew the PEM had been overwritten, handed the lost key back to the verifier from outside, and asserted the resulting mismatch as expected behaviour. In the real incident nobody had that key to hand back — that is the entire point. It now passes verify_tree nothing and expects OK.

Three others (missing_key_is_partial, no_key_at_all, cli_partial_exit_6) harvested their "missing key" condition from the destructive overwrite — one comment reads "the only remaining defect is the 3 unverifiable A-records". That condition is real and worth testing, so it is now created deliberately via _strip_pubkeys_except rather than as a side effect of a bug.

Reported from bosun forensics, 2026-07-17.