verify_signature_mismatch on 2.7.6 agent traces survives persist 0.4.4 (2.7.legacy in SUPPORTED_SCHEMA_VERSIONS) — legacy-canonical envelopes not routed through legacy verifier

[emooreatx]

Symptom

Production agent on 2.7.6-stable (CIRISAgent commit c237e4bf) emits 2-field
legacy canonical signatures. Lens 422s every batch with
verify_signature_mismatch, even after persist was bumped to 0.4.4 with
2.7.legacy added to SUPPORTED_SCHEMA_VERSIONS = ['2.7.0', '2.7.9', '2.7.legacy'].

Bridge / Caddy stats from this morning's persist 0.4.4 deploy:

pre-deploy:   {422: 1190}        ← 1190 rejected/window
post-deploy:  {422: 7, 502: 3}   ← 3-min sample, same rate (~2/min)

Sample rejected lines (post-deploy):

WARNING:accord_api:PERSIST_DELEGATE_REJECT class=ValueError msg=verify_signature_mismatch body_sha256_prefix=dcc1037f3d885748 body_bytes=9952
WARNING:accord_api:PERSIST_DELEGATE_REJECT class=ValueError msg=verify_signature_mismatch body_sha256_prefix=dd67f6646c3b3da8 body_bytes=9952
WARNING:accord_api:PERSIST_DELEGATE_REJECT class=ValueError msg=verify_signature_mismatch body_sha256_prefix=77f8d4be52965901 body_bytes=9952
WARNING:accord_api:PERSIST_DELEGATE_REJECT class=ValueError msg=verify_signature_mismatch body_sha256_prefix=cd1123a715fe85c4 body_bytes=9952

Uniform body_bytes=9952 across every rejection — same envelope shape, same
trace_level=detailed batches of 2 events each.

Source confirmed

Source IP 144.202.55.195 is one operator's CIRIS agent ("Scout1", running
2.7.6-stable). The agent's local audit DB shows it signs with key_id
agent-1ee871dcf31b. The key is internally consistent and locally verifiable
— the agent's own verify_trace (using the same canonical it signed with)
passes. The mismatch is purely between what the agent canonicalizes and what
persist canonicalizes for verification.

The agent's own source documents the migration:
ciris_adapters/ciris_accord_metrics/services.py:308-337 (CIRISAgent main):

Migration history:

• <= 2.7.8.8 used a 2-field legacy canonical {"components", "trace_level"}
  to match what lens-legacy api/accord_api.py::verify_trace_signature
  accepted. Persist v0.1.15 implements the spec's 9-field canonical;
  legacy 2-field signatures fail verify_strict with HTTP 422
  verify_signature_mismatch against the new typed verify path.
• Persist's try-both fallback (CIRISPersist issue) accepts BOTH shapes
  during the migration window — so flipping the agent doesn't gate
  persist, and persist's fallback doesn't gate the agent.

The 9-field canonical landed in 2.7.8.9 / CIRISAgent#710. This operator's
agent is on 2.7.6 — three patches before the fix. They have not flipped to
9-field. CIRIS is decentralized; we cannot push an upgrade to other operators'
agents on their behalf.

Why we believe this is a lens issue (not just upstream of lens)

api/accord_api.py:1843-1888
delegates raw body bytes to engine.receive_and_persist(body) without telling
persist which schema version the envelope is shaped to. Persist 0.4.4 added
2.7.legacy to SUPPORTED_SCHEMA_VERSIONS, but the router that picks
which canonicalizer to use isn't reaching the legacy path for these envelopes
— either because:

1. Persist's auto-selection logic doesn't recognize the 2-field shape as
   2.7.legacy (treats it as 2.7.9 and fails strict verify), or
2. Lens needs to inspect the envelope (e.g. the trace_schema_version field
   is missing or 2.7.0) and pass an explicit hint into
   receive_and_persist(body, schema_version="2.7.legacy").

Either way, the observable failure surface is /accord/events — that's
lens's API. Persist already shipped what it said it would; the operational
gap (the try-both behavior the agent code's comment promised) isn't yet
materializing in lens's reject stream.

Reproduction

# From any host that can reach lens.ciris-services-1.ai
# Send a body with 2-field legacy canonical:
#   signed_payload = {"components": [...], "trace_level": "detailed"}
#   signature = ed25519(json.dumps(signed_payload, sort_keys=True, separators=(",",":")))
# Wrap in 9-field BatchEnvelope (which IS what 2.7.6 emits — see CIRISAgent
# services.py:to_dict() shipping all 9 fields envelope-side, but only 2
# fields canonical-side)
curl -s -X POST https://lens.ciris-services-1.ai/lens-api/api/v1/accord/events \
  -H "Content-Type: application/json" \
  --data-binary @legacy_canonical_envelope.json
# expect 422 verify_signature_mismatch

Or just observe Scout1's traffic (144.202.55.195) for 1 min — ~2-3 envelopes
will land, all rejected.

Hardening ask (separate from the immediate fix)

The agent-side comment notes deliberately:

"trace_schema_version is NOT included [in the canonical], so adding it on
the agent side breaks signature verification for every trace."

That omission is the root cause of this whole class of break. A future
trace-format RFC should include trace_schema_version as the first field of
the signed canonical bytes, so both sides can route to the right
canonicalizer based on declared version, and a strict-verifier can reject
"unknown version" with a useful error instead of generic
verify_signature_mismatch. Persist would still own which versions it
supports; lens would still own routing; the difference is verification fails
with verify: unsupported schema_version=X.Y.Z instead of by accident.

Filing as a separate issue is fine — but worth tracking under this one for
context.

Expected resolution paths (not prescriptive — just enumerating)

1. Persist routing fix: persist 0.4.5 auto-routes legacy-shape envelopes
   to the 2.7.legacy verifier path. Lens unchanged. Recommended if persist's
   SUPPORTED_SCHEMA_VERSIONS is meant to be authoritative.
2. Lens schema sniffing: lens inspects the envelope, picks the schema
   version (or tries both), and passes a hint into
   receive_and_persist(body, schema_version=...). Persist API change
   required.
3. Federation registration sweep: confirm agent-1ee871dcf31b (and any
   other 2.7.6-era operator key) is in cirislens.federation_keys. If
   they're missing, the message would be verify: unknown key rather than
   verify_signature_mismatch — but worth ruling out.

What we (CIRISManager) verified locally

• Restarting the source agent (ciris-scout-remote-test-dahrb9 on
  144.202.55.195) did NOT change the rejection cadence. The agent does not
  self-register on boot; its on-disk signing key files are unchanged
  (agent_signing.key from Jan 19, agent_signing.master.key from
  Apr 28 — both untouched by reboot). Same key, same canonical, same
  rejection.
• Crime scene preserved: no agent code changes, no key rotations, no manual
  registrations attempted. We're standing by for guidance from this issue
  before touching anything.

cc CIRISPersist for the routing-layer angle.

---

🤖 Generated with Claude Code