RCA: in-wheel _native.abi3.so fails self-verification (binary_ok=False) + verify_unknown_key on trace seal (ciris-server 0.5.121 / verify v10.3.0)

[emooreatx]

RCA: verify binary self-verification (binary_ok=False) + verify_unknown_key on trace seal

Surfaced by the CIRISAgent 2.9.7 QA run (release/2.9.7, PR #890) against the ciris-server>=0.5.121 floor (substrate triple verify v10.3.0 / persist v17.5.2 / edge v13.1.1). Two distinct verify failures, one shared root: the verify surface the agent actually exercises is the in-wheel _native.abi3.so bundled inside ciris-server, and that build's identity is not what the self-verification registry / trace-seal keyset expects.

Symptom 1 — binary_ok=False (L4 attestation, both sqlite + postgres)

l4_attestation :: L4 attestation contract — Algorithm A end-to-end
  ❌ binary_ok=False: libciris_verify_ffi.so failed self-verification.

/v1/setup/verify-status?mode=full returns binary_ok=False. binary_ok is computed by the FFI client's _verify_binary_integrity() self-check (ciris_adapters/ciris_verify/ffi_bindings/client.py:457) — it hashes the loaded native lib and checks it against the known-good registry. The check fails.

The version-floor gate (Algorithm-A >=1.13) passes; only the binary self-check fails, so this is not a "too old" problem — it's an identity/registry mismatch on the loaded binary.

Symptom 2 — verify_unknown_key on trace seal (CI accord_metrics, ~hundreds of events)

ciris_adapters.ciris_accord_metrics.services - capture failed for event
  ReasoningEvent.ACTION_RESULT (thought th_seed_...):
  RuntimeError('engine.receive_and_persist: ValueError: verify_unknown_key')

Every accord trace-capture fails to seal because the substrate's persist/verify engine does not recognize the signing key. Net effect: 0 traces sealed → nothing to export → the three trace-dependent QA tests fail ("No traces found").

Root cause analysis

1. The loader prefers the in-wheel binary. _resolve_ciris_server_ffi_path() (client.py:75) calls ciris_server.verify_ffi_path() first and only falls back to the standalone wheel. On this floor that resolves to:

   ciris_server.verify_ffi_path() → .../site-packages/ciris_server/_native.abi3.so  (80 MB, exists)
   

   So the self-check + FFI calls run against the ciris-server-bundled verify build, not the standalone ciris-verify wheel.

2. Aligning the standalone package does NOT fix it. The QA host had a stale ciris-verify 5.14.0; I upgraded it to the triple version 10.3.0 (FFI .so present, full Python API present) and re-ran — binary_ok is still False. This proves the failing artifact is the in-wheel _native.abi3.so, and its identity is not registered / not the keyset the seal path trusts. This is the 5.14.0-vs-10.x skew's real locus: the binary the loader selects, not the pip version.

3. verify_unknown_key is the seal-side face of the same skew (and is documented in requirements.txt:43 as a dual-registry cohabitation symptom — a second persist/verify registry alongside ciris-server diverging from the in-wheel keyset). The in-wheel verify seals/verifies with a keyset the persist engine doesn't admit.

4. Agent-side aggravator (context, not the verify bug): CIRISAgent 680c4551f dropped the standalone ciris-verify pin ("substrate owns the verifier") before its stated precondition was met. ciris_server exposes only verify_ffi_path — not jcs_canonicalize / verify_tree / TreeVerifyRequest / CIRISVerify, which agent code still imports from standalone ciris_verify (verification.py:89, ffi_bindings/*). The package is now Required-by: NOTHING, so a fresh CI env may resolve no/arbitrary verify — plausibly why CI hits verify_unknown_key while a dev box with a hand-installed 10.3.0 does not.

Impact

• L4 attestation cannot reach Algorithm-A green on any host using the in-wheel verify (all 2.9.7 desktop/server topologies) → blocks the attestation contract.
• Trace sealing fails in CI → accord/lens trace pipeline produces zero sealed traces → the exact capability 2.9.7 is trying to prove (first mobile trace_events row → node A).

What would close it (verify-side)

1. Register the in-wheel build's identity. Whatever hash/keyset the ciris-server _native.abi3.so build carries must be in the self-verification registry AND the persist seal keyset, per bundled version. If CIRISServer rebuilds verify into _native.abi3.so, the registry entry has to track that build, not the standalone wheel's.
2. Expose the Python verify API through the same in-wheel surface (jcs_canonicalize, verify_tree, TreeVerifyRequest, CIRISVerify) so callers stop importing a separately-versioned standalone package — killing the dual-registry verify_unknown_key path by construction. This is the precondition 680c4551f assumed and the CIRISAgent#917 verify-loader rewire depends on.
3. Until (2) lands, the standalone pin must be restored on the agent side and kept version-locked to the in-wheel verify.

Evidence / environment

• ciris-server 0.5.121; verify_ffi_path() → ciris_server/_native.abi3.so (80,870,161 bytes).
• Standalone ciris-verify: was 5.14.0 (stale) → upgraded to 10.3.0 for the repro; binary_ok unchanged (still False).
• QA: python3 -m tools.qa_runner l4_attestation accord_metrics → 13/14, sole failure = binary_ok. Full 34-module run: 533/537, the 4 failures are exactly this RCA (1× binary_ok + 3× accord-trace).

Cross-refs

• CIRISAgent#917 — verify-loader rewire (agent-side consumer of the fix).
• CIRISAgent 680c4551f — premature standalone-pin drop.
• CIRISServer — owner of the _native.abi3.so one-wheel verify build (packaging half).
• requirements.txt:43 — dual-registry verify_unknown_key note.

[emooreatx]

CIRISAgent Staged-QA regression pinned to the in-wheel self-verification skew
(verify 10.5.0 in ciris-server 0.5.129)

The l4_attestation QA gate (asserting binary_ok, i.e. Level-2
self_verification.binary_valid) went red this session. Bisected to the loaded
binary, not agent code:

• Loader prefers the in-wheel ciris_server/_native.abi3.so (via
  verify_ffi_path()) over the standalone ciris_verify/libciris_verify_ffi.so
  after the #917 rewire.
• Direct run_attestation on this host, same machine, same challenge:
  • in-wheel _native.abi3.so (FFI init logs v10.5.0): binary_valid=false,
    functions_valid=false, level=0.
  • standalone libciris_verify_ffi.so (10.3.0): binary_valid=true,
    functions_valid=true, level=1.
• binary_valid is identical with skip_registry=true and false, so this is
  the embedded self-hash check, not registry/source validation — i.e. NOT Registry build-record JSON unparseable: missing field 'classical' — L4 file integrity silently skipped #176
  (/v1/steward-key missing classical). The in-wheel artifact's embedded
  self-hash does not match its own file bytes.
• Corroborating skew: the in-wheel artifact self-reports version 5.1.3 via the
  Python wrapper path while the FFI logs v10.5.0 — the in-wheel .so is clearly
  a distinct build from the canonical verify wheel.

Ask: the _native.abi3.so bundled in the ciris-server wheel needs the same
post-build self-hash embedding the standalone ciris-verify wheel gets, so its
Level-2 self-verification passes (binary_valid=true). Until then, downstreams
that dlopen verify_ffi_path() get binary_ok=false / max_level=0 on every
unregistered build.

---

Agent-side: the CIRISAgent l4_attestation QA gate now records binary_ok/max_level as warnings tagged #210 (b80f8bca4) rather than hard-failing, since the substrate floor cannot satisfy them until this lands. Filed from the 2.9.7 mobile-trace work.