From df745a3b0973654447cc52e221280376c46b346a Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 06:56:51 +0000 Subject: [PATCH 01/37] refactor: remove private subc runtime dependency Make mc-host own wire, authentication, discovery, managed clients, component lifecycle, and direct test fixtures so repository consumers use one fail-closed boundary. --- .beads/interactions.jsonl | 1 + .beads/issues.jsonl | 3 +- ARCHITECTURE.md | 6 +- Cargo.lock | 236 +- Cargo.toml | 12 - assets/magic-context.schema.json | 2 +- bun.lock | 4 +- crates/mc-host/Cargo.toml | 10 +- crates/mc-host/benches/ipc_budget.rs | 6 +- crates/mc-host/src/auth.rs | 1007 ++++++ crates/mc-host/src/broca/subprocess.rs | 4 +- crates/mc-host/src/client.rs | 2380 ++++++++++++++ crates/mc-host/src/config.rs | 33 +- crates/mc-host/src/connection.rs | 114 +- crates/mc-host/src/connection_file.rs | 382 +++ crates/mc-host/src/control.rs | 19 +- crates/mc-host/src/dispatch.rs | 19 +- crates/mc-host/src/frame_channel.rs | 3 +- .../src/frame_channel/contract_tests.rs | 3 +- crates/mc-host/src/instance.rs | 4 +- crates/mc-host/src/lib.rs | 20 + crates/mc-host/src/lifecycle.rs | 10 +- crates/mc-host/src/tcp_frame_channel.rs | 15 +- crates/mc-host/src/transport_negotiation.rs | 3 - crates/mc-host/src/transport_provider.rs | 2 +- crates/mc-host/src/wire.rs | 561 +++- crates/mc-host/tests/client.rs | 462 +++ crates/mc-host/tests/host_roundtrip.rs | 13 +- crates/mc-host/tests/instance_security.rs | 66 +- crates/mc-host/tests/lifecycle.rs | 63 +- crates/mc-host/tests/perf_budget_runner.rs | 2 + .../mc-host/tests/support/fake_transport.rs | 2 +- crates/mc-host/tests/support/mod.rs | 8 +- .../mc-host/tests/support/perf_measurement.rs | 2 +- crates/mc-host/tests/support/raw_client.rs | 59 +- crates/mc-host/tests/transport_negotiation.rs | 132 +- crates/mc-module/Cargo.toml | 26 +- .../mc-module/examples/direct_host_fixture.rs | 615 ++++ crates/mc-module/src/dispatch.rs | 458 +++ crates/mc-module/src/historian.rs | 285 +- crates/mc-module/src/historian_producer.rs | 2499 ++++++--------- crates/mc-module/src/lib.rs | 2826 ++++++++++------- crates/mc-module/src/main.rs | 51 - crates/mc-module/src/prompt_surface.rs | 19 +- crates/mc-module/src/session_resolver.rs | 208 +- .../tests/boundary_counter_durability.rs | 81 +- crates/mc-module/tests/broca_roundtrip.rs | 1547 +-------- crates/mc-module/tests/direct_host.rs | 460 +++ crates/mc-module/tests/host_adapter.rs | 157 + crates/mc-module/tests/prepared_output.rs | 282 ++ crates/mc-module/tests/real_daemon.rs | 657 ---- crates/mc-module/tests/support/direct_host.rs | 386 +++ crates/mc-module/tests/support/mod.rs | 4 + .../claims-backfill/v84-process-crash.json | 4 +- docs/mc-host-wire-protocol.md | 256 +- docs/subc-api-surface-inventory-2026-08-17.md | 41 +- packages/cli/package.json | 1 + packages/cli/src/commands/doctor-authority.ts | 8 +- packages/cli/src/commands/migrate-session.ts | 4 +- packages/cli/src/lib/logs-opencode.test.ts | 13 +- packages/e2e-tests/README.md | 74 +- packages/e2e-tests/mode-manifest.json | 4 +- packages/e2e-tests/mutations/fm-oc-5.json | 4 +- .../mutations/rust-ctx-reduce-roundtrip.json | 2 +- .../mutations/rust-historian-producer.json | 27 +- packages/e2e-tests/package.json | 2 +- .../scripts/check-rust-prerequisites.test.ts | 79 +- .../scripts/check-rust-prerequisites.ts | 145 +- .../e2e-tests/scripts/run-rust-fm-mutation.ts | 21 +- .../run-rust-historian-producer-mutation.ts | 85 +- packages/e2e-tests/src/harness.ts | 84 +- .../src/opencode-runner/spawn.test.ts | 105 + .../e2e-tests/src/opencode-runner/spawn.ts | 414 +-- packages/e2e-tests/src/rust-harness.ts | 252 +- .../e2e-tests/src/rust-runner/fake-broca.ts | 206 -- .../src/rust-runner/hermetic-mc-host.test.ts | 340 ++ .../src/rust-runner/hermetic-mc-host.ts | 1005 ++++++ .../src/rust-runner/hermetic-subc.test.ts | 25 - .../src/rust-runner/hermetic-subc.ts | 865 ----- .../e2e-tests/src/rust-scenario-support.ts | 25 +- packages/e2e-tests/src/test-db.ts | 5 +- .../e2e-tests/tests/cache-invariants.test.ts | 6 +- .../tests/deferred-compaction-marker.test.ts | 6 +- .../tests/long-running-session.test.ts | 10 +- .../e2e-tests/tests/overflow-recovery.test.ts | 2 +- .../tests/rust-cold-start-drop-seed.test.ts | 4 - .../tests/rust-ctx-reduce-roundtrip.test.ts | 6 +- .../tests/rust-duplicate-tool-use-id.test.ts | 2 +- packages/e2e-tests/tests/rust-fm-oc-1.test.ts | 3 +- packages/e2e-tests/tests/rust-fm-oc-2.test.ts | 3 +- packages/e2e-tests/tests/rust-fm-oc-3.test.ts | 5 +- packages/e2e-tests/tests/rust-fm-oc-4.test.ts | 3 +- packages/e2e-tests/tests/rust-fm-oc-5.test.ts | 5 +- packages/e2e-tests/tests/rust-fm-oc-6.test.ts | 3 +- .../tests/rust-historian-producer.test.ts | 192 +- .../tests/rust-multi-frame-delta-perf.test.ts | 2 +- .../tests/rust-park-self-heal.test.ts | 8 +- .../tests/rust-removal-self-heal.test.ts | 2 +- packages/e2e-tests/tests/rust-smoke.test.ts | 18 +- .../tests/rust-tail-mutation-readopt.test.ts | 2 +- .../e2e-tests/tests/session-isolation.test.ts | 4 +- packages/e2e-tests/tsconfig.json | 1 + packages/pi-plugin/PARITY.md | 2 +- packages/plugin/scripts/drive-preseed.ts | 8 +- .../scripts/mc-host-client-boundary.test.ts | 258 +- ...ransport.ts => probe-mc-host-transport.ts} | 22 +- .../retrieval-benchmark/privacy.test.ts | 4 +- .../plugin/scripts/smoke-mc-host-client.ts | 6 +- .../plugin/scripts/smoke-mc-host-synapse.ts | 38 +- packages/plugin/src/config/index.test.ts | 3 +- packages/plugin/src/config/index.ts | 3 +- .../src/config/project-security.test.ts | 2 +- .../plugin/src/config/schema/magic-context.ts | 2 +- .../memory/embedding-synapse.test.ts | 20 +- .../magic-context/memory/embedding-synapse.ts | 6 +- .../magic-context/smart-notes/wake-plane.ts | 4 +- .../plugin/src/hooks/magic-context/hook.ts | 6 +- .../magic-context/module-state-sync.test.ts | 6 +- .../magic-context/module-transport.test.ts | 172 +- .../hooks/magic-context/module-transport.ts | 55 +- .../magic-context/rust-mode-transform.test.ts | 4 +- packages/plugin/src/index.ts | 5 +- .../src/plugin/dream-timer-module-client.ts | 2 +- .../src/plugin/embedding-routing.test.ts | 3 +- .../plugin/src/plugin/embedding-routing.ts | 3 +- .../src/shared/mc-host-client/client.test.ts | 189 +- .../src/shared/mc-host-client/client.ts | 156 +- .../mc-host-client/connection-file.test.ts | 9 +- .../shared/mc-host-client/connection-file.ts | 8 +- .../shared/mc-host-client/connection.test.ts | 2 +- .../src/shared/mc-host-client/connection.ts | 32 +- .../src/shared/mc-host-client/errors.ts | 38 +- .../shared/mc-host-client/frame-channel.ts | 2 +- .../plugin/src/shared/mc-host-client/index.ts | 14 +- .../mc-host-client/tcp-frame-channel.ts | 10 +- .../test-support/adversarial-scenarios.ts | 8 +- .../test-support/frame-channel-contract.ts | 8 +- .../mc-host-client/test-support/test-util.ts | 12 +- .../transport-negotiation.test.ts | 51 +- .../mc-host-client/transport-negotiation.ts | 56 - .../mc-host-client/transport-provider.ts | 10 +- .../plugin/src/shared/mc-host-client/types.ts | 8 +- packages/plugin/src/shared/redaction.test.ts | 17 +- 143 files changed, 13751 insertions(+), 8125 deletions(-) create mode 100644 crates/mc-host/src/auth.rs create mode 100644 crates/mc-host/src/client.rs create mode 100644 crates/mc-host/src/connection_file.rs create mode 100644 crates/mc-host/tests/client.rs create mode 100644 crates/mc-module/examples/direct_host_fixture.rs create mode 100644 crates/mc-module/src/dispatch.rs delete mode 100644 crates/mc-module/src/main.rs create mode 100644 crates/mc-module/tests/direct_host.rs create mode 100644 crates/mc-module/tests/host_adapter.rs create mode 100644 crates/mc-module/tests/prepared_output.rs delete mode 100644 crates/mc-module/tests/real_daemon.rs create mode 100644 crates/mc-module/tests/support/direct_host.rs create mode 100644 crates/mc-module/tests/support/mod.rs create mode 100644 packages/e2e-tests/src/opencode-runner/spawn.test.ts delete mode 100644 packages/e2e-tests/src/rust-runner/fake-broca.ts create mode 100644 packages/e2e-tests/src/rust-runner/hermetic-mc-host.test.ts create mode 100644 packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts delete mode 100644 packages/e2e-tests/src/rust-runner/hermetic-subc.test.ts delete mode 100644 packages/e2e-tests/src/rust-runner/hermetic-subc.ts rename packages/plugin/scripts/{probe-subc-transport.ts => probe-mc-host-transport.ts} (97%) diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index d2477cbfe..71e90c6f1 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -70,3 +70,4 @@ {"id":"int-957b99e48af7d10119aa3146ba385e7e","kind":"field_change","created_at":"2026-08-23T21:07:58.638698062Z","actor":"AhravDutta","issue_id":"magic-context-ymc.1","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} {"id":"int-adf3e410823be79e2f65962c00ee00ac","kind":"field_change","created_at":"2026-08-24T13:41:22.526618118Z","actor":"AhravDutta","issue_id":"magic-context-ymc.3","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} {"id":"int-fd51b14afbf32b43a54c33c665d47e22","kind":"field_change","created_at":"2026-08-24T20:56:41.646974108Z","actor":"AhravDutta","issue_id":"magic-context-c50.12","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Compile matrix passed at main tip 574569d5; inventory matches evidence in docs/evidence/subc-compiler-closure/ (87 rows: 83 exact/3 changed/1 private-unknown). Completeness gate on c50.4 cleared."}} +{"id":"int-5eed966629542898b21c355dc0b781fb","kind":"field_change","created_at":"2026-08-25T06:56:29.585492666Z","actor":"AhravDutta","issue_id":"magic-context-c50.4","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Implemented direct mc-host boundary: host-owned Rust wire/auth/discovery/client, direct McHandler adapter and historian, host-owned TS API, direct Rust/E2E fixtures, dependency/docs closure; workspace, TS, E2E, specialist and ponytail reviews complete."}} diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index a281ac996..35ecdc4db 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,4 @@ +{"_type":"issue","id":"magic-context-8nz","title":"Build cross-harness prose steering","description":"Implement shared plain-prose policy and adapters for Pi, Claude Code, OpenCode, Kilo Code, and Codex. Add broad deterministic and harness-contract tests; validate installed hook configurations. User requires final-output rewrite where supported and bounded Stop/SubagentStop retries otherwise.","status":"closed","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T20:46:16Z","created_by":"AhravDutta","updated_at":"2026-08-24T21:00:25Z","started_at":"2026-08-24T20:46:25Z","closed_at":"2026-08-24T21:00:25Z","close_reason":"Installed prose-steering across Pi, OpenCode, Kilo, Claude Code, and Codex; published Pi sync snapshot 2026-08-24T20-59-09-401Z-3e487759; copied and verified three remote hosts.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-cjs","title":"Wire recordDispositionEventInCurrentTransaction into a host command","description":"PR #24 round-45 (comment 3843919627): recordDispositionEventInCurrentTransaction is the only entry point for rejected/quarantined/explicit-stale/explicit-disputed dispositions, and claim-visibility-policy's hard-hide matrix treats them as authoritative — but no production call site exists in packages/plugin or packages/pi-plugin. The hard-hide/quarantine/reject branch is exercised only by unit tests. Decide the host surface (a /ctx-dispute or /ctx-quarantine command mirroring /ctx-approve's confirmation flow, or dreamer-driven assertion) and wire it. Related: magic-context-x84 (MODULE-authority decision channel) will need the same events to flow to native readers.","status":"open","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T14:02:10Z","created_by":"AhravDutta","updated_at":"2026-08-24T14:02:10Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-x84","title":"MODULE-authority native rendering needs a claim-policy decision channel","description":"Under MODULE memory authority, the native render path (mc-store load_active_memories and the m1 lane) selects active/permanent unexpired rows with no claim-policy or verification predicate, and the TypeScript state-sync policy filter never applies because memory sections are omitted while the module owns the lane (module-state-sync.ts omitAuthorityMemorySections). A /ctx_memory write under MODULE authority therefore creates an active, unverified native row that the next native transform injects automatically — a CANDIDATE bypasses the v86 visibility ladder entirely in rust mode.\n\nRaised by codex review on PR #24 (inline comment 3840295420). Deferred from the PR because the fix needs an authority-model design decision, not a mechanical patch: while MODULE owns the lane, TypeScript (the policy authority) has no channel to push per-row eligibility without violating the one-writer-per-pool rule that fences the memories section (mc-store apply gates PREPARING/MODULE/DRAINING).\n\nCandidate shapes to evaluate:\n1. A dedicated policy sidecar section in state_sync (eligibility verdicts keyed by native row id) that the apply lane accepts even under MODULE authority, since it carries no row content — keeps Rust free of policy derivation.\n2. Mirror auto_eligible into mc_memories via the module-to-TS changefeed round trip: TS adjudicates module-created rows on the reverse sync and pushes the verdict back; native render gains a WHERE auto_eligible = 1 predicate. Requires a default for not-yet-adjudicated rows (fail closed = candidate-invisible until adjudicated; fail open = today's behavior).\n3. Derive a conservative native predicate from existing columns (verification_status) — rejected on first pass: USER_EXPLICIT-taint rows are auto-eligible without verification, so this under-renders legitimate content.\n\nAcceptance: under MODULE authority, a freshly written unverified memory does not render into the native m0/m1 lanes until the policy authority marks it eligible; explicit get keeps working with trust labels; no policy derivation logic moves into Rust.\n\n## Context\nPR #24 v86 claim trust policy follow-up","status":"open","priority":1,"issue_type":"bug","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T02:22:20Z","created_by":"AhravDutta","updated_at":"2026-08-24T02:22:20Z","comments":[{"id":"01a032c6-1a8c-7ba0-9c23-7bcd93599c95","issue_id":"magic-context-x84","author":"AhravDutta","text":"New review evidence (PR #24 comment 3841517352): in MODULE authority mode omitAuthorityMemorySections=true also omits the memories_delete_ids lane and the filtered replacement snapshot, so a module-created candidate or a row quarantined/rejected after mirror-back stays automatically injected for the whole ownership period. The decision channel must either keep policy revocations crossing the boundary in authority mode (e.g. send the delete lane even when snapshot sections are omitted) or move the policy evaluator natively.","created_at":"2026-08-24T07:57:19Z"},{"id":"01a033d3-3d07-76c4-9a40-568a4351264c","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-41 evidence (PR #24 comment 3843504733): while MODULE owns memories, module-state-sync.ts:1662 suppresses the full policy replacement/delete list; a native ctx_memory row is active+unverified in the module store and its mirrored CANDIDATE decision never flows back — McStore::load_memory_render_snapshot and the native search/get readers select active rows with no claim-policy predicate, so the row stays injectable and searchable indefinitely. The decision channel must either propagate policy visibility separately (preserving single-writer content ownership) or enforce mirrored decisions in the native readers.","created_at":"2026-08-24T12:51:17Z"},{"id":"01a034ea-fa65-7ed8-b0b1-a5cf5cbea034","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-57 evidence (PR #24 comment 3846037059): while authorityState is MODULE, omitAuthorityMemorySections suppresses the policy-filtered snapshot AND its deletion/reclassification updates; the authority seed copies raw rows and native render/search select active rows with no policy predicate — a fresh unverified CANDIDATE or a later-quarantined/contradicted/rejected row stays injectable through native Rust-mode surfaces. The decision channel must stay synchronized without transferring row ownership.","created_at":"2026-08-24T17:56:50Z"},{"id":"01a0351d-751f-7db2-a49f-5d5f32937cfb","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-60 evidence (PR #24 comment 3846478393): with omitAuthorityMemorySections both allMemories and incrementalMemories are empty, so the sync's policy filter never adjudicates native ctx_memory-created rows; mc-store renders active/permanent rows with no policy predicate, so a fresh unverified CANDIDATE enters native m0/m1 and search immediately. Reviewer proposes carrying host-computed eligibility to the native store or failing closed for native-created rows before rendering.","created_at":"2026-08-24T18:51:58Z"}],"dependency_count":0,"dependent_count":0,"comment_count":4} {"_type":"issue","id":"magic-context-a52","title":"Add bounded Synapse queue and latency instruments","description":"## Context\n\n`magic-context-515` measured the embedding path and found the transport floor is sub-ms while\n`embed.query` and `embed.result` tails sit at 100ms-1s, caused by fail-fast query admission plus\nsynchronized client retries and fixed 50ms result polling. The two follow-ups that must change that\nmechanism -- `magic-context-s64` (bounded queue/backoff/fast-first polling) and `magic-context-ioi`\n(concurrency topology) -- cannot attribute a tail today: the host exposes **zero** counters, so a\np95 improvement cannot be split into \"waited for the CPU permit\", \"inference was slow\", or \"client\nslept on a `retry_after`\". This task adds the smallest fixed-cost, fixed-cardinality instrument set\nthat makes that attribution possible, and nothing more.\n\n**There is no metrics sink in this repo.** No `prometheus`/`opentelemetry`/`metrics` dependency\nexists (`Cargo.toml`, `crates/mc-host/Cargo.toml`), and `HealthReport.metrics`\n(`crates/mc-host/src/handler.rs:178`) is dropped by the health probe\n(`crates/mc-host/src/runtime.rs:955-975`, \"the report itself is informational\"). Scope is therefore\nan in-process snapshot read by tests, by `health()`, and by the bench example's stderr dump. Do not\nadd an exporter, a metrics crate, or a wire op.\n\n## Current State\n\n```rust\n// crates/mc-host/src/synapse/mod.rs:164-179 — no instrument state exists\nstruct SynapseInner {\n config: Option\u003cSynapseConfig\u003e, limits: SynapseLimits, state: Mutex\u003cLaneState\u003e, jobs: JobTable,\n cpu: Arc\u003ctokio::sync::Semaphore\u003e, // 1 permit, FIFO\n query_admission: Arc\u003ctokio::sync::Semaphore\u003e, // 1 permit, fail-fast\n tracker: TaskTracker, closing: CancellationToken,\n}\n```\n\n```rust\n// crates/mc-host/src/synapse/mod.rs:478-479 — rejection is invisible\nlet Ok(query_permit) = Arc::clone(\u0026self.inner.query_admission).try_acquire_owned() else {\n return app_error(\"queue_full\", \"query admission capacity is exhausted\");\n};\n```\n\n```rust\n// crates/mc-host/src/synapse/mod.rs:499-515 — unbounded-duration wait, unmeasured\nlet permit = tokio::select! { biased;\n () = inner.closing.cancelled() =\u003e { let _ = tx.send(Err(QueryFault::Cancelled)); return; }\n () = tx.closed() =\u003e return,\n () = tokio::time::sleep_until(deadline) =\u003e { let _ = tx.send(Err(QueryFault::Timeout)); return; }\n permit = Arc::clone(\u0026inner.cpu).acquire_owned() =\u003e permit,\n};\n```\n\n`spawn_batch_worker` (mod.rs:638-651) has the same unmeasured wait; `handle_result`\n(mod.rs:729-733) discards which `PollOutcome` it served; six sites return `queue_full`\n(mod.rs:373, 479, 621, 725, 821, 851) with no way to tell them apart. Depth is already cheap:\n`Semaphore::available_permits()` and the `by_seq`/`queued_text_bytes`/`retained_result_bytes` fields\nin `jobs.rs:134-141`; `admit_charged` already walks `by_seq` per admission (jobs.rs:405-410), so an\nO(jobs\u003c=128) snapshot walk is no new cost class.\n\n## Desired State\n\nOne `SynapseMetrics` struct of plain atomics inside `SynapseInner`, a `snapshot()` reader, and a fixed\nset of recording sites. **Simplicity gate:** no observer trait (one implementation), no config knob, no\nbucket configuration, no reset API, no new module -- this mirrors `Supervisor::metrics()` /\n`SupervisorMetrics` (`crates/mc-host/src/broca/supervisor.rs:203-213, 271-291`), the pattern that\nalready exists in this crate for exactly this need.\n\n**Instruments (complete, fixed set).** All counters `AtomicU64`, `Ordering::Relaxed`, monotonic\nsince process start; consumers take deltas. No string labels anywhere; every \"dimension\" is a\ncompile-time array index, so series count stays below 128 scalars and is independent of traffic, job\nids, request keys, or text.\n\n| Instrument | Kind | Unit | Dimension (fixed) | Site |\n|---|---|---|---|---|\n| `cpu_wait.query` | histogram | ms | none | mod.rs:499 select -\u003e grant arm |\n| `cpu_wait.batch` | histogram | ms | none | mod.rs:641 select -\u003e grant arm |\n| `cpu_hold.query` | histogram | ms | none | CPU permit grant -\u003e blocking-task completion; includes blocking-pool queue + native inference |\n| `cpu_hold.batch` | histogram | ms | none | grant (mod.rs:648) -\u003e after the `settle_inference` match (mod.rs:670-688); includes blocking-pool queue + native inference |\n| `inference.query` | histogram | ms | none | immediately around `backend.embed` inside the blocking closure |\n| `inference.batch` | histogram | ms | none | immediately around `backend.embed` inside the blocking closure |\n| `cpu_wait_outcome.query` | counter[4] | count | granted, timeout, waiter_gone, cancelled_or_closed | mod.rs:499-518 arms |\n| `cpu_wait_outcome.batch` | counter[3] | count | granted, cancelled, closed | mod.rs:641-649 arms |\n| `queue_full` | counter[6] | count | parse_reservation_unsatisfiable, parse_resident_exhausted, parse_coverage_short, query_admission, job_admission, result_page_resident | one helper, all six sites |\n| `poll_outcome` | counter[7] | count | restarted, key_mismatch, bad_cursor, failed, pending_queued, pending_running, page | mod.rs:729-733 |\n| depth gauges | computed on read | count / bytes | free_cpu_permits, free_query_permits, jobs_active, jobs_retained, queued_text_bytes, retained_result_bytes | `snapshot()` only |\n\n`query_admission_rejected` is `queue_full[query_admission]`; do not add a second counter for it.\n\n**Histogram semantics.** Fixed edges in ms: `[1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000]`\nplus one overflow bucket = 13 `AtomicU64`, each observation lands in exactly one bucket\n(`edge[i-1] \u003c= v \u003c edge[i]`), plus `count` and `sum_us` (microseconds, saturating) so the mean is\nexact and two snapshots merge additively. Edges at 50/100/200 are chosen deliberately: they straddle\nthe 50ms `retry_after_ms` quantum (`mod.rs:81`) and the client's 100ms `queue_full` backoff\n(`packages/plugin/src/features/magic-context/memory/embedding-synapse.ts:1453-1454`) that s64 must\nremove. **The host computes no percentiles** -- exact percentiles keep coming from the load\ngenerator's sorted vectors (`docs/perf/mc-host-baseline.md`, \"Percentiles computed over all requests\nof one run (exact, sorted vector)\"). Host buckets answer *where the time went*, not *how long*.\n\n**Cost bound to verify, not assume.** Per request: a bounded number of `Instant::now()` reads and\nrelaxed `fetch_add`; per poll: one fixed counter update. Memory is six fixed histograms plus fixed\noutcome counters, under 1 KiB, with zero per-request allocation and zero per-job metric state. The\nmodel-free overhead experiment decides whether these reads and shared atomics are acceptable; no\nnanosecond estimate is treated as evidence.\n\n**Fast path.** Start with the existing in-process snapshot pattern and fixed relaxed atomics; do not\nadd observer dispatch or dynamic labels. Treat its cost as a hypothesis until the model-free overhead\nrun. The lane-unavailable paths return before instruments are touched. If fresh evidence shows the\nclock reads or shared atomics are material, the minimum fallback is one optional snapshot pointer and\none predictable guard per site, not a new framework.\n\n## Required skills and execution order\n\n1. `/systems-design:metrics-audit` owns names, units, fixed dimensions, bucket boundaries, cardinality, mergeability, privacy, and consumer usefulness.\n2. `/quantitative-analysis:statistics-and-benchmarking-discipline` fixes which observations the instruments represent. Attempt and logical-request metrics, service and wait time, and terminal outcomes must not be conflated.\n3. `/performance:perf-pipeline` ensures instruments answer the queueing/concurrency decisions rather than adding generic telemetry, and coordinates the model-free overhead baseline.\n4. `/systems-design:bounded-design` verifies metric storage and recording work are fixed-capacity, non-blocking, allocation-free on request paths, and safe under overload.\n5. `/quantitative-analysis:benchmark-experiment-design` is required for the before/after overhead claim, including process-level units, A/A path validation, replication, and stopping.\n6. `/testing:test-strategy` chooses bucket-boundary, exact-attribution, concurrency, snapshot, and privacy tests. `/implementation:rust-implementation` applies the minimal in-crate snapshot pattern.\n\n## Implementation Guidance\n\n### Files to Modify\n| File | Change | Rationale |\n|---|---|---|\n| `crates/mc-host/src/synapse/mod.rs` | add private fixed metrics/snapshot/histogram state; field on `SynapseInner`; initialize in both constructors; record at the table sites; expose `metrics()`; extend `health()` | all recording sites live here |\n| `crates/mc-host/src/synapse/jobs.rs` | `pub fn depth(\u0026self) -\u003e JobDepth` after `key_is_retained:319`, reusing `lock_jobs` and `Job::is_completed:130` | table internals are private |\n| `crates/mc-host/examples/synapse_host.rs` | print one `serde_json` snapshot line to stderr on shutdown (and every `--stats-secs N` if passed) | only cross-process consumer; no wire op needed |\n| `crates/mc-host/tests/synapse_metrics.rs` (new) | snapshot contract tests | mirrors `tests/broca_supervisor.rs:80` |\n| `crates/mc-host/tests/synapse_protocol.rs:142` | extend the existing `queue_full` case with the counter assertion | reuse, don't duplicate the fixture |\n| `docs/perf/mc-host-baseline.md` | short \"Synapse instrument semantics\" subsection: names, units, bucket edges, what `cpu_hold` includes | the measurement contract doc s64/ioi cite |\n\n### Patterns to Follow\n- `SupervisorMetrics` + `Supervisor::metrics()` (`broca/supervisor.rs:203-291`) -- snapshot struct built from `available_permits()` + table sizes. Copy this shape.\n- `HealthReport.metrics` JSON keys with `_ms` suffixes and tests asserting them: `crates/mc-module/src/lib.rs:294-302` and `lib.rs:16679-16688`.\n- One funnel per policy: add `fn queue_full(inner, reason, message) -\u003e RequestOutcome` next to `app_error` (mod.rs:417) and route all six sites through it, so a future `queue_full` site cannot silently escape counting.\n\n### Blast Radius\n`SynapseInner` is private; `SynapseComponent::new`/`ready_with_engine` signatures do not change, so\n`examples/synapse_host.rs`, `tests/support/synapse.rs`, and every existing synapse test compile\nuntouched. `health()` gains a `metrics` payload on the `Ok` arm (previously `HealthReport::ok()`,\nmetrics `None`) -- `tests/synapse_bundle.rs:156` only asserts `status`, so it stays green. No wire\nchange: `protocol::decode_request`, `docs/mc-host-wire-protocol.md`, `tests/protocol_vectors.rs`,\nand the TS client are out of scope.\n\n## Testing Strategy\n1. Bucket unit test in `mod.rs` `#[cfg(test)]`: values at every edge and one overflow land in the\n documented bucket; `count == sum(buckets)`; `sum_us` matches the observations.\n2. `queue_full` attribution: the existing overload case (`tests/synapse_protocol.rs:142`) bumps\n `queue_full[query_admission]` exactly once and creates no job; a saturated job table bumps\n `queue_full[job_admission]` and `jobs_active == max_queued_jobs`.\n3. Wait/hold/service attribution: a gate-controlled engine via `ready_with_engine` blocks before and\n inside inference; snapshots distinguish `cpu_wait.query`, `cpu_hold.query`, and `inference.query`,\n with exactly one granted outcome. Repeat for batch. Assert bucket membership and counter identity,\n never wall-clock equality.\n4. Poll accounting: `pending_queued`, `page`, `restarted` each increment exactly once per\n `embed.result` call, including the multi-page path (no double counting per item).\n5. Contract test: snapshot JSON key set and units are stable, and the payload contains no job id,\n request key, content hash, or text (metrics-audit privacy rule).\n\n### Benchmark overhead gate\nRe-run the model-free tiny-engine arm from `magic-context-515` (see memory\n`embedding-machinery-baseline-21bd53d0`) before and after, 3 open-loop runs each, same host/commit\ncontract as `docs/perf/mc-host-baseline.md`. Gate: median throughput delta and `embed.query` p99\ndelta must be within the measured run-to-run spread, and the report must state that spread (n=3\nmedians cannot resolve \u003c1% honestly -- claim \"no detectable regression\", not \"0.4% faster\"). If a\nregression exceeds the spread, drop the clock reads from the poll path first, then revert.\n\n## Related Work\n| Task | Relationship |\n|---|---|\n| `magic-context-s64` | blocked by this: needs `cpu_wait.*` + `queue_full[*]` + `poll_outcome[*]` to prove the 100ms staircase is gone |\n| `magic-context-ioi` | blocked by this: needs `cpu_wait`/`cpu_hold` split and depth gauges to judge a concurrency topology |\n| `magic-context-18r` | consumer: `cpu_hold.batch` bounds the duplicate-preprocessing win |\n| `magic-context-09u` | consumer: client-side yield fix, verified against the same counters |\n| `magic-context-515` | closed origin of all four |\n| `magic-context-09q` | parent enrichment umbrella |\n\nWiring: `bd dep \u003cnew-id\u003e --blocks magic-context-s64` and `bd dep \u003cnew-id\u003e --blocks magic-context-ioi`.\n\n## Acceptance Criteria\n- [ ] All instrument groups in the table exist with the documented units and bucket edges, and\n `SynapseComponent::metrics()` returns them in one allocation-free snapshot (depth gauges\n computed on read).\n- [ ] Every `queue_full` return in `synapse/mod.rs` routes through the counting helper (grep shows no\n bare `app_error(\"queue_full\", ...)` left in the file).\n- [ ] Snapshot carries no unbounded or sensitive value: no job id, request key, content hash, text,\n or per-item state; series count is a compile-time constant.\n- [ ] `health()` exposes the snapshot as JSON and one test asserts its key/unit set.\n- [ ] Tests 1-5 above pass; `bun run test:rust` and `cargo clippy --workspace --all-targets -- -D warnings`\n and `cargo fmt --check` are clean.\n- [ ] Overhead gate reported in `docs/perf/mc-host-baseline.md` with n, spread, and verdict.\n\n## Stop Conditions (do not exceed)\nStop when s64 and ioi can split a tail into wait / hold / rejection / poll from one snapshot, and the\noverhead gate passes. Explicitly out of scope: any exporter or metrics crate; a new wire op or\nprotocol version bump; plugin-side instrumentation; host-computed percentiles; per-key, per-tenant, or\nper-job labels; configurable buckets; reset/clear APIs; dashboards or alarms (no monitoring backend\nexists to consume them).\n\n## Pointers\n`docs/perf/mc-host-baseline.md` (measurement contract), memory\n`embedding-machinery-baseline-21bd53d0` (the 515 evidence), `crates/mc-host/src/broca/supervisor.rs:203-291`\n(snapshot pattern), `crates/mc-module/src/lib.rs:294-302` (health metrics JSON precedent),\n`crates/mc-host/src/synapse/jobs.rs:34-63` (`AdmitOutcome`/`PollOutcome` variant sets that pin the\ncounter dimensions).\n","acceptance_criteria":"Fixed-cardinality allocation-free Synapse snapshots distinguish permit wait, permit hold, native inference, rejection reason, poll outcome, and queue/job depth with documented units and buckets; queueing/concurrency tasks can reconcile them with wire outcomes; model-free overhead evidence passes human review without an exporter or wire operation.","status":"open","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-23T20:32:58Z","created_by":"AhravDutta","updated_at":"2026-08-23T20:36:31Z","labels":["mc-host","observability","performance","synapse"],"dependencies":[{"issue_id":"magic-context-a52","depends_on_id":"magic-context-515","type":"discovered-from","created_at":"2026-08-23T20:32:57Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":0,"dependent_count":2,"comment_count":0} @@ -39,7 +40,7 @@ {"_type":"issue","id":"magic-context-c50.1","title":"Spike: inventory subc API surface actually used","description":"Inventory the subc API surface used by mc-module and the TypeScript transport callers, then use that inventory to plan a direct port to the mc-host SDK.\n\n## Decision\nThe project has no legacy callers or deployed users. Do not preserve the subc API through compatible local shim crates. Rewrite mc-module and TypeScript boundaries against the mc-host protocol and owned types.\n\n## Scope\n- Keep the existing inventory and evidence for semantic coverage.\n- Identify every caller that must change for mc-host.\n- Define the smallest owned Rust and TypeScript SDK surface.\n- Record explicit removals of subc compatibility crates, aliases, and adapter-only types.\n\n## Acceptance\n- Inventory is complete.\n- Direct mc-host boundary design is recorded.\n- No shim-crate decision remains.\n- c50.4 has an implementation-ready port plan.","design":"Companion evidence doc: docs/subc-api-surface-inventory-2026-08-17.md. Reproduction: docs/evidence/verify-rust-surface.py, docs/evidence/verify-ts-surface.py, docs/evidence/subc-surface-probe/. Blocked compiler closure tracked as magic-context-c50.12 (gates c50.4).","acceptance_criteria":"Complete API inventory plus direct mc-host SDK port plan; no compatible shim-crate decision remains.","notes":"Supersedes the earlier shim decision. Inventory remains useful, but direct mc-host port is now required because there are no legacy callers.","status":"in_progress","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:33:32Z","started_at":"2026-08-17T04:28:35Z","dependencies":[{"issue_id":"magic-context-c50.1","depends_on_id":"magic-context-c50","type":"parent-child","created_at":"2026-08-17T03:51:11Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-c50.2","title":"Design minimal wire protocol + connection handshake","description":"Single-module host needs: connection file discovery (subc-connection.json in dataDir/cortexkit/run — keep path so TS side barely changes), framing (length-prefixed JSON is fine; docs/rust-mode-transport-overhead-2026-08-10.md shows codec cost is negligible), request/response correlation, route/session semantics mc-module relies on (OpenedRoute/RouteHandle epoch checks), timeouts, restart/reconnect story (TS side already has backoff + lease-wait on store open). Use published subc-protocol 0.10.0 source as reference for message shapes. Cut: flow credits, multi-module routing, admission classes (keep enum stubs if shimming).","acceptance_criteria":"Protocol doc: frame format, handshake, message set, error/reconnect semantics","notes":"Planning artifact: docs/plans/2026-08-17-0550-docs-minimal-wire-protocol-plan.md (implementation-ready; deepened 2026-08-17). Non-interactive document review: 0 fixes applied; 3 proposed fixes and 6 decisions remain (8 P1, 1 P2; 0 FYI). Cross-model review was skipped because provider identity could not be attested.\nNormative artifact implemented at docs/mc-host-wire-protocol.md. Review gates and final verification passed at close.\nFinal verification: all R1-R16 and AE1-AE13 mapped; conformance scenario matrix V1-V44 (count grew during PR #4 review rounds; the matrix in docs/mc-host-wire-protocol.md Section 14 is authoritative, not this note); JSON/HMAC/header vectors independently recomputed; HTML render and pi-lens Markdown diagnostics clean. Final invariant-test-review and test-strategy passed.\nDependency on magic-context-c50.1 reclassified blocks -\u003e related at close: c50.1's residual compiler-closure completeness risk is tracked by magic-context-c50.12, so downstream resolution must not treat this doc task as gating on it.","status":"closed","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","updated_at":"2026-08-18T00:40:09Z","started_at":"2026-08-17T05:37:41Z","closed_at":"2026-08-17T15:53:51Z","close_reason":"Normative protocol implemented at docs/mc-host-wire-protocol.md; inventory contradiction corrected; downstream retry/counter/dependency obligations recorded; review and document gates pass. Forced only because c50.1 remains open for its separate compiler-closure risk tracked by c50.12.","dependencies":[{"issue_id":"magic-context-c50.2","depends_on_id":"magic-context-c50.1","type":"related","created_at":"2026-08-18T00:40:08Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.2","depends_on_id":"magic-context-c50","type":"parent-child","created_at":"2026-08-17T03:51:11Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"magic-context-c50.3","title":"Implement mc-host daemon crate","description":"New crate crates/mc-host: tokio unix/TCP socket listener, writes connection file, spawns/links mc-module handler directly (no dynamic module loading), correlation dispatch, graceful shutdown + connection-file cleanup, single-instance lock (cortexkit-lease exists in forked commons). Reuse cortexkit-paths for data dirs.","acceptance_criteria":"Daemon starts, writes connection file, echoes a round-trip request; unit tests","notes":"Planning artifact: docs/plans/2026-08-18-0125-feat-mc-host-daemon-core-plan.md (implementation-ready; deepened 2026-08-18). Confirmed c50.3 acceptance is the complete protocol-conformant generic host core required by docs/mc-host-wire-protocol.md, not only an echo path. Host-owned acceptance includes secure instance/publication, authentication/framing, control and global route lifecycle, bounded dispatch/settlement, admission/health/shutdown, and host conformance proof. McHandler adaptation and production ck-mc wiring remain owned by magic-context-c50.4; dependency direction is mc-module -\u003e mc-host and mc-host exports no subc-* types. Non-interactive ce-doc-review: 0 fixes applied; 2 proposed fixes, 9 decisions, and 2 FYI observations remain (10 actionable items at P1). Cross-model pass did not run because the host serving family could not be attested.\nImplementation complete on feat/mc-host-daemon-core. Added crates/mc-host with secure descriptor-relative instance lifecycle, pinned published auth/wire/control graph, bounded global route/request ownership, first-terminal settlement, Ping/Pong capability disabled by default, tracked shutdown, and independent raw-client conformance suites. Final focused evidence: cargo fmt -p mc-host -- --check; cargo check -p mc-host --all-targets; cargo clippy -p mc-host --all-targets -- -D warnings; cargo test -p mc-host (140 tests incl. doctest); cargo metadata --no-deps; cargo check --workspace --all-targets. mc-host has no mc-module dependency and resolves subc-protocol 0.10.0, subc-transport 0.5.0, subc-control 0.1.1. Full workspace fmt remains blocked only by pre-existing crates/mc-module/src/historian_validate.rs formatting drift. Requested rust-code-reviewer/invariant-test-review/ponytail-review passes run before commit; findings addressed and re-reviewed.\nFinal post-review evidence: 144 mc-host tests/doctests pass. Final rust-code-reviewer verdict: No actionable findings. Final invariant-test-review verdict: No blocking findings. Final ponytail-review verdict: Lean already. Ship. Full workspace check passes using locally restored sibling compatibility sources; full workspace fmt still reports only pre-existing crates/mc-module/src/historian_validate.rs formatting drift.","status":"closed","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","updated_at":"2026-08-18T07:10:57Z","started_at":"2026-08-18T01:09:10Z","closed_at":"2026-08-18T07:10:57Z","close_reason":"Implemented and verified protocol-conformant generic mc-host core; final requested reviews clean.","dependencies":[{"issue_id":"magic-context-c50.3","depends_on_id":"magic-context-c50","type":"parent-child","created_at":"2026-08-17T03:51:11Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.3","depends_on_id":"magic-context-c50.2","type":"blocks","created_at":"2026-08-17T03:52:13Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":1,"dependent_count":5,"comment_count":0} -{"_type":"issue","id":"magic-context-c50.4","title":"Port mc-module directly to mc-host SDK","description":"Port mc-module and TypeScript transport callers directly to the mc-host SDK.\n\n## Goal\nThe new host protocol is the only supported module boundary. Remove subc compatibility shims and stale subc caller APIs.\n\n## Scope\n- Rewrite mc-module imports and boundary types against mc-host-owned contracts.\n- Rewrite TypeScript module-transport, embedding, wake-plane, and related callers against the new client.\n- Remove local `subc-shim` crates, Cargo patches, compatibility aliases, and adapter-only translation layers.\n- Preserve protocol semantics that are part of the new mc-host contract: checked correlation exhaustion, typed errors, epochs, health probes, and shutdown behavior.\n- Replace unsupported resolver paths with direct host routes or explicit typed absence.\n\n## Acceptance\n- `cargo test --workspace` passes with mc-module included.\n- TypeScript transport and e2e tests use mc-host directly.\n- No production `subc_*` compatibility crate or shim remains.\n- Unsupported old daemon/API versions fail closed rather than selecting a compatibility path.","acceptance_criteria":"Workspace and e2e tests pass with mc-module using mc-host directly; no subc compatibility crate, shim, or adapter remains.","notes":"No subc compatibility layer. Rewrite callers against mc-host-owned types and fail closed on unsupported old APIs.","status":"open","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:33:32Z","dependencies":[{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50.3","type":"blocks","created_at":"2026-08-17T03:52:13Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50","type":"parent-child","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50.12","type":"blocks","created_at":"2026-08-17T05:22:01Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"magic-context-c50.4","title":"Port mc-module directly to mc-host SDK","description":"Port mc-module and TypeScript transport callers directly to the mc-host SDK.\n\n## Goal\nThe new host protocol is the only supported module boundary. Remove subc compatibility shims and stale subc caller APIs.\n\n## Scope\n- Rewrite mc-module imports and boundary types against mc-host-owned contracts.\n- Rewrite TypeScript module-transport, embedding, wake-plane, and related callers against the new client.\n- Remove local `subc-shim` crates, Cargo patches, compatibility aliases, and adapter-only translation layers.\n- Preserve protocol semantics that are part of the new mc-host contract: checked correlation exhaustion, typed errors, epochs, health probes, and shutdown behavior.\n- Replace unsupported resolver paths with direct host routes or explicit typed absence.\n\n## Acceptance\n- `cargo test --workspace` passes with mc-module included.\n- TypeScript transport and e2e tests use mc-host directly.\n- No production `subc_*` compatibility crate or shim remains.\n- Unsupported old daemon/API versions fail closed rather than selecting a compatibility path.","acceptance_criteria":"Workspace and e2e tests pass with mc-module using mc-host directly; no subc compatibility crate, shim, or adapter remains.","notes":"No subc compatibility layer. Rewrite callers against mc-host-owned types and fail closed on unsupported old APIs.","status":"closed","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","updated_at":"2026-08-25T06:56:30Z","started_at":"2026-08-24T22:53:13Z","closed_at":"2026-08-25T06:56:30Z","close_reason":"Implemented direct mc-host boundary: host-owned Rust wire/auth/discovery/client, direct McHandler adapter and historian, host-owned TS API, direct Rust/E2E fixtures, dependency/docs closure; workspace, TS, E2E, specialist and ponytail reviews complete.","dependencies":[{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50.3","type":"blocks","created_at":"2026-08-17T03:52:13Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50","type":"parent-child","created_at":"2026-08-17T03:51:12Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-c50.4","depends_on_id":"magic-context-c50.12","type":"blocks","created_at":"2026-08-17T05:22:01Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"magic-context-c50","title":"Hand-rolled Rust module host: replace private subc daemon","description":"Rust mode is blocked: crates/mc-module needs subc-* SDK crates from the private github.com/cortexkit/subconscious repo (crates.io copies are stale: protocol 0.10 vs needed 0.12; subc-core/subc-jsonc unpublished), and the subc daemon binary itself is a separate unreleased product. All heavy MC logic (transform, historian, store) already exists in Rust in this repo — only the hosting daemon + SDK glue is missing. Hand-roll a minimal single-module host: our own daemon binary linking mc-module logic directly, our own thin wire protocol to the TS plugin, replacing @cortexkit/subc-client. We do NOT need subc's generality (multi-module routing, flow credits, epochs across modules) — one module, one consumer.","status":"open","priority":1,"issue_type":"epic","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T03:50:43Z","created_by":"AhravDutta","updated_at":"2026-08-17T03:50:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-3q5.5","title":"U5: Stage-decomposed benchmark harness","description":"Phase 1 — Measure. THIS UNIT IS THE GATE: U5 gates all ranking/model/layout/ANN changes. U30, U15, U16, U17, U18, U19, U20, U23, and U26 must pass its regression mode before landing. Its baseline must exist BEFORE any Phase 3/4 ranking-affecting change lands.\n\n## Goal\nOne harness that measures retrieval quality and stage-level latency, and serves as the regression gate.\n\n## Governing constraints (quoted)\n- R27: stage-level timing decomposition — \"query inference, generation lookup, filter construction, vector scan, top-K, metadata hydration, fusion, reranking, packing\"; reports \"Recall@10/50, MRR, nDCG@10, p50/p95 latency, decoded bytes per query, and index build time\"; \"fixture-judged relevance labels (shadow result disagreement is not a label)\".\n- R28: \"Hardware-level bets (AoSoA layout, int8/f16/Matryoshka-prefix cascades, GEMM micro-batching, mmap layouts) land only on measured wins against the row-major f32 exact baseline with stable tie-breaking, reporting recall and ranking quality for every approximate variant.\"\n- KTD17: \"nDCG@10 regression tolerance \u003c= 2 percentage points versus the stored snapshot, averaged over 3 runs, with no single run below baseline - 5 points; Recall@50 uses the same 2/5-point tolerance pair; p95 latency tolerance is +10% versus snapshot. Store the tolerance values themselves as versioned config next to the snapshot so tightening them later is a reviewed change, not a silent drift.\"\n- Benchmark labels use canonical claim, revision, and retrieval-document identities built in U4. Legacy result IDs are rejected.\n- KD6/Success Criteria: the U5 measurement RECORDS whether TS-only plus Phase 0/1 already met the latency and recall thresholds — an auditability record, not a stop condition.\n\n## Key files\n- packages/plugin/scripts/benchmark-retrieval.ts (new), reusing patterns from packages/plugin/scripts/benchmark-message-fts.ts; Rust side hooks later via U10's criterion benches.\n\n## Approach\n1. Metrics: Recall@10/50, MRR, nDCG@10, reranker lift, duplicate rate, context tokens per useful result, p50/p95 latency, decoded bytes per query, index build time.\n2. Stage decomposition: a labeled timer around each pipeline seam, recording BOTH inclusive and exclusive span duration. Lexical and dense lanes run in parallel, so account for overlapping spans on the CRITICAL PATH, not by summing every stage's elapsed time — a naive sum over parallel spans would either exceed wall-clock time or misattribute a regression to the wrong stage.\n3. Parameter matrix: corpus 1K-1M, dims 128-1024, K 5-100, filter selectivity 0.1%-100%, cold/warm page cache, concurrency 1-8; run on ARM NEON and x86 AVX2 hosts.\n4. Baseline the current pipeline per-source and fused BEFORE any Phase 3/4 change lands; store results as versioned JSON snapshots for regression comparison. Record the TS-only-vs-thresholds measurement (KD6).\n5. Acceptance policy per KTD17 (quoted above); tolerance values live as versioned config next to the snapshot.\n6. Relevance labels resolve directly to claims, revisions, and retrieval documents. No legacy-vocabulary adapter or simulated migration path exists.\n\n## Test scenarios\n- Harness on the seeded synthetic corpus reproduces identical quality metrics across two runs (determinism).\n- Inclusive stage-span durations sum to within tolerance of end-to-end time; EXCLUSIVE (critical-path) durations are used for regression attribution; a fixture with overlapping lexical/dense lanes does not fail the accounting check purely from parallelism.\n- Regression mode exits nonzero when nDCG@10 or Recall@50 breaches the KTD17 2/5-point rule vs a stored snapshot, or p95 latency exceeds +10%; passes when within tolerance across 3 runs.\n- A result whose storage identity changed across the claims migration (same canonical relevance identity, different underlying row shape) scores as present, not as a false recall loss.\n\n## Verification\n- Harness runs clean on the fixture corpus in CI-sized configuration; baseline snapshot and its versioned tolerance config committed together.\n\n","design":"Focused implementation plan: docs/plans/2026-08-19-0158-feat-stage-decomposed-benchmark-plan.md. Parent architecture plan remains unchanged. Planning decisions: one modular command-facing harness; per-call observation of real retrieval and packing paths; portable quality baselines separated from exact-host latency baselines; TS-only quality target_status is not_specified; sparse v1 supports judged-regression detection but measured-win claims require the denser reviewed release from magic-context-u51.","acceptance_criteria":"Deterministic stage and quality baselines use canonical post-cutover identities; no benchmark code accepts legacy result IDs; regression gate passes.","notes":"Benchmark gate remains. It measures the new claims-native retrieval contract directly; no legacy-ID adapter or old-pipeline baseline is a runtime dependency.","status":"in_progress","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:23:51Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:33:37Z","started_at":"2026-08-19T01:17:43Z","labels":["benchmark-gate","phase-1"],"dependencies":[{"issue_id":"magic-context-3q5.5","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:23:50Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.5","depends_on_id":"magic-context-3q5.4","type":"blocks","created_at":"2026-08-17T02:38:13Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":1,"dependent_count":9,"comment_count":0} {"_type":"issue","id":"magic-context-3q5.6","title":"U30: Query-purpose default flip (benchmark-gated)","description":"Phase 1 — Measure. BENCHMARK-GATED: U5 gates all ranking/model/layout/ANN changes; this is a ranking-affecting change and must pass U5 regression mode before landing (KTD20 makes the dependency explicit).\n\n## Goal\nunifiedSearch embeds queries as queries by default, landed against a measured baseline rather than ahead of it.\n\n## Governing constraints (quoted)\n- R24: \"The query-vs-passage purpose is honored end to end: unifiedSearch defaults to query purpose, and the local provider applies purpose templates instead of ignoring the parameter.\" (The provider-side template work is U24; this unit flips the default.)\n- R27: the judged benchmark gates ranking changes.\n- KTD20: \"U3's default embed-purpose change ships only after U5's benchmark can compare the pre-change (passage-default) and post-change (query-default) paths. A ranking-affecting change landing before the judged baseline exists would let a real regression become the new reference undetected.\"\n\n## Key files\n- packages/plugin/src/features/magic-context/memory/embedding.ts\n- packages/plugin/src/features/magic-context/search.ts (+ co-located tests)\n\n## Approach\n1. Change unifiedSearch's default embed purpose to \"query\" (the embedText fallback currently embeds as passage); the shadow-measurement path uses \"query\" too so comparisons stay fair.\n2. Run the U5 harness comparing the pre-change (passage-default) and post-change (query-default) paths on the judged corpus BEFORE merging; a regression beyond KTD17's tolerance blocks the change.\n\n## Test scenarios\n- unifiedSearch without an embedQuery override calls the provider with purpose \"query\" (spy on provider); existing callers passing wrappers keep their behavior.\n- Shadow measurement records under the same purpose as primary.\n- Benchmark comparison: the U5 harness run comparing passage-default against query-default purpose is committed alongside this unit's PR, showing no regression beyond KTD17's tolerance.\n\n## Verification\n- bun test; U5 benchmark comparison attached to the PR.\n","design":"Focused implementation plan: docs/plans/2026-08-19-1526-fix-query-purpose-default-plan.md. Keep generic embedText passage-default; make only unifiedSearch's missing-override path request query purpose; preserve OpenCode, Pi, explicit, automatic, and shadow wrappers. Capture a scoreable pre-change baseline before the code change, require fallback-purpose evidence plus KTD17 regression output, and commit provenance-bound reports. U5 and magic-context-4s1 remain landing prerequisites. Non-interactive doc review left two gate decisions and two proposed corrections for user adjudication.","notes":"Implemented on branch fix/query-purpose-search-default per docs/plans/2026-08-19-1526-fix-query-purpose-default-plan.md. unifiedSearch fallback now requests purpose query; embedText stays passage-default; provider-boundary tests (AE1-AE4) in search.test.ts/embedding.test.ts/search-measurement.test.ts. U5 evidence: docs/evidence/retrieval-benchmark/u30-query-purpose/ (3 pre-passage + 3 post-query reports, regression quality-only with gate.unblocked=true, manifest binds commits/purposes/digests). Baseline republished from the pre-change runs. PR: https://github.com/ahrav/magic-context/pull/13","status":"in_progress","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:23:51Z","created_by":"AhravDutta","updated_at":"2026-08-19T23:23:24Z","started_at":"2026-08-19T15:14:59Z","labels":["benchmark-gated","phase-1"],"dependencies":[{"issue_id":"magic-context-3q5.6","depends_on_id":"magic-context-3q5.5","type":"blocks","created_at":"2026-08-17T02:38:29Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.6","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:23:51Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.6","depends_on_id":"magic-context-4s1","type":"blocks","created_at":"2026-08-19T15:37:12Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 63aa77d95..0e24027a6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -12,8 +12,8 @@ Magic Context is an `@opencode-ai/plugin` (entry `src/index.ts`) that rewrites t - **Hidden subagents** (`historian`, `historian-editor`, `dreamer`, `sidekick`) do the heavy LLM work out of band; the transform itself does no LLM calls. - **Runtime SQLite backend** (`src/shared/sqlite.ts`): `bun:sqlite` under Bun, `node:sqlite` (`DatabaseSync`) under Node (Pi) and Electron (Desktop). The non-Bun branch adds a savepoint-aware `transaction()` shim and `readonly`→`readOnly` mapping; otherwise identical. No native module, no prebuild. - **Pi parity:** `packages/pi-plugin/` mirrors OpenCode semantics, importing shared core from `@magic-context/core`. Intentional divergences are tracked in `packages/pi-plugin/PARITY.md`. Provider prefixes are translated between canonical (OpenCode) and Pi configuration models via `src/shared/harness-provider-map.ts` at configuration read/write edges to keep shared model configurations portable. Pi implements session state inheritance on branch clone forks by copying filtered tags, compartments, and pending marker states, whereas OpenCode `/fork` does not yet inherit context state. -- **Rust module and host migration.** A harness-agnostic Rust workspace in `crates/` re-implements the cache-stability transform and autonomous historian. The current `ck-mc` production path still runs under the subconscious daemon (`subc`), while `crates/mc-host/` now provides the reusable directly linked host runtime that owns secure publication, authentication, wire v2 framing, global routes, bounded dispatch, and shutdown. The `McHandler` adapter, production binary wiring, and TypeScript client cutover remain staged follow-up work. -- **Experimental Rust runtime mode.** Gated by `transform_mode: "rust"`, currently routes the entire Magic Context transform pipeline for a project through the ck-mc Rust module over subc. The TypeScript layer serves as a coordinator that manages state sync, ordinal tracking, and Last Known Good (LKG) fallbacks. +- **Direct Rust host boundary.** A harness-agnostic Rust workspace in `crates/` implements the cache-stability transform and autonomous historian. `crates/mc-host/` owns secure publication, authentication, wire v2 framing, global routes, bounded dispatch, managed clients, and shutdown. `McHandler`, Synapse, and Broca compose directly under that host with no provider process or private `subc-*` runtime dependency. Production launcher wiring remains follow-up work. +- **Experimental Rust runtime mode.** Gated by `transform_mode: "rust"`, routes the Magic Context transform pipeline through the direct mc-host boundary. TypeScript remains coordinator for state sync, ordinal tracking, and Last Known Good (LKG) fallbacks. ## Layers @@ -164,7 +164,7 @@ Background maintenance (V2: per-task cron scheduling). A process-wide 15-min tim Migration v84 keeps `memories` as current production reader projection while claims hold durable semantic history. OpenCode, Pi, dreamer, historian, identity repair, and module-mirror adapters own their outer immediate transactions and call dependency-light `memory/storage-memory-claims.ts`; kernel composes claim revisions/evidence, non-cascading `legacy_memory_claims` crosswalk, `memories` projection side effects, operation envelope, `claim_change_outbox`, one project-generation bump, and cursor/checkpoint changes atomically. `memory/storage-memory-projection.ts` remains leaf dependency, avoiding facade/kernel cycles. Search, mural, dreamer task gates, injection, and module serialization still read only `memories`; current-claim reads are independent reconciliation/U8-U9 surfaces, never production fan-out reads. -Ship OpenCode plugin, Pi plugin, CLI, and `ck-mc` from same release before any process applies v84. Transaction-scoped claims capability makes held-open legacy semantic writers fail; v84 also takes over pending v22 identity work. Production nonempty eager cutoff remains zero until full calibration is reviewed, so lazy recovery uses `magic-context doctor --check-claims-backfill` then `magic-context doctor --retry-claims-backfill` and restarts all harnesses. Pre-commit migration death reruns from complete v82; post-commit death rolls v84 forward; fence refuses downgrade. Back up with SQLite backup API, or stop writers, checkpoint, and retain `context.db`, `context.db-wal`, and `context.db-shm` together. Archive/delete retire compatibility rows while immutable claims remain; neither promises erasure. Erasure needs future privileged purge protocol. +Ship OpenCode plugin, Pi plugin, CLI, and the directly linked mc-host component from same release before any process applies v84. Transaction-scoped claims capability makes held-open legacy semantic writers fail; v84 also takes over pending v22 identity work. Production nonempty eager cutoff remains zero until full calibration is reviewed, so lazy recovery uses `magic-context doctor --check-claims-backfill` then `magic-context doctor --retry-claims-backfill` and restarts all harnesses. Pre-commit migration death reruns from complete v82; post-commit death rolls v84 forward; fence refuses downgrade. Back up with SQLite backup API, or stop writers, checkpoint, and retain `context.db`, `context.db-wal`, and `context.db-shm` together. Archive/delete retire compatibility rows while immutable claims remain; neither promises erasure. Erasure needs future privileged purge protocol. ## Session modes diff --git a/Cargo.lock b/Cargo.lock index fafb7e230..9176b3459 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -301,12 +301,6 @@ dependencies = [ "fs2", ] -[[package]] -name = "cortexkit-paths" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d14cf5bd9d4b76fcf4d5380550dd11f9191598309ecaacc4805e317b2d6deb90" - [[package]] name = "cortexkit-store" version = "0.1.0" @@ -632,16 +626,6 @@ dependencies = [ "winapi", ] -[[package]] -name = "fs4" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e72ed92b67c146290f88e9c89d60ca163ea417a446f61ffd7b72df3e7f1dfd5" -dependencies = [ - "rustix", - "windows-sys", -] - [[package]] name = "futures-core" version = "0.3.32" @@ -919,15 +903,6 @@ version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc04a4c58212d57930a24bf47d3fa87485264a3a054e9c10e042eb373573ad3c" -[[package]] -name = "matchers" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" -dependencies = [ - "regex-automata", -] - [[package]] name = "matrixmultiply" version = "0.3.11" @@ -961,9 +936,7 @@ dependencies = [ "serde", "serde_json", "sha2", - "subc-control 0.1.1", - "subc-protocol 0.10.0", - "subc-transport 0.5.0", + "subtle", "tempfile", "tokio", "tokio-util", @@ -973,6 +946,7 @@ dependencies = [ name = "mc-module" version = "0.1.0" dependencies = [ + "async-trait", "chrono", "chrono-tz", "cortexkit-lease", @@ -987,13 +961,9 @@ dependencies = [ "serde", "serde_json", "sha2", - "subc-client-rs", - "subc-control 0.1.2", - "subc-core", - "subc-protocol 0.12.0", - "subc-transport 0.5.1", "tempfile", "tokio", + "tokio-util", ] [[package]] @@ -1118,15 +1088,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "nu-ansi-term" -version = "0.50.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" -dependencies = [ - "windows-sys", -] - [[package]] name = "num-complex" version = "0.4.6" @@ -1436,15 +1397,6 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" -[[package]] -name = "rlimit" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f35ee2729c56bb610f6dba436bf78135f728b7373bdffae2ec815b2d3eb98cc3" -dependencies = [ - "libc", -] - [[package]] name = "rusqlite" version = "0.32.1" @@ -1566,15 +1518,6 @@ dependencies = [ "digest", ] -[[package]] -name = "sharded-slab" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" -dependencies = [ - "lazy_static", -] - [[package]] name = "shlex" version = "2.0.1" @@ -1649,112 +1592,6 @@ version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" -[[package]] -name = "subc-client-rs" -version = "0.3.1" -dependencies = [ - "async-trait", - "serde", - "serde_json", - "subc-control 0.1.2", - "subc-protocol 0.12.0", - "subc-transport 0.5.1", - "tokio", - "tokio-util", -] - -[[package]] -name = "subc-control" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "226b6eb603c230bfdee64b9f1ce9e032523dbecde2eb9977e609163b15f8fbbf" -dependencies = [ - "serde", - "serde_json", - "subc-protocol 0.10.0", -] - -[[package]] -name = "subc-control" -version = "0.1.2" -dependencies = [ - "serde", - "serde_json", - "subc-protocol 0.12.0", -] - -[[package]] -name = "subc-core" -version = "0.3.1" -dependencies = [ - "cortexkit-paths", - "fs4", - "getrandom 0.2.17", - "rlimit", - "serde", - "serde_json", - "subc-control 0.1.2", - "subc-jsonc", - "subc-protocol 0.12.0", - "subc-transport 0.5.1", - "terminal_size", - "tokio", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "subc-jsonc" -version = "0.0.0" - -[[package]] -name = "subc-protocol" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d13feabcd80d43ea9e12819f0e5b28f06c2be21de475bc0d0b7fe97ec27d7c6f" -dependencies = [ - "serde", - "serde_json", -] - -[[package]] -name = "subc-protocol" -version = "0.12.0" -dependencies = [ - "serde", - "serde_json", -] - -[[package]] -name = "subc-transport" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcc815910039de920bb680edd1a2c5743c86fe7a51e24c83c1db94da950874ab" -dependencies = [ - "getrandom 0.2.17", - "hmac", - "serde", - "serde_json", - "sha2", - "subc-protocol 0.10.0", - "subtle", - "tokio", -] - -[[package]] -name = "subc-transport" -version = "0.5.1" -dependencies = [ - "getrandom 0.2.17", - "hmac", - "serde", - "serde_json", - "sha2", - "subc-protocol 0.12.0", - "subtle", - "tokio", -] - [[package]] name = "subtle" version = "2.6.1" @@ -1796,16 +1633,6 @@ dependencies = [ "windows-sys", ] -[[package]] -name = "terminal_size" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874" -dependencies = [ - "rustix", - "windows-sys", -] - [[package]] name = "thiserror" version = "2.0.20" @@ -1826,15 +1653,6 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "thread_local" -version = "1.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070" -dependencies = [ - "cfg-if", -] - [[package]] name = "tiktoken-rs" version = "0.11.0" @@ -1941,21 +1759,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" dependencies = [ "pin-project-lite", - "tracing-attributes", "tracing-core", ] -[[package]] -name = "tracing-attributes" -version = "0.1.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.118", -] - [[package]] name = "tracing-core" version = "0.1.36" @@ -1963,36 +1769,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" dependencies = [ "once_cell", - "valuable", -] - -[[package]] -name = "tracing-log" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" -dependencies = [ - "log", - "once_cell", - "tracing-core", -] - -[[package]] -name = "tracing-subscriber" -version = "0.3.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" -dependencies = [ - "matchers", - "nu-ansi-term", - "once_cell", - "regex-automata", - "sharded-slab", - "smallvec", - "thread_local", - "tracing", - "tracing-core", - "tracing-log", ] [[package]] @@ -2028,12 +1804,6 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e" -[[package]] -name = "valuable" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" - [[package]] name = "vcpkg" version = "0.2.15" diff --git a/Cargo.toml b/Cargo.toml index 7e2e0f1e0..32ccd51ce 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,7 +1,3 @@ -# Magic Context Rust workspace — the harness-agnostic MC module (CK-in / CK-out) -# that runs under the subc daemon. Lives in-repo alongside the bun `packages/` -# (the session/tooling is cwd-bound, and the build needs this repo's cache-stability -# context); the dashboard's Tauri crate is its own standalone build, excluded here. [workspace] resolver = "2" members = ["crates/mc-core", "crates/mc-store", "crates/mc-host", "crates/mc-module", "crates/mc-tokenizer"] @@ -19,14 +15,6 @@ cortexkit-store = { path = "../commons/crates/cortexkit-store" } cortexkit-store-types = { path = "../commons/crates/cortexkit-store-types" } cortexkit-lease = { path = "../commons/crates/cortexkit-lease" } -# subc wire contract + module SDK. Path-deps to the same sibling source so -# subc-protocol/transport unify with subc-client-rs's transitive use (no [patch]). -subc-protocol = { path = "../subconscious/crates/subc-protocol" } -subc-control = { path = "../subconscious/crates/subc-control" } -subc-transport = { path = "../subconscious/crates/subc-transport" } -subc-client-rs = { path = "../subconscious/crates/subc-client-rs" } -subc-core = { path = "../subconscious/crates/subc-core" } - # Internal crates. mc-core = { path = "crates/mc-core" } mc-store = { path = "crates/mc-store" } diff --git a/assets/magic-context.schema.json b/assets/magic-context.schema.json index fa5dcdf65..bcbaf2070 100644 --- a/assets/magic-context.schema.json +++ b/assets/magic-context.schema.json @@ -35,7 +35,7 @@ }, "transform_mode": { "default": "ts", - "description": "Experimental: routes the entire Magic Context runtime for the project through the ck-mc Rust module over subc (requires user-level `subc` config); \"ts\" is the current TypeScript pipeline.", + "description": "Experimental: routes the project through the direct mc-host Rust runtime (requires user-level host connection config); \"ts\" is the current TypeScript pipeline.", "type": "string", "enum": [ "ts", diff --git a/bun.lock b/bun.lock index c3057096e..4e693c0d2 100644 --- a/bun.lock +++ b/bun.lock @@ -18,6 +18,7 @@ "@biomejs/biome": "^2.5.1", "@types/better-sqlite3": "^7.6.13", "@types/node": "^22.20.0", + "bun-types": "^1.3.11", "typescript": "^5.8.0", }, }, @@ -57,7 +58,6 @@ "name": "@cortexkit/opencode-magic-context-e2e", "version": "0.0.0", "dependencies": { - "@cortexkit/subc-client": "0.4.1", "@opencode-ai/sdk": "^1.15.13", }, "devDependencies": { @@ -369,8 +369,6 @@ "@cortexkit/retina-local-fs": ["@cortexkit/retina-local-fs@workspace:packages/retina-local-fs"], - "@cortexkit/subc-client": ["@cortexkit/subc-client@0.4.1", "", {}, "sha512-kHLx5L/iefnbR/fiETQsCGGCegoQ/rcr70J7r/RfPakci6G1lEcTcK5V67o6WGWs4IOizkCM2tPa4ew8cp1tzA=="], - "@cspotcode/source-map-support": ["@cspotcode/source-map-support@0.8.1", "", { "dependencies": { "@jridgewell/trace-mapping": "0.3.9" } }, "sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw=="], "@ctrl/tinycolor": ["@ctrl/tinycolor@4.2.0", "", {}, "sha512-kzyuwOAQnXJNLS9PSyrk0CWk35nWJW/zl/6KvnTBMFK65gm7U1/Z5BqjxeapjZCIhQcM/DsrEmcbRwDyXyXK4A=="], diff --git a/crates/mc-host/Cargo.toml b/crates/mc-host/Cargo.toml index 674f10672..58025d3bc 100644 --- a/crates/mc-host/Cargo.toml +++ b/crates/mc-host/Cargo.toml @@ -4,15 +4,12 @@ version = "0.1.0" edition = "2021" license = "MIT" publish = false +description = "Directly linked host for Magic Context, Synapse, and Broca components." [dependencies] -# The host implements the published wire authority; module crates continue to -# use the workspace's sibling dependencies for local SDK co-development. -subc-protocol = "=0.10.0" -subc-transport = "=0.5.0" -subc-control = "=0.1.1" - serde = { workspace = true } +hmac = "0.12" +subtle = "2" serde_json = { workspace = true } sha2 = { workspace = true } fastembed = { version = "=6.0.0", default-features = false, features = ["ort-load-dynamic"] } @@ -23,7 +20,6 @@ rustix = { version = "1.1.4", features = ["fs", "process", "thread"] } getrandom = "0.2" [dev-dependencies] -hmac = "0.12" tempfile = "3" tokio = { workspace = true, features = ["test-util", "signal"] } # `stdio` only serves the broca_subprocess fixture that replaces its own diff --git a/crates/mc-host/benches/ipc_budget.rs b/crates/mc-host/benches/ipc_budget.rs index 8451bd614..39afe51a2 100644 --- a/crates/mc-host/benches/ipc_budget.rs +++ b/crates/mc-host/benches/ipc_budget.rs @@ -1366,11 +1366,7 @@ fn aggregate(run_dir: &Path) -> Result { "exchanges_per_sec": output.exchanges_per_sec(), "clock_bracket_ns": output.clock_bracket_ns, "batches": batches, - "exchanges_per_batch": if batches == 0 { - 0 - } else { - output.total_exchanges / batches - }, + "exchanges_per_batch": output.total_exchanges.checked_div(batches).unwrap_or(0), }); let manifest_results = a .manifest diff --git a/crates/mc-host/src/auth.rs b/crates/mc-host/src/auth.rs new file mode 100644 index 000000000..6f46ae065 --- /dev/null +++ b/crates/mc-host/src/auth.rs @@ -0,0 +1,1007 @@ +//! Host-owned three-message authentication handshake. + +use std::{error::Error, fmt, future::Future, io, time::Duration}; + +use hmac::{Hmac, Mac}; +use serde::{de::DeserializeOwned, Deserialize, Serialize}; +use sha2::Sha256; +use subtle::ConstantTimeEq; +use tokio::{ + io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}, + time, +}; + +use crate::connection_file::{ConnectionInfo, DAEMON_ID_LEN, MIN_KEY_LEN}; + +pub const NONCE_LEN: usize = 32; +pub const PROOF_LEN: usize = 32; +pub const MAX_AUTH_MESSAGE_LEN: u32 = 4096; +pub const SERVER_PROOF_DOMAIN: &str = "subc-server-v1"; +pub const CLIENT_AUTH_DOMAIN: &str = "subc-client-v1"; +pub const DEFAULT_CLIENT_ROLE: &str = "client"; +pub const WATCHDOG_CLIENT_ROLE: &str = "watchdog"; + +type HmacSha256 = Hmac; + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ClientHello { + pub client_nonce: [u8; NONCE_LEN], + pub role: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ServerProof { + pub daemon_id: [u8; DAEMON_ID_LEN], + pub server_nonce: [u8; NONCE_LEN], + pub daemon_ver: String, + pub server_proof: [u8; PROOF_LEN], +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ClientAuth { + pub client_auth: [u8; PROOF_LEN], +} + +/// The outcome of a successful handshake. +/// +/// WHAT THIS PROVES: the peer possesses the connection key, and (client side) +/// that the daemon does too. Nothing more. +/// +/// WHAT `role` IS NOT: it is a string the CLIENT SENT, echoed back unverified. +/// The handshake never checks it against anything, so it carries no authority -- +/// any peer holding the key can claim any role. It exists so a caller can tell +/// self-issued traffic (the daemon's own watchdog probe) from real clients when +/// REPORTING, and it must never decide admission, capacity, or privilege. +/// +/// A type called `Authenticated` invites reading every field as attested. Only +/// the possession of the key is. Module identity, which IS attested, travels a +/// different path entirely (spawn nonces validated at route.open). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Authenticated { + pub role: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuthStage { + ClientHello, + ServerProof, + ClientAuth, +} + +#[derive(Debug)] +pub enum AuthError { + Io { + stage: AuthStage, + source: io::Error, + }, + Timeout { + stage: AuthStage, + deadline: Duration, + }, + UnexpectedEof { + stage: AuthStage, + expected: usize, + actual: usize, + }, + MessageTooLarge { + stage: AuthStage, + len: u32, + max: u32, + }, + JsonEncode { + stage: AuthStage, + source: serde_json::Error, + }, + JsonDecode { + stage: AuthStage, + source: serde_json::Error, + }, + Random(getrandom::Error), + KeyTooShort { + len: usize, + min: usize, + }, + InvalidServerProof, + DaemonIdMismatch, + InvalidClientAuth, +} + +pub fn compute_proof( + key: &[u8], + domain: &str, + client_nonce: &[u8; NONCE_LEN], + server_nonce: &[u8; NONCE_LEN], + daemon_id: &[u8], +) -> [u8; PROOF_LEN] { + let mut mac = HmacSha256::new_from_slice(key).expect("HMAC accepts keys of any length"); + mac.update(domain.as_bytes()); + mac.update(client_nonce); + mac.update(server_nonce); + mac.update(daemon_id); + mac.finalize().into_bytes().into() +} + +/// An absolute handshake deadline. Every per-stage read/write recomputes the time +/// remaining until `at`, so the WHOLE handshake (length byte + body, across all +/// stages, plus error teardown) is bounded by a single wall-clock budget. Passing +/// a bare `Duration` to each step instead would let a slow peer spend the full +/// budget on every length read AND every body read — multiplying the real bound. +#[derive(Clone, Copy)] +struct Deadline { + at: time::Instant, + total: Duration, +} + +impl Deadline { + fn starting_now(total: Duration) -> Self { + Self { + at: time::Instant::now() + total, + total, + } + } + + /// Time left until the deadline, or `Timeout` if it has already elapsed. + fn remaining(&self, stage: AuthStage) -> Result { + let remaining = self.at.saturating_duration_since(time::Instant::now()); + if remaining.is_zero() { + Err(AuthError::Timeout { + stage, + deadline: self.total, + }) + } else { + Ok(remaining) + } + } + + /// Time left until the deadline, clamped to zero — for best-effort teardown + /// that must not outlive the handshake budget. + fn remaining_or_zero(&self) -> Duration { + self.at.saturating_duration_since(time::Instant::now()) + } +} + +pub async fn authenticate_server( + stream: &mut S, + key: &[u8], + daemon_id: &[u8; DAEMON_ID_LEN], + daemon_ver: &str, + deadline: Duration, +) -> Result +where + S: AsyncRead + AsyncWrite + Unpin, +{ + let deadline = Deadline::starting_now(deadline); + let result = authenticate_server_inner(stream, key, daemon_id, daemon_ver, deadline).await; + if result.is_err() { + // Bound teardown by the SAME absolute deadline so a failed handshake (and + // the unauthenticated-handshake slot it holds) is released promptly instead + // of waiting out another full budget. + let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; + } + result +} + +async fn authenticate_server_inner( + stream: &mut S, + key: &[u8], + daemon_id: &[u8; DAEMON_ID_LEN], + daemon_ver: &str, + deadline: Deadline, +) -> Result +where + S: AsyncRead + AsyncWrite + Unpin, +{ + validate_key(key)?; + + let hello: ClientHello = read_message(stream, AuthStage::ClientHello, deadline).await?; + let server_nonce = random_nonce()?; + let server_proof = compute_proof( + key, + SERVER_PROOF_DOMAIN, + &hello.client_nonce, + &server_nonce, + daemon_id, + ); + + write_message( + stream, + AuthStage::ServerProof, + &ServerProof { + daemon_id: *daemon_id, + server_nonce, + daemon_ver: daemon_ver.to_owned(), + server_proof, + }, + deadline, + ) + .await?; + + let client_auth: ClientAuth = read_message(stream, AuthStage::ClientAuth, deadline).await?; + let expected_client_auth = compute_proof( + key, + CLIENT_AUTH_DOMAIN, + &hello.client_nonce, + &server_nonce, + daemon_id, + ); + if !constant_time_eq(&expected_client_auth, &client_auth.client_auth) { + return Err(AuthError::InvalidClientAuth); + } + + Ok(Authenticated { role: hello.role }) +} + +pub async fn authenticate_client( + stream: &mut S, + conn: &ConnectionInfo, + deadline: Duration, +) -> Result<(), AuthError> +where + S: AsyncRead + AsyncWrite + Unpin, +{ + authenticate_client_with_role(stream, conn, deadline, DEFAULT_CLIENT_ROLE).await +} + +pub async fn authenticate_client_with_role( + stream: &mut S, + conn: &ConnectionInfo, + deadline: Duration, + role: &str, +) -> Result<(), AuthError> +where + S: AsyncRead + AsyncWrite + Unpin, +{ + let deadline = Deadline::starting_now(deadline); + let result = authenticate_client_inner(stream, conn, deadline, role).await; + if result.is_err() { + let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; + } + result +} + +async fn authenticate_client_inner( + stream: &mut S, + conn: &ConnectionInfo, + deadline: Deadline, + role: &str, +) -> Result<(), AuthError> +where + S: AsyncRead + AsyncWrite + Unpin, +{ + validate_key(&conn.key)?; + + let client_nonce = random_nonce()?; + write_message( + stream, + AuthStage::ClientHello, + &ClientHello { + client_nonce, + role: role.to_owned(), + }, + deadline, + ) + .await?; + + let server_proof: ServerProof = read_message(stream, AuthStage::ServerProof, deadline).await?; + let expected_server_proof = compute_proof( + &conn.key, + SERVER_PROOF_DOMAIN, + &client_nonce, + &server_proof.server_nonce, + &server_proof.daemon_id, + ); + if !constant_time_eq(&expected_server_proof, &server_proof.server_proof) { + return Err(AuthError::InvalidServerProof); + } + if server_proof.daemon_id != conn.daemon_id { + return Err(AuthError::DaemonIdMismatch); + } + + let client_auth = compute_proof( + &conn.key, + CLIENT_AUTH_DOMAIN, + &client_nonce, + &server_proof.server_nonce, + &server_proof.daemon_id, + ); + write_message( + stream, + AuthStage::ClientAuth, + &ClientAuth { client_auth }, + deadline, + ) + .await +} + +fn validate_key(key: &[u8]) -> Result<(), AuthError> { + if key.len() < MIN_KEY_LEN { + return Err(AuthError::KeyTooShort { + len: key.len(), + min: MIN_KEY_LEN, + }); + } + Ok(()) +} + +fn random_nonce() -> Result<[u8; NONCE_LEN], AuthError> { + let mut nonce = [0u8; NONCE_LEN]; + getrandom::getrandom(&mut nonce).map_err(AuthError::Random)?; + Ok(nonce) +} + +/// Both directions of this comparison are fenced, verified by mutation rather than +/// assumed, because a proof check has the failure mode where a suite proves only +/// that it can say NO. +/// +/// ALWAYS-FALSE (no proof ever verifies, every connection in the fleet refused) is +/// caught by the handshake integration tests and by two bootstrap tests -- named +/// for key rotation and singleton probing, so this is coverage carried by tests +/// about something else. Narrowing either would remove it silently. +/// +/// ALWAYS-TRUE is caught by `foreign_server_reused_port_never_receives_client_auth` +/// -- the case where a client must refuse a server that cannot produce the proof. +/// Named for the refusal, and it holds that direction directly. +/// +/// The construction feeding it is pinned separately by +/// `committed_wire_vectors_pin_the_proof_construction`, which reddens for a constant +/// proof AND for one that folds only part of its input -- the second matters because +/// a partial-input proof still produces different outputs for different inputs, so +/// any distinctness assertion passes it while a proof minted against one daemon +/// verifies against another. +fn constant_time_eq(expected: &[u8; PROOF_LEN], actual: &[u8; PROOF_LEN]) -> bool { + expected.as_slice().ct_eq(actual.as_slice()).into() +} + +async fn read_message( + stream: &mut S, + stage: AuthStage, + deadline: Deadline, +) -> Result +where + S: AsyncRead + Unpin, + T: DeserializeOwned, +{ + // Both the length read and the body read recompute the time remaining against + // the same absolute deadline, so the two together cannot exceed the budget. + let mut len_bytes = [0u8; 4]; + read_exact_deadline(stream, &mut len_bytes, stage, deadline).await?; + let len = u32::from_le_bytes(len_bytes); + if len > MAX_AUTH_MESSAGE_LEN { + return Err(AuthError::MessageTooLarge { + stage, + len, + max: MAX_AUTH_MESSAGE_LEN, + }); + } + + let mut json = vec![0u8; len as usize]; + if !json.is_empty() { + read_exact_deadline(stream, &mut json, stage, deadline).await?; + } + serde_json::from_slice(&json).map_err(|source| AuthError::JsonDecode { stage, source }) +} + +async fn write_message( + stream: &mut S, + stage: AuthStage, + value: &T, + deadline: Deadline, +) -> Result<(), AuthError> +where + S: AsyncWrite + Unpin, + T: Serialize, +{ + let json = + serde_json::to_vec(value).map_err(|source| AuthError::JsonEncode { stage, source })?; + let len = u32::try_from(json.len()).map_err(|_| AuthError::MessageTooLarge { + stage, + len: u32::MAX, + max: MAX_AUTH_MESSAGE_LEN, + })?; + if len > MAX_AUTH_MESSAGE_LEN { + return Err(AuthError::MessageTooLarge { + stage, + len, + max: MAX_AUTH_MESSAGE_LEN, + }); + } + + write_all_deadline(stream, &len.to_le_bytes(), stage, deadline).await?; + write_all_deadline(stream, &json, stage, deadline).await +} + +async fn read_exact_deadline( + stream: &mut S, + buf: &mut [u8], + stage: AuthStage, + deadline: Deadline, +) -> Result<(), AuthError> +where + S: AsyncRead + Unpin, +{ + let remaining = deadline.remaining(stage)?; + let expected = buf.len(); + with_timeout(stage, remaining, async { + let mut actual = 0; + while actual < expected { + let read = stream.read(&mut buf[actual..]).await?; + if read == 0 { + return Err(ReadExactError::UnexpectedEof { actual }); + } + actual += read; + } + Ok(()) + }) + .await + .map_err(|err| match err { + DeadlineIoError::Io(source) => AuthError::Io { stage, source }, + DeadlineIoError::Timeout => AuthError::Timeout { + stage, + deadline: deadline.total, + }, + DeadlineIoError::UnexpectedEof { actual } => AuthError::UnexpectedEof { + stage, + expected, + actual, + }, + }) +} + +async fn write_all_deadline( + stream: &mut S, + buf: &[u8], + stage: AuthStage, + deadline: Deadline, +) -> Result<(), AuthError> +where + S: AsyncWrite + Unpin, +{ + let remaining = deadline.remaining(stage)?; + timeout_io(stage, remaining, deadline.total, stream.write_all(buf)).await +} + +async fn timeout_io( + stage: AuthStage, + remaining: Duration, + total: Duration, + future: F, +) -> Result +where + F: Future>, +{ + match time::timeout(remaining, future).await { + Ok(Ok(value)) => Ok(value), + Ok(Err(source)) => Err(AuthError::Io { stage, source }), + Err(_) => Err(AuthError::Timeout { + stage, + deadline: total, + }), + } +} + +async fn with_timeout( + _stage: AuthStage, + deadline: Duration, + future: F, +) -> Result<(), DeadlineIoError> +where + F: Future>, +{ + match time::timeout(deadline, future).await { + Ok(Ok(())) => Ok(()), + Ok(Err(ReadExactError::Io(source))) => Err(DeadlineIoError::Io(source)), + Ok(Err(ReadExactError::UnexpectedEof { actual })) => { + Err(DeadlineIoError::UnexpectedEof { actual }) + } + Err(_) => Err(DeadlineIoError::Timeout), + } +} + +#[derive(Debug)] +enum ReadExactError { + Io(io::Error), + UnexpectedEof { actual: usize }, +} + +impl From for ReadExactError { + fn from(source: io::Error) -> Self { + Self::Io(source) + } +} + +#[derive(Debug)] +enum DeadlineIoError { + Io(io::Error), + Timeout, + UnexpectedEof { actual: usize }, +} + +impl fmt::Display for AuthError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Io { stage, source } => write!(f, "auth {stage:?} I/O error: {source}"), + Self::Timeout { stage, deadline } => { + write!(f, "auth {stage:?} timed out after {deadline:?}") + } + Self::UnexpectedEof { + stage, + expected, + actual, + } => write!( + f, + "auth {stage:?} ended early: expected {expected} bytes, got {actual}" + ), + Self::MessageTooLarge { stage, len, max } => write!( + f, + "auth {stage:?} message length {len} exceeds hard cap {max}" + ), + Self::JsonEncode { stage, source } => { + write!(f, "auth {stage:?} JSON encode error: {source}") + } + Self::JsonDecode { stage, source } => { + write!(f, "auth {stage:?} JSON decode error: {source}") + } + Self::Random(source) => write!(f, "auth random generation failed: {source}"), + Self::KeyTooShort { len, min } => { + write!(f, "auth key is too short: {len} bytes, need at least {min}") + } + Self::InvalidServerProof => write!(f, "invalid server auth proof"), + Self::DaemonIdMismatch => write!(f, "server daemon_id did not match connection file"), + Self::InvalidClientAuth => write!(f, "invalid client auth proof"), + } + } +} + +impl Error for AuthError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Io { source, .. } => Some(source), + Self::JsonEncode { source, .. } | Self::JsonDecode { source, .. } => Some(source), + Self::Random(_) => None, + Self::Timeout { .. } + | Self::UnexpectedEof { .. } + | Self::MessageTooLarge { .. } + | Self::KeyTooShort { .. } + | Self::InvalidServerProof + | Self::DaemonIdMismatch + | Self::InvalidClientAuth => None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use tokio::{ + io::{duplex, DuplexStream}, + task::yield_now, + time::advance, + }; + + const TEST_DAEMON_VER: &str = "mc-host-auth-test-1"; + const TEST_ROLE: &str = "client"; + + /// The TypeScript client asserts its handshake against the same fixed + /// vectors (`packages/plugin/src/shared/mc-host-client/auth.test.ts`), so + /// they form a cross-language contract: changing the domain separator, + /// the field order, or the MAC breaks the build here, where the change is + /// being made, instead of surfacing as a handshake failure against a peer + /// that has not been rebuilt. + #[test] + fn committed_wire_vectors_pin_the_proof_construction() { + fn unhex(hex: &str) -> Vec { + assert!(hex.len().is_multiple_of(2), "odd-length hex"); + (0..hex.len()) + .step_by(2) + .map(|i| u8::from_str_radix(&hex[i..i + 2], 16).expect("hex byte")) + .collect() + } + + let key = unhex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f"); + let client_nonce: [u8; NONCE_LEN] = unhex(&"ab".repeat(NONCE_LEN)) + .try_into() + .expect("client nonce length"); + let server_nonce: [u8; NONCE_LEN] = unhex(&"cd".repeat(NONCE_LEN)) + .try_into() + .expect("server nonce length"); + let daemon_id = unhex("000102030405060708090a0b0c0d0e0f"); + + for (domain, expected) in [ + ( + SERVER_PROOF_DOMAIN, + "ea06076a980bc7558e45017df86de89f3d2fc09861f8460795dea31eadf40527", + ), + ( + CLIENT_AUTH_DOMAIN, + "a3bc64784dbd94c4f52799e0c66f7b2b8183aa4655594630245c5d4a2fa387a9", + ), + ] { + let proof = compute_proof(&key, domain, &client_nonce, &server_nonce, &daemon_id); + let actual: String = proof.iter().map(|byte| format!("{byte:02x}")).collect(); + assert_eq!( + actual, expected, + "proof construction changed for domain {domain}: the committed \ + cross-language wire vectors no longer describe this implementation" + ); + } + } + + async fn write_auth_json(stream: &mut DuplexStream, value: &T) + where + T: Serialize, + { + let body = serde_json::to_vec(value).expect("encode auth json"); + assert!( + body.len() <= MAX_AUTH_MESSAGE_LEN as usize, + "test helper auth message over cap" + ); + stream + .write_all(&(body.len() as u32).to_le_bytes()) + .await + .expect("write auth length"); + stream.write_all(&body).await.expect("write auth body"); + } + + async fn read_auth_json(stream: &mut DuplexStream) -> T + where + T: DeserializeOwned, + { + let mut len_bytes = [0u8; 4]; + stream + .read_exact(&mut len_bytes) + .await + .expect("read auth length"); + let len = u32::from_le_bytes(len_bytes); + assert!( + len <= MAX_AUTH_MESSAGE_LEN, + "test helper received auth message over cap" + ); + let mut body = vec![0u8; len as usize]; + stream.read_exact(&mut body).await.expect("read auth body"); + serde_json::from_slice(&body).expect("decode auth json") + } + + /// Write only the 4-byte length prefix of an auth message, withholding + /// the body — stalls the peer mid-message so the within-stage deadline + /// can be exercised. + async fn write_auth_len_only(stream: &mut DuplexStream, value: &T) + where + T: Serialize, + { + let body = serde_json::to_vec(value).expect("encode auth json"); + stream + .write_all(&(body.len() as u32).to_le_bytes()) + .await + .expect("write auth length"); + } + + #[tokio::test(start_paused = true)] + async fn authenticate_server_deadline_is_absolute_across_handshake() { + let key = vec![0x5a; MIN_KEY_LEN]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let deadline = Duration::from_millis(100); + let stage_delay = Duration::from_millis(60); + let (mut client, mut server) = duplex(4096); + + let server_task = tokio::spawn(async move { + authenticate_server(&mut server, &key, &daemon_id, TEST_DAEMON_VER, deadline).await + }); + + yield_now().await; + assert!(!server_task.is_finished()); + + advance(stage_delay).await; + write_auth_json( + &mut client, + &ClientHello { + client_nonce: [0x11; NONCE_LEN], + role: TEST_ROLE.to_owned(), + }, + ) + .await; + yield_now().await; + + let server_proof: ServerProof = read_auth_json(&mut client).await; + assert_eq!(server_proof.daemon_id, daemon_id); + assert_eq!(server_proof.daemon_ver, TEST_DAEMON_VER); + assert!(!server_task.is_finished()); + + advance(stage_delay).await; + yield_now().await; + assert!(server_task.is_finished()); + + let err = server_task + .await + .expect("server task should join") + .expect_err("server handshake should time out once the total deadline elapses"); + assert!(matches!( + err, + AuthError::Timeout { + stage: AuthStage::ClientAuth, + .. + } + )); + } + + #[tokio::test(start_paused = true)] + async fn server_deadline_spans_length_and_body_within_one_stage() { + // The bug this guards: applying the timeout independently to the + // length read and the body read lets a single stage consume ~2x the + // budget. Here the client sends the ClientHello length prefix late, + // then withholds the body until the absolute deadline has passed. + let key = vec![0x5a; MIN_KEY_LEN]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let deadline = Duration::from_millis(100); + let (mut client, mut server) = duplex(4096); + + let server_task = tokio::spawn(async move { + authenticate_server(&mut server, &key, &daemon_id, TEST_DAEMON_VER, deadline).await + }); + + yield_now().await; + advance(Duration::from_millis(60)).await; + write_auth_len_only( + &mut client, + &ClientHello { + client_nonce: [0x11; NONCE_LEN], + role: TEST_ROLE.to_owned(), + }, + ) + .await; + yield_now().await; + assert!(!server_task.is_finished()); + + // Cross the absolute deadline (60 + 50 > 100) without sending the body. + advance(Duration::from_millis(50)).await; + yield_now().await; + assert!( + server_task.is_finished(), + "body read must share the handshake deadline, not get a fresh window" + ); + let err = server_task + .await + .expect("join") + .expect_err("must time out at ClientHello body"); + assert!(matches!( + err, + AuthError::Timeout { + stage: AuthStage::ClientHello, + .. + } + )); + } + + /// Drives one full handshake against `authenticate_server` and returns + /// the server's `ServerProof` message. + async fn complete_handshake(key: &[u8], daemon_id: [u8; DAEMON_ID_LEN]) -> ServerProof { + let (mut client, mut server) = duplex(4096); + let key_owned = key.to_vec(); + let server_task = tokio::spawn(async move { + authenticate_server( + &mut server, + &key_owned, + &daemon_id, + TEST_DAEMON_VER, + Duration::from_secs(5), + ) + .await + }); + + let client_nonce = [0x11u8; NONCE_LEN]; + write_auth_json( + &mut client, + &ClientHello { + client_nonce, + role: TEST_ROLE.to_owned(), + }, + ) + .await; + let server_proof: ServerProof = read_auth_json(&mut client).await; + let client_auth = compute_proof( + key, + CLIENT_AUTH_DOMAIN, + &client_nonce, + &server_proof.server_nonce, + &server_proof.daemon_id, + ); + write_auth_json(&mut client, &ClientAuth { client_auth }).await; + let authenticated = server_task + .await + .expect("join") + .expect("handshake completes"); + assert_eq!(authenticated.role, TEST_ROLE); + server_proof + } + + #[tokio::test] + async fn repeated_handshakes_receive_fresh_server_nonces() { + let key = vec![0x5a; MIN_KEY_LEN]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let first = complete_handshake(&key, daemon_id).await; + let second = complete_handshake(&key, daemon_id).await; + assert_ne!( + first.server_nonce, second.server_nonce, + "server nonces must be fresh per handshake, never replayed" + ); + assert_ne!( + first.server_proof, second.server_proof, + "a fresh nonce must produce a fresh proof" + ); + } + + #[tokio::test] + async fn wrong_client_proof_is_rejected_and_error_carries_no_secrets() { + let key = vec![0x5a; MIN_KEY_LEN]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let (mut client, mut server) = duplex(4096); + let key_task = key.clone(); + let server_task = tokio::spawn(async move { + authenticate_server( + &mut server, + &key_task, + &daemon_id, + TEST_DAEMON_VER, + Duration::from_secs(5), + ) + .await + }); + + write_auth_json( + &mut client, + &ClientHello { + client_nonce: [0x11; NONCE_LEN], + role: TEST_ROLE.to_owned(), + }, + ) + .await; + let _proof: ServerProof = read_auth_json(&mut client).await; + write_auth_json( + &mut client, + &ClientAuth { + client_auth: [0u8; PROOF_LEN], + }, + ) + .await; + + let err = server_task + .await + .expect("join") + .expect_err("wrong proof must be rejected"); + assert!(matches!(err, AuthError::InvalidClientAuth)); + let key_decimals = format!("{:?}", key); + for rendered in [format!("{err}"), format!("{err:?}")] { + assert!( + !rendered.contains(&key_decimals), + "auth errors must not leak key bytes: {rendered}" + ); + } + } + + #[tokio::test] + async fn over_cap_auth_message_is_rejected_before_allocation() { + let key = vec![0x5a; MIN_KEY_LEN]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let (mut client, mut server) = duplex(4096); + let server_task = tokio::spawn(async move { + authenticate_server( + &mut server, + &key, + &daemon_id, + TEST_DAEMON_VER, + Duration::from_secs(5), + ) + .await + }); + + client + .write_all(&(MAX_AUTH_MESSAGE_LEN + 1).to_le_bytes()) + .await + .expect("write oversize length"); + let err = server_task + .await + .expect("join") + .expect_err("over-cap message must be rejected"); + assert!(matches!( + err, + AuthError::MessageTooLarge { + stage: AuthStage::ClientHello, + len, + max: MAX_AUTH_MESSAGE_LEN, + } if len == MAX_AUTH_MESSAGE_LEN + 1 + )); + } + + async fn rejected_server_sends_no_client_auth( + server_daemon_id: [u8; DAEMON_ID_LEN], + valid_proof: bool, + expected: fn(&AuthError) -> bool, + ) { + let key = vec![0x5a; MIN_KEY_LEN]; + let expected_daemon_id = [0x6b; DAEMON_ID_LEN]; + let conn = ConnectionInfo { + schema: crate::connection_file::SCHEMA_VERSION, + wire_version: crate::wire::PROTOCOL_VERSION, + endpoints: vec![crate::connection_file::Endpoint { + host: "127.0.0.1".to_owned(), + port: 1, + }], + key: key.clone(), + daemon_id: expected_daemon_id, + pid: 1, + daemon_ver: TEST_DAEMON_VER.to_owned(), + }; + let (mut server, mut client) = duplex(4096); + let task = tokio::spawn(async move { + authenticate_client(&mut client, &conn, Duration::from_secs(5)).await + }); + let hello: ClientHello = read_auth_json(&mut server).await; + let server_nonce = [0x22; NONCE_LEN]; + let server_proof = if valid_proof { + compute_proof( + &key, + SERVER_PROOF_DOMAIN, + &hello.client_nonce, + &server_nonce, + &server_daemon_id, + ) + } else { + [0; PROOF_LEN] + }; + write_auth_json( + &mut server, + &ServerProof { + daemon_id: server_daemon_id, + server_nonce, + daemon_ver: TEST_DAEMON_VER.to_owned(), + server_proof, + }, + ) + .await; + let err = task.await.expect("join").expect_err("server rejected"); + assert!(expected(&err), "unexpected error: {err}"); + let mut byte = [0u8; 1]; + assert_eq!( + server.read(&mut byte).await.expect("read after rejection"), + 0 + ); + } + + #[tokio::test] + async fn invalid_server_proof_sends_no_client_auth() { + rejected_server_sends_no_client_auth([0x6b; DAEMON_ID_LEN], false, |err| { + matches!(err, AuthError::InvalidServerProof) + }) + .await; + } + + #[tokio::test] + async fn daemon_id_mismatch_sends_no_client_auth() { + rejected_server_sends_no_client_auth([0x7c; DAEMON_ID_LEN], true, |err| { + matches!(err, AuthError::DaemonIdMismatch) + }) + .await; + } + + #[tokio::test] + async fn short_key_is_rejected_before_any_read() { + let key = vec![0x5a; MIN_KEY_LEN - 1]; + let daemon_id = [0x6b; DAEMON_ID_LEN]; + let (_client, mut server) = duplex(64); + let err = authenticate_server( + &mut server, + &key, + &daemon_id, + TEST_DAEMON_VER, + Duration::from_secs(1), + ) + .await + .expect_err("short key must be rejected"); + assert!(matches!( + err, + AuthError::KeyTooShort { + len, + min: MIN_KEY_LEN + } if len == MIN_KEY_LEN - 1 + )); + } +} diff --git a/crates/mc-host/src/broca/subprocess.rs b/crates/mc-host/src/broca/subprocess.rs index 274f4eb28..1ec20707a 100644 --- a/crates/mc-host/src/broca/subprocess.rs +++ b/crates/mc-host/src/broca/subprocess.rs @@ -30,8 +30,8 @@ use super::backend::{ /// (R17): a harness child inheriting these could reconnect to the daemon as /// the supervised module itself. pub const HOST_LAUNCH_IDENTITY_VARS: [&str; 2] = [ - subc_protocol::SUBC_MODULE_ID_ENV, - subc_protocol::SUBC_LAUNCH_NONCE_ENV, + crate::wire::SUBC_MODULE_ID_ENV, + crate::wire::SUBC_LAUNCH_NONCE_ENV, ]; /// Immutable copy of the daemon-startup environment (R17): provider diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs new file mode 100644 index 000000000..1a7135296 --- /dev/null +++ b/crates/mc-host/src/client.rs @@ -0,0 +1,2380 @@ +//! Managed Rust consumer for one authenticated mc-host generation. +//! +//! This module owns discovery, authentication, mandatory negotiation, +//! correlation allocation, framing, liveness, route epochs, bounded queues, +//! cancellation, and cleanup. Raw frame types never cross the public API. + +use std::{ + collections::{HashMap, HashSet}, + error::Error, + fmt, + path::Path, + sync::{ + atomic::{AtomicBool, AtomicU8, Ordering}, + Arc, Mutex, MutexGuard, Weak, + }, + time::Duration, +}; + +use serde_json::Value; +use tokio::{ + io::{AsyncRead, AsyncReadExt, AsyncWriteExt}, + net::{tcp::OwnedReadHalf, tcp::OwnedWriteHalf, TcpStream}, + sync::{mpsc, oneshot}, + task::JoinHandle, + time::{timeout_at, Instant}, +}; +use tokio_util::sync::CancellationToken; + +use crate::{ + auth::authenticate_client, + connection_file::{read_for_client, ConnectionInfo, DAEMON_ID_LEN}, + handler::{RouteHandle, RouteIdentity, RouteTarget, TargetKind}, + transport_negotiation::{ + decode_negotiate_response, NegotiateResponse, TransportOffer, NEGOTIATION_VERSION, + TRANSPORT_TCP, + }, + wire::{ + decode_header, encode_owned_frame, pure_header_flags, EnvelopeHeader, Flags, FrameId, + FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, PROTOCOL_VERSION, + }, +}; + +/// Total deadline for dial, authentication, and mandatory negotiation. +pub const CLIENT_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(2); +/// Deadline for a frame after its first header byte. Idle header waits are unbounded. +pub const CLIENT_FRAME_TIMEOUT: Duration = Duration::from_secs(30); +/// Absolute deadline for one route-open operation, including retries. +pub const CLIENT_ROUTE_OPEN_TIMEOUT: Duration = Duration::from_secs(30); +/// Default absolute deadline for one request. +pub const CLIENT_REQUEST_TIMEOUT: Duration = Duration::from_secs(30); +/// Absolute deadline for owner shutdown. +pub const CLIENT_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(5); +/// Owner-wide pending request cap. +pub const CLIENT_MAX_PENDING_REQUESTS: usize = 1_024; +/// Owner-wide live stream cap. +pub const CLIENT_MAX_LIVE_STREAMS: usize = 64; +/// Per-stream item queue. Saturation cancels only that stream. +pub const CLIENT_STREAM_QUEUE_ITEMS: usize = 16; +/// Ordinary writer slots. Reserved controls do not consume these slots. +pub const CLIENT_DATA_QUEUE_FRAMES: usize = 256; +/// Reserved pure-header Pong, Cancel, and Goodbye slots. +pub const CLIENT_CONTROL_QUEUE_FRAMES: usize = 32; +/// Shared queued-byte cap charged by both ordinary and reserved control frames. +pub const CLIENT_QUEUED_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; +/// Owner-wide bytes retained in pending stream queues. +pub const CLIENT_RETAINED_RESPONSE_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; + +const NEGOTIATION_CORRELATION: u64 = 1; +const FIRST_APPLICATION_CORRELATION: u64 = 2; +const MAX_ERROR_CODE_BYTES: usize = 128; +const MAX_ERROR_MESSAGE_BYTES: usize = 512; + +/// Exact send-outcome classifications used by recovery policy. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SendOutcome { + /// Request bytes provably never reached the writer. + NotSent, + /// Some request bytes may have reached the peer without a terminal. + OutcomeUnknown, + /// Matching host terminal was observed. + Terminal, +} + +impl SendOutcome { + /// Stable spelling used by cross-language recovery policy. + pub const fn as_str(self) -> &'static str { + match self { + Self::NotSent => "not_sent", + Self::OutcomeUnknown => "outcome_unknown", + Self::Terminal => "terminal", + } + } +} + +impl fmt::Display for SendOutcome { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.as_str()) + } +} + +/// Managed call failure. Formatting never includes payload or identity data. +#[derive(Clone, PartialEq, Eq)] +pub struct CallError { + outcome: SendOutcome, + code: String, + message: String, +} + +impl CallError { + fn new(outcome: SendOutcome, code: impl Into, message: impl Into) -> Self { + Self { + outcome, + code: bounded_code(&code.into()), + message: bounded_text(&message.into(), MAX_ERROR_MESSAGE_BYTES), + } + } + + fn local(outcome: SendOutcome, code: &'static str, message: &'static str) -> Self { + Self::new(outcome, code, message) + } + + fn host_terminal(body: &[u8]) -> Self { + let code = serde_json::from_slice::(body) + .ok() + .and_then(|value| value.get("code")?.as_str().map(str::to_owned)) + .map(|code| bounded_code(&code)) + .unwrap_or_else(|| "remote_error".to_owned()); + // Peer text may echo a request, credential, or identity. Preserve the + // stable code but never retain the raw terminal message. + Self::new( + SendOutcome::Terminal, + code, + "host returned a terminal error (message redacted)", + ) + } + + /// Send classification. + pub const fn outcome(&self) -> SendOutcome { + self.outcome + } + + /// Stable bounded error code. + pub fn code(&self) -> &str { + &self.code + } + + /// Bounded redacted message. + pub fn message(&self) -> &str { + &self.message + } +} + +impl fmt::Debug for CallError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("CallError") + .field("outcome", &self.outcome) + .field("code", &self.code) + .field("message", &self.message) + .finish() + } +} + +impl fmt::Display for CallError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}: {} ({})", self.outcome, self.message, self.code) + } +} + +impl Error for CallError {} + +/// Discovery, authentication, negotiation, or owner-lifecycle failure. +#[derive(Clone, PartialEq, Eq)] +pub struct ClientError { + code: &'static str, + message: &'static str, +} + +impl ClientError { + fn new(code: &'static str, message: &'static str) -> Self { + Self { code, message } + } + + /// Stable failure code. + pub const fn code(&self) -> &'static str { + self.code + } +} + +impl fmt::Debug for ClientError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ClientError") + .field("code", &self.code) + .field("message", &self.message) + .finish() + } +} + +impl fmt::Display for ClientError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} ({})", self.message, self.code) + } +} + +impl Error for ClientError {} + +/// One successful unary response. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Response { + /// Response bytes. The client does not interpret application payloads. + pub body: Vec, + /// Whether the host marked the body as binary. + pub binary: bool, +} + +/// One ordered streaming response item. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StreamItem { + /// Item bytes. + pub body: Vec, + /// Whether the host marked the item as binary. + pub binary: bool, +} + +/// Per-request deadline and cancellation controls. +#[derive(Debug, Clone)] +pub struct RequestOptions { + /// Total operation budget. Queueing, publication, and terminal wait share it. + pub timeout: Duration, + /// Optional caller cancellation token. + pub cancellation: Option, +} + +impl Default for RequestOptions { + fn default() -> Self { + Self { + timeout: CLIENT_REQUEST_TIMEOUT, + cancellation: None, + } + } +} + +/// Managed connection to one authenticated daemon generation. +pub struct Client { + inner: Arc, +} + +impl fmt::Debug for Client { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Client") + .field("closed", &self.inner.closed.load(Ordering::Acquire)) + .finish_non_exhaustive() + } +} + +impl Client { + /// Securely discovers, authenticates, and negotiates one TCP generation. + /// + /// Discovery validates one descriptor-anchored snapshot before any dial. + pub async fn connect(path: impl AsRef) -> Result { + let info = read_for_client(path) + .map_err(|_| ClientError::new("discovery_failed", "secure discovery failed"))?; + Self::connect_info(info).await + } + + async fn connect_info(info: ConnectionInfo) -> Result { + let endpoint = info + .endpoints + .first() + .ok_or_else(|| ClientError::new("discovery_failed", "secure discovery failed"))? + .clone(); + let deadline = Instant::now() + CLIENT_HANDSHAKE_TIMEOUT; + let mut stream = timeout_at( + deadline, + TcpStream::connect((endpoint.host.as_str(), endpoint.port)), + ) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("dial_failed", "daemon dial failed"))?; + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + return Err(ClientError::new( + "handshake_timeout", + "client handshake timed out", + )); + } + timeout_at(deadline, authenticate_client(&mut stream, &info, remaining)) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| { + ClientError::new("authentication_failed", "daemon authentication failed") + })?; + negotiate_tcp(&mut stream, deadline).await?; + + let (read, write) = stream.into_split(); + let (data_tx, data_rx) = mpsc::channel(CLIENT_DATA_QUEUE_FRAMES); + let (control_tx, control_rx) = mpsc::channel(CLIENT_CONTROL_QUEUE_FRAMES); + let inner = Arc::new(Inner { + daemon_id: info.daemon_id, + closed: AtomicBool::new(false), + retired: AtomicBool::new(false), + cancel: CancellationToken::new(), + correlations: Mutex::new(Correlations::new(FIRST_APPLICATION_CORRELATION)), + admission: Mutex::new(()), + pending: Mutex::new(HashMap::new()), + streams: Mutex::new(0), + routes: Mutex::new(HashSet::new()), + queue_budget: Arc::new(ByteCounter::new(CLIENT_QUEUED_BYTES)), + retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), + data_tx, + control_tx, + close_lock: tokio::sync::Mutex::new(()), + reader: tokio::sync::Mutex::new(None), + writer: tokio::sync::Mutex::new(None), + }); + let writer_inner = Arc::clone(&inner); + let writer = tokio::spawn(async move { + writer_loop(writer_inner, write, data_rx, control_rx).await; + }); + let reader_inner = Arc::clone(&inner); + let reader = tokio::spawn(async move { + reader_loop(reader_inner, read).await; + }); + *inner.writer.lock().await = Some(writer); + *inner.reader.lock().await = Some(reader); + Ok(Self { inner }) + } + + /// Authenticated daemon ID from the secure discovery and proof transcript. + pub fn daemon_id(&self) -> [u8; DAEMON_ID_LEN] { + self.inner.daemon_id + } + + /// Opens a full `(channel, epoch)` route under one absolute 30-second deadline. + pub async fn open_route( + &self, + target: RouteTarget, + identity: RouteIdentity, + ) -> Result { + if self.inner.closed.load(Ordering::Acquire) { + return Err(CallError::local( + SendOutcome::NotSent, + "client_closed", + "client is closed", + )); + } + let body = route_open_body(&target, &identity)?; + let deadline = Instant::now() + CLIENT_ROUTE_OPEN_TIMEOUT; + let mut backoff = Duration::from_millis(25); + loop { + let response = self + .inner + .unary( + RouteHandle { + channel: 0, + epoch: 0, + }, + body.clone(), + deadline, + None, + ) + .await; + match response { + Ok(response) => { + let handle = parse_route_open(&response.body)?; + lock_unpoisoned(&self.inner.routes).insert(handle); + return Ok(handle); + } + Err(error) + if error.outcome == SendOutcome::Terminal + && matches!( + error.code.as_str(), + "unknown_module" + | "module_reloading" + | "target_unavailable" + | "module_timeout" + ) + && Instant::now() < deadline => + { + let remaining = deadline.saturating_duration_since(Instant::now()); + tokio::time::sleep(backoff.min(remaining)).await; + backoff = (backoff * 2).min(Duration::from_millis(500)); + } + Err(error) => return Err(error), + } + } + } + + /// Sends one unary request. The body is never replayed. + pub async fn request( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result { + self.require_route(route)?; + let deadline = Instant::now() + options.timeout; + self.inner + .unary(route, body, deadline, options.cancellation) + .await + } + + /// Starts one bounded streaming request. + pub async fn request_stream( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result { + self.require_route(route)?; + self.inner.start_stream(route, body, options) + } + + /// Cancels one correlation on exactly the supplied route epoch. + pub fn cancel(&self, route: RouteHandle, correlation: u64) -> Result<(), CallError> { + self.require_route(route)?; + self.inner + .cancel_key(PendingKey::new(route, correlation), "cancelled")?; + Ok(()) + } + + /// Idempotently closes one exact route generation. + pub async fn close_route(&self, route: RouteHandle) -> Result<(), ClientError> { + if !self.inner.settle_route(route) { + return Ok(()); + } + let deadline = Instant::now() + CLIENT_SHUTDOWN_TIMEOUT; + self.inner + .send_control_wait(FrameType::Goodbye, FrameId::routed(route, 0), deadline) + .await + } + + /// Rejects new work, closes routes, settles pending calls, and joins I/O tasks. + pub async fn close(&self) -> Result<(), ClientError> { + let deadline = Instant::now() + CLIENT_SHUTDOWN_TIMEOUT; + let _close = timeout_at(deadline, self.inner.close_lock.lock()) + .await + .map_err(|_| { + self.inner.retire("shutdown_timeout"); + ClientError::new("shutdown_timeout", "client shutdown timed out") + })?; + let mut guard = CloseGuard::new(&self.inner); + let already_closed = self.inner.closed.swap(true, Ordering::AcqRel); + let mut result = Ok(()); + if !already_closed { + self.inner.settle_all("owner_close"); + let routes: Vec<_> = lock_unpoisoned(&self.inner.routes).drain().collect(); + for route in routes { + if self + .inner + .send_control_wait(FrameType::Goodbye, FrameId::routed(route, 0), deadline) + .await + .is_err() + { + result = Err(ClientError::new( + "shutdown_timeout", + "client shutdown timed out", + )); + break; + } + } + if result.is_ok() + && self + .inner + .send_control_wait(FrameType::Goodbye, FrameId::control(0), deadline) + .await + .is_err() + { + result = Err(ClientError::new( + "shutdown_timeout", + "client shutdown timed out", + )); + } + self.inner.cancel.cancel(); + } + if !self.inner.join_tasks_until(deadline).await { + result = Err(ClientError::new( + "shutdown_timeout", + "client shutdown timed out", + )); + } + guard.disarm(); + result + } + + fn require_route(&self, route: RouteHandle) -> Result<(), CallError> { + if self.inner.closed.load(Ordering::Acquire) { + return Err(CallError::local( + SendOutcome::NotSent, + "client_closed", + "client is closed", + )); + } + if !lock_unpoisoned(&self.inner.routes).contains(&route) { + return Err(CallError::local( + SendOutcome::NotSent, + "route_not_live", + "route is not live on this generation", + )); + } + Ok(()) + } +} + +impl Drop for Client { + fn drop(&mut self) { + self.inner.retire("owner_drop"); + } +} + +struct CloseGuard<'a> { + inner: &'a Inner, + armed: bool, +} + +impl<'a> CloseGuard<'a> { + const fn new(inner: &'a Inner) -> Self { + Self { inner, armed: true } + } + + fn disarm(&mut self) { + self.armed = false; + } +} + +impl Drop for CloseGuard<'_> { + fn drop(&mut self) { + if self.armed { + self.inner.retire("owner_close_dropped"); + } + } +} + +/// Consumer for one bounded stream. Dropping it emits a best-effort Cancel. +pub struct ResponseStream { + inner: Weak, + key: PendingKey, + correlation: u64, + items: mpsc::Receiver, + terminal: Option>>, + finished: bool, +} + +impl fmt::Debug for ResponseStream { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ResponseStream") + .field("correlation", &self.correlation) + .field("finished", &self.finished) + .finish_non_exhaustive() + } +} + +impl ResponseStream { + /// Correlation used by this stream. + pub const fn correlation(&self) -> u64 { + self.correlation + } + + /// Returns next ordered item, `None` after StreamEnd, or terminal error. + pub async fn next(&mut self) -> Result, CallError> { + if self.finished { + return Ok(None); + } + if let Ok(item) = self.items.try_recv() { + return Ok(Some(item.into_public())); + } + let Some(terminal) = self.terminal.as_mut() else { + self.finished = true; + return Ok(None); + }; + enum Next { + Item(ChargedItem), + ItemsClosed, + Terminal(Result, oneshot::error::RecvError>), + } + let next = tokio::select! { + biased; + item = self.items.recv() => match item { + Some(item) => Next::Item(item), + None => Next::ItemsClosed, + }, + result = terminal => Next::Terminal(result), + }; + match next { + Next::Item(item) => Ok(Some(item.into_public())), + Next::ItemsClosed => { + let Some(terminal) = self.terminal.take() else { + self.finished = true; + return Err(retired_error(SendOutcome::OutcomeUnknown)); + }; + let result = terminal + .await + .unwrap_or_else(|_| Err(retired_error(SendOutcome::OutcomeUnknown))); + self.finished = true; + result.map(|()| None) + } + Next::Terminal(result) => { + self.finished = true; + self.terminal = None; + result + .unwrap_or_else(|_| Err(retired_error(SendOutcome::OutcomeUnknown))) + .map(|()| None) + } + } + } + + /// Cancels the stream once. Cleanup remains epoch- and correlation-scoped. + pub fn cancel(&mut self) -> Result<(), CallError> { + if self.finished { + return Ok(()); + } + self.finished = true; + if let Some(inner) = self.inner.upgrade() { + inner.cancel_key(self.key, "cancelled")?; + } + Ok(()) + } +} + +impl Drop for ResponseStream { + fn drop(&mut self) { + let _ = self.cancel(); + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +struct PendingKey { + channel: u16, + epoch: u32, + corr: u64, +} + +impl PendingKey { + fn new(route: RouteHandle, corr: u64) -> Self { + Self { + channel: route.channel, + epoch: route.epoch, + corr, + } + } + + fn route(self) -> RouteHandle { + RouteHandle { + channel: self.channel, + epoch: self.epoch, + } + } +} + +const QUEUED: u8 = 0; +const WRITING: u8 = 1; +const WRITTEN: u8 = 2; +const CANCELLED: u8 = 3; + +struct PendingState { + publish: Arc, + kind: PendingKind, +} + +enum PendingKind { + Unary(oneshot::Sender>), + Stream { + items: mpsc::Sender, + terminal: oneshot::Sender>, + }, +} + +struct Inner { + daemon_id: [u8; DAEMON_ID_LEN], + closed: AtomicBool, + retired: AtomicBool, + cancel: CancellationToken, + correlations: Mutex, + admission: Mutex<()>, + pending: Mutex>, + streams: Mutex, + routes: Mutex>, + queue_budget: Arc, + retained_budget: Arc, + data_tx: mpsc::Sender, + control_tx: mpsc::Sender, + close_lock: tokio::sync::Mutex<()>, + reader: tokio::sync::Mutex>>, + writer: tokio::sync::Mutex>>, +} + +impl Inner { + async fn unary( + self: &Arc, + route: RouteHandle, + body: Vec, + deadline: Instant, + cancellation: Option, + ) -> Result { + let (tx, rx) = oneshot::channel(); + let (key, publish) = self.admit(route, body, PendingKind::Unary(tx), deadline)?; + let mut guard = UnaryAdmissionGuard::new(Arc::clone(self), key); + let cancelled = cancellation.unwrap_or_default(); + let result = tokio::select! { + biased; + result = rx => result.unwrap_or_else(|_| Err(retired_error(classify(&publish)))), + () = cancelled.cancelled() => { + let outcome = self.cancel_key(key, "cancelled").err().map_or_else(|| classify(&publish), |error| error.outcome); + Err(CallError::local(outcome, "cancelled", "request was cancelled")) + } + () = tokio::time::sleep_until(deadline) => { + let outcome = self.cancel_key(key, "deadline_expired").err().map_or_else(|| classify(&publish), |error| error.outcome); + Err(CallError::local(outcome, "deadline_expired", "request deadline expired")) + } + }; + guard.disarm(); + result + } + + fn start_stream( + self: &Arc, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result { + { + let mut streams = lock_unpoisoned(&self.streams); + if *streams >= CLIENT_MAX_LIVE_STREAMS { + return Err(CallError::local( + SendOutcome::NotSent, + "stream_capacity", + "live stream capacity exhausted", + )); + } + *streams += 1; + } + let (item_tx, item_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let deadline = Instant::now() + options.timeout; + let admitted = self.admit( + route, + body, + PendingKind::Stream { + items: item_tx, + terminal: terminal_tx, + }, + deadline, + ); + let (key, _publish) = match admitted { + Ok(value) => value, + Err(error) => { + *lock_unpoisoned(&self.streams) -= 1; + return Err(error); + } + }; + if let Some(cancel) = options.cancellation { + let weak = Arc::downgrade(self); + tokio::spawn(async move { + tokio::select! { + () = cancel.cancelled() => { + if let Some(inner) = weak.upgrade() { + let _ = inner.cancel_key(key, "cancelled"); + } + } + () = tokio::time::sleep_until(deadline) => { + if let Some(inner) = weak.upgrade() { + let _ = inner.cancel_key(key, "deadline_expired"); + } + } + } + }); + } else { + let weak = Arc::downgrade(self); + tokio::spawn(async move { + tokio::time::sleep_until(deadline).await; + if let Some(inner) = weak.upgrade() { + let _ = inner.cancel_key(key, "deadline_expired"); + } + }); + } + Ok(ResponseStream { + inner: Arc::downgrade(self), + key, + correlation: key.corr, + items: item_rx, + terminal: Some(terminal_rx), + finished: false, + }) + } + + fn admit( + &self, + route: RouteHandle, + body: Vec, + kind: PendingKind, + deadline: Instant, + ) -> Result<(PendingKey, Arc), CallError> { + if self.closed.load(Ordering::Acquire) || self.retired.load(Ordering::Acquire) { + return Err(CallError::local( + SendOutcome::NotSent, + "connection_retired", + "connection generation is retired", + )); + } + if Instant::now() >= deadline { + return Err(CallError::local( + SendOutcome::NotSent, + "deadline_expired", + "request deadline expired before admission", + )); + } + let _admission = lock_unpoisoned(&self.admission); + let mut pending = lock_unpoisoned(&self.pending); + if self.closed.load(Ordering::Acquire) || self.retired.load(Ordering::Acquire) { + return Err(CallError::local( + SendOutcome::NotSent, + "connection_retired", + "connection generation is retired", + )); + } + if route + != (RouteHandle { + channel: 0, + epoch: 0, + }) + && !lock_unpoisoned(&self.routes).contains(&route) + { + return Err(CallError::local( + SendOutcome::NotSent, + "route_not_live", + "route is not live on this generation", + )); + } + if Instant::now() >= deadline { + return Err(CallError::local( + SendOutcome::NotSent, + "deadline_expired", + "request deadline expired before admission", + )); + } + if pending.len() >= CLIENT_MAX_PENDING_REQUESTS { + return Err(CallError::local( + SendOutcome::NotSent, + "pending_capacity", + "pending request capacity exhausted", + )); + } + let mut correlations = lock_unpoisoned(&self.correlations); + let corr = correlations.allocate().ok_or_else(|| { + CallError::local( + SendOutcome::NotSent, + "correlations_exhausted", + "correlation space exhausted after u64::MAX", + ) + })?; + let key = PendingKey::new(route, corr); + let publish = Arc::new(AtomicU8::new(QUEUED)); + let frame = match encode_data_frame( + route, + corr, + body, + Arc::clone(&publish), + &self.queue_budget, + deadline, + ) { + Ok(frame) => frame, + Err(error) => { + correlations.restore(corr); + return Err(error); + } + }; + pending.insert( + key, + PendingState { + publish: Arc::clone(&publish), + kind, + }, + ); + if self.data_tx.try_send(frame).is_err() { + pending.remove(&key); + correlations.restore(corr); + return Err(CallError::local( + SendOutcome::NotSent, + "writer_queue_full", + "writer data queue is full", + )); + } + Ok((key, publish)) + } + + fn cancel_key(&self, key: PendingKey, code: &'static str) -> Result<(), CallError> { + let state = lock_unpoisoned(&self.pending).remove(&key); + let Some(state) = state else { + return Ok(()); + }; + let outcome = if state + .publish + .compare_exchange(QUEUED, CANCELLED, Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + SendOutcome::NotSent + } else { + SendOutcome::OutcomeUnknown + }; + self.finish_pending( + state, + Err(CallError::local(outcome, code, "request stopped")), + ); + if outcome == SendOutcome::OutcomeUnknown { + self.send_control( + FrameType::Cancel, + FrameId { + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + None, + )?; + } + Ok(()) + } + + fn send_control( + &self, + ty: FrameType, + id: FrameId, + ack: Option>, + ) -> Result<(), CallError> { + if self.retired.load(Ordering::Acquire) { + return Err(retired_error(SendOutcome::NotSent)); + } + let bytes = encode_owned_frame(ty, pure_header_flags(), id, Vec::new()).map_err(|_| { + CallError::local( + SendOutcome::NotSent, + "encode_failed", + "control encode failed", + ) + })?; + let charge = self.queue_budget.charge(bytes.len()).ok_or_else(|| { + self.retire("control_capacity_exhausted"); + CallError::local( + SendOutcome::Terminal, + "control_capacity_exhausted", + "reserved control admission exhausted", + ) + })?; + let frame = QueuedFrame { + bytes, + charge, + publish: None, + ack, + deadline: Instant::now() + CLIENT_FRAME_TIMEOUT, + }; + if self.control_tx.try_send(frame).is_err() { + self.retire("control_capacity_exhausted"); + return Err(CallError::local( + SendOutcome::Terminal, + "control_capacity_exhausted", + "reserved control admission exhausted", + )); + } + Ok(()) + } + + async fn send_control_wait( + &self, + ty: FrameType, + id: FrameId, + deadline: Instant, + ) -> Result<(), ClientError> { + let (tx, rx) = oneshot::channel(); + self.send_control(ty, id, Some(tx)).map_err(|_| { + ClientError::new( + "control_capacity_exhausted", + "client control admission failed", + ) + })?; + timeout_at(deadline, rx) + .await + .map_err(|_| ClientError::new("shutdown_timeout", "client shutdown timed out"))? + .map_err(|_| ClientError::new("connection_retired", "connection retired")) + } + + fn dispatch( + self: &Arc, + header: EnvelopeHeader, + body: Vec, + charge: Option, + ) { + match header.ty { + FrameType::Ping => { + let _ = self.send_control(FrameType::Pong, FrameId::control(header.corr), None); + } + FrameType::Goodbye if header.channel == 0 => self.retire("connection_goodbye"), + FrameType::Goodbye => { + let route = RouteHandle { + channel: header.channel, + epoch: header.epoch, + }; + self.settle_route(route); + } + FrameType::Push => {} + FrameType::Response | FrameType::Error | FrameType::StreamEnd => { + let key = PendingKey { + channel: header.channel, + epoch: header.epoch, + corr: header.corr, + }; + let state = lock_unpoisoned(&self.pending).remove(&key); + let Some(state) = state else { + return; + }; + drop(charge); + match state.kind { + PendingKind::Unary(tx) => { + let result = match header.ty { + FrameType::Response => Ok(Response { + body, + binary: header.flags.is_binary(), + }), + FrameType::Error => Err(CallError::host_terminal(&body)), + FrameType::StreamEnd => Err(CallError::local( + SendOutcome::Terminal, + "unexpected_stream", + "unary request received stream terminal", + )), + _ => unreachable!(), + }; + let _ = tx.send(result); + } + PendingKind::Stream { terminal, .. } => { + let terminal_result = match header.ty { + FrameType::StreamEnd => Ok(()), + FrameType::Error => Err(CallError::host_terminal(&body)), + FrameType::Response => Err(CallError::local( + SendOutcome::Terminal, + "unexpected_response", + "stream received unary response terminal", + )), + _ => unreachable!(), + }; + let _ = terminal.send(terminal_result); + self.release_stream(); + } + } + } + FrameType::StreamData => { + let key = PendingKey { + channel: header.channel, + epoch: header.epoch, + corr: header.corr, + }; + let mut pending = lock_unpoisoned(&self.pending); + let Some(state) = pending.get_mut(&key) else { + return; + }; + match &mut state.kind { + PendingKind::Unary(_) => { + let state = pending.remove(&key).expect("entry exists"); + drop(pending); + self.finish_pending( + state, + Err(CallError::local( + SendOutcome::Terminal, + "unexpected_stream", + "unary request received stream data", + )), + ); + let _ = self.send_control( + FrameType::Cancel, + FrameId { + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + None, + ); + } + PendingKind::Stream { items, .. } => { + let Some(charge) = charge else { + drop(pending); + self.retire("response_memory_exhausted"); + return; + }; + let item = ChargedItem { + body, + binary: header.flags.is_binary(), + _charge: charge, + }; + if items.try_send(item).is_err() { + let state = pending.remove(&key).expect("entry exists"); + drop(pending); + self.finish_pending( + state, + Err(CallError::local( + SendOutcome::Terminal, + "stream_saturated", + "stream consumer queue saturated", + )), + ); + let _ = self.send_control( + FrameType::Cancel, + FrameId { + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + None, + ); + } + } + } + } + _ => self.retire("protocol_violation"), + } + } + + fn finish_pending(&self, state: PendingState, result: Result) { + match state.kind { + PendingKind::Unary(tx) => { + let _ = tx.send(result); + } + PendingKind::Stream { terminal, .. } => { + let terminal_result = result.map(|_| ()); + let _ = terminal.send(terminal_result); + self.release_stream(); + } + } + } + + fn release_stream(&self) { + let mut streams = lock_unpoisoned(&self.streams); + *streams = streams.saturating_sub(1); + } + + fn settle_route(&self, route: RouteHandle) -> bool { + let pending = { + let _admission = lock_unpoisoned(&self.admission); + let mut pending = lock_unpoisoned(&self.pending); + if !lock_unpoisoned(&self.routes).remove(&route) { + return false; + } + let keys: Vec<_> = pending + .keys() + .copied() + .filter(|key| key.route() == route) + .collect(); + keys.into_iter() + .filter_map(|key| pending.remove(&key).map(|state| (key, state))) + .collect::>() + }; + for (key, state) in pending { + let outcome = cancel_classification(&state.publish); + self.finish_pending( + state, + Err(CallError::local(outcome, "route_gone", "request stopped")), + ); + if outcome == SendOutcome::OutcomeUnknown { + let _ = self.send_control( + FrameType::Cancel, + FrameId { + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + None, + ); + } + } + true + } + + fn settle_all(&self, code: &'static str) { + let pending = { + let _admission = lock_unpoisoned(&self.admission); + std::mem::take(&mut *lock_unpoisoned(&self.pending)) + }; + for (_, state) in pending { + let outcome = cancel_classification(&state.publish); + self.finish_pending( + state, + Err(CallError::local( + outcome, + code, + "connection generation closed", + )), + ); + } + } + + fn retire(&self, code: &'static str) { + if self.retired.swap(true, Ordering::AcqRel) { + return; + } + self.closed.store(true, Ordering::Release); + self.settle_all(code); + lock_unpoisoned(&self.routes).clear(); + self.cancel.cancel(); + } + + async fn join_tasks_until(&self, deadline: Instant) -> bool { + loop { + let writer_finished = self + .writer + .lock() + .await + .as_ref() + .is_none_or(JoinHandle::is_finished); + let reader_finished = self + .reader + .lock() + .await + .as_ref() + .is_none_or(JoinHandle::is_finished); + if writer_finished && reader_finished { + break; + } + if Instant::now() >= deadline { + if let Some(task) = self.writer.lock().await.as_ref() { + task.abort(); + } + if let Some(task) = self.reader.lock().await.as_ref() { + task.abort(); + } + break; + } + tokio::task::yield_now().await; + } + let within_deadline = Instant::now() < deadline; + if let Some(task) = self.writer.lock().await.take() { + let _ = task.await; + } + if let Some(task) = self.reader.lock().await.take() { + let _ = task.await; + } + within_deadline + } +} + +struct UnaryAdmissionGuard { + inner: Arc, + key: PendingKey, + armed: bool, +} + +impl UnaryAdmissionGuard { + const fn new(inner: Arc, key: PendingKey) -> Self { + Self { + inner, + key, + armed: true, + } + } + + fn disarm(&mut self) { + self.armed = false; + } +} + +impl Drop for UnaryAdmissionGuard { + fn drop(&mut self) { + if self.armed { + let _ = self.inner.cancel_key(self.key, "caller_dropped"); + } + } +} + +struct Correlations { + next: Option, +} + +impl Correlations { + const fn new(first: u64) -> Self { + Self { next: Some(first) } + } + + fn allocate(&mut self) -> Option { + let current = self.next?; + self.next = current.checked_add(1); + Some(current) + } + + fn restore(&mut self, correlation: u64) { + if self.next == correlation.checked_add(1) + || (correlation == u64::MAX && self.next.is_none()) + { + self.next = Some(correlation); + } + } +} + +struct ByteCounter { + cap: usize, + used: Mutex, +} + +impl ByteCounter { + const fn new(cap: usize) -> Self { + Self { + cap, + used: Mutex::new(0), + } + } + + fn charge(self: &Arc, bytes: usize) -> Option { + let mut used = lock_unpoisoned(&self.used); + let next = used.checked_add(bytes)?; + if next > self.cap { + return None; + } + *used = next; + Some(ByteCharge { + owner: Arc::downgrade(self), + bytes, + }) + } + + #[cfg(test)] + fn used(&self) -> usize { + *lock_unpoisoned(&self.used) + } +} + +struct ByteCharge { + owner: Weak, + bytes: usize, +} + +impl Drop for ByteCharge { + fn drop(&mut self) { + if let Some(owner) = self.owner.upgrade() { + let mut used = lock_unpoisoned(&owner.used); + *used = used.saturating_sub(self.bytes); + } + } +} + +struct ChargedItem { + body: Vec, + binary: bool, + _charge: ByteCharge, +} + +impl ChargedItem { + fn into_public(self) -> StreamItem { + StreamItem { + body: self.body, + binary: self.binary, + } + } +} + +struct QueuedFrame { + bytes: Vec, + charge: ByteCharge, + publish: Option>, + ack: Option>, + deadline: Instant, +} + +async fn writer_loop( + inner: Arc, + mut write: OwnedWriteHalf, + mut data_rx: mpsc::Receiver, + mut control_rx: mpsc::Receiver, +) { + loop { + let frame = if let Ok(frame) = control_rx.try_recv() { + frame + } else { + tokio::select! { + biased; + () = inner.cancel.cancelled() => break, + frame = control_rx.recv() => match frame { + Some(frame) => frame, + None => match data_rx.recv().await { Some(frame) => frame, None => break }, + }, + frame = data_rx.recv() => match frame { + Some(frame) => frame, + None => match control_rx.recv().await { Some(frame) => frame, None => break }, + }, + } + }; + if frame + .publish + .as_ref() + .is_some_and(|state| !claim_for_write(state)) + { + continue; + } + let written = tokio::select! { + biased; + () = inner.cancel.cancelled() => break, + result = timeout_at(frame.deadline, write.write_all(&frame.bytes)) => result, + }; + if !matches!(written, Ok(Ok(()))) { + inner.retire("write_failed"); + break; + } + if let Some(state) = &frame.publish { + state.store(WRITTEN, Ordering::Release); + } + if let Some(ack) = frame.ack { + let _ = ack.send(()); + } + drop(frame.charge); + } + let _ = write.shutdown().await; +} + +async fn reader_loop(inner: Arc, mut read: OwnedReadHalf) { + loop { + let frame = match read_active_frame(&mut read, &inner).await { + Ok(Some(frame)) => frame, + Ok(None) => { + inner.retire("eof"); + break; + } + Err(()) => { + inner.retire("protocol_violation"); + break; + } + }; + inner.dispatch(frame.header, frame.body, frame.charge); + if inner.retired.load(Ordering::Acquire) { + break; + } + } +} + +struct InboundFrame { + header: EnvelopeHeader, + body: Vec, + charge: Option, +} + +async fn read_active_frame( + read: &mut R, + inner: &Arc, +) -> Result, ()> { + let mut header_bytes = [0u8; HEADER_LEN]; + let first = tokio::select! { + biased; + () = inner.cancel.cancelled() => return Err(()), + result = read.read(&mut header_bytes[..1]) => result.map_err(|_| ())?, + }; + if first == 0 { + return Ok(None); + } + let deadline = Instant::now() + CLIENT_FRAME_TIMEOUT; + read_exact_until(read, &mut header_bytes[1..], deadline, &inner.cancel).await?; + let header = decode_header(&header_bytes).map_err(|_| ())?; + validate_inbound(&header)?; + let charge = if header.len == 0 { + None + } else { + inner.retained_budget.charge(header.len as usize) + }; + let mut body = Vec::new(); + if let Some(_charge) = charge.as_ref() { + body.resize(header.len as usize, 0); + read_exact_until(read, &mut body, deadline, &inner.cancel).await?; + } else if header.len > 0 { + drain_until(read, header.len as usize, deadline, &inner.cancel).await?; + return Err(()); + } + Ok(Some(InboundFrame { + header, + body, + charge, + })) +} + +async fn read_exact_until( + read: &mut R, + buf: &mut [u8], + deadline: Instant, + cancel: &CancellationToken, +) -> Result<(), ()> { + let mut offset = 0; + while offset < buf.len() { + let count = tokio::select! { + biased; + () = cancel.cancelled() => return Err(()), + result = timeout_at(deadline, read.read(&mut buf[offset..])) => result.map_err(|_| ())?.map_err(|_| ())?, + }; + if count == 0 { + return Err(()); + } + offset += count; + } + Ok(()) +} + +async fn drain_until( + read: &mut R, + mut remaining: usize, + deadline: Instant, + cancel: &CancellationToken, +) -> Result<(), ()> { + let mut scratch = [0u8; 8192]; + while remaining > 0 { + let take = remaining.min(scratch.len()); + read_exact_until(read, &mut scratch[..take], deadline, cancel).await?; + remaining -= take; + } + Ok(()) +} + +fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { + if header.ver != PROTOCOL_VERSION || header.len > MAX_BODY_LEN { + return Err(()); + } + match header.ty { + FrameType::Response | FrameType::Error | FrameType::StreamData | FrameType::StreamEnd => { + if header.corr == 0 { + return Err(()); + } + } + FrameType::Push => { + if header.channel == 0 || header.epoch == 0 { + return Err(()); + } + } + FrameType::Ping => { + if header.channel != 0 || header.epoch != 0 || header.corr == 0 { + return Err(()); + } + } + FrameType::Goodbye => { + if header.corr != 0 || (header.channel != 0 && header.epoch == 0) { + return Err(()); + } + } + _ => return Err(()), + } + if header.ty.is_pure_header() && (header.flags != pure_header_flags() || header.len != 0) { + return Err(()); + } + Ok(()) +} + +fn encode_data_frame( + route: RouteHandle, + corr: u64, + body: Vec, + publish: Arc, + budget: &Arc, + deadline: Instant, +) -> Result { + let bytes = encode_owned_frame( + FrameType::Request, + Flags::new(false, Priority::Interactive, false), + FrameId::routed(route, corr), + body, + ) + .map_err(|_| { + CallError::local( + SendOutcome::NotSent, + "body_too_large", + "request body exceeds wire limit", + ) + })?; + let charge = budget.charge(bytes.len()).ok_or_else(|| { + CallError::local( + SendOutcome::NotSent, + "queued_byte_capacity", + "shared queued-byte capacity exhausted", + ) + })?; + Ok(QueuedFrame { + bytes, + charge, + publish: Some(publish), + ack: None, + deadline, + }) +} + +async fn negotiate_tcp(stream: &mut TcpStream, deadline: Instant) -> Result<(), ClientError> { + let body = serde_json::to_vec(&serde_json::json!({ + "op": "transport.negotiate", + "negotiation_version": NEGOTIATION_VERSION, + "offers": [{"transport": TRANSPORT_TCP, "capability_version": 1}] + })) + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + let bytes = encode_owned_frame( + FrameType::Request, + Flags::new(false, Priority::Interactive, false), + FrameId::control(NEGOTIATION_CORRELATION), + body, + ) + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + timeout_at(deadline, stream.write_all(&bytes)) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + let frame = read_setup_frame(stream, deadline).await?; + if frame.header.ty != FrameType::Response + || frame.header.channel != 0 + || frame.header.epoch != 0 + || frame.header.corr != NEGOTIATION_CORRELATION + { + return Err(ClientError::new( + "negotiation_failed", + "transport negotiation failed closed", + )); + } + let offers = [TransportOffer { + transport: TRANSPORT_TCP.to_owned(), + capability_version: 1, + parameters: None, + }]; + let selection = decode_negotiate_response(&frame.body, &offers).map_err(|_| { + ClientError::new("negotiation_failed", "transport negotiation failed closed") + })?; + if !matches!(selection, NegotiateResponse::Tcp { reason: None }) { + return Err(ClientError::new( + "negotiation_failed", + "transport negotiation failed closed", + )); + } + Ok(()) +} + +async fn read_setup_frame( + stream: &mut TcpStream, + deadline: Instant, +) -> Result { + let mut header_bytes = [0u8; HEADER_LEN]; + timeout_at(deadline, stream.read_exact(&mut header_bytes)) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + let header = decode_header(&header_bytes) + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + if header.len > MAX_BODY_LEN { + return Err(ClientError::new( + "negotiation_failed", + "transport negotiation failed", + )); + } + let mut body = vec![0u8; header.len as usize]; + timeout_at(deadline, stream.read_exact(&mut body)) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + Ok(InboundFrame { + header, + body, + charge: None, + }) +} + +fn route_open_body(target: &RouteTarget, identity: &RouteIdentity) -> Result, CallError> { + let project_root = identity.project_root.to_str().ok_or_else(|| { + CallError::local( + SendOutcome::NotSent, + "invalid_identity", + "route identity path is not UTF-8", + ) + })?; + let kind = match target.kind { + TargetKind::ToolProvider => "tool_provider", + TargetKind::ManagementSurface => "management_surface", + }; + let mut request = serde_json::json!({ + "op": "route.open", + "target": {"kind": kind, "module_id": target.module_id}, + "identity": { + "project_root": project_root, + "harness": identity.harness, + "session": identity.session + }, + "consumer_capabilities": identity.consumer_capabilities, + "admission_facts": identity.admission_facts + }); + if let (Some(module_id), Some(launch_nonce)) = ( + identity.consumer_module_id.as_ref(), + identity.consumer_launch_nonce.as_ref(), + ) { + request["consumer_identity"] = serde_json::json!({ + "module_id": module_id, + "launch_nonce": launch_nonce + }); + } + serde_json::to_vec(&request).map_err(|_| { + CallError::local( + SendOutcome::NotSent, + "invalid_identity", + "route-open request could not be encoded", + ) + }) +} + +fn parse_route_open(body: &[u8]) -> Result { + let value = serde_json::from_slice::(body).map_err(|_| { + CallError::local( + SendOutcome::Terminal, + "invalid_route_response", + "host returned an invalid route-open response", + ) + })?; + if value.get("op").and_then(Value::as_str) != Some("route.open") { + return Err(CallError::local( + SendOutcome::Terminal, + "invalid_route_response", + "host returned an invalid route-open response", + )); + } + let channel = value + .get("route_channel") + .and_then(Value::as_u64) + .and_then(|value| u16::try_from(value).ok()) + .filter(|value| *value != 0) + .ok_or_else(|| { + CallError::local( + SendOutcome::Terminal, + "invalid_route_response", + "host returned an invalid route-open response", + ) + })?; + let epoch = value + .get("route_epoch") + .and_then(Value::as_u64) + .and_then(|value| u32::try_from(value).ok()) + .filter(|value| *value != 0) + .ok_or_else(|| { + CallError::local( + SendOutcome::Terminal, + "invalid_route_response", + "host returned an invalid route-open response", + ) + })?; + Ok(RouteHandle { channel, epoch }) +} + +fn claim_for_write(state: &AtomicU8) -> bool { + state + .compare_exchange(QUEUED, WRITING, Ordering::AcqRel, Ordering::Acquire) + .is_ok() +} + +fn classify(state: &AtomicU8) -> SendOutcome { + match state.load(Ordering::Acquire) { + QUEUED | CANCELLED => SendOutcome::NotSent, + WRITING | WRITTEN => SendOutcome::OutcomeUnknown, + _ => SendOutcome::OutcomeUnknown, + } +} + +fn cancel_classification(state: &AtomicU8) -> SendOutcome { + if state + .compare_exchange(QUEUED, CANCELLED, Ordering::AcqRel, Ordering::Acquire) + .is_ok() + { + SendOutcome::NotSent + } else { + classify(state) + } +} + +fn retired_error(outcome: SendOutcome) -> CallError { + CallError::local( + outcome, + "generation_retired", + "connection generation retired", + ) +} + +fn lock_unpoisoned(mutex: &Mutex) -> MutexGuard<'_, T> { + mutex + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + +fn bounded_code(code: &str) -> String { + let code = code + .chars() + .filter(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '_' | '-' | '.')) + .take(MAX_ERROR_CODE_BYTES) + .collect::(); + if code.is_empty() { + "remote_error".to_owned() + } else { + code + } +} + +fn bounded_text(text: &str, max: usize) -> String { + text.chars().take(max).collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::wire::response_flags; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + fn test_inner( + queued_bytes: usize, + ) -> ( + Arc, + mpsc::Receiver, + mpsc::Receiver, + ) { + let (data_tx, data_rx) = mpsc::channel(CLIENT_DATA_QUEUE_FRAMES); + let (control_tx, control_rx) = mpsc::channel(CLIENT_CONTROL_QUEUE_FRAMES); + ( + Arc::new(Inner { + daemon_id: [0; DAEMON_ID_LEN], + closed: AtomicBool::new(false), + retired: AtomicBool::new(false), + cancel: CancellationToken::new(), + correlations: Mutex::new(Correlations::new(FIRST_APPLICATION_CORRELATION)), + admission: Mutex::new(()), + pending: Mutex::new(HashMap::new()), + streams: Mutex::new(0), + routes: Mutex::new(HashSet::from([route(1), route(2)])), + queue_budget: Arc::new(ByteCounter::new(queued_bytes)), + retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), + data_tx, + control_tx, + close_lock: tokio::sync::Mutex::new(()), + reader: tokio::sync::Mutex::new(None), + writer: tokio::sync::Mutex::new(None), + }), + data_rx, + control_rx, + ) + } + + fn route(epoch: u32) -> RouteHandle { + RouteHandle { channel: 7, epoch } + } + + fn unary_sender() -> (PendingKind, oneshot::Receiver>) { + let (tx, rx) = oneshot::channel(); + (PendingKind::Unary(tx), rx) + } + + async fn ack_controls(mut rx: mpsc::Receiver, count: usize) { + for _ in 0..count { + let mut frame = rx.recv().await.expect("control frame"); + frame + .ack + .take() + .expect("close control has ack") + .send(()) + .ok(); + } + } + + #[test] + fn max_correlation_is_used_once_then_exhausted() { + let mut correlations = Correlations::new(u64::MAX); + assert_eq!(correlations.allocate(), Some(u64::MAX)); + assert_eq!(correlations.allocate(), None); + correlations.restore(u64::MAX); + assert_eq!(correlations.allocate(), Some(u64::MAX)); + assert_eq!(correlations.allocate(), None); + } + + #[tokio::test] + async fn real_admission_exhausts_after_max_without_second_charge_or_frame() { + let (inner, data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.correlations).next = Some(u64::MAX); + let deadline = Instant::now() + Duration::from_secs(1); + let (first_kind, _first_rx) = unary_sender(); + let (key, _) = inner + .admit(route(1), Vec::new(), first_kind, deadline) + .expect("u64::MAX is admitted once"); + assert_eq!(key.corr, u64::MAX); + let charged = inner.queue_budget.used(); + assert_eq!(data_rx.len(), 1); + + let (second_kind, _second_rx) = unary_sender(); + let error = inner + .admit(route(1), Vec::new(), second_kind, deadline) + .expect_err("correlation space is exhausted"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "correlations_exhausted"); + assert_eq!(data_rx.len(), 1); + assert_eq!(inner.queue_budget.used(), charged); + + inner.retire("test_done"); + drop(data_rx); + assert_eq!(inner.queue_budget.used(), 0); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn close_wins_against_admission_blocked_on_pending() { + let (inner, data_rx, control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let client = Arc::new(Client { + inner: Arc::clone(&inner), + }); + let pending = lock_unpoisoned(&inner.pending); + let closer = { + let client = Arc::clone(&client); + tokio::spawn(async move { client.close().await }) + }; + while !inner.closed.load(Ordering::Acquire) { + std::thread::yield_now(); + } + loop { + match inner.admission.try_lock() { + Err(std::sync::TryLockError::WouldBlock) => break, + Err(std::sync::TryLockError::Poisoned(_)) => panic!("admission lock poisoned"), + Ok(guard) => drop(guard), + } + std::thread::yield_now(); + } + let admission = { + let inner = Arc::clone(&inner); + tokio::task::spawn_blocking(move || { + let (kind, _rx) = unary_sender(); + inner.admit( + route(1), + b"must-not-write".to_vec(), + kind, + Instant::now() + Duration::from_secs(1), + ) + }) + }; + drop(pending); + let acknowledger = tokio::spawn(ack_controls(control_rx, 3)); + + closer.await.unwrap().expect("close completes"); + acknowledger.await.unwrap(); + let error = admission + .await + .unwrap() + .expect_err("close wins admission ordering"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "connection_retired"); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + assert!(data_rx.is_empty(), "losing admission queues no write"); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn exact_route_close_wins_against_admission_blocked_on_pending() { + let (inner, data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let pending = lock_unpoisoned(&inner.pending); + let closer = { + let inner = Arc::clone(&inner); + tokio::task::spawn_blocking(move || inner.settle_route(route(1))) + }; + loop { + match inner.admission.try_lock() { + Err(std::sync::TryLockError::WouldBlock) => break, + Err(std::sync::TryLockError::Poisoned(_)) => panic!("admission lock poisoned"), + Ok(guard) => drop(guard), + } + std::thread::yield_now(); + } + let admission = { + let inner = Arc::clone(&inner); + tokio::task::spawn_blocking(move || { + let (kind, _rx) = unary_sender(); + inner.admit( + route(1), + b"must-not-write".to_vec(), + kind, + Instant::now() + Duration::from_secs(1), + ) + }) + }; + drop(pending); + + assert!(closer.await.unwrap(), "exact route was live"); + let error = admission + .await + .unwrap() + .expect_err("closed route rejects admission"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "route_not_live"); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + assert!(data_rx.is_empty(), "losing admission queues no write"); + assert!(lock_unpoisoned(&inner.routes).contains(&route(2))); + } + + #[tokio::test] + async fn admission_winning_is_settled_by_close() { + let (inner, data_rx, control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let deadline = Instant::now() + Duration::from_secs(1); + let (kind, rx) = unary_sender(); + let (_key, publish) = inner + .admit(route(1), b"admitted".to_vec(), kind, deadline) + .expect("admission wins"); + let client = Client { + inner: Arc::clone(&inner), + }; + let acknowledger = tokio::spawn(ack_controls(control_rx, 3)); + + client.close().await.expect("close completes"); + acknowledger.await.unwrap(); + let error = rx.await.unwrap().expect_err("close settles admitted work"); + assert_eq!(error.code(), "owner_close"); + assert_eq!(classify(&publish), SendOutcome::NotSent); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + assert_eq!(data_rx.len(), 1, "admission queued exactly one frame"); + } + + #[tokio::test] + async fn cancel_winning_queued_prevents_writer_claim_and_frame() { + let (inner, data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let deadline = Instant::now() + Duration::from_secs(1); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit(route(1), b"must-not-send".to_vec(), kind, deadline) + .expect("admitted"); + inner.cancel_key(key, "cancelled").expect("cancel queued"); + let error = rx.await.expect("settled").expect_err("cancelled"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(publish.load(Ordering::Acquire), CANCELLED); + assert!(!claim_for_write(&publish)); + assert!(control_rx.try_recv().is_err(), "not-sent needs no Cancel"); + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let mut peer = TcpStream::connect(listener.local_addr().unwrap()) + .await + .unwrap(); + let (socket, _) = listener.accept().await.unwrap(); + let (_read, write) = socket.into_split(); + let writer_inner = Arc::clone(&inner); + let writer = tokio::spawn(async move { + writer_loop(writer_inner, write, data_rx, control_rx).await; + }); + let mut byte = [0u8; 1]; + assert!( + tokio::time::timeout(Duration::from_millis(50), peer.read(&mut byte)) + .await + .is_err(), + "cancel-winning queued request must write no frame bytes" + ); + assert_eq!(inner.queue_budget.used(), 0); + inner.cancel.cancel(); + writer.await.unwrap(); + } + + #[tokio::test] + async fn writer_winning_cancel_is_outcome_unknown_and_queues_cancel() { + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit( + route(1), + b"possibly-sent".to_vec(), + kind, + Instant::now() + Duration::from_secs(1), + ) + .expect("admitted"); + assert!(claim_for_write(&publish), "writer wins QUEUED CAS"); + inner.cancel_key(key, "cancelled").expect("cancel writing"); + let error = rx.await.expect("settled").expect_err("cancelled"); + assert_eq!(error.outcome(), SendOutcome::OutcomeUnknown); + let cancel = control_rx.recv().await.expect("Cancel queued"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + assert_eq!(publish.load(Ordering::Acquire), WRITING); + drop(data_rx.recv().await); + drop(cancel); + assert_eq!(inner.queue_budget.used(), 0); + } + + #[tokio::test] + async fn dropped_unary_future_cleans_pending_and_possibly_sent_request() { + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let task_inner = Arc::clone(&inner); + let request = tokio::spawn(async move { + task_inner + .unary( + route(1), + b"stalled-peer".to_vec(), + Instant::now() + Duration::from_secs(60), + None, + ) + .await + }); + let frame = data_rx.recv().await.expect("request admitted"); + let publish = frame.publish.as_ref().expect("data publication state"); + assert!( + claim_for_write(publish), + "simulate stalled writer after claim" + ); + request.abort(); + assert!(request.await.expect_err("request aborted").is_cancelled()); + + assert!(lock_unpoisoned(&inner.pending).is_empty()); + let cancel = control_rx.recv().await.expect("possibly-sent Cancel"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + drop(frame); + drop(cancel); + assert_eq!(inner.queue_budget.used(), 0); + } + + #[tokio::test] + async fn dropped_close_retires_and_repeated_close_joins_tasks() { + let (inner, _data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let writer_cancel = inner.cancel.clone(); + *inner.writer.lock().await = Some(tokio::spawn(async move { + writer_cancel.cancelled().await; + })); + let reader_cancel = inner.cancel.clone(); + *inner.reader.lock().await = Some(tokio::spawn(async move { + reader_cancel.cancelled().await; + })); + let client = Arc::new(Client { + inner: Arc::clone(&inner), + }); + let closing = { + let client = Arc::clone(&client); + tokio::spawn(async move { client.close().await }) + }; + while !inner.closed.load(Ordering::Acquire) { + tokio::task::yield_now().await; + } + closing.abort(); + assert!(closing.await.expect_err("close aborted").is_cancelled()); + assert!(inner.retired.load(Ordering::Acquire)); + assert!(inner.cancel.is_cancelled()); + + timeout_at(Instant::now() + Duration::from_secs(1), client.close()) + .await + .expect("second close bounded") + .expect("second close succeeds"); + assert!(inner.writer.lock().await.is_none()); + assert!(inner.reader.lock().await.is_none()); + } + + #[tokio::test] + async fn data_capacity_spares_control_reserve_and_does_not_burn_correlation() { + let (inner, data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let deadline = Instant::now() + Duration::from_secs(1); + let mut receivers = Vec::new(); + for _ in 0..CLIENT_DATA_QUEUE_FRAMES { + let (kind, rx) = unary_sender(); + inner + .admit(route(1), Vec::new(), kind, deadline) + .expect("data slot"); + receivers.push(rx); + } + let next_before = lock_unpoisoned(&inner.correlations).next; + let (kind, _rx) = unary_sender(); + let error = inner + .admit(route(1), Vec::new(), kind, deadline) + .expect_err("257th data frame is rejected"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "writer_queue_full"); + assert_eq!(lock_unpoisoned(&inner.correlations).next, next_before); + + inner + .send_control(FrameType::Pong, FrameId::control(99), None) + .expect("reserved control remains available"); + let pong = control_rx.recv().await.expect("queued Pong"); + assert_eq!(pong.bytes[5], FrameType::Pong as u8); + drop(pong); + assert_eq!(data_rx.len(), CLIENT_DATA_QUEUE_FRAMES); + + inner.retire("test_done"); + drop(data_rx); + drop(control_rx); + drop(receivers); + assert_eq!(inner.queue_budget.used(), 0); + } + + #[tokio::test] + async fn control_exhaustion_retires_and_releases_all_queued_bytes() { + let (inner, data_rx, control_rx) = test_inner(CLIENT_QUEUED_BYTES); + for corr in 1..=CLIENT_CONTROL_QUEUE_FRAMES as u64 { + inner + .send_control(FrameType::Pong, FrameId::control(corr), None) + .expect("reserved slot"); + } + let error = inner + .send_control(FrameType::Pong, FrameId::control(99), None) + .expect_err("33rd control retires generation"); + assert_eq!(error.code(), "control_capacity_exhausted"); + assert!(inner.retired.load(Ordering::Acquire)); + drop(data_rx); + drop(control_rx); + assert_eq!(inner.queue_budget.used(), 0); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + } + + #[tokio::test] + async fn data_and_control_charge_one_shared_byte_cap() { + let (inner, data_rx, control_rx) = test_inner(HEADER_LEN * 2); + inner + .send_control(FrameType::Pong, FrameId::control(1), None) + .expect("first header"); + let (kind, _rx) = unary_sender(); + inner + .admit( + route(1), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(1), + ) + .expect("data header uses remaining shared bytes"); + assert_eq!(inner.queue_budget.used(), HEADER_LEN * 2); + assert!(inner + .send_control(FrameType::Pong, FrameId::control(2), None) + .is_err()); + drop(data_rx); + drop(control_rx); + assert_eq!(inner.queue_budget.used(), 0); + } + + #[tokio::test] + async fn stale_epoch_terminal_cannot_settle_reused_channel() { + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (kind, mut rx) = unary_sender(); + let (key, _) = inner + .admit( + route(2), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(1), + ) + .expect("admit current epoch"); + drop(data_rx.recv().await); + inner.dispatch( + EnvelopeHeader { + len: 0, + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, true), + channel: key.channel, + epoch: 1, + corr: key.corr, + }, + Vec::new(), + None, + ); + assert!(matches!( + rx.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + )); + assert!(lock_unpoisoned(&inner.pending).contains_key(&key)); + inner.retire("test_done"); + } + + #[tokio::test] + async fn saturated_stream_fails_alone_and_queues_cancel() { + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let (stream_key, _) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + }, + Instant::now() + Duration::from_secs(1), + ) + .expect("stream admitted"); + drop(data_rx.recv().await); + let (unary_kind, mut unary_rx) = unary_sender(); + let (unary_key, _) = inner + .admit( + route(1), + Vec::new(), + unary_kind, + Instant::now() + Duration::from_secs(1), + ) + .expect("unrelated unary admitted"); + drop(data_rx.recv().await); + + for _ in 0..=CLIENT_STREAM_QUEUE_ITEMS { + let charge = inner.retained_budget.charge(1).expect("retained byte"); + inner.dispatch( + EnvelopeHeader { + len: 1, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: stream_key.channel, + epoch: stream_key.epoch, + corr: stream_key.corr, + }, + vec![1], + Some(charge), + ); + } + let error = terminal_rx + .await + .expect("terminal sender") + .expect_err("saturated stream fails"); + assert_eq!(error.code(), "stream_saturated"); + let cancel = control_rx.recv().await.expect("stream Cancel"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + + inner.dispatch( + EnvelopeHeader { + len: 0, + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, true), + channel: unary_key.channel, + epoch: unary_key.epoch, + corr: unary_key.corr, + }, + Vec::new(), + None, + ); + assert!(unary_rx.try_recv().expect("unary settled").is_ok()); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + inner.retire("test_done"); + } + + #[tokio::test(start_paused = true)] + async fn idle_header_is_unbounded_then_partial_frame_has_one_deadline() { + let (inner, _data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (mut peer, mut reader) = tokio::io::duplex(64); + let task_inner = Arc::clone(&inner); + let task = tokio::spawn(async move { read_active_frame(&mut reader, &task_inner).await }); + + tokio::time::advance(Duration::from_secs(3_600)).await; + tokio::task::yield_now().await; + assert!( + !task.is_finished(), + "idle first-header wait must be unbounded" + ); + + peer.write_all(&[0]).await.expect("first header byte"); + tokio::time::advance(CLIENT_FRAME_TIMEOUT - Duration::from_millis(1)).await; + tokio::task::yield_now().await; + assert!(!task.is_finished(), "partial frame keeps original deadline"); + tokio::time::advance(Duration::from_millis(1)).await; + tokio::task::yield_now().await; + assert!(task.await.expect("reader task").is_err()); + } + + #[test] + fn queue_and_retained_charges_release_exactly() { + let budget = Arc::new(ByteCounter::new(10)); + let first = budget.charge(7).expect("first charge"); + assert!(budget.charge(4).is_none()); + assert_eq!(budget.used(), 7); + drop(first); + assert_eq!(budget.used(), 0); + } + + #[test] + fn epoch_is_part_of_pending_key() { + let old = PendingKey::new( + RouteHandle { + channel: 7, + epoch: 1, + }, + 9, + ); + let current = PendingKey::new( + RouteHandle { + channel: 7, + epoch: 2, + }, + 9, + ); + assert_ne!(old, current); + } + + #[test] + fn terminal_formatting_redacts_peer_message_and_body() { + let sentinel = "CANARY-CREDENTIAL-PAYLOAD-93ff"; + let body = serde_json::to_vec(&serde_json::json!({ + "code": "stable_code", + "message": sentinel + })) + .expect("serialize"); + let error = CallError::host_terminal(&body); + let rendered = format!("{error:?} {error}"); + assert_eq!(error.outcome(), SendOutcome::Terminal); + assert_eq!(error.code(), "stable_code"); + assert!(!rendered.contains(sentinel)); + } + + #[test] + fn outcome_spellings_are_exact() { + assert_eq!(SendOutcome::NotSent.as_str(), "not_sent"); + assert_eq!(SendOutcome::OutcomeUnknown.as_str(), "outcome_unknown"); + assert_eq!(SendOutcome::Terminal.as_str(), "terminal"); + } +} diff --git a/crates/mc-host/src/config.rs b/crates/mc-host/src/config.rs index 1f08558e7..5c173cfef 100644 --- a/crates/mc-host/src/config.rs +++ b/crates/mc-host/src/config.rs @@ -8,7 +8,11 @@ use std::path::PathBuf; use std::time::Duration; -use crate::wire::MAX_BODY_LEN; +use crate::auth::{ServerProof, MAX_AUTH_MESSAGE_LEN, NONCE_LEN, PROOF_LEN}; +use crate::connection_file::{ + ConnectionInfo, Endpoint, DAEMON_ID_LEN, KEY_LEN, MAX_CONNECTION_FILE_LEN, SCHEMA_VERSION, +}; +use crate::wire::{HEADER_LEN, MAX_BODY_LEN, PROTOCOL_VERSION}; /// One maximum inbound body, one maximum encoded outbound frame, and one /// maximum request-scratch reservation must coexist: a handler can stream @@ -21,8 +25,7 @@ pub const MIN_RESIDENT_BYTES: u64 = /// Capacity reserved exclusively for encoded output, preventing an admitted /// request from consuming the permits its own terminal needs. -pub(crate) const EGRESS_RESERVED_BYTES: u64 = - MAX_BODY_LEN as u64 + subc_protocol::HEADER_LEN as u64; +pub(crate) const EGRESS_RESERVED_BYTES: u64 = MAX_BODY_LEN as u64 + HEADER_LEN as u64; /// Capacity reserved exclusively for request scratch and request-derived /// ownership: parser transients, query text held by a worker, queued batch @@ -286,30 +289,30 @@ impl HostConfig { // the values ("0" is one char, "255" is three), so sizing must use // worst-case fills or a validated daemon_ver can exceed the caps at // runtime depending on the generated bytes. - let auth_message_bytes = serde_json::to_vec(&subc_transport::ServerProof { - daemon_id: [u8::MAX; subc_transport::DAEMON_ID_LEN], - server_nonce: [u8::MAX; subc_transport::NONCE_LEN], + let auth_message_bytes = serde_json::to_vec(&ServerProof { + daemon_id: [u8::MAX; DAEMON_ID_LEN], + server_nonce: [u8::MAX; NONCE_LEN], daemon_ver: self.daemon_ver.clone(), - server_proof: [u8::MAX; subc_transport::PROOF_LEN], + server_proof: [u8::MAX; PROOF_LEN], }) .expect("fixed auth shape serializes") .len(); - let connection_file_bytes = serde_json::to_vec_pretty(&subc_transport::ConnectionInfo { - schema: subc_transport::SCHEMA_VERSION, - wire_version: Some(subc_protocol::PROTOCOL_VERSION), - endpoints: vec![subc_transport::Endpoint { + let connection_file_bytes = serde_json::to_vec_pretty(&ConnectionInfo { + schema: SCHEMA_VERSION, + wire_version: PROTOCOL_VERSION, + endpoints: vec![Endpoint { host: "127.0.0.1".to_owned(), port: u16::MAX, }], - key: vec![u8::MAX; subc_transport::KEY_LEN], - daemon_id: [u8::MAX; subc_transport::DAEMON_ID_LEN], + key: vec![u8::MAX; KEY_LEN], + daemon_id: [u8::MAX; DAEMON_ID_LEN], pid: u32::MAX, daemon_ver: self.daemon_ver.clone(), }) .expect("fixed publication shape serializes") .len(); - if auth_message_bytes > subc_transport::MAX_AUTH_MESSAGE_LEN as usize - || connection_file_bytes > 65_536 + if auth_message_bytes > MAX_AUTH_MESSAGE_LEN as usize + || connection_file_bytes > MAX_CONNECTION_FILE_LEN { return Err(ConfigError::DaemonVerTooLarge { auth_message_bytes, diff --git a/crates/mc-host/src/connection.rs b/crates/mc-host/src/connection.rs index e7ce786d7..ed33e1e3d 100644 --- a/crates/mc-host/src/connection.rs +++ b/crates/mc-host/src/connection.rs @@ -9,7 +9,7 @@ use std::collections::HashMap; use std::sync::atomic::Ordering; use std::sync::{Arc, Mutex}; -use subc_protocol::FrameType; +use crate::wire::{FrameType, HEADER_LEN}; use tokio::net::TcpStream; use tokio::sync::OwnedSemaphorePermit; use tokio::time::{timeout_at, Instant}; @@ -145,7 +145,7 @@ pub async fn run_connection( handshake_permit: OwnedSemaphorePermit, ) { let _ = stream.set_nodelay(true); - let auth = subc_transport::authenticate_server( + let auth = crate::auth::authenticate_server( &mut stream, shared.auth_key.bytes(), &shared.daemon_id, @@ -426,11 +426,7 @@ async fn read_loop( return ReadExit::Peer; } watermark = corr; - // An over-cap channel-0 Request is still a consumer request: - // it commits the generation to TCP, and during candidate - // setup it is a protocol failure (protocol §7.7.5), exactly - // like the requests that pass the cap. - if commit_transport(setup).is_err() { + if !transport_ready(setup) { return ReadExit::Peer; } // Off-reader like the other rejections (contended egress @@ -482,11 +478,7 @@ async fn read_loop( if header.epoch == 0 { return ReadExit::Peer; } - // A routed request is application-bearing: it - // commits the generation to TCP, and during - // candidate setup it is a protocol failure - // (protocol §7.7.5). - if commit_transport(setup).is_err() { + if !transport_ready(setup) { return ReadExit::Peer; } dispatch_request(shared, gen, frame).await; @@ -498,7 +490,7 @@ async fn read_loop( if header.channel == 0 || header.epoch == 0 || header.corr == 0 { return ReadExit::Peer; } - if matches!(setup.state, TransportState::CandidateSetup) { + if !transport_ready(setup) { return ReadExit::Peer; } handle_cancel(gen, (header.channel, header.epoch, header.corr)); @@ -555,7 +547,7 @@ async fn read_loop( if header.epoch == 0 || header.corr != 0 { return ReadExit::Peer; } - if matches!(setup.state, TransportState::CandidateSetup) { + if !transport_ready(setup) { return ReadExit::Peer; } // The Closing transition is synchronous, so any later @@ -624,9 +616,10 @@ async fn handle_control( } action => action, }; - // Non-negotiation channel-0 requests commit the generation to TCP; - // during candidate setup they are a protocol failure (protocol §7.7.5). - if commit_transport(setup).is_err() { + if !matches!( + setup.state, + TransportState::TcpCommitted | TransportState::ProviderActive + ) { return ControlFlow::Close(ReadExit::Peer); } @@ -743,14 +736,8 @@ enum ControlFlow { /// [`run_candidate_setup`]; the bootstrap observes them all as /// [`TransportState::CandidateSetup`]. enum TransportState { - /// Authenticated bootstrap; nothing selected or committed yet. BootstrapTcp, - /// The generation is committed to TCP. `late_used` is false only while - /// the one allowed late negotiation (a first negotiation arriving after - /// a non-negotiation request implicitly committed TCP) remains - /// available; a negotiation that itself selects TCP consumes the - /// allowance, so any further negotiation is a protocol failure. - TcpCommitted { late_used: bool }, + TcpCommitted, /// A candidate is being activated; the bootstrap accepts no further /// requests. CandidateSetup, @@ -795,18 +782,11 @@ pub(crate) struct CandidateHandoff { io: Mutex>>, } -/// The first consumer request that is not `transport.negotiate` commits the -/// generation to TCP (protocol §7.7); during candidate setup any such -/// request is a protocol failure. -fn commit_transport(setup: &mut ConnectionSetup) -> Result<(), ()> { - match setup.state { - TransportState::BootstrapTcp => { - setup.state = TransportState::TcpCommitted { late_used: false }; - Ok(()) - } - TransportState::CandidateSetup => Err(()), - TransportState::TcpCommitted { .. } | TransportState::ProviderActive => Ok(()), - } +fn transport_ready(setup: &ConnectionSetup) -> bool { + matches!( + setup.state, + TransportState::TcpCommitted | TransportState::ProviderActive + ) } async fn handle_negotiate( @@ -816,16 +796,8 @@ async fn handle_negotiate( decoded: Result, setup: &mut ConnectionSetup, ) -> ControlFlow { - match setup.state { - // Selection is sticky (§7.7.5): a repeated negotiation, or any - // negotiation while a candidate is being set up or after promotion, - // is a protocol failure. - TransportState::CandidateSetup - | TransportState::ProviderActive - | TransportState::TcpCommitted { late_used: true } => { - return ControlFlow::Close(ReadExit::Peer); - } - TransportState::BootstrapTcp | TransportState::TcpCommitted { late_used: false } => {} + if !matches!(setup.state, TransportState::BootstrapTcp) { + return ControlFlow::Close(ReadExit::Peer); } let request = match decoded { Ok(request) => request, @@ -863,29 +835,8 @@ async fn handle_negotiate( }) { return ControlFlow::Close(ReadExit::Peer); } - if matches!(setup.state, TransportState::TcpCommitted { .. }) { - setup.state = TransportState::TcpCommitted { late_used: true }; - return respond_tcp( - shared, - gen, - corr, - request.negotiation_version, - Some(FallbackReason::ConnectionInUse), - ) - .await; - } if request.negotiation_version != NEGOTIATION_VERSION { - // Negotiation itself selected TCP, so the late-negotiation - // allowance (implicit-commit-first only, §7.7.5) is consumed. - setup.state = TransportState::TcpCommitted { late_used: true }; - return respond_tcp( - shared, - gen, - corr, - request.negotiation_version, - Some(FallbackReason::NegotiationVersionMismatch), - ) - .await; + return ControlFlow::Close(ReadExit::Peer); } // The first serveable offer in client preference order wins. let mut capability_mismatch = false; @@ -915,7 +866,6 @@ async fn handle_negotiate( shared, gen, corr, - request.negotiation_version, selected, provider, offer.parameters.clone(), @@ -938,9 +888,7 @@ async fn handle_negotiate( } else { None }; - // Negotiation itself selected TCP: the late-negotiation allowance is - // consumed, so a repeated negotiation is a protocol failure (§7.7.5). - setup.state = TransportState::TcpCommitted { late_used: true }; + setup.state = TransportState::TcpCommitted; respond_tcp(shared, gen, corr, request.negotiation_version, reason).await } @@ -985,7 +933,6 @@ async fn grant_candidate( shared: &Arc>, gen: &Arc, corr: u64, - negotiation_version: u32, selected: SelectedTransport, provider: Arc, offer_parameters: Option, @@ -1046,7 +993,7 @@ async fn grant_candidate( activation_token: token, descriptor, }, - negotiation_version, + NEGOTIATION_VERSION, TCP_CAPABILITY_VERSION, ) { Ok(body) => body, @@ -1213,7 +1160,7 @@ async fn send_candidate_response( written_tx: Option>, ) -> Result<(), ()> { let deadline = handoff.sender.admission_deadline(); - let frame_bytes = u32::try_from(body.len() + subc_protocol::HEADER_LEN).map_err(|_| ())?; + let frame_bytes = u32::try_from(body.len() + HEADER_LEN).map_err(|_| ())?; let charge = tokio::select! { biased; () = handoff.root.cancelled() => return Err(()), @@ -1276,10 +1223,7 @@ async fn reserve_catalog_frame( id: FrameId, body: &[u8], ) -> Result { - let frame_bytes = body - .len() - .checked_add(subc_protocol::HEADER_LEN) - .ok_or(())?; + let frame_bytes = body.len().checked_add(HEADER_LEN).ok_or(())?; let charged_bytes = u32::try_from(frame_bytes).map_err(|_| ())?; let charge = tokio::select! { biased; @@ -1459,19 +1403,21 @@ mod tests { use std::time::Duration; use tokio::io::{AsyncReadExt, DuplexStream}; + use crate::wire::decode_header; + #[derive(Clone, Copy)] enum FencedProducer { Catalog, CapacityRejection, } - async fn read_frame_from(stream: &mut DuplexStream) -> subc_protocol::EnvelopeHeader { - let mut header_bytes = [0; subc_protocol::HEADER_LEN]; + async fn read_frame_from(stream: &mut DuplexStream) -> crate::wire::EnvelopeHeader { + let mut header_bytes = [0; HEADER_LEN]; stream .read_exact(&mut header_bytes) .await .expect("frame header"); - let header = subc_protocol::decode_header(&header_bytes).expect("valid frame header"); + let header = decode_header(&header_bytes).expect("valid frame header"); let mut body = vec![0; header.len as usize]; stream.read_exact(&mut body).await.expect("frame body"); header @@ -1582,7 +1528,7 @@ mod tests { #[tokio::test] async fn cached_catalog_clone_holds_one_full_frame_charge() { let body = br#"{"op":"catalog.list","modules":[]}"#; - let frame_bytes = subc_protocol::HEADER_LEN + body.len(); + let frame_bytes = HEADER_LEN + body.len(); let budget = crate::wire::ByteBudget::new(frame_bytes as u64); let generation = CancellationToken::new(); let (server, client) = tokio::io::duplex(64); @@ -1618,7 +1564,7 @@ mod tests { .expect("catalog frame reservation"); assert_eq!(budget.available(), 0); - assert_eq!(&frame.bytes[subc_protocol::HEADER_LEN..], body); + assert_eq!(&frame.bytes[HEADER_LEN..], body); drop(frame); assert_eq!(budget.available(), frame_bytes); diff --git a/crates/mc-host/src/connection_file.rs b/crates/mc-host/src/connection_file.rs new file mode 100644 index 000000000..e3d02d37a --- /dev/null +++ b/crates/mc-host/src/connection_file.rs @@ -0,0 +1,382 @@ +//! Host-owned connection-file schema and secure discovery. +//! +//! Reads stay anchored to open directory and file descriptors. The path is +//! traversed without following links, the owner-only regular file is bounded +//! before JSON parsing, and a canonical-name replacement during the read is +//! rejected. Connection key bytes never appear in formatting or errors. + +use std::{ + error::Error, + ffi::OsString, + fmt, io, + path::{Component, Path, PathBuf}, +}; + +use rustix::{ + fd::OwnedFd, + fs::{openat, Mode, OFlags, CWD}, +}; +use serde::{Deserialize, Serialize}; + +use crate::{ + instance::{is_safe_ancestor, is_secure_regular, read_all_fd, S_IFDIR, S_IFMT}, + wire::PROTOCOL_VERSION, +}; + +pub const SCHEMA_VERSION: u32 = 1; +pub const MIN_KEY_LEN: usize = 32; +pub const KEY_LEN: usize = 32; +pub const DAEMON_ID_LEN: usize = 16; +pub const MAX_CONNECTION_FILE_LEN: usize = 65_536; + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Endpoint { + pub host: String, + pub port: u16, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ConnectionInfo { + pub schema: u32, + pub wire_version: u8, + pub endpoints: Vec, + pub key: Vec, + pub daemon_id: [u8; DAEMON_ID_LEN], + pub pid: u32, + pub daemon_ver: String, +} + +impl fmt::Debug for ConnectionInfo { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ConnectionInfo") + .field("schema", &self.schema) + .field("wire_version", &self.wire_version) + .field("endpoints", &self.endpoints) + .field("key", &format_args!("<{} bytes redacted>", self.key.len())) + .field("daemon_id", &self.daemon_id) + .field("pid", &self.pid) + .field("daemon_ver", &self.daemon_ver) + .finish() + } +} + +impl ConnectionInfo { + pub fn validate(&self) -> Result<(), ConnectionFileError> { + if self.schema != SCHEMA_VERSION { + return Err(ConnectionFileError::UnsupportedSchema { + schema: self.schema, + supported: SCHEMA_VERSION, + }); + } + if self.wire_version != PROTOCOL_VERSION { + return Err(ConnectionFileError::WireVersionMismatch { + file: self.wire_version, + supported: PROTOCOL_VERSION, + }); + } + let endpoint = self + .endpoints + .first() + .ok_or(ConnectionFileError::Invalid("missing endpoint"))?; + if endpoint.host != "127.0.0.1" || endpoint.port == 0 { + return Err(ConnectionFileError::Invalid("invalid loopback endpoint")); + } + if self.key.len() != KEY_LEN { + return Err(ConnectionFileError::InvalidKeyLength { + len: self.key.len(), + expected: KEY_LEN, + }); + } + if self.daemon_ver.is_empty() { + return Err(ConnectionFileError::Invalid("empty daemon version")); + } + Ok(()) + } +} + +#[derive(Debug)] +pub enum ConnectionFileError { + InvalidPath { + path: PathBuf, + }, + Io { + op: &'static str, + path: PathBuf, + source: io::Error, + }, + Insecure { + path: PathBuf, + }, + TooLarge { + path: PathBuf, + max: usize, + }, + Replaced { + path: PathBuf, + }, + Json { + path: PathBuf, + source: serde_json::Error, + }, + UnsupportedSchema { + schema: u32, + supported: u32, + }, + WireVersionMismatch { + file: u8, + supported: u8, + }, + Invalid(&'static str), + InvalidKeyLength { + len: usize, + expected: usize, + }, +} + +impl fmt::Display for ConnectionFileError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidPath { path } => { + write!(f, "invalid connection file path {}", path.display()) + } + Self::Io { op, path, source } => { + write!(f, "connection file {op} failed for {}: {source}", path.display()) + } + Self::Insecure { path } => write!( + f, + "refusing insecure connection file at {}: wrong type, owner, mode, or link count", + path.display() + ), + Self::TooLarge { path, max } => write!( + f, + "connection file {} exceeds {max} byte limit", + path.display() + ), + Self::Replaced { path } => { + write!(f, "connection file {} changed while reading", path.display()) + } + Self::Json { path, source } => write!( + f, + "connection file JSON read failed for {}: {source}", + path.display() + ), + Self::UnsupportedSchema { schema, supported } => write!( + f, + "unsupported connection file schema {schema}; expected {supported}" + ), + Self::WireVersionMismatch { file, supported } => write!( + f, + "connection file wire version {file} does not match supported wire version {supported}" + ), + Self::Invalid(reason) => write!(f, "invalid connection file: {reason}"), + Self::InvalidKeyLength { len, expected } => write!( + f, + "connection file key is {len} bytes; expected exactly {expected}" + ), + } + } +} + +impl Error for ConnectionFileError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Io { source, .. } => Some(source), + Self::Json { source, .. } => Some(source), + _ => None, + } + } +} + +/// Reads one secure connection-file snapshot. Validation completes before a +/// caller can use its endpoint, so invalid versions never reach a dial. +pub fn read_for_client(path: impl AsRef) -> Result { + let path = path.as_ref(); + let (parent, name) = open_parent(path)?; + let fd = open_file(&parent, &name, path)?; + let before = checked_stat(&fd, path)?; + if before.st_size < 0 || before.st_size as u64 > MAX_CONNECTION_FILE_LEN as u64 { + return Err(ConnectionFileError::TooLarge { + path: path.to_path_buf(), + max: MAX_CONNECTION_FILE_LEN, + }); + } + let bytes = read_all_fd(&fd, MAX_CONNECTION_FILE_LEN).map_err(|source| { + if source.kind() == io::ErrorKind::InvalidData { + ConnectionFileError::TooLarge { + path: path.to_path_buf(), + max: MAX_CONNECTION_FILE_LEN, + } + } else { + io_error("read", path, source) + } + })?; + let after = rustix::fs::fstat(&fd) + .map_err(|source| io_error("fstat_after_read", path, source.into()))?; + let current = open_file(&parent, &name, path)?; + let current = checked_stat(¤t, path)?; + if !same_snapshot(&before, &after) || !same_snapshot(&before, ¤t) { + return Err(ConnectionFileError::Replaced { + path: path.to_path_buf(), + }); + } + + let info = serde_json::from_slice::(&bytes).map_err(|source| { + ConnectionFileError::Json { + path: path.to_path_buf(), + source, + } + })?; + info.validate()?; + Ok(info) +} + +fn open_parent(path: &Path) -> Result<(OwnedFd, OsString), ConnectionFileError> { + let name = path + .file_name() + .filter(|name| !name.is_empty()) + .ok_or_else(|| ConnectionFileError::InvalidPath { + path: path.to_path_buf(), + })? + .to_os_string(); + let parent = path + .parent() + .ok_or_else(|| ConnectionFileError::InvalidPath { + path: path.to_path_buf(), + })?; + let flags = OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::RDONLY | OFlags::CLOEXEC; + let mut current = openat( + CWD, + if parent.is_absolute() { "/" } else { "." }, + flags, + Mode::empty(), + ) + .map_err(|source| io_error("open_anchor", path, source.into()))?; + validate_directory(¤t, path, false)?; + + for component in parent.components() { + let Component::Normal(component) = component else { + if matches!(component, Component::RootDir | Component::CurDir) { + continue; + } + return Err(ConnectionFileError::InvalidPath { + path: path.to_path_buf(), + }); + }; + current = openat(¤t, component, flags, Mode::empty()) + .map_err(|source| io_error("open_parent", path, source.into()))?; + validate_directory(¤t, path, false)?; + } + validate_directory(¤t, path, true)?; + Ok((current, name)) +} + +fn validate_directory( + fd: &OwnedFd, + path: &Path, + require_private: bool, +) -> Result<(), ConnectionFileError> { + let stat = + rustix::fs::fstat(fd).map_err(|source| io_error("fstat_parent", path, source.into()))?; + let directory = (stat.st_mode & S_IFMT) == S_IFDIR; + let private = stat.st_uid == rustix::process::geteuid().as_raw() && stat.st_mode & 0o077 == 0; + if !directory || !is_safe_ancestor(&stat) || (require_private && !private) { + return Err(ConnectionFileError::Insecure { + path: path.to_path_buf(), + }); + } + Ok(()) +} + +fn open_file( + parent: &OwnedFd, + name: &OsString, + path: &Path, +) -> Result { + openat( + parent, + name, + OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .map_err(|source| io_error("open", path, source.into())) +} + +fn checked_stat(fd: &OwnedFd, path: &Path) -> Result { + let stat = rustix::fs::fstat(fd).map_err(|source| io_error("fstat", path, source.into()))?; + if !is_secure_regular(&stat) { + return Err(ConnectionFileError::Insecure { + path: path.to_path_buf(), + }); + } + Ok(stat) +} + +fn same_snapshot(left: &rustix::fs::Stat, right: &rustix::fs::Stat) -> bool { + left.st_dev == right.st_dev + && left.st_ino == right.st_ino + && left.st_size == right.st_size + && left.st_mtime == right.st_mtime + && left.st_mtime_nsec == right.st_mtime_nsec + && left.st_ctime == right.st_ctime + && left.st_ctime_nsec == right.st_ctime_nsec +} + +fn io_error(op: &'static str, path: &Path, source: io::Error) -> ConnectionFileError { + ConnectionFileError::Io { + op, + path: path.to_path_buf(), + source, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn info() -> ConnectionInfo { + ConnectionInfo { + schema: SCHEMA_VERSION, + wire_version: PROTOCOL_VERSION, + endpoints: vec![Endpoint { + host: "127.0.0.1".to_owned(), + port: 1, + }], + key: vec![7; KEY_LEN], + daemon_id: [8; DAEMON_ID_LEN], + pid: 9, + daemon_ver: "test".to_owned(), + } + } + + #[test] + fn strict_wire_version_rejects_missing_null_string_and_other() { + let valid = serde_json::to_value(info()).expect("serialize"); + for version in [ + None, + Some(serde_json::Value::Null), + Some(serde_json::json!("2")), + Some(serde_json::json!(1)), + ] { + let mut candidate = valid.clone(); + let object = candidate.as_object_mut().expect("object"); + match version { + Some(value) => { + object.insert("wire_version".to_owned(), value); + } + None => { + object.remove("wire_version"); + } + } + assert!(serde_json::from_value::(candidate) + .and_then(|info| info.validate().map_err(serde::de::Error::custom)) + .is_err()); + } + } + + #[test] + fn debug_redacts_key() { + let rendered = format!("{:?}", info()); + assert!(rendered.contains("redacted")); + assert!(!rendered.contains("7, 7")); + } +} diff --git a/crates/mc-host/src/control.rs b/crates/mc-host/src/control.rs index 1de2e1b2c..a09ae7282 100644 --- a/crates/mc-host/src/control.rs +++ b/crates/mc-host/src/control.rs @@ -18,9 +18,8 @@ pub const CODE_SERVER_BUSY: &str = "server_busy"; pub const CODE_CANCELLED: &str = "cancelled"; pub const CODE_INTERNAL_ERROR: &str = "internal_error"; -pub const OP_ROUTE_OPEN: &str = subc_control::ops::ROUTE_OPEN; -pub const OP_CATALOG_LIST: &str = subc_control::ops::CATALOG_LIST; -/// `subc_control` does not publish this operation. +pub const OP_ROUTE_OPEN: &str = "route.open"; +pub const OP_CATALOG_LIST: &str = "catalog.list"; pub const OP_HOST_SHUTDOWN: &str = "host.shutdown"; pub const OP_TRANSPORT_NEGOTIATE: &str = crate::transport_negotiation::OP_TRANSPORT_NEGOTIATE; @@ -581,10 +580,18 @@ fn serialize_catalog_response( Ok(writer.buf.into_boxed_slice()) } -/// Tagged `route.open` success response, built from the published control -/// shape so its tag and field names cannot drift. +#[derive(serde::Serialize)] +#[serde(tag = "op")] +enum ClientControlResponse { + #[serde(rename = "route.open")] + RouteOpen { + route_channel: u16, + route_epoch: u32, + }, +} + pub fn route_open_response_json(channel: u16, epoch: u32) -> Vec { - serde_json::to_vec(&subc_control::ClientControlResponse::RouteOpen { + serde_json::to_vec(&ClientControlResponse::RouteOpen { route_channel: channel, route_epoch: epoch, }) diff --git a/crates/mc-host/src/dispatch.rs b/crates/mc-host/src/dispatch.rs index 1b2950567..bf55581c0 100644 --- a/crates/mc-host/src/dispatch.rs +++ b/crates/mc-host/src/dispatch.rs @@ -9,7 +9,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use subc_protocol::FrameType; +use crate::wire::{Flags, FrameType, HEADER_LEN}; use tokio::sync::oneshot; use tokio::time::{timeout, timeout_at, Instant}; use tokio_util::sync::CancellationToken; @@ -125,7 +125,7 @@ async fn charged_error_body( if gen.writer.is_retired() || gen.token.is_cancelled() { return Err(()); } - let frame_bytes = u32::try_from(body_len + subc_protocol::HEADER_LEN).map_err(|_| ())?; + let frame_bytes = u32::try_from(body_len + HEADER_LEN).map_err(|_| ())?; let deadline = gen.writer.admission_deadline(); let charge = tokio::select! { biased; @@ -142,7 +142,7 @@ async fn charged_error_body( // Header spare capacity up front: an exactly sized buffer would force // `encode_owned_frame`'s reserve to reallocate, transiently retaining // two near-maximum bodies against one charge. - Vec::with_capacity(body_len + subc_protocol::HEADER_LEN), + Vec::with_capacity(body_len + HEADER_LEN), code, message, ); @@ -184,7 +184,7 @@ pub async fn emit_frame( budget: &crate::wire::ByteBudget, gen: &GenerationCore, ty: FrameType, - flags: subc_protocol::Flags, + flags: Flags, id: FrameId, body: Vec, ) -> Result<(), ()> { @@ -198,7 +198,7 @@ pub async fn emit_frame( let charge = if body.is_empty() { crate::wire::ByteCharge::none() } else { - let frame_bytes = u32::try_from(body.len() + subc_protocol::HEADER_LEN).map_err(|_| ())?; + let frame_bytes = u32::try_from(body.len() + HEADER_LEN).map_err(|_| ())?; tokio::select! { biased; () = gen.token.cancelled() => return Err(()), @@ -236,7 +236,7 @@ pub async fn emit_frame( async fn emit_reserved_frame( gen: &GenerationCore, ty: FrameType, - flags: subc_protocol::Flags, + flags: Flags, id: FrameId, body: OutputBuffer, deadline: Instant, @@ -413,7 +413,7 @@ impl StreamSink { { return Err(StreamClosed); } - let bytes = u32::try_from(max_len + subc_protocol::HEADER_LEN).map_err(|_| StreamClosed)?; + let bytes = u32::try_from(max_len + HEADER_LEN).map_err(|_| StreamClosed)?; let deadline = self.gen.writer.admission_deadline(); let charge = tokio::select! { biased; @@ -434,7 +434,7 @@ impl StreamSink { return Err(StreamClosed); } Ok(OutputBuffer { - body: Vec::with_capacity(max_len + subc_protocol::HEADER_LEN), + body: Vec::with_capacity(max_len + HEADER_LEN), charge, max_len, }) @@ -567,8 +567,7 @@ pub async fn handle_host_shutdown( return; } let deadline = gen.writer.admission_deadline(); - let frame_bytes = - u32::try_from(body.len() + subc_protocol::HEADER_LEN).expect("fixed-size body"); + let frame_bytes = u32::try_from(body.len() + HEADER_LEN).expect("fixed-size body"); let charge = tokio::select! { biased; () = gen.token.cancelled() => return, diff --git a/crates/mc-host/src/frame_channel.rs b/crates/mc-host/src/frame_channel.rs index 6dbd9f823..2586e1225 100644 --- a/crates/mc-host/src/frame_channel.rs +++ b/crates/mc-host/src/frame_channel.rs @@ -19,8 +19,9 @@ use std::future::Future; -use subc_protocol::EnvelopeHeader; use tokio::sync::mpsc; + +use crate::wire::EnvelopeHeader; use tokio::time::{timeout_at, Duration, Instant}; use tokio_util::sync::CancellationToken; diff --git a/crates/mc-host/src/frame_channel/contract_tests.rs b/crates/mc-host/src/frame_channel/contract_tests.rs index b4dc45d48..5a9761190 100644 --- a/crates/mc-host/src/frame_channel/contract_tests.rs +++ b/crates/mc-host/src/frame_channel/contract_tests.rs @@ -11,8 +11,9 @@ use std::future::Future; use std::sync::{Arc, Mutex}; -use subc_protocol::{EnvelopeHeader, Flags, FrameType}; use tokio::time::{Duration, Instant}; + +use crate::wire::{EnvelopeHeader, Flags, FrameType}; use tokio_util::sync::CancellationToken; use crate::frame_channel::{FrameReceiver, FrameSender, InboundEvent, OutboundFrame, ReadClose}; diff --git a/crates/mc-host/src/instance.rs b/crates/mc-host/src/instance.rs index a8f8d15d9..2c8a79471 100644 --- a/crates/mc-host/src/instance.rs +++ b/crates/mc-host/src/instance.rs @@ -16,7 +16,7 @@ use rustix::fs::{ flock, fsync, mkdirat, openat, renameat, unlinkat, AtFlags, FlockOperation, Mode, OFlags, CWD, }; -use subc_transport::{ConnectionInfo, Endpoint, DAEMON_ID_LEN, KEY_LEN, SCHEMA_VERSION}; +use crate::connection_file::{ConnectionInfo, Endpoint, DAEMON_ID_LEN, KEY_LEN, SCHEMA_VERSION}; /// Canonical publication name inside the runtime directory (protocol §4.1). pub const CONNECTION_FILE_NAME: &str = "subc-connection.json"; @@ -216,7 +216,7 @@ impl InstanceGuard { pub fn publish(&mut self, port: u16, daemon_ver: &str) -> Result<(), InstanceError> { let info = ConnectionInfo { schema: SCHEMA_VERSION, - wire_version: Some(subc_protocol::PROTOCOL_VERSION), + wire_version: crate::wire::PROTOCOL_VERSION, endpoints: vec![Endpoint { host: "127.0.0.1".to_owned(), port, diff --git a/crates/mc-host/src/lib.rs b/crates/mc-host/src/lib.rs index 2a6227e0c..f05a3b31d 100644 --- a/crates/mc-host/src/lib.rs +++ b/crates/mc-host/src/lib.rs @@ -7,9 +7,12 @@ // scoped `allow` and a safety justification. #![deny(unsafe_code)] +pub mod auth; pub mod broca; +pub mod client; pub mod composite; pub mod config; +pub mod connection_file; pub mod handler; pub mod lifecycle; pub mod synapse; @@ -29,10 +32,27 @@ mod runtime; mod tcp_frame_channel; mod wire; +pub use auth::{ + authenticate_client, authenticate_client_with_role, authenticate_server, compute_proof, + AuthError, AuthStage, Authenticated, ClientAuth, ClientHello, ServerProof, CLIENT_AUTH_DOMAIN, + DEFAULT_CLIENT_ROLE, MAX_AUTH_MESSAGE_LEN, NONCE_LEN, PROOF_LEN, SERVER_PROOF_DOMAIN, + WATCHDOG_CLIENT_ROLE, +}; +pub use client::{ + CallError, Client, ClientError, RequestOptions, Response, ResponseStream, SendOutcome, + StreamItem, CLIENT_CONTROL_QUEUE_FRAMES, CLIENT_DATA_QUEUE_FRAMES, CLIENT_FRAME_TIMEOUT, + CLIENT_HANDSHAKE_TIMEOUT, CLIENT_MAX_LIVE_STREAMS, CLIENT_MAX_PENDING_REQUESTS, + CLIENT_QUEUED_BYTES, CLIENT_REQUEST_TIMEOUT, CLIENT_RETAINED_RESPONSE_BYTES, + CLIENT_ROUTE_OPEN_TIMEOUT, CLIENT_SHUTDOWN_TIMEOUT, CLIENT_STREAM_QUEUE_ITEMS, +}; pub use composite::{ CompositeComponent, PrimaryComponent, SecondaryComponent, ShutdownError, StaticComposite, }; pub use config::{ConfigError, HostConfig, HostInit, HostLimits, HostTiming, LivenessPolicy}; +pub use connection_file::{ + read_for_client as read_connection_file, ConnectionFileError, ConnectionInfo, Endpoint, + DAEMON_ID_LEN, KEY_LEN, MAX_CONNECTION_FILE_LEN, MIN_KEY_LEN, SCHEMA_VERSION, +}; pub use handler::{ BindOutcome, HealthReport, HealthStatus, InitError, ManifestSnapshot, McHostHandler, OutputBuffer, RequestCtx, RequestOutcome, ResourceDeclaration, RouteClass, RouteHandle, diff --git a/crates/mc-host/src/lifecycle.rs b/crates/mc-host/src/lifecycle.rs index bc8f5e59b..fb76737fe 100644 --- a/crates/mc-host/src/lifecycle.rs +++ b/crates/mc-host/src/lifecycle.rs @@ -13,7 +13,7 @@ use std::time::{Duration, SystemTime}; use rustix::fd::OwnedFd; use rustix::fs::{flock, openat, unlinkat, AtFlags, FlockOperation, Mode, OFlags}; -use subc_transport::ConnectionInfo; +use crate::connection_file::{ConnectionInfo, KEY_LEN}; use crate::instance::{ flock_bounded, flock_exclusive_bounded, hex, io_err, is_safe_ancestor, is_secure_regular, @@ -507,16 +507,12 @@ fn instance_lock_free(dir: &OwnedFd, dir_path: &Path) -> Result Option { let info: ConnectionInfo = serde_json::from_slice(bytes).ok()?; info.validate().ok()?; - info.validate_wire_version(subc_protocol::PROTOCOL_VERSION) - .ok()?; + let endpoint = info.endpoints.first()?; - // `validate()` accepts any key of at least the minimum length; this - // profile and its clients (protocol §4.1) require exactly KEY_LEN bytes, - // so a longer key is a publication no conforming client would accept. if endpoint.host != "127.0.0.1" || endpoint.port == 0 || info.daemon_ver.is_empty() - || info.key.len() != subc_transport::KEY_LEN + || info.key.len() != KEY_LEN { return None; } diff --git a/crates/mc-host/src/tcp_frame_channel.rs b/crates/mc-host/src/tcp_frame_channel.rs index 4a3be92db..4302331d9 100644 --- a/crates/mc-host/src/tcp_frame_channel.rs +++ b/crates/mc-host/src/tcp_frame_channel.rs @@ -12,9 +12,11 @@ //! frame; semantic rejection with trustworthy identity flows through the //! settlement path in `dispatch` instead (protocol §6.3). -use subc_protocol::{ +#[cfg(test)] +use crate::wire::Flags; +use crate::wire::{ decode_header, AdmissionClass, DecodeError, EnvelopeHeader, FrameType, FROZEN_PREFIX_LEN, - HEADER_LEN, + HEADER_LEN, PROTOCOL_VERSION, }; use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader}; use tokio::time::{timeout_at, Duration, Instant}; @@ -69,7 +71,7 @@ impl TcpFrameChannel { ) -> ( FrameSender, Self, - impl std::future::Future + Send, + std::pin::Pin + Send>>, ) where W: AsyncWrite + Send + Unpin + 'static, @@ -82,7 +84,7 @@ impl TcpFrameChannel { cancel: read_cancel, pending_drain: None, }; - let task = write_frames(write, queue, frame_deadline); + let task = Box::pin(write_frames(write, queue, frame_deadline)); (sender, receiver, task) } } @@ -167,7 +169,7 @@ where cancel, ) .await?; - if header_bytes[4] != subc_protocol::PROTOCOL_VERSION { + if header_bytes[4] != PROTOCOL_VERSION { return Err(ReadClose::Corrupt("unsupported version")); } read_exact_deadline( @@ -474,7 +476,7 @@ impl crate::frame_channel::contract_tests::PeerDriver for TcpPeer { async fn send_frame( &mut self, ty: FrameType, - flags: subc_protocol::Flags, + flags: Flags, id: crate::wire::FrameId, body: Vec, ) { @@ -514,7 +516,6 @@ impl crate::frame_channel::contract_tests::PeerDriver for TcpPeer { mod tests { use super::*; use std::io; - use subc_protocol::PROTOCOL_VERSION; use tokio::io::{duplex, AsyncWriteExt}; use crate::wire::{encode_frame, response_flags, FrameId}; diff --git a/crates/mc-host/src/transport_negotiation.rs b/crates/mc-host/src/transport_negotiation.rs index 77dd3c94c..c9fded47b 100644 --- a/crates/mc-host/src/transport_negotiation.rs +++ b/crates/mc-host/src/transport_negotiation.rs @@ -116,7 +116,6 @@ pub enum FallbackReason { Unavailable, NegotiationVersionMismatch, CapabilityVersionMismatch, - ConnectionInUse, } impl FallbackReason { @@ -125,7 +124,6 @@ impl FallbackReason { Self::Unavailable => "unavailable", Self::NegotiationVersionMismatch => "negotiation_version_mismatch", Self::CapabilityVersionMismatch => "capability_version_mismatch", - Self::ConnectionInUse => "connection_in_use", } } @@ -134,7 +132,6 @@ impl FallbackReason { "unavailable" => Some(Self::Unavailable), "negotiation_version_mismatch" => Some(Self::NegotiationVersionMismatch), "capability_version_mismatch" => Some(Self::CapabilityVersionMismatch), - "connection_in_use" => Some(Self::ConnectionInUse), _ => None, } } diff --git a/crates/mc-host/src/transport_provider.rs b/crates/mc-host/src/transport_provider.rs index db0f31419..51b11228e 100644 --- a/crates/mc-host/src/transport_provider.rs +++ b/crates/mc-host/src/transport_provider.rs @@ -146,7 +146,7 @@ pub struct TransportProviders { } impl TransportProviders { - /// Test seam: a registry with injected providers beside implicit TCP. + /// The registry retains built-in TCP when tests inject providers. /// Provider-authored `transport()` and `capability_version()` run once, /// here: negotiation lookups on a connection's read loop touch only the /// snapshot, so a slow or blocking metadata method cannot stall reads diff --git a/crates/mc-host/src/wire.rs b/crates/mc-host/src/wire.rs index 34fe9c790..9bbd75b3e 100644 --- a/crates/mc-host/src/wire.rs +++ b/crates/mc-host/src/wire.rs @@ -1,11 +1,360 @@ //! The connection engine and its transports share frame encoding, protocol //! size caps, and aggregate resident-byte accounting. +//! +//! ```text +//! offset size field type purpose +//! 0 4 len u32 # of BODY bytes after this 21-byte header +//! 4 1 ver u8 envelope version +//! 5 1 type u8 frame kind (see FrameType) +//! 6 1 flags u8 bit0 BINARY · bits1-2 PRIORITY · bit3 LAST · bits4-5 ADMISSION · bits6-7 reserved +//! 7 2 channel u16 route slot; 0 = the host itself +//! 9 4 epoch u32 per-slot binding epoch; 0 on channel 0 +//! 13 8 corr u64 correlation id; CANCEL carries the target call's corr +//! 21 -> body +//! ``` +//! +//! Little-endian. **Frozen prefix:** `len` (u32 @ 0) and `ver` (u8 @ 4) keep +//! fixed meaning and position in every future version; `decode_header` +//! enforces that discipline. + +use std::{error::Error, fmt, sync::Arc}; -use std::sync::Arc; - -use subc_protocol::{EnvelopeHeader, Flags, FrameType, Priority, HEADER_LEN, MAX_FRAME_BODY_LEN}; use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +/// Envelope protocol version this build speaks. +pub const PROTOCOL_VERSION: u8 = 2; + +/// Fixed header length for `PROTOCOL_VERSION` 2. +pub const HEADER_LEN: usize = 21; + +/// Bytes of the frozen prefix (`len` u32 + `ver` u8) that are stable across +/// every envelope version. +pub const FROZEN_PREFIX_LEN: usize = 5; + +/// Maximum frame body accepted before allocation (64 MiB). +pub const MAX_FRAME_BODY_LEN: u32 = 64 * 1024 * 1024; + +/// Env var naming the module id a supervised child registers under. +/// Canonical version-2 vocabulary (KTD8): the name is protocol surface and +/// must stay byte-identical. +pub const SUBC_MODULE_ID_ENV: &str = "SUBC_MODULE_ID"; + +/// Env var carrying the one-time launch nonce injected into a spawned +/// reserved module. Canonical version-2 vocabulary (KTD8). +pub const SUBC_LAUNCH_NONCE_ENV: &str = "SUBC_LAUNCH_NONCE"; + +/// Frame kind (`type` byte at offset 5). +/// +/// `CANCEL`, `PING`, `PONG`, and `GOODBYE` are pure-header frames (`len == 0`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +pub enum FrameType { + Request = 0, + Response = 1, + Push = 2, + StreamData = 3, + StreamEnd = 4, + Error = 5, + Cancel = 6, + Ping = 7, + Pong = 8, + Hello = 9, + HelloAck = 10, + Goodbye = 11, +} + +impl FrameType { + /// Map the raw `type` byte to a `FrameType`, or `None` if unknown. + pub fn from_u8(b: u8) -> Option { + Some(match b { + 0 => Self::Request, + 1 => Self::Response, + 2 => Self::Push, + 3 => Self::StreamData, + 4 => Self::StreamEnd, + 5 => Self::Error, + 6 => Self::Cancel, + 7 => Self::Ping, + 8 => Self::Pong, + 9 => Self::Hello, + 10 => Self::HelloAck, + 11 => Self::Goodbye, + _ => return None, + }) + } + + pub fn is_pure_header(self) -> bool { + matches!(self, Self::Cancel | Self::Ping | Self::Pong | Self::Goodbye) + } +} + +/// Scheduling priority carried in `flags` bits 1-2. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +pub enum Priority { + Passive = 0, + Interactive = 1, + Background = 2, +} + +impl Priority { + fn from_bits(bits: u8) -> Option { + Some(match bits { + 0 => Self::Passive, + 1 => Self::Interactive, + 2 => Self::Background, + _ => return None, + }) + } +} + +/// Admission behavior carried in `flags` bits 4-5. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +pub enum AdmissionClass { + Normal = 0, + Expedite = 1, + Sheddable = 2, +} + +impl AdmissionClass { + fn from_bits(bits: u8) -> Option { + Some(match bits { + 0 => Self::Normal, + 1 => Self::Expedite, + 2 => Self::Sheddable, + _ => return None, + }) + } +} + +const FLAG_BINARY: u8 = 0b0000_0001; // bit 0 +const FLAG_PRIORITY_MASK: u8 = 0b0000_0110; // bits 1-2 +const FLAG_PRIORITY_SHIFT: u8 = 1; +const FLAG_LAST: u8 = 0b0000_1000; // bit 3 +const FLAG_ADMISSION_MASK: u8 = 0b0011_0000; // bits 4-5 +const FLAG_ADMISSION_SHIFT: u8 = 4; +const FLAG_RESERVED_MASK: u8 = 0b1100_0000; // bits 6-7 must be zero + +/// The `flags` byte (offset 6): binary, priority, last, admission, then +/// reserved bits. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Flags(pub u8); + +impl Flags { + /// Build flags with the default [`AdmissionClass::Normal`] class. + pub fn new(binary: bool, priority: Priority, last: bool) -> Self { + let mut b = 0u8; + if binary { + b |= FLAG_BINARY; + } + b |= (priority as u8) << FLAG_PRIORITY_SHIFT; + if last { + b |= FLAG_LAST; + } + Flags(b) + } + + /// Body is raw bytes (bulk lane) rather than JSON-RPC. + pub fn is_binary(self) -> bool { + self.0 & FLAG_BINARY != 0 + } + + /// Final frame of a streamed message. + pub fn is_last(self) -> bool { + self.0 & FLAG_LAST != 0 + } + + /// Decode the priority bits, or `None` if they hold a reserved value. + pub fn priority(self) -> Option { + Priority::from_bits((self.0 & FLAG_PRIORITY_MASK) >> FLAG_PRIORITY_SHIFT) + } + + /// Decode the admission-class bits, or `None` if they hold `0b11`. + pub fn admission_class(self) -> Option { + AdmissionClass::from_bits((self.0 & FLAG_ADMISSION_MASK) >> FLAG_ADMISSION_SHIFT) + } + + /// True if either reserved bit (6-7) is set. + pub fn has_reserved_bits(self) -> bool { + self.0 & FLAG_RESERVED_MASK != 0 + } +} + +/// A decoded envelope header. The body is the `len` bytes that follow it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct EnvelopeHeader { + /// Number of body bytes after the header. + pub len: u32, + /// Envelope version. + pub ver: u8, + /// Frame kind. + pub ty: FrameType, + /// Flag bits. + pub flags: Flags, + /// Sender-local route slot; 0 is the control channel. + pub channel: u16, + /// Sender-local binding epoch; 0 is reserved for the control channel. + pub epoch: u32, + /// Correlation id. + pub corr: u64, +} + +impl EnvelopeHeader { + /// Serialize the header to its fixed 21-byte little-endian form. + pub fn encode(&self) -> [u8; HEADER_LEN] { + let mut buf = [0u8; HEADER_LEN]; + buf[0..4].copy_from_slice(&self.len.to_le_bytes()); + buf[4] = self.ver; + buf[5] = self.ty as u8; + buf[6] = self.flags.0; + buf[7..9].copy_from_slice(&self.channel.to_le_bytes()); + buf[9..13].copy_from_slice(&self.epoch.to_le_bytes()); + buf[13..21].copy_from_slice(&self.corr.to_le_bytes()); + buf + } +} + +/// Why a header could not be decoded. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DecodeError { + /// Fewer than `FROZEN_PREFIX_LEN` bytes — cannot even read `len`/`ver`. + TooShortForPrefix { have: usize }, + /// `ver` is not a version this build understands. + UnsupportedVersion { ver: u8 }, + /// Version known but fewer than its header length is present. + TooShortForHeader { have: usize, need: usize }, + /// `type` byte is not a known `FrameType`. + UnknownFrameType { byte: u8 }, + /// A reserved flag bit (6-7) is set. + ReservedFlagBits { flags: u8 }, + /// Priority bits 1-2 hold the reserved value `0b11`. + ReservedPriorityBits { flags: u8 }, + /// Admission bits 4-5 hold the reserved value `0b11`. + ReservedAdmissionClass { flags: u8 }, + /// SHEDDABLE is set on a frame type that must be delivered. + SheddableIllegalFrameType { ty: FrameType, flags: u8 }, + /// Channel 0 carried an epoch other than its reserved epoch 0. + NonzeroEpochOnControlChannel { epoch: u32 }, + /// A pure-header frame declared body bytes. + PureHeaderFrameWithBody { ty: FrameType, len: u32 }, +} + +impl fmt::Display for DecodeError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::TooShortForPrefix { have } => { + write!(f, "header shorter than frozen prefix: have {have} bytes") + } + Self::UnsupportedVersion { ver } => write!(f, "unsupported envelope version {ver}"), + Self::TooShortForHeader { have, need } => { + write!( + f, + "header too short for version: have {have} bytes, need {need}" + ) + } + Self::UnknownFrameType { byte } => write!(f, "unknown frame type byte {byte}"), + Self::ReservedFlagBits { flags } => { + write!(f, "reserved flag bits set in flags 0b{flags:08b}") + } + Self::ReservedPriorityBits { flags } => { + write!(f, "reserved priority bits set in flags 0b{flags:08b}") + } + Self::ReservedAdmissionClass { flags } => { + write!(f, "reserved admission class set in flags 0b{flags:08b}") + } + Self::SheddableIllegalFrameType { ty, flags } => write!( + f, + "SHEDDABLE admission class is illegal on {ty:?} in flags 0b{flags:08b}" + ), + Self::NonzeroEpochOnControlChannel { epoch } => { + write!(f, "control channel carried nonzero epoch {epoch}") + } + Self::PureHeaderFrameWithBody { ty, len } => { + write!( + f, + "pure-header frame {ty:?} declared non-zero body length {len}" + ) + } + } + } +} + +impl Error for DecodeError {} + +/// How many header bytes a given envelope version occupies. Driven by the +/// frozen prefix: read `ver`, then learn the full header length here. +fn header_len_for_version(ver: u8) -> Option { + match ver { + PROTOCOL_VERSION => Some(HEADER_LEN), + _ => None, + } +} + +/// Decode an envelope header from the front of `bytes`, following the +/// frozen-prefix discipline: +/// 1. need at least the 5-byte prefix to read `len` + `ver`; +/// 2. dispatch the full header length on `ver`; +/// 3. need the full header present; then parse the rest. +/// +/// Never panics on malformed input — returns a typed [`DecodeError`]. +pub fn decode_header(bytes: &[u8]) -> Result { + if bytes.len() < FROZEN_PREFIX_LEN { + return Err(DecodeError::TooShortForPrefix { have: bytes.len() }); + } + let ver = bytes[4]; + let need = header_len_for_version(ver).ok_or(DecodeError::UnsupportedVersion { ver })?; + if bytes.len() < need { + return Err(DecodeError::TooShortForHeader { + have: bytes.len(), + need, + }); + } + + let len = u32::from_le_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]); + let ty = + FrameType::from_u8(bytes[5]).ok_or(DecodeError::UnknownFrameType { byte: bytes[5] })?; + let flags = Flags(bytes[6]); + if flags.has_reserved_bits() { + return Err(DecodeError::ReservedFlagBits { flags: bytes[6] }); + } + if flags.priority().is_none() { + return Err(DecodeError::ReservedPriorityBits { flags: bytes[6] }); + } + let admission_class = flags + .admission_class() + .ok_or(DecodeError::ReservedAdmissionClass { flags: bytes[6] })?; + if admission_class == AdmissionClass::Sheddable + && !matches!(ty, FrameType::Push | FrameType::StreamData) + { + return Err(DecodeError::SheddableIllegalFrameType { + ty, + flags: bytes[6], + }); + } + if ty.is_pure_header() && len != 0 { + return Err(DecodeError::PureHeaderFrameWithBody { ty, len }); + } + let channel = u16::from_le_bytes([bytes[7], bytes[8]]); + let epoch = u32::from_le_bytes([bytes[9], bytes[10], bytes[11], bytes[12]]); + if channel == 0 && epoch != 0 { + return Err(DecodeError::NonzeroEpochOnControlChannel { epoch }); + } + let corr = u64::from_le_bytes([ + bytes[13], bytes[14], bytes[15], bytes[16], bytes[17], bytes[18], bytes[19], bytes[20], + ]); + + Ok(EnvelopeHeader { + len, + ver, + ty, + flags, + channel, + epoch, + corr, + }) +} + /// Interoperability body maximum: exactly 64 MiB (protocol §6.3). pub const MAX_BODY_LEN: u32 = MAX_FRAME_BODY_LEN; @@ -194,7 +543,7 @@ pub fn encode_frame( })?; let header = EnvelopeHeader { len, - ver: subc_protocol::PROTOCOL_VERSION, + ver: PROTOCOL_VERSION, ty, flags, channel: id.channel, @@ -222,7 +571,7 @@ pub fn encode_owned_frame( let len = u32::try_from(body_len).map_err(|_| EncodeError { body_len })?; let header = EnvelopeHeader { len, - ver: subc_protocol::PROTOCOL_VERSION, + ver: PROTOCOL_VERSION, ty, flags, channel: id.channel, @@ -260,7 +609,7 @@ pub fn encode_split_frame( let len = u32::try_from(body_len).map_err(|_| EncodeError { body_len })?; let header = EnvelopeHeader { len, - ver: subc_protocol::PROTOCOL_VERSION, + ver: PROTOCOL_VERSION, ty, flags, channel: id.channel, @@ -286,6 +635,206 @@ pub fn pure_header_flags() -> Flags { mod tests { use super::*; + fn hdr(len: u32, ty: FrameType, flags: Flags, channel: u16, corr: u64) -> EnvelopeHeader { + hdr_with_epoch(len, ty, flags, channel, u32::from(channel != 0), corr) + } + + fn hdr_with_epoch( + len: u32, + ty: FrameType, + flags: Flags, + channel: u16, + epoch: u32, + corr: u64, + ) -> EnvelopeHeader { + EnvelopeHeader { + len, + ver: PROTOCOL_VERSION, + ty, + flags, + channel, + epoch, + corr, + } + } + + #[test] + fn canonical_env_names_are_pinned() { + assert_eq!(SUBC_MODULE_ID_ENV, "SUBC_MODULE_ID"); + assert_eq!(SUBC_LAUNCH_NONCE_ENV, "SUBC_LAUNCH_NONCE"); + } + + #[test] + fn round_trip_request() { + let h = hdr( + 1234, + FrameType::Request, + Flags::new(false, Priority::Interactive, false), + 42, + 0xDEAD_BEEF_0000_0001, + ); + let decoded = decode_header(&h.encode()).unwrap(); + assert_eq!(h, decoded); + } + + #[test] + fn round_trip_all_frame_types() { + for b in 0u8..=11 { + let ty = FrameType::from_u8(b).unwrap(); + let h = hdr(0, ty, Flags::new(false, Priority::Passive, false), 0, 0); + assert_eq!(decode_header(&h.encode()).unwrap().ty, ty); + } + assert_eq!(FrameType::from_u8(12), None); + } + + #[test] + fn little_endian_and_frozen_prefix_layout() { + let h = hdr_with_epoch( + 0x0403_0201, + FrameType::Request, + Flags(0), + 0x0605, + 0x0a09_0807, + 0x1211_100f_0e0d_0c0b, + ); + let buf = h.encode(); + assert_eq!(&buf[0..4], &[1, 2, 3, 4]); + assert_eq!(buf[4], PROTOCOL_VERSION); + assert_eq!(&buf[7..9], &[5, 6]); + assert_eq!(&buf[9..13], &[7, 8, 9, 10]); + assert_eq!(&buf[13..21], &[11, 12, 13, 14, 15, 16, 17, 18]); + assert_eq!(buf.len(), HEADER_LEN); + } + + #[test] + fn reject_truncated_headers_and_unsupported_versions() { + assert_eq!( + decode_header(&[0, 0, 0, 0]), + Err(DecodeError::TooShortForPrefix { have: 4 }) + ); + let mut b = [0u8; 10]; + b[4] = PROTOCOL_VERSION; + assert_eq!( + decode_header(&b), + Err(DecodeError::TooShortForHeader { + have: 10, + need: HEADER_LEN + }) + ); + let mut b = [0u8; HEADER_LEN]; + b[4] = 1; + assert_eq!( + decode_header(&b), + Err(DecodeError::UnsupportedVersion { ver: 1 }) + ); + } + + #[test] + fn reject_unknown_frame_type_and_reserved_flag_encodings() { + let mut b = [0u8; HEADER_LEN]; + b[4] = PROTOCOL_VERSION; + b[5] = 99; + assert_eq!( + decode_header(&b), + Err(DecodeError::UnknownFrameType { byte: 99 }) + ); + + let mut b = [0u8; HEADER_LEN]; + b[4] = PROTOCOL_VERSION; + b[5] = FrameType::Request as u8; + b[6] = 0b1000_0000; // reserved bit 7 set + assert_eq!( + decode_header(&b), + Err(DecodeError::ReservedFlagBits { flags: 0b1000_0000 }) + ); + + b[6] = 0b0000_0110; // priority bits 1-2 hold reserved 0b11 + assert_eq!( + decode_header(&b), + Err(DecodeError::ReservedPriorityBits { flags: 0b0000_0110 }) + ); + + b[6] = 0b0011_0000; // admission bits 4-5 hold reserved 0b11 + assert_eq!( + decode_header(&b), + Err(DecodeError::ReservedAdmissionClass { flags: 0b0011_0000 }) + ); + } + + #[test] + fn reject_pure_header_frame_with_body_len() { + let h = hdr( + 1, + FrameType::Ping, + Flags::new(false, Priority::Passive, false), + 0, + 1, + ); + assert_eq!( + decode_header(&h.encode()), + Err(DecodeError::PureHeaderFrameWithBody { + ty: FrameType::Ping, + len: 1 + }) + ); + } + + #[test] + fn epoch_boundaries_round_trip_and_control_channel_epoch_is_reserved() { + for (channel, epoch) in [(0, 0), (1, 1), (u16::MAX, u32::MAX)] { + let h = hdr_with_epoch( + 0, + FrameType::Request, + Flags::new(false, Priority::Passive, false), + channel, + epoch, + 9, + ); + assert_eq!(decode_header(&h.encode()).unwrap(), h); + } + let h = hdr_with_epoch( + 0, + FrameType::Request, + Flags::new(false, Priority::Passive, false), + 0, + u32::MAX, + 2, + ); + assert_eq!( + decode_header(&h.encode()), + Err(DecodeError::NonzeroEpochOnControlChannel { epoch: u32::MAX }) + ); + } + + #[test] + fn sheddable_rejected_on_every_illegal_frame_type() { + // AdmissionClass::Sheddable = 0b10 at bits 4-5. + let flags = Flags(Flags::new(false, Priority::Passive, false).0 | 0b0010_0000); + assert_eq!(flags.admission_class(), Some(AdmissionClass::Sheddable)); + for ty in [ + FrameType::Request, + FrameType::Response, + FrameType::StreamEnd, + FrameType::Error, + FrameType::Cancel, + FrameType::Ping, + FrameType::Pong, + FrameType::Hello, + FrameType::HelloAck, + FrameType::Goodbye, + ] { + let h = hdr(0, ty, flags, 1, 2); + assert_eq!( + decode_header(&h.encode()), + Err(DecodeError::SheddableIllegalFrameType { ty, flags: flags.0 }) + ); + } + for ty in [FrameType::Push, FrameType::StreamData] { + let h = hdr_with_epoch(0, ty, flags, 1, 1, 0); + assert_eq!(decode_header(&h.encode()).unwrap().flags, flags); + } + } + #[test] fn capacity_separates_permanent_from_transient_exhaustion() { let budget = ByteBudget::new(100); diff --git a/crates/mc-host/tests/client.rs b/crates/mc-host/tests/client.rs new file mode 100644 index 000000000..cda3a54dc --- /dev/null +++ b/crates/mc-host/tests/client.rs @@ -0,0 +1,462 @@ +mod support; + +use std::{path::PathBuf, time::Duration}; + +#[cfg(unix)] +use std::os::unix::fs::{symlink, PermissionsExt}; + +use mc_host::{ + auth::authenticate_server, + connection_file::{ConnectionInfo, Endpoint}, + Client, LivenessPolicy, RequestOptions, RouteIdentity, RouteTarget, SendOutcome, TargetKind, +}; +use support::{ + mode_body, + raw_client::{self, FLAGS_RESPONSE_TEXT_LAST, TY_ERROR, TY_RESPONSE}, + TestHost, LINKED_MODULE_ID, +}; + +fn target() -> RouteTarget { + RouteTarget { + module_id: LINKED_MODULE_ID.to_owned(), + kind: TargetKind::ToolProvider, + } +} + +fn identity(session: &str) -> RouteIdentity { + RouteIdentity { + project_root: PathBuf::from("/tmp/mc-host-client-test"), + harness: "client-test".to_owned(), + session: session.to_owned(), + consumer_module_id: None, + consumer_launch_nonce: None, + consumer_capabilities: Vec::new(), + admission_facts: None, + } +} + +#[tokio::test] +async fn authenticates_negotiates_routes_unary_and_closes() { + let host = TestHost::start().await; + let client = Client::connect(host.publication_path()) + .await + .expect("managed client connects"); + assert_eq!( + client.daemon_id().as_slice(), + host.info.daemon_id.as_slice() + ); + + let route = client + .open_route(target(), identity("happy")) + .await + .expect("route opens"); + let body = mode_body(serde_json::json!({"mode": "echo", "value": 7})); + let response = client + .request(route, body.clone(), RequestOptions::default()) + .await + .expect("unary response"); + assert_eq!(response.body, body); + + client.close_route(route).await.expect("route closes"); + client.close().await.expect("client closes"); + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn host_terminal_is_typed_and_redacted() { + let host = TestHost::start().await; + let client = Client::connect(host.publication_path()).await.unwrap(); + let route = client + .open_route(target(), identity("terminal")) + .await + .unwrap(); + let sentinel = "CANARY-TERMINAL-BODY-7f31"; + let error = client + .request( + route, + mode_body(serde_json::json!({ + "mode": "error", + "code": "stable_failure", + "message": sentinel + })), + RequestOptions::default(), + ) + .await + .expect_err("host returns Error terminal"); + assert_eq!(error.outcome(), SendOutcome::Terminal); + assert_eq!(error.code(), "stable_failure"); + let rendered = format!("{error:?} {error}"); + assert!(!rendered.contains(sentinel)); + + client.close().await.unwrap(); + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn stream_order_and_slow_consumer_do_not_block_ping_or_unary() { + let host = TestHost::start_with(|config| { + config.liveness = Some(LivenessPolicy { + ping_interval: Duration::from_millis(20), + pong_deadline: Duration::from_millis(80), + invalidate_on_missed: true, + }); + }) + .await; + let client = Client::connect(host.publication_path()).await.unwrap(); + let route = client + .open_route(target(), identity("stream-ping")) + .await + .unwrap(); + let mut stream = client + .request_stream( + route, + mode_body(serde_json::json!({"mode": "stream_then_hang", "items": 2})), + RequestOptions { + timeout: Duration::from_secs(2), + cancellation: None, + }, + ) + .await + .unwrap(); + + let first = stream.next().await.unwrap().expect("first item"); + let second = stream.next().await.unwrap().expect("second item"); + assert_eq!( + serde_json::from_slice::(&first.body).unwrap()["item"], + 0 + ); + assert_eq!( + serde_json::from_slice::(&second.body).unwrap()["item"], + 1 + ); + + tokio::time::sleep(Duration::from_millis(150)).await; + let body = mode_body(serde_json::json!({"mode": "echo", "value": "unrelated"})); + let response = client + .request(route, body.clone(), RequestOptions::default()) + .await + .expect("Ping/Pong and slow stream do not block unary"); + assert_eq!(response.body, body); + stream.cancel().expect("stream cancellation"); + + client.close().await.unwrap(); + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn caller_cancellation_is_correlation_scoped() { + let host = TestHost::start().await; + let client = Client::connect(host.publication_path()).await.unwrap(); + let route = client + .open_route(target(), identity("cancel")) + .await + .unwrap(); + let cancel = mc_host::CancellationToken::new(); + let trigger = cancel.clone(); + let request = client.request( + route, + mode_body(serde_json::json!({"mode": "await_cancel"})), + RequestOptions { + timeout: Duration::from_secs(2), + cancellation: Some(cancel), + }, + ); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(20)).await; + trigger.cancel(); + }); + let error = request.await.expect_err("caller cancellation wins"); + assert!(matches!( + error.outcome(), + SendOutcome::NotSent | SendOutcome::OutcomeUnknown + )); + + let body = mode_body(serde_json::json!({"mode": "echo", "after": "cancel"})); + let response = client + .request(route, body.clone(), RequestOptions::default()) + .await + .expect("later request remains independent"); + assert_eq!(response.body, body); + + client.close().await.unwrap(); + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn request_deadline_is_one_absolute_owner_and_honors_overrides() { + let host = TestHost::start().await; + let client = Client::connect(host.publication_path()).await.unwrap(); + let route = client + .open_route(target(), identity("deadline")) + .await + .unwrap(); + + let error = client + .request( + route, + mode_body(serde_json::json!({"mode": "slow", "ms": 100})), + RequestOptions { + timeout: Duration::from_millis(20), + cancellation: None, + }, + ) + .await + .expect_err("short caller deadline wins"); + assert_eq!(error.outcome(), SendOutcome::OutcomeUnknown); + assert_eq!(error.code(), "deadline_expired"); + + let response = client + .request( + route, + mode_body(serde_json::json!({"mode": "slow", "ms": 20})), + RequestOptions { + timeout: Duration::from_millis(200), + cancellation: None, + }, + ) + .await + .expect("longer caller deadline is honored"); + assert_eq!(response.body, b"slow-done"); + + client.close().await.unwrap(); + host.shutdown_gracefully().await; +} + +#[cfg(unix)] +#[tokio::test] +async fn invalid_discovery_matrix_never_dials() { + let root = tempfile::tempdir().unwrap(); + let publication = root.path().join("connection.json"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + let valid = serde_json::json!({ + "schema": 1, + "wire_version": 2, + "endpoints": [{"host": "127.0.0.1", "port": port}], + "key": vec![7; 32], + "daemon_id": vec![8; 16], + "pid": 1, + "daemon_ver": "test" + }); + + async fn rejected_without_dial( + listener: &tokio::net::TcpListener, + publication: &std::path::Path, + ) { + let error = Client::connect(publication) + .await + .expect_err("discovery rejects before dial"); + assert_eq!(error.code(), "discovery_failed"); + assert!( + tokio::time::timeout(Duration::from_millis(10), listener.accept()) + .await + .is_err() + ); + } + + let write = |value: &serde_json::Value| { + std::fs::write(&publication, serde_json::to_vec(value).unwrap()).unwrap(); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o600)).unwrap(); + }; + + let mut missing = valid.clone(); + missing.as_object_mut().unwrap().remove("wire_version"); + write(&missing); + rejected_without_dial(&listener, &publication).await; + + let mut unsupported = valid.clone(); + unsupported["wire_version"] = serde_json::json!(1); + write(&unsupported); + rejected_without_dial(&listener, &publication).await; + + std::fs::write(&publication, b"not-json").unwrap(); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o600)).unwrap(); + rejected_without_dial(&listener, &publication).await; + + std::fs::write( + &publication, + vec![b'x'; mc_host::MAX_CONNECTION_FILE_LEN + 1], + ) + .unwrap(); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o600)).unwrap(); + rejected_without_dial(&listener, &publication).await; + + write(&valid); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o640)).unwrap(); + rejected_without_dial(&listener, &publication).await; + + let target = root.path().join("target.json"); + std::fs::rename(&publication, &target).unwrap(); + symlink(&target, &publication).unwrap(); + rejected_without_dial(&listener, &publication).await; +} + +#[cfg(unix)] +#[tokio::test] +async fn managed_client_negotiation_failures_retire_socket_without_application_frame() { + enum Reply { + Error(&'static str), + Response { corr: u64, body: &'static [u8] }, + } + + let cases = [ + ( + "legacy unsupported_operation", + Reply::Error("unsupported_operation"), + ), + ("connection_in_use", Reply::Error("connection_in_use")), + ( + "first application response", + Reply::Response { + corr: 2, + body: br#"{"op":"route.open","route_channel":7,"route_epoch":1}"#, + }, + ), + ( + "malformed selection", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp"}}"#, + }, + ), + ( + "version mismatch", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","negotiation_version":2,"selected":{"transport":"tcp","capability_version":1}}"#, + }, + ), + ( + "duplicate root op", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1}}"#, + }, + ), + ( + "duplicate root version", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","negotiation_version":1,"negotiation_version":1,"selected":{"transport":"tcp","capability_version":1}}"#, + }, + ), + ( + "duplicate nested transport", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","transport":"tcp","capability_version":1}}"#, + }, + ), + ( + "tcp fallback reason", + Reply::Response { + corr: 1, + body: br#"{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"unavailable"}"#, + }, + ), + ]; + + for (name, reply) in cases { + let root = tempfile::tempdir().unwrap(); + std::fs::set_permissions(root.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + let publication = root.path().join("connection.json"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let key = vec![0x5a; 32]; + let daemon_id = [0x3c; 16]; + let info = ConnectionInfo { + schema: 1, + wire_version: 2, + endpoints: vec![Endpoint { + host: "127.0.0.1".to_owned(), + port: listener.local_addr().unwrap().port(), + }], + key: key.clone(), + daemon_id, + pid: std::process::id(), + daemon_ver: "fake-peer".to_owned(), + }; + std::fs::write(&publication, serde_json::to_vec(&info).unwrap()).unwrap(); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o600)).unwrap(); + + let peer = tokio::spawn(async move { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let (mut socket, _) = listener.accept().await.unwrap(); + authenticate_server( + &mut socket, + &key, + &daemon_id, + "fake-peer", + Duration::from_secs(1), + ) + .await + .expect("managed client authenticates"); + let mut header = [0u8; raw_client::HEADER_LEN]; + socket + .read_exact(&mut header) + .await + .expect("negotiation header"); + let frame = raw_client::decode_header(&header); + assert_eq!(frame.ty, raw_client::TY_REQUEST, "{name}"); + assert_eq!(frame.corr, 1, "{name}"); + let mut request_body = vec![0; frame.len as usize]; + socket + .read_exact(&mut request_body) + .await + .expect("negotiation body"); + assert_eq!( + serde_json::from_slice::(&request_body).unwrap()["op"], + "transport.negotiate", + "{name}" + ); + + let (ty, corr, body) = match reply { + Reply::Error(code) => ( + TY_ERROR, + 1, + serde_json::to_vec(&serde_json::json!({ + "code": code, + "message": "peer-controlled sentinel" + })) + .unwrap(), + ), + Reply::Response { corr, body } => (TY_RESPONSE, corr, body.to_vec()), + }; + let mut response = + raw_client::header(body.len() as u32, ty, FLAGS_RESPONSE_TEXT_LAST, 0, 0, corr); + response.extend_from_slice(&body); + socket.write_all(&response).await.expect("selection reply"); + + let mut byte = [0u8; 1]; + let closed = tokio::time::timeout(Duration::from_secs(1), socket.read(&mut byte)) + .await + .expect("client retires socket") + .expect("socket read"); + assert_eq!(closed, 0, "{name}: no application frame after rejection"); + }); + + let error = Client::connect(&publication) + .await + .expect_err(&format!("{name}: connect must fail")); + assert_eq!(error.code(), "negotiation_failed", "{name}"); + peer.await.unwrap(); + } +} + +#[tokio::test] +async fn close_rejects_new_sends() { + let host = TestHost::start().await; + let client = Client::connect(host.publication_path()).await.unwrap(); + let route = client + .open_route(target(), identity("close")) + .await + .unwrap(); + client.close().await.unwrap(); + let error = client + .request(route, b"after-close".to_vec(), RequestOptions::default()) + .await + .expect_err("closed client rejects sends"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "client_closed"); + host.shutdown_gracefully().await; +} diff --git a/crates/mc-host/tests/host_roundtrip.rs b/crates/mc-host/tests/host_roundtrip.rs index b008b5df1..ec3cfd776 100644 --- a/crates/mc-host/tests/host_roundtrip.rs +++ b/crates/mc-host/tests/host_roundtrip.rs @@ -322,25 +322,24 @@ async fn degraded_storage_is_an_application_error_not_a_disconnect() { host.shutdown_gracefully().await; } -/// A raw client that never negotiates commits TCP with its first request and -/// observes no negotiation timeout, even one far past the setup deadline. +/// A negotiated TCP connection remains live after the optional-candidate setup +/// deadline has elapsed. #[tokio::test] -async fn a_legacy_first_request_commits_tcp_without_negotiation_timeout() { +async fn negotiated_connection_stays_live_beyond_setup_deadline() { let host = TestHost::start_with(|config| { config.timing.transport_setup_deadline = Duration::from_millis(100); }) .await; let mut client = host.client().await; - // Idle well past the candidate setup deadline before the first request. tokio::time::sleep(Duration::from_millis(400)).await; let (channel, epoch) = client - .route_open(LINKED_MODULE_ID, ROOT, "opencode", "legacy-omission") + .route_open(LINKED_MODULE_ID, ROOT, "opencode", "negotiated") .await - .expect("route.open as the first request"); + .expect("route.open after negotiation"); let corr = client.next_corr(); - let body = support::echo_body("legacy"); + let body = support::echo_body("negotiated"); client .send_frame(TY_REQUEST, FLAGS_INTERACTIVE, channel, epoch, corr, &body) .await diff --git a/crates/mc-host/tests/instance_security.rs b/crates/mc-host/tests/instance_security.rs index b4b6222c8..acd2f7117 100644 --- a/crates/mc-host/tests/instance_security.rs +++ b/crates/mc-host/tests/instance_security.rs @@ -80,9 +80,14 @@ async fn publication_is_an_owner_only_regular_file_in_an_owner_only_dir() { #[tokio::test] async fn discovery_validates_the_publication_the_way_a_client_must() { let host = TestHost::start().await; + let owned = mc_host::read_connection_file(host.publication_path()) + .expect("host-owned discovery accepts publication"); + assert_eq!(owned.wire_version, 2); + assert_eq!(owned.key.len(), 32); + let info = raw_client::discover(&host.publication_path()).expect("valid publication"); assert_eq!(info.schema, 1); - assert_eq!(info.wire_version, Some(2)); + assert_eq!(info.wire_version, 2); assert_eq!(info.host, "127.0.0.1"); assert_ne!(info.port, 0); assert_eq!(info.key.len(), 32); @@ -98,6 +103,65 @@ async fn discovery_validates_the_publication_the_way_a_client_must() { raw_client::discover(&loose).is_err(), "an insecure mode must fail client validation" ); + assert!(mc_host::read_connection_file(&loose).is_err()); + + let oversized = host.runtime_dir().join("oversized.json"); + std::fs::write(&oversized, vec![b' '; mc_host::MAX_CONNECTION_FILE_LEN + 1]) + .expect("write oversized publication"); + std::fs::set_permissions(&oversized, std::fs::Permissions::from_mode(0o600)) + .expect("owner-only oversized publication"); + assert!(mc_host::read_connection_file(&oversized).is_err()); + + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn discovery_requires_numeric_wire_version_two() { + let host = TestHost::start().await; + let original: serde_json::Value = + serde_json::from_slice(&std::fs::read(host.publication_path()).expect("read publication")) + .expect("parse publication"); + + for (name, value) in [ + ("missing", None), + ("null", Some(serde_json::Value::Null)), + ("string", Some(serde_json::json!("2"))), + ("other", Some(serde_json::json!(1))), + ] { + let path = host.runtime_dir().join(format!("{name}.json")); + let mut candidate = original.clone(); + let object = candidate.as_object_mut().expect("object"); + match value { + Some(value) => { + object.insert("wire_version".to_owned(), value); + } + None => { + object.remove("wire_version"); + } + } + std::fs::write(&path, serde_json::to_vec(&candidate).expect("serialize")) + .expect("write candidate"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)) + .expect("owner-only candidate"); + assert!( + mc_host::read_connection_file(&path).is_err(), + "{name} wire_version must fail discovery" + ); + } + + host.shutdown_gracefully().await; +} + +#[tokio::test] +async fn discovery_rejects_symlink_and_hard_link_publications() { + let host = TestHost::start().await; + let symlink = host.runtime_dir().join("symlink.json"); + std::os::unix::fs::symlink(host.publication_path(), &symlink).expect("create symlink"); + assert!(mc_host::read_connection_file(&symlink).is_err()); + + let hard_link = host.runtime_dir().join("hard-link.json"); + std::fs::hard_link(host.publication_path(), &hard_link).expect("create hard link"); + assert!(mc_host::read_connection_file(&hard_link).is_err()); host.shutdown_gracefully().await; } diff --git a/crates/mc-host/tests/lifecycle.rs b/crates/mc-host/tests/lifecycle.rs index 9cda59876..4461c0c22 100644 --- a/crates/mc-host/tests/lifecycle.rs +++ b/crates/mc-host/tests/lifecycle.rs @@ -296,7 +296,7 @@ async fn saturated_connection_capacity_closes_after_authentication() { // ServerProof must arrive before the post-authentication capacity close; // an accept-time capacity check would make this connect fail earlier. - let mut second = raw_client::RawClient::connect(&host.info) + let mut second = raw_client::RawClient::connect_setup_only(&host.info) .await .expect("the proof exchange completes before promotion is refused"); assert!( @@ -413,10 +413,8 @@ async fn ping_and_consumer_correlations_do_not_cross_settle() { .await .expect("route"); - // route_open consumes correlation 1; the test leaves correlation 2 - // pending until the host sends Ping correlation 2. let corr = client.next_corr(); - assert_eq!(corr, 2); + assert_eq!(corr, 3); client .send_frame( TY_REQUEST, @@ -1534,37 +1532,38 @@ async fn a_dying_requester_cannot_strand_the_stop() { host.shutdown().await.expect("graceful shutdown"); } -/// A second shutdown request on the same connection (after the first) also -/// settles: an already-committed latch answers without a second commit. #[tokio::test] -async fn shutdown_after_commit_reports_success_again() { +async fn pipelined_shutdown_requests_on_one_connection_both_settle() { let host = TestHost::start().await; let mut client = host.client().await; - let first = client - .control(&serde_json::json!({"op": "host.shutdown"})) - .await - .expect("first shutdown"); - let second = client - .control(&serde_json::json!({"op": "host.shutdown"})) - .await - .expect("second shutdown"); - // Both requests are admitted before the commit, but their per-request - // tasks race for latch ownership, so the responses can arrive in either - // order; collect by correlation instead of assuming wire order. - // Whichever attempt commits, the other settles through the - // already-committed branch without a second commit. - let deadline = tokio::time::Instant::now() + BUDGET; - let mut settled: std::collections::HashMap = - std::collections::HashMap::new(); - while settled.len() < 2 { - let remaining = deadline.saturating_duration_since(tokio::time::Instant::now()); - let frame = client.frame_within(remaining).await.expect("settled"); - if frame.ty != TY_PING && (frame.corr == first || frame.corr == second) { - settled.insert(frame.corr, frame); - } - } + let body = br#"{"op":"host.shutdown"}"#; + let first = client.next_corr(); + let second = client.next_corr(); + let mut wire = raw_client::header( + body.len() as u32, + TY_REQUEST, + FLAGS_INTERACTIVE, + 0, + 0, + first, + ); + wire.extend_from_slice(body); + wire.extend_from_slice(&raw_client::header( + body.len() as u32, + TY_REQUEST, + FLAGS_INTERACTIVE, + 0, + 0, + second, + )); + wire.extend_from_slice(body); + client.send_raw(&wire).await.expect("pipeline shutdowns"); + for corr in [first, second] { - let response = settled.get(&corr).expect("both correlations settle"); + let (_, response) = client + .frames_until_corr(corr, BUDGET) + .await + .expect("shutdown response"); assert_eq!(response.ty, TY_RESPONSE); assert_eq!(response.json()["op"], "host.shutdown"); } @@ -1714,7 +1713,7 @@ async fn shutdown_during_candidate_setup_reaps_both_channels() { }) .await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let corr = client .control(&serde_json::json!({ "op": "transport.negotiate", diff --git a/crates/mc-host/tests/perf_budget_runner.rs b/crates/mc-host/tests/perf_budget_runner.rs index 19e2b2eb6..876ee4b16 100644 --- a/crates/mc-host/tests/perf_budget_runner.rs +++ b/crates/mc-host/tests/perf_budget_runner.rs @@ -3,6 +3,8 @@ //! preserved open-loop arrival schedules, missed-slot accounting under a //! full in-flight queue, and throughput-arm rate separation. +#![allow(clippy::duplicate_mod)] + #[path = "support/raw_client.rs"] mod raw_client; diff --git a/crates/mc-host/tests/support/fake_transport.rs b/crates/mc-host/tests/support/fake_transport.rs index 70f659da8..6d12b66ec 100644 --- a/crates/mc-host/tests/support/fake_transport.rs +++ b/crates/mc-host/tests/support/fake_transport.rs @@ -51,7 +51,7 @@ impl FakeProvider { ) } - /// Registry containing only this provider beside implicit TCP. + /// The registry includes this provider and built-in TCP only. pub fn registry(provider: &Arc) -> TransportProviders { TransportProviders::with_injected(vec![Arc::clone(provider) as Arc]) } diff --git a/crates/mc-host/tests/support/mod.rs b/crates/mc-host/tests/support/mod.rs index 88cdd36d5..0467b5421 100644 --- a/crates/mc-host/tests/support/mod.rs +++ b/crates/mc-host/tests/support/mod.rs @@ -666,7 +666,13 @@ impl TestHost { pub async fn client(&self) -> raw_client::RawClient { raw_client::RawClient::connect(&self.info) .await - .expect("authenticated connection") + .expect("negotiated connection") + } + + pub async fn setup_client(&self) -> raw_client::RawClient { + raw_client::RawClient::connect_setup_only(&self.info) + .await + .expect("setup-only connection") } /// Signals shutdown and returns the runtime's result. diff --git a/crates/mc-host/tests/support/perf_measurement.rs b/crates/mc-host/tests/support/perf_measurement.rs index 4596234c8..3560b4f16 100644 --- a/crates/mc-host/tests/support/perf_measurement.rs +++ b/crates/mc-host/tests/support/perf_measurement.rs @@ -159,7 +159,7 @@ pub fn open_loop_interval_ns(rate_per_sec: u64) -> Result { // A truncated interval silently raises the actual arrival rate while // every manifest retains the requested label; only exactly // representable rates are honest. - if 1_000_000_000u64 % rate_per_sec != 0 { + if !1_000_000_000u64.is_multiple_of(rate_per_sec) { return Err(format!( "offered rate {rate_per_sec}/s has no exact nanosecond interval; \ choose a rate that divides 1e9" diff --git a/crates/mc-host/tests/support/raw_client.rs b/crates/mc-host/tests/support/raw_client.rs index 517f989fd..3c7b030d4 100644 --- a/crates/mc-host/tests/support/raw_client.rs +++ b/crates/mc-host/tests/support/raw_client.rs @@ -171,7 +171,7 @@ pub struct Discovered { pub pid: u64, pub daemon_ver: String, pub schema: u64, - pub wire_version: Option, + pub wire_version: u64, } /// Validates and reads a publication the way a conforming client must @@ -201,11 +201,12 @@ pub fn discover(path: &Path) -> Result { if schema != 1 { return Err(format!("unsupported schema {schema}")); } - let wire_version = json.get("wire_version").and_then(serde_json::Value::as_u64); - if let Some(version) = wire_version { - if version != u64::from(WIRE_VERSION) { - return Err(format!("wire version {version} is not 2")); - } + let wire_version = json + .get("wire_version") + .and_then(serde_json::Value::as_u64) + .ok_or("missing or invalid wire_version")?; + if wire_version != u64::from(WIRE_VERSION) { + return Err(format!("wire version {wire_version} is not 2")); } let endpoints = json["endpoints"].as_array().ok_or("missing endpoints")?; @@ -314,13 +315,24 @@ pub struct RawClient { } impl RawClient { - /// Completes the three-message handshake, verifying the server proof and - /// daemon ID before sending `ClientAuth` (protocol §5.2). pub async fn connect(info: &Discovered) -> Result { Self::connect_with_role(info, "client").await } pub async fn connect_with_role(info: &Discovered, role: &str) -> Result { + let mut client = Self::connect_setup_only_with_role(info, role).await?; + client.negotiate_tcp().await?; + Ok(client) + } + + pub async fn connect_setup_only(info: &Discovered) -> Result { + Self::connect_setup_only_with_role(info, "client").await + } + + pub async fn connect_setup_only_with_role( + info: &Discovered, + role: &str, + ) -> Result { let mut stream = TcpStream::connect((info.host.as_str(), info.port)) .await .map_err(|err| err.to_string())?; @@ -381,6 +393,37 @@ impl RawClient { }) } + async fn negotiate_tcp(&mut self) -> Result<(), String> { + let corr = self + .control(&serde_json::json!({ + "op": "transport.negotiate", + "negotiation_version": 1, + "offers": [{"transport": "tcp", "capability_version": 1}] + })) + .await + .map_err(|err| err.to_string())?; + let (skipped, frame) = self.frames_until_corr(corr, Duration::from_secs(5)).await?; + if !skipped.is_empty() { + return Err("unexpected frame before transport selection".to_owned()); + } + if frame.ty != TY_RESPONSE + || frame.channel != 0 + || frame.epoch != 0 + || frame.flags != FLAGS_RESPONSE_TEXT_LAST + { + return Err("invalid transport selection frame".to_owned()); + } + let expected = serde_json::json!({ + "op": "transport.negotiate", + "negotiation_version": 1, + "selected": {"transport": "tcp", "capability_version": 1} + }); + if frame.json() != expected { + return Err("invalid TCP transport selection".to_owned()); + } + Ok(()) + } + /// Allocates the next monotonic consumer correlation. pub fn next_corr(&mut self) -> u64 { self.next_corr += 1; diff --git a/crates/mc-host/tests/transport_negotiation.rs b/crates/mc-host/tests/transport_negotiation.rs index 5d4b47fd6..11f6308b1 100644 --- a/crates/mc-host/tests/transport_negotiation.rs +++ b/crates/mc-host/tests/transport_negotiation.rs @@ -157,7 +157,6 @@ fn version_mismatches_encode_the_documented_tcp_fallback_reasons() { "capability_version_mismatch", FallbackReason::CapabilityVersionMismatch, ), - ("connection_in_use", FallbackReason::ConnectionInUse), ] { assert_eq!(FallbackReason::parse(name), Some(expected)); assert_eq!(expected.as_str(), name); @@ -746,7 +745,7 @@ async fn grant_over( host: &TestHost, peers: &mut tokio::sync::mpsc::UnboundedReceiver, ) -> (raw_client::RawClient, RawCandidate, String) { - let mut client = host.client().await; + let mut client = host.setup_client().await; let frame = control_response(&mut client, &negotiate_body(fake_and_tcp_offers())).await; assert_eq!(frame.ty, TY_RESPONSE, "grant expected: {frame:?}"); let json = frame.json(); @@ -801,7 +800,7 @@ async fn commit_ok(candidate: &mut RawCandidate) { #[tokio::test] async fn tcp_only_selection_is_exact_and_the_generation_serves_requests() { let host = TestHost::start().await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let frame = control_response(&mut client, &negotiate_body(tcp_only_offers())).await; assert_eq!(frame.ty, TY_RESPONSE); @@ -840,7 +839,7 @@ async fn tcp_only_selection_is_exact_and_the_generation_serves_requests() { #[tokio::test] async fn unprovided_non_tcp_offer_selects_tcp_with_unavailable() { let host = TestHost::start().await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let offers = serde_json::json!([ {"transport": "shm", "capability_version": 1, "parameters": {}}, @@ -860,34 +859,25 @@ async fn unprovided_non_tcp_offer_selects_tcp_with_unavailable() { } #[tokio::test] -async fn version_mismatches_select_tcp_with_their_exact_reason() { - // Unsupported negotiation version, parsed under version-1 grammar. +async fn negotiation_version_mismatch_retires_but_capability_mismatch_falls_back() { let host = TestHost::start().await; - let mut client = host.client().await; - let body = serde_json::json!({ - "op": "transport.negotiate", - "negotiation_version": 2, - "offers": [{"transport": "tcp", "capability_version": 1}] - }); - let frame = control_response(&mut client, &body).await; - assert_eq!(frame.ty, TY_RESPONSE); - let json = frame.json(); - // The response echoes the request's grammar version (§7.7.2), not the - // host's. Stamping the host's version would make this fallback reason - // unconsumable: the peer's decoder requires its own version, so it would - // reject the response and fail closed instead of retaining TCP (R8). - assert_eq!(json["negotiation_version"], 2); - assert_eq!(json["selected"]["transport"], "tcp"); - assert_eq!(json["reason"], "negotiation_version_mismatch"); - let frame = control_response(&mut client, &serde_json::json!({"op": "catalog.list"})).await; - assert_eq!(frame.ty, TY_RESPONSE, "the generation stays usable"); + let mut client = host.setup_client().await; + client + .control(&serde_json::json!({ + "op": "transport.negotiate", + "negotiation_version": 2, + "offers": [{"transport": "tcp", "capability_version": 1}] + })) + .await + .expect("send mismatched negotiation"); + assert!(client.closed_within(HOST_BUDGET).await); host.shutdown_gracefully().await; // The fake transport is installed at capability 2 but offered at 1. let (provider, _peers) = FakeProvider::install(2, serde_json::json!({}), 64 * 1024); let registry = FakeProvider::registry(&provider); let host = TestHost::start_with(move |config| config.transport_providers = registry).await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let frame = control_response(&mut client, &negotiate_body(fake_and_tcp_offers())).await; assert_eq!(frame.ty, TY_RESPONSE); let json = frame.json(); @@ -900,44 +890,64 @@ async fn version_mismatches_select_tcp_with_their_exact_reason() { } #[tokio::test] -async fn late_negotiation_reports_connection_in_use_once_then_retires() { - let host = TestHost::start().await; - let mut client = host.client().await; - - let frame = control_response(&mut client, &serde_json::json!({"op": "catalog.list"})).await; - assert_eq!(frame.ty, TY_RESPONSE, "application traffic commits TCP"); +async fn application_before_negotiation_retires_without_side_effects() { + for body in [ + serde_json::json!({"op": "catalog.list"}), + serde_json::json!({"op": "host.shutdown"}), + serde_json::json!({ + "op": "route.open", + "target": {"kind": "tool_provider", "module_id": LINKED_MODULE_ID}, + "identity": {"project_root": ROOT, "harness": "opencode", "session": "setup"} + }), + serde_json::json!({"op": "unknown.operation"}), + ] { + let host = TestHost::start().await; + let mut client = host.setup_client().await; + client.control(&body).await.expect("send setup violation"); + assert!(client.closed_within(HOST_BUDGET).await); + assert!(host.handler.binds().is_empty()); - let frame = control_response(&mut client, &negotiate_body(tcp_only_offers())).await; - assert_eq!(frame.ty, TY_RESPONSE); - let json = frame.json(); - assert_eq!(json["selected"]["transport"], "tcp"); - assert_eq!(json["reason"], "connection_in_use"); + let mut negotiated = host.client().await; + let frame = + control_response(&mut negotiated, &serde_json::json!({"op": "catalog.list"})).await; + assert_eq!(frame.ty, TY_RESPONSE); + host.shutdown_gracefully().await; + } - let frame = control_response(&mut client, &serde_json::json!({"op": "catalog.list"})).await; - assert_eq!( - frame.ty, TY_RESPONSE, - "one late negotiation keeps the generation" - ); + let host = TestHost::start().await; + let mut client = host.setup_client().await; + client + .send_frame(TY_REQUEST, FLAGS_INTERACTIVE, 1, 1, 1, b"{}") + .await + .expect("send routed setup violation"); + assert!(client.closed_within(HOST_BUDGET).await); + assert!(host.handler.binds().is_empty()); + host.shutdown_gracefully().await; - let _ = client - .control(&negotiate_body(tcp_only_offers())) + let host = TestHost::start().await; + let mut client = host.setup_client().await; + client + .send_frame(TY_REQUEST, FLAGS_INTERACTIVE, 0, 0, 1, &vec![b' '; 65_537]) .await - .expect("send repeated negotiation"); - assert!( - client.closed_within(HOST_BUDGET).await, - "a repeated negotiation is a protocol failure" - ); + .expect("send oversized setup violation"); + assert!(client.closed_within(HOST_BUDGET).await); + assert!(host.handler.binds().is_empty()); + host.shutdown_gracefully().await; +} +#[tokio::test] +async fn normal_raw_client_returns_after_tcp_negotiation() { + let host = TestHost::start().await; + let mut client = host.client().await; + let frame = control_response(&mut client, &serde_json::json!({"op": "catalog.list"})).await; + assert_eq!(frame.ty, TY_RESPONSE); host.shutdown_gracefully().await; } #[tokio::test] async fn repeated_negotiation_after_negotiated_tcp_selection_retires() { - // A negotiation that itself selects TCP consumes the late-negotiation - // allowance: the allowance exists only for a first negotiation arriving - // after an implicit TCP commit (§7.7.5). let host = TestHost::start().await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let frame = control_response(&mut client, &negotiate_body(tcp_only_offers())).await; assert_eq!(frame.ty, TY_RESPONSE); @@ -980,7 +990,7 @@ async fn stalled_provider_prepare_fails_setup_within_the_deadline() { config.timing.transport_setup_deadline = Duration::from_millis(100); }) .await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let _ = client .control(&negotiate_body(fake_and_tcp_offers())) .await @@ -1002,7 +1012,7 @@ async fn duplicate_key_negotiation_settles_with_its_terminal_and_retires() { // path, never the generic rejection that would commit TCP and leave the // generation usable (§7.7.1). let host = TestHost::start().await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let dup = r#"{"op":"transport.negotiate","negotiation_version":1,"offers":[{"transport":"tcp","capability_version":1,"parameters":{"a":1,"a":2}}]}"#; let corr = client.next_corr(); @@ -1024,7 +1034,7 @@ async fn duplicate_key_negotiation_settles_with_its_terminal_and_retires() { #[tokio::test] async fn malformed_negotiation_settles_with_its_terminal_and_never_reaches_dispatch() { let host = TestHost::start().await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let missing_tcp = serde_json::json!({ "op": "transport.negotiate", @@ -1052,7 +1062,7 @@ async fn injected_grant_activates_commits_and_serves_application_traffic() { let registry = FakeProvider::registry(&provider); let host = TestHost::start_with(move |config| config.transport_providers = registry).await; - let mut client = host.client().await; + let mut client = host.setup_client().await; let frame = control_response(&mut client, &negotiate_body(fake_and_tcp_offers())).await; let json = frame.json(); assert_eq!(json["descriptor"], serde_json::json!({"kind": "fake"})); @@ -1185,7 +1195,7 @@ async fn ktd9_attachment_failures_fail_closed_before_any_candidate_exists() { ProviderFailure::StaleDescriptor, ] { provider.fail_next(failure); - let mut client = host.client().await; + let mut client = host.setup_client().await; let _ = client .control(&negotiate_body(fake_and_tcp_offers())) .await @@ -1310,7 +1320,7 @@ async fn liveness_hands_off_from_bootstrap_to_candidate_around_the_grant() { }); }) .await; - let mut client = host.client().await; + let mut client = host.setup_client().await; // Bootstrap liveness runs before negotiation. let ping = client @@ -1508,7 +1518,7 @@ async fn max_connections_bounds_prepared_candidates_and_failure_releases_them() // The setup retains the sole connection permit: a second authenticated // connection is refused while the candidate is prepared. - let mut second = raw_client::RawClient::connect(&host.info) + let mut second = raw_client::RawClient::connect_setup_only(&host.info) .await .expect("handshake completes"); assert!( @@ -1572,7 +1582,7 @@ async fn sentinel_provider_data_stays_off_diagnostic_surfaces() { // A malformed negotiation whose unknown field name carries the sentinel // gets the bounded terminal with no sentinel bytes echoed. - let mut client = host.client().await; + let mut client = host.setup_client().await; let body = format!( r#"{{"op":"transport.negotiate","negotiation_version":1,"offers":[{{"transport":"tcp","capability_version":1}}],"{SENTINEL}":1}}"# ); @@ -1588,7 +1598,7 @@ async fn sentinel_provider_data_stays_off_diagnostic_surfaces() { // Offer parameters carrying the sentinel never reach the fallback // response. - let mut client = host.client().await; + let mut client = host.setup_client().await; let offers = serde_json::json!([ {"transport": "shm", "capability_version": 1, "parameters": {"secret": SENTINEL}}, {"transport": "tcp", "capability_version": 1} diff --git a/crates/mc-module/Cargo.toml b/crates/mc-module/Cargo.toml index aba51dbce..4ad3482e1 100644 --- a/crates/mc-module/Cargo.toml +++ b/crates/mc-module/Cargo.toml @@ -3,36 +3,34 @@ name = "mc-module" version = "0.1.0" edition = "2021" publish = false -description = "The Magic Context subc module: harness-agnostic cache-stability transform (CK-in/CK-out), backed by the single-writer mc-store, served over the subc wire." +autobins = false +description = "Magic Context host component: harness-agnostic cache-stability transform backed by the single-writer mc-store." [lib] name = "mc_module" path = "src/lib.rs" -[[bin]] -# Fleet convention: supervised binaries carry the ck-* prefix so they group in -# Activity Monitor. Crate/package stays mc-module and module_id stays -# "magic-context" — only the built executable name changes. -name = "ck-mc" -path = "src/main.rs" +[[example]] +name = "direct_host_fixture" +path = "examples/direct_host_fixture.rs" +required-features = ["direct-host-fixture"] [dependencies] mc-core = { workspace = true } mc-store = { workspace = true } mc-tokenizer = { path = "../mc-tokenizer" } +mc-host = { path = "../mc-host" } cortexkit-store-types = { workspace = true } cortexkit-store = { workspace = true } cortexkit-lease = { workspace = true } -subc-protocol = { workspace = true } -subc-control = { workspace = true } -subc-transport = { workspace = true } -subc-client-rs = { workspace = true } +async-trait = "0.1" serde = { workspace = true } serde_json = { workspace = true } # Provider context-overflow error patterns (scheduler unit). Plain regex (no # lookarounds in those patterns); fancy-regex stays confined to mc-tokenizer. regex = "1" tokio = { workspace = true } +tokio-util = { version = "0.7", features = ["rt"] } sha2 = { workspace = true } chrono = { version = "0.4", default-features = false, features = ["clock"] } @@ -52,12 +50,12 @@ chrono = { version = "0.4", default-features = false, features = ["clock"] } # Do NOT add drive-fault to a default feature set, and do NOT add new fault arms # without the matching absence-proof note here and the fault-shape tests in lib.rs. drive-fault = [] +# The `direct-host-fixture` feature enables a test-only process fixture, not a production launch surface. +direct-host-fixture = [] [dev-dependencies] -subc-core = { workspace = true } chrono-tz = "0.10" -# Dev-only host for the authenticated round-trip test; production mc-module must never link mc-host. commentlint: allow(JUDGE) -mc-host = { path = "../mc-host" } +tokio = { workspace = true, features = ["signal"] } # Enable the store's seed helpers for mc-module's compose tests (populate memories + # mutations). The feature is gated so the writers never reach production builds. mc-store = { workspace = true, features = ["test-support"] } diff --git a/crates/mc-module/examples/direct_host_fixture.rs b/crates/mc-module/examples/direct_host_fixture.rs new file mode 100644 index 000000000..2fc1e7653 --- /dev/null +++ b/crates/mc-module/examples/direct_host_fixture.rs @@ -0,0 +1,615 @@ +//! Test-only directly linked mc-host process fixture. + +#![forbid(unsafe_code)] + +#[cfg(unix)] +mod unix { + use std::error::Error; + use std::fs; + use std::io::{self, Write}; + use std::os::unix::fs::{FileTypeExt, PermissionsExt}; + use std::path::{Path, PathBuf}; + use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::{Arc, Mutex}; + use std::time::Duration; + + use mc_host::broca::backend::{ + BackendError, BackendEvent, BackendFuture, BackendRequest, BackendTerminal, ErrorClass, + EventSink, FinishReason, LlmExecutionBackend, + }; + use mc_host::broca::BrocaComponent; + use mc_host::synapse::inference::InferenceError; + use mc_host::synapse::{EmbeddingEngine, LaneInfo, SynapseComponent, SynapseLimits}; + use mc_host::{CancellationToken, HostConfig, HostInit, StaticComposite}; + use serde::{Deserialize, Serialize}; + use sha2::Digest; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::{UnixListener, UnixStream}; + + const CONTROL_FILE: &str = "direct-host-control.sock"; + const STORE_FILE: &str = "mc-store.db"; + const MAX_CONTROL_LINE: usize = 64 * 1024; + const READY_TIMEOUT: Duration = Duration::from_secs(30); + const CATALOG: [&str; 3] = ["magic-context", "synapse", "broca"]; + + #[derive(Debug, Clone, Copy)] + enum NextBehavior { + Success, + Block, + Failure, + } + + #[derive(Default)] + struct BackendCounters { + started: AtomicU64, + completed: AtomicU64, + blocked: AtomicU64, + active_blocked: AtomicU64, + released: AtomicU64, + failed: AtomicU64, + cancelled: AtomicU64, + } + + #[derive(Serialize)] + struct CounterSnapshot { + started: u64, + completed: u64, + blocked: u64, + released: u64, + failed: u64, + cancelled: u64, + } + + impl BackendCounters { + fn snapshot(&self) -> CounterSnapshot { + CounterSnapshot { + started: self.started.load(Ordering::SeqCst), + completed: self.completed.load(Ordering::SeqCst), + blocked: self.blocked.load(Ordering::SeqCst), + released: self.released.load(Ordering::SeqCst), + failed: self.failed.load(Ordering::SeqCst), + cancelled: self.cancelled.load(Ordering::SeqCst), + } + } + } + + struct ControlledBackend { + next: Mutex, + release: Arc, + shutdown: CancellationToken, + counters: Arc, + } + + impl ControlledBackend { + fn new(shutdown: CancellationToken) -> Arc { + Arc::new(Self { + next: Mutex::new(NextBehavior::Success), + release: Arc::new(tokio::sync::Semaphore::new(0)), + shutdown, + counters: Arc::new(BackendCounters::default()), + }) + } + + fn set_next(&self, behavior: NextBehavior) { + *self.next.lock().expect("fixture backend behavior mutex") = behavior; + } + + fn release_blocked(&self) -> bool { + if !take_blocked_slot(&self.counters) { + return false; + } + self.counters.released.fetch_add(1, Ordering::SeqCst); + self.release.add_permits(1); + true + } + + fn terminal_error(message: &str) -> BackendTerminal { + BackendTerminal::Failed(BackendError { + class: ErrorClass::Permanent, + message: message.to_owned(), + retry_after_secs: None, + provider_code: Some("fixture_terminal".to_owned()), + }) + } + } + + fn take_blocked_slot(counters: &BackendCounters) -> bool { + counters + .active_blocked + .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |active| { + active.checked_sub(1) + }) + .is_ok() + } + + impl LlmExecutionBackend for ControlledBackend { + fn execute( + &self, + _request: BackendRequest, + events: EventSink, + cancel: CancellationToken, + ) -> BackendFuture { + self.counters.started.fetch_add(1, Ordering::SeqCst); + let behavior = std::mem::replace( + &mut *self.next.lock().expect("fixture backend behavior mutex"), + NextBehavior::Success, + ); + let release = Arc::clone(&self.release); + let shutdown = self.shutdown.clone(); + let counters = Arc::clone(&self.counters); + Box::pin(async move { + match behavior { + NextBehavior::Success => { + events.emit(BackendEvent::AssistantText { + text: "fixture-success".to_owned(), + finish_reason: None, + }); + counters.completed.fetch_add(1, Ordering::SeqCst); + BackendTerminal::Completed { + finish_reason: FinishReason::Completed, + } + } + NextBehavior::Failure => { + counters.failed.fetch_add(1, Ordering::SeqCst); + ControlledBackend::terminal_error("fixture requested typed failure") + } + NextBehavior::Block => { + counters.blocked.fetch_add(1, Ordering::SeqCst); + counters.active_blocked.fetch_add(1, Ordering::SeqCst); + tokio::select! { + biased; + () = shutdown.cancelled() => { + take_blocked_slot(&counters); + counters.cancelled.fetch_add(1, Ordering::SeqCst); + ControlledBackend::terminal_error("fixture shutting down") + } + () = cancel.cancelled() => { + take_blocked_slot(&counters); + counters.cancelled.fetch_add(1, Ordering::SeqCst); + ControlledBackend::terminal_error("fixture run cancelled") + } + permit = release.acquire() => { + permit.expect("fixture release semaphore stays open").forget(); + events.emit(BackendEvent::AssistantText { + text: "fixture-released".to_owned(), + finish_reason: None, + }); + counters.completed.fetch_add(1, Ordering::SeqCst); + BackendTerminal::Completed { + finish_reason: FinishReason::Completed, + } + } + } + } + } + }) + } + } + + struct DeterministicEngine; + + impl EmbeddingEngine for DeterministicEngine { + fn embed(&self, texts: &[&str]) -> Result>, InferenceError> { + Ok(texts + .iter() + .map(|text| { + let digest = sha2::Sha256::digest(text.as_bytes()); + let mut vector: Vec = digest[..8] + .iter() + .map(|byte| f32::from(*byte) + 1.0) + .collect(); + let norm = vector.iter().map(|value| value * value).sum::().sqrt(); + for value in &mut vector { + *value /= norm; + } + vector + }) + .collect()) + } + } + + fn synapse_component() -> SynapseComponent { + let limits = SynapseLimits::default(); + SynapseComponent::ready_with_engine( + LaneInfo { + model: "direct-host-fixture".to_owned(), + fingerprint: "a2b4c6d8e0f01234a2b4c6d8e0f01234a2b4c6d8e0f01234a2b4c6d8e0f01234" + .to_owned(), + table_epoch: 1, + dims: 8, + max_tokens: 512, + max_text_bytes: limits.max_text_bytes, + provenance: serde_json::json!({"source": "direct host fixture"}), + recommended_rows: 16, + recommended_token_budget: 8_192, + }, + Arc::new(DeterministicEngine), + limits, + ) + } + + #[derive(Deserialize)] + #[serde(deny_unknown_fields)] + struct ControlRequest { + id: u64, + command: ControlCommand, + } + + #[derive(Deserialize)] + #[serde(deny_unknown_fields)] + struct ControlId { + id: u64, + command: serde::de::IgnoredAny, + } + + #[derive(Deserialize)] + #[serde(tag = "name", rename_all = "kebab-case", deny_unknown_fields)] + enum ControlCommand { + BackendSuccess, + BlockNextCall, + ReleaseBlockedCall, + TypedFailure, + Counters, + GracefulShutdown, + } + + #[derive(Serialize)] + struct ControlResponse { + id: Option, + ok: bool, + #[serde(skip_serializing_if = "Option::is_none")] + result: Option, + #[serde(skip_serializing_if = "Option::is_none")] + error: Option, + } + + #[derive(Serialize)] + struct ControlError { + code: &'static str, + message: &'static str, + } + + #[derive(Serialize)] + #[serde(untagged)] + enum ControlResult { + Ack { accepted: bool }, + Counters(CounterSnapshot), + } + + enum BoundedLine { + Line(Vec), + Oversized, + } + + async fn read_bounded_line(stream: &mut UnixStream) -> io::Result> { + let mut line = Vec::with_capacity(256); + let mut oversized = false; + let mut byte = [0u8; 1]; + loop { + let read = stream.read(&mut byte).await?; + if read == 0 { + return if line.is_empty() && !oversized { + Ok(None) + } else if oversized { + Ok(Some(BoundedLine::Oversized)) + } else { + Ok(Some(BoundedLine::Line(line))) + }; + } + if byte[0] == b'\n' { + return if oversized { + Ok(Some(BoundedLine::Oversized)) + } else { + if line.last() == Some(&b'\r') { + line.pop(); + } + Ok(Some(BoundedLine::Line(line))) + }; + } + if line.len() < MAX_CONTROL_LINE { + line.push(byte[0]); + } else { + oversized = true; + } + } + } + + async fn write_response( + stream: &mut UnixStream, + response: &ControlResponse, + ) -> io::Result<()> { + let bytes = serde_json::to_vec(response).expect("fixture response serializes"); + if bytes.len() > MAX_CONTROL_LINE { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "fixture response exceeded control cap", + )); + } + stream.write_all(&bytes).await?; + stream.write_all(b"\n").await + } + + fn rejected(id: Option, code: &'static str) -> ControlResponse { + ControlResponse { + id, + ok: false, + result: None, + error: Some(ControlError { + code, + message: "control request rejected", + }), + } + } + + async fn handle_control_connection( + mut stream: UnixStream, + backend: Arc, + shutdown: CancellationToken, + ) -> io::Result<()> { + loop { + let line = tokio::select! { + biased; + line = read_bounded_line(&mut stream) => line?, + () = shutdown.cancelled() => return Ok(()), + }; + let Some(line) = line else { + return Ok(()); + }; + let response = match line { + BoundedLine::Oversized => rejected(None, "request_too_large"), + BoundedLine::Line(line) => match serde_json::from_slice::(&line) { + Ok(request) => { + let (result, stop) = match request.command { + ControlCommand::BackendSuccess => { + backend.set_next(NextBehavior::Success); + (ControlResult::Ack { accepted: true }, false) + } + ControlCommand::BlockNextCall => { + backend.set_next(NextBehavior::Block); + (ControlResult::Ack { accepted: true }, false) + } + ControlCommand::ReleaseBlockedCall => ( + ControlResult::Ack { + accepted: backend.release_blocked(), + }, + false, + ), + ControlCommand::TypedFailure => { + backend.set_next(NextBehavior::Failure); + (ControlResult::Ack { accepted: true }, false) + } + ControlCommand::Counters => { + (ControlResult::Counters(backend.counters.snapshot()), false) + } + ControlCommand::GracefulShutdown => { + (ControlResult::Ack { accepted: true }, true) + } + }; + let response = ControlResponse { + id: Some(request.id), + ok: true, + result: Some(result), + error: None, + }; + write_response(&mut stream, &response).await?; + if stop { + shutdown.cancel(); + return Ok(()); + } + continue; + } + Err(error) => { + let code = if error.to_string().contains("unknown variant") { + "unknown_command" + } else { + "malformed_request" + }; + let id = serde_json::from_slice::(&line) + .ok() + .map(|probe| { + let _ = probe.command; + probe.id + }); + rejected(id, code) + } + }, + }; + write_response(&mut stream, &response).await?; + } + } + + async fn run_control_server( + listener: UnixListener, + backend: Arc, + shutdown: CancellationToken, + accepting: tokio::sync::oneshot::Sender<()>, + ) -> io::Result<()> { + let _ = accepting.send(()); + let mut connections = tokio::task::JoinSet::new(); + loop { + tokio::select! { + biased; + () = shutdown.cancelled() => break, + accepted = listener.accept() => { + let (stream, _) = accepted?; + let backend = Arc::clone(&backend); + let shutdown = shutdown.clone(); + connections.spawn(async move { + let _ = handle_control_connection(stream, backend, shutdown).await; + }); + } + Some(_) = connections.join_next(), if !connections.is_empty() => {} + } + } + while connections.join_next().await.is_some() {} + Ok(()) + } + + fn prepare_state_root(path: &Path) -> Result<(), Box> { + if path.as_os_str().is_empty() { + return Err("state root must not be empty".into()); + } + if path.exists() { + let metadata = fs::symlink_metadata(path)?; + if metadata.file_type().is_symlink() || !metadata.is_dir() { + return Err("state root must be a real directory".into()); + } + } else { + fs::create_dir_all(path)?; + } + fs::set_permissions(path, fs::Permissions::from_mode(0o700))?; + if fs::symlink_metadata(path)?.permissions().mode() & 0o777 != 0o700 { + return Err("state root is not owner-only".into()); + } + Ok(()) + } + + fn bind_control_socket(path: &Path) -> Result> { + if let Ok(metadata) = fs::symlink_metadata(path) { + if !metadata.file_type().is_socket() { + return Err("control path exists and is not a socket".into()); + } + fs::remove_file(path)?; + } + let listener = UnixListener::bind(path)?; + fs::set_permissions(path, fs::Permissions::from_mode(0o600))?; + if fs::symlink_metadata(path)?.permissions().mode() & 0o777 != 0o600 { + return Err("control socket is not owner-only".into()); + } + Ok(listener) + } + + fn storage_init(root: &Path) -> HostInit { + let descriptor = cortexkit_store_types::StorageDescriptor { + module_id: "magic-context".to_owned(), + storage_namespace: "mc_cache".to_owned(), + isolation: cortexkit_store_types::Isolation::Module, + backend: cortexkit_store_types::StorageBackend::Sqlite { + path: root.join(STORE_FILE).to_string_lossy().into_owned(), + }, + }; + HostInit { + subc_capabilities: Vec::new(), + storage: Some(serde_json::to_value(descriptor).expect("storage descriptor serializes")), + } + } + + async fn wait_for_publication( + publication: &Path, + host: &mut tokio::task::JoinHandle>, + ) -> Result<(), Box> { + let deadline = tokio::time::Instant::now() + READY_TIMEOUT; + loop { + if let Ok(info) = mc_host::read_connection_file(publication) { + if info.wire_version != 2 { + return Err("fixture published an unsupported wire version".into()); + } + return Ok(()); + } + if host.is_finished() { + return match host.await? { + Ok(()) => Err("host exited before readiness".into()), + Err(error) => Err(Box::new(error)), + }; + } + if tokio::time::Instant::now() >= deadline { + return Err("host did not publish before readiness deadline".into()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + + fn state_root_arg() -> Result> { + let mut args = std::env::args_os().skip(1); + match (args.next(), args.next(), args.next()) { + (Some(flag), Some(path), None) if flag == "--state-root" => Ok(path.into()), + _ => Err("usage: direct_host_fixture --state-root ".into()), + } + } + + pub async fn run() -> Result<(), Box> { + let root = state_root_arg()?; + prepare_state_root(&root)?; + let control_path = root.join(CONTROL_FILE); + let listener = bind_control_socket(&control_path)?; + + let shutdown = CancellationToken::new(); + let backend = ControlledBackend::new(shutdown.clone()); + let (accepting_tx, accepting_rx) = tokio::sync::oneshot::channel(); + let control_shutdown = shutdown.clone(); + let control_backend = Arc::clone(&backend); + let control_task = tokio::spawn(async move { + run_control_server(listener, control_backend, control_shutdown, accepting_tx).await + }); + accepting_rx + .await + .map_err(|_| "control server failed to start")?; + + let publication = + mc_host::runtime_dir_path(Some(&root))?.join(mc_host::CONNECTION_FILE_NAME); + let composite = StaticComposite::new( + mc_module::McHandler::new_with_connection_file(Some(publication.clone())), + synapse_component(), + BrocaComponent::new(backend), + )?; + let config = HostConfig { + data_dir: Some(root.clone()), + daemon_ver: "mc-module/direct-host-fixture".to_owned(), + init: storage_init(&root), + ..Default::default() + }; + let host_shutdown = shutdown.clone(); + let mut host = + tokio::spawn(async move { mc_host::run(composite, config, host_shutdown).await }); + + let signal_shutdown = shutdown.clone(); + let signal_task = tokio::spawn(async move { + let mut signal = + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) + .expect("SIGTERM handler installs"); + if signal.recv().await.is_some() { + signal_shutdown.cancel(); + } + }); + + let ready = wait_for_publication(&publication, &mut host).await; + if ready.is_ok() { + let record = serde_json::json!({ + "status": "ready", + "wire_version": 2, + "catalog": CATALOG, + }); + let mut stdout = io::stdout().lock(); + serde_json::to_writer(&mut stdout, &record)?; + stdout.write_all(b"\n")?; + stdout.flush()?; + } else { + shutdown.cancel(); + } + + let host_result = host.await?; + shutdown.cancel(); + signal_task.abort(); + let _ = signal_task.await; + control_task.await??; + match fs::remove_file(&control_path) { + Ok(()) => {} + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => return Err(error.into()), + } + ready?; + host_result?; + Ok(()) + } +} + +#[cfg(unix)] +#[tokio::main(flavor = "multi_thread", worker_threads = 4)] +async fn main() -> Result<(), Box> { + unix::run().await +} + +#[cfg(not(unix))] +fn main() { + compile_error!("direct_host_fixture is Unix-only"); +} diff --git a/crates/mc-module/src/dispatch.rs b/crates/mc-module/src/dispatch.rs new file mode 100644 index 000000000..89374009e --- /dev/null +++ b/crates/mc-module/src/dispatch.rs @@ -0,0 +1,458 @@ +use std::fmt; +use std::io::{self, Write}; +use std::sync::Arc; + +use serde_json::{Map, Value}; + +/// Maximum body accepted by the version-2 wire contract. +pub const MAX_WIRE_BODY_BYTES: usize = 64 * 1024 * 1024; + +/// Successful response body prepared without an encoded output buffer. +#[derive(Clone)] +pub struct PreparedOutput { + source: PreparedSource, + #[cfg(test)] + encoded_for_test: Arc>>, +} + +#[derive(Clone)] +enum PreparedSource { + Json(Arc), + Exact(Arc>), + Transform(Arc), +} + +struct TransformSegments { + envelope: Map, + messages: Vec, +} + +#[derive(Clone)] +enum PreparedSegmentSource { + Exact(Arc<[u8]>), + Served(crate::transform::ServedMessage), +} + +/// One immutable, already encoded transform message. +#[derive(Clone)] +pub struct PreparedSegment { + source: PreparedSegmentSource, + measured_len: usize, +} + +impl PreparedSegment { + pub fn exact(bytes: Arc<[u8]>) -> Self { + let measured_len = bytes.len(); + Self { + source: PreparedSegmentSource::Exact(bytes), + measured_len, + } + } + + pub(crate) fn served(message: crate::transform::ServedMessage) -> Self { + let measured_len = message.canonical_bytes().len(); + Self { + source: PreparedSegmentSource::Served(message), + measured_len, + } + } + + /// Constructs a deliberately inconsistent segment for length-check tests. + #[doc(hidden)] + pub fn inconsistent_for_test(bytes: Arc<[u8]>, measured_len: usize) -> Self { + Self { + source: PreparedSegmentSource::Exact(bytes), + measured_len, + } + } + + fn bytes(&self) -> &[u8] { + match &self.source { + PreparedSegmentSource::Exact(bytes) => bytes, + PreparedSegmentSource::Served(message) => message.canonical_bytes(), + } + } +} + +impl fmt::Debug for PreparedSegment { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PreparedSegment") + .field("bytes_len", &self.bytes().len()) + .field("measured_len", &self.measured_len) + .finish() + } +} + +impl PreparedOutput { + /// Retains a JSON value for counting and serialization after reservation. + pub fn json(value: Value) -> Self { + Self { + source: PreparedSource::Json(Arc::new(value)), + #[cfg(test)] + encoded_for_test: Arc::new(std::sync::OnceLock::new()), + } + } + + /// Retains existing encoded bytes without copying them into an output body. + pub fn cached_bytes(bytes: Vec) -> Self { + Self { + source: PreparedSource::Exact(Arc::new(bytes)), + #[cfg(test)] + encoded_for_test: Arc::new(std::sync::OnceLock::new()), + } + } + + pub fn transform_segments( + envelope: Value, + messages: Vec, + ) -> Result { + let Value::Object(envelope) = envelope else { + return Err(PreparedOutputError::InvalidTransformEnvelope); + }; + if envelope.get("ck_messages") != Some(&Value::Null) { + return Err(PreparedOutputError::InvalidTransformEnvelope); + } + Ok(Self { + source: PreparedSource::Transform(Arc::new(TransformSegments { envelope, messages })), + #[cfg(test)] + encoded_for_test: Arc::new(std::sync::OnceLock::new()), + }) + } + + /// Measures this immutable source exactly before output reservation. + pub fn measure(&self) -> Result, PreparedOutputError> { + let len = match &self.source { + PreparedSource::Json(value) => measure_json(value)?, + PreparedSource::Exact(bytes) => checked_body_len([bytes.len()])?, + PreparedSource::Transform(segments) => measure_transform(segments)?, + }; + Ok(MeasuredOutput { output: self, len }) + } +} + +#[cfg(test)] +impl std::ops::Deref for PreparedOutput { + type Target = [u8]; + + fn deref(&self) -> &Self::Target { + self.encoded_for_test + .get_or_init(|| { + let measured = self.measure().expect("test response must measure"); + let mut bytes = Vec::with_capacity(measured.len()); + measured + .write_to(&mut bytes) + .expect("test response must serialize"); + bytes + }) + .as_slice() + } +} + +#[cfg(test)] +impl AsRef<[u8]> for PreparedOutput { + fn as_ref(&self) -> &[u8] { + self + } +} + +#[cfg(test)] +impl PartialEq> for PreparedOutput { + fn eq(&self, other: &Vec) -> bool { + &**self == other.as_slice() + } +} + +impl fmt::Debug for PreparedOutput { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let kind = match self.source { + PreparedSource::Json(_) => "json", + PreparedSource::Exact(_) => "exact", + PreparedSource::Transform(_) => "transform", + }; + f.debug_struct("PreparedOutput") + .field("kind", &kind) + .finish() + } +} + +/// Successful, typed-error, and streamed dispatch settlements. +pub enum PreparedOutcome { + Response(PreparedOutput), + Error { code: String, message: String }, + Streamed, +} + +impl fmt::Debug for PreparedOutcome { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Response(body) => f.debug_tuple("Response").field(body).finish(), + Self::Error { code, message } => f + .debug_struct("Error") + .field("code_len", &code.len()) + .field("message_len", &message.len()) + .finish(), + Self::Streamed => f.write_str("Streamed"), + } + } +} + +/// Exact measurement tied to the immutable source that produced it. +pub struct MeasuredOutput<'a> { + output: &'a PreparedOutput, + len: usize, +} + +impl MeasuredOutput<'_> { + pub fn len(&self) -> usize { + self.len + } + + pub fn is_empty(&self) -> bool { + self.len == 0 + } + + /// Writes into a caller-reserved destination and verifies exact length. + pub fn write_to(&self, destination: &mut W) -> Result { + let mut destination = BoundedWriter::new(destination, self.len); + match &self.output.source { + PreparedSource::Json(value) => { + serde_json::to_writer(&mut destination, value) + .map_err(PreparedOutputError::Serialize)?; + } + PreparedSource::Exact(bytes) => destination.write_all(bytes)?, + PreparedSource::Transform(segments) => write_transform(segments, &mut destination)?, + } + let written = destination.written(); + if written != self.len { + return Err(PreparedOutputError::LengthMismatch { + measured: self.len, + written, + }); + } + Ok(written) + } +} + +#[derive(Debug)] +pub enum PreparedOutputError { + BodyTooLarge { len: usize, max: usize }, + LengthOverflow, + InvalidTransformEnvelope, + Serialize(serde_json::Error), + Write(io::Error), + LengthMismatch { measured: usize, written: usize }, +} + +impl fmt::Display for PreparedOutputError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::BodyTooLarge { len, max } => { + write!(f, "prepared body length {len} exceeds wire cap {max}") + } + Self::LengthOverflow => f.write_str("prepared body length overflowed"), + Self::InvalidTransformEnvelope => { + f.write_str("transform envelope must contain a null ck_messages field") + } + Self::Serialize(error) => write!(f, "prepared JSON serialization failed: {error}"), + Self::Write(error) => write!(f, "prepared body write failed: {error}"), + Self::LengthMismatch { measured, written } => write!( + f, + "prepared body length mismatch: measured {measured}, wrote {written}" + ), + } + } +} + +impl std::error::Error for PreparedOutputError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Serialize(error) => Some(error), + Self::Write(error) => Some(error), + Self::BodyTooLarge { .. } + | Self::LengthOverflow + | Self::InvalidTransformEnvelope + | Self::LengthMismatch { .. } => None, + } + } +} + +impl From for PreparedOutputError { + fn from(error: io::Error) -> Self { + Self::Write(error) + } +} + +fn checked_body_len( + lengths: impl IntoIterator, +) -> Result { + let mut total = 0usize; + for len in lengths { + total = total + .checked_add(len) + .ok_or(PreparedOutputError::LengthOverflow)?; + } + if total > MAX_WIRE_BODY_BYTES { + return Err(PreparedOutputError::BodyTooLarge { + len: total, + max: MAX_WIRE_BODY_BYTES, + }); + } + Ok(total) +} + +fn measure_json(value: &Value) -> Result { + let mut counter = CountingWriter::default(); + let result = serde_json::to_writer(&mut counter, value).map_err(PreparedOutputError::Serialize); + finish_count(counter, result) +} + +fn finish_count( + counter: CountingWriter, + result: Result<(), PreparedOutputError>, +) -> Result { + match counter.failure { + Some(CountFailure::Overflow) => Err(PreparedOutputError::LengthOverflow), + Some(CountFailure::TooLarge(len)) => Err(PreparedOutputError::BodyTooLarge { + len, + max: MAX_WIRE_BODY_BYTES, + }), + None => { + result?; + Ok(counter.len) + } + } +} + +fn measure_transform(segments: &TransformSegments) -> Result { + let mut counter = CountingWriter::default(); + let result = write_transform_envelope(segments, &mut counter, |counter, message| { + counter.add_len(message.measured_len) + }); + finish_count(counter, result) +} + +fn write_transform( + segments: &TransformSegments, + destination: &mut W, +) -> Result<(), PreparedOutputError> { + write_transform_envelope(segments, destination, |destination, message| { + destination.write_all(message.bytes())?; + Ok(()) + }) +} + +fn write_transform_envelope( + segments: &TransformSegments, + destination: &mut W, + mut write_message: impl FnMut(&mut W, &PreparedSegment) -> Result<(), PreparedOutputError>, +) -> Result<(), PreparedOutputError> { + destination.write_all(b"{")?; + for (index, (key, value)) in segments.envelope.iter().enumerate() { + if index > 0 { + destination.write_all(b",")?; + } + serde_json::to_writer(&mut *destination, key).map_err(PreparedOutputError::Serialize)?; + destination.write_all(b":")?; + if key == "ck_messages" { + destination.write_all(b"[")?; + for (message_index, message) in segments.messages.iter().enumerate() { + if message_index > 0 { + destination.write_all(b",")?; + } + write_message(destination, message)?; + } + destination.write_all(b"]")?; + } else { + serde_json::to_writer(&mut *destination, value) + .map_err(PreparedOutputError::Serialize)?; + } + } + destination.write_all(b"}")?; + Ok(()) +} + +#[derive(Debug, Clone, Copy)] +enum CountFailure { + Overflow, + TooLarge(usize), +} + +#[derive(Default)] +struct CountingWriter { + len: usize, + failure: Option, +} + +impl CountingWriter { + fn add_len(&mut self, len: usize) -> Result<(), PreparedOutputError> { + let Some(next) = self.len.checked_add(len) else { + self.failure = Some(CountFailure::Overflow); + return Err(PreparedOutputError::LengthOverflow); + }; + if next > MAX_WIRE_BODY_BYTES { + self.failure = Some(CountFailure::TooLarge(next)); + return Err(PreparedOutputError::BodyTooLarge { + len: next, + max: MAX_WIRE_BODY_BYTES, + }); + } + self.len = next; + Ok(()) + } +} + +impl Write for CountingWriter { + fn write(&mut self, bytes: &[u8]) -> io::Result { + self.add_len(bytes.len()) + .map_err(|error| io::Error::new(io::ErrorKind::FileTooLarge, error.to_string()))?; + Ok(bytes.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +struct BoundedWriter<'a, W> { + inner: &'a mut W, + max_len: usize, + written: usize, +} + +impl<'a, W> BoundedWriter<'a, W> { + fn new(inner: &'a mut W, max_len: usize) -> Self { + Self { + inner, + max_len, + written: 0, + } + } + + fn written(&self) -> usize { + self.written + } +} + +impl Write for BoundedWriter<'_, W> { + fn write(&mut self, bytes: &[u8]) -> io::Result { + if bytes.len() > self.max_len - self.written { + return Err(io::Error::new( + io::ErrorKind::WriteZero, + "prepared body exceeded measured length", + )); + } + let written = self.inner.write(bytes)?; + if written > bytes.len() { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "destination reported an invalid write length", + )); + } + self.written += written; + Ok(written) + } + + fn flush(&mut self) -> io::Result<()> { + self.inner.flush() + } +} diff --git a/crates/mc-module/src/historian.rs b/crates/mc-module/src/historian.rs index 3ec554b7b..4f8194f31 100644 --- a/crates/mc-module/src/historian.rs +++ b/crates/mc-module/src/historian.rs @@ -19,7 +19,7 @@ use mc_store::{ use crate::historian_producer::{ attach_cleanup, ErrorClass, ErrorClassification, HistorianProducer, HistorianProducerError, - ProducerOutput, RunHandle, RunState, + HistorianSendOutcome, ProducerOutput, RunHandle, RunState, }; use crate::historian_validate::{ validate_historian_output, HistorianChunk, HistorianValidationError, StoredCompartmentRange, @@ -758,7 +758,7 @@ impl From for HistorianDriveError { } } -#[subc_client_rs::async_trait] +#[async_trait::async_trait] pub trait HistorianProducerDriver: Send { async fn bind_session(&mut self, session_id: &str) -> Result<(), HistorianProducerError>; async fn start( @@ -805,6 +805,9 @@ pub trait HistorianProducerDriver: Send { } async fn status(&mut self, run_id: &str) -> Result; async fn cancel(&mut self, run_id: &str) -> Result<(), HistorianProducerError>; + async fn close_attempt(&mut self) -> Result<(), HistorianProducerError> { + self.close().await + } async fn close(&mut self) -> Result<(), HistorianProducerError>; /// Delete the provider session on every terminal path. The default calls close() /// for compatibility with older test producers, while production producers override @@ -814,7 +817,7 @@ pub trait HistorianProducerDriver: Send { } } -#[subc_client_rs::async_trait] +#[async_trait::async_trait] impl HistorianProducerDriver for HistorianProducer { async fn bind_session(&mut self, session_id: &str) -> Result<(), HistorianProducerError> { HistorianProducer::bind_session(self, session_id.to_string()); @@ -890,6 +893,10 @@ impl HistorianProducerDriver for HistorianProducer { HistorianProducer::cancel(self, run_id).await } + async fn close_attempt(&mut self) -> Result<(), HistorianProducerError> { + HistorianProducer::close_attempt(self).await + } + async fn close(&mut self) -> Result<(), HistorianProducerError> { HistorianProducer::close(self).await } @@ -1074,6 +1081,14 @@ fn decide_producer_failure( now_ms: i64, default_failure_backoff_at_ms: i64, ) -> ProducerFailureDecision { + if err.is_cross_incarnation_unknown() { + *all_failures_permanent = false; + return ProducerFailureDecision { + try_next_model: false, + failure_backoff_at_ms: default_failure_backoff_at_ms, + detail_prefix: None, + }; + } if let Some(classification) = err.classification() { // The producer owns classification. Once a class tag is present, the consumer // branches only on that field and its structured retry-after sibling; provider @@ -1221,14 +1236,14 @@ fn log_cleanup_failure( } } -/// True when a cancel could not prove the run's harness process group -/// stopped. The host reports this as the `teardown_unconfirmed` request -/// error, which the wire producer surfaces verbatim as a `Subc` body code. -fn cancel_left_work_unconfirmed(result: &Result<(), HistorianProducerError>) -> bool { - matches!( - result, - Err(HistorianProducerError::Subc(body)) if body.code == "teardown_unconfirmed" - ) +fn cancellation_confirmed_stopped(result: &Result<(), HistorianProducerError>) -> bool { + match result { + Ok(()) => true, + Err(error) => { + error.send_outcome() == Some(HistorianSendOutcome::Terminal) + && error.code() != Some("teardown_unconfirmed") + } + } } pub async fn run_historian_firing

( @@ -1307,11 +1322,12 @@ where )), ), )?; - let close_result = producer.close().await; if decision.try_next_model { - log_cleanup_failure(request.session_id, "close", &close_result); + let cleanup = producer.close_attempt().await; + log_cleanup_failure(request.session_id, "attempt close", &cleanup); continue; } + let close_result = producer.close().await; return Err(HistorianDriveError::Producer(attach_cleanup( err, close_result, @@ -1386,19 +1402,16 @@ where )), ), )?; - let close_result = producer.close().await; - // Fallback requires the failed attempt to actually be over: - // a cancel that reports `teardown_unconfirmed` means the - // host could not prove the provider descendant stopped, and - // starting the next model would run a second billable - // producer beside it under a fresh firing sequence. The - // firing ends here instead; the durable abandon above keeps - // the state recoverable and backoff-gated. - if decision.try_next_model && !cancel_left_work_unconfirmed(&cancel_result) { + // Fallback requires typed proof that the failed attempt is over. + // Transport failures and uncertain send outcomes cannot prove + // the cancellation reached and stopped the provider run. + if decision.try_next_model && cancellation_confirmed_stopped(&cancel_result) { + let cleanup = producer.close_attempt().await; log_cleanup_failure(request.session_id, "cancel", &cancel_result); - log_cleanup_failure(request.session_id, "close", &close_result); + log_cleanup_failure(request.session_id, "attempt close", &cleanup); continue; } + let close_result = producer.close().await; return Err(HistorianDriveError::Producer(attach_cleanup( attach_cleanup(err, cancel_result, "cancel"), close_result, @@ -1429,19 +1442,25 @@ where completion_now_ms: request.completion_now_ms, publication_fence: request.publication_fence, }); - log_cleanup_failure(request.session_id, "close", &producer.close().await); let row_version = match publish_result { Ok(row_version) => row_version, Err(HistorianDriveError::Validation(err)) => { // Validation rejection is model-local output failure. Exhaust the // configured fallback chain before returning the final rejection. if has_eligible_model(&request.model_chain[index + 1..], &auth_blocked_providers) { + let cleanup = producer.close_attempt().await; + log_cleanup_failure(request.session_id, "attempt close", &cleanup); continue; } + log_cleanup_failure(request.session_id, "close", &producer.close().await); return Err(HistorianDriveError::Validation(err)); } - Err(err) => return Err(err), + Err(err) => { + log_cleanup_failure(request.session_id, "close", &producer.close().await); + return Err(err); + } }; + log_cleanup_failure(request.session_id, "close", &producer.close().await); return Ok(HistorianDriveOutcome::Completed(HistorianRunSuccess { row_version, producer_session_id, @@ -2132,7 +2151,9 @@ mod tests { observed_systems: Vec, await_run_ids: Vec, cancels: Vec, + attempt_closes: usize, closes: usize, + connection_closed: bool, on_await_output: Option>, } @@ -2168,7 +2189,7 @@ mod tests { } } - #[subc_client_rs::async_trait] + #[async_trait::async_trait] impl HistorianProducerDriver for ScriptedProducer { async fn bind_session(&mut self, session_id: &str) -> Result<(), HistorianProducerError> { self.observed_sessions.push(session_id.to_string()); @@ -2182,6 +2203,14 @@ mod tests { _prompt: &str, model: &str, ) -> Result { + if self.connection_closed { + return Err(HistorianProducerError::tagged_call( + "connection_closed", + "managed producer connection is closed", + ErrorClass::Permanent, + None, + )); + } self.observed_sessions.push(session_id.to_string()); self.observed_systems.push(system.to_string()); self.observed_starts @@ -2215,8 +2244,14 @@ mod tests { self.cancel_results.pop_front().unwrap_or(Ok(())) } + async fn close_attempt(&mut self) -> Result<(), HistorianProducerError> { + self.attempt_closes += 1; + self.close_results.pop_front().unwrap_or(Ok(())) + } + async fn close(&mut self) -> Result<(), HistorianProducerError> { self.closes += 1; + self.connection_closed = true; self.close_results.pop_front().unwrap_or(Ok(())) } } @@ -2582,6 +2617,42 @@ mod tests { assert_eq!(state.firing_seq, 1); } + #[tokio::test] + async fn cross_incarnation_unknown_records_completion_backoff_without_fallback() { + fn completed_at() -> i64 { + 10_000 + } + + let dir = tempfile::tempdir().unwrap(); + let store = store(dir.path()); + seed_prior_compartment(&store); + let chunk = historian_chunk(); + let prior = prior_ranges(); + let models = vec!["prov/model-a".to_owned(), "prov/model-b".to_owned()]; + let mut producer = ScriptedProducer::default().with_start(Err( + HistorianProducerError::CrossIncarnationUnknown { + daemon_changed: true, + identity_changed: false, + }, + )); + let mut request = fire_request(&store, "placeholder prompt", &models, &chunk, &prior); + request.completion_now_ms = completed_at; + + let error = run_historian_firing(&mut producer, request) + .await + .unwrap_err(); + + assert!(matches!( + error, + HistorianDriveError::Producer(HistorianProducerError::CrossIncarnationUnknown { .. }) + )); + assert_eq!(producer.observed_starts.len(), 1); + assert_eq!(producer.closes, 1); + let historian = store.load("ses").unwrap().meta.historian; + assert_eq!(historian.state, HistorianPhase::Idle); + assert_eq!(historian.failure_backoff_at_ms, Some(10_876)); + } + #[tokio::test] async fn permanent_class_advances_chain_immediately() { let dir = tempfile::tempdir().unwrap(); @@ -2591,7 +2662,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; let mut producer = ScriptedProducer::default() - .with_start(Err(HistorianProducerError::tagged_subc( + .with_start(Err(HistorianProducerError::tagged_call( "provider_error", "model id does not exist", ErrorClass::Permanent, @@ -2614,6 +2685,45 @@ mod tests { }; assert_eq!(success.model, "prov/model-b"); assert_eq!(producer.observed_starts.len(), 2); + assert_eq!(producer.attempt_closes, 1); + assert_eq!(producer.closes, 1); + assert!(producer.connection_closed); + } + + #[tokio::test] + async fn output_failure_closes_attempt_routes_but_keeps_connection_for_fallback() { + let dir = tempfile::tempdir().unwrap(); + let store = store(dir.path()); + seed_prior_compartment(&store); + let chunk = historian_chunk(); + let prior = prior_ranges(); + let models = vec!["prov/model-a".to_string(), "other/model-b".to_string()]; + let mut producer = ScriptedProducer::default() + .with_start(Ok(run_handle("run-1"))) + .with_output(Err(HistorianProducerError::tagged_call( + "provider_error", + "provider unavailable", + ErrorClass::Transient, + None, + ))) + .with_start(Ok(run_handle("run-2"))) + .with_output(Ok(producer_output(historian_xml("fallback output")))); + + let outcome = run_historian_firing( + &mut producer, + fire_request(&store, "placeholder prompt", &models, &chunk, &prior), + ) + .await + .expect("fallback uses still-open managed connection"); + + let HistorianDriveOutcome::Completed(success) = outcome else { + panic!("expected fallback completion"); + }; + assert_eq!(success.model, "other/model-b"); + assert_eq!(producer.observed_starts.len(), 2); + assert_eq!(producer.attempt_closes, 1); + assert_eq!(producer.closes, 1); + assert!(producer.connection_closed); } #[tokio::test] @@ -2625,13 +2735,13 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string(), "other/model-b".to_string()]; let mut producer = ScriptedProducer::default() - .with_start(Err(HistorianProducerError::tagged_subc( + .with_start(Err(HistorianProducerError::tagged_call( "provider_error", "model a is permanently unavailable", ErrorClass::Permanent, None, ))) - .with_start(Err(HistorianProducerError::tagged_subc( + .with_start(Err(HistorianProducerError::tagged_call( "provider_error", "model b is permanently unavailable", ErrorClass::Permanent, @@ -2668,7 +2778,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string()]; let mut producer = - ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_subc( + ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_call( "provider_error", "short retry-after should not shorten our schedule", ErrorClass::Transient, @@ -2700,7 +2810,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string()]; let mut producer = - ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_subc( + ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_call( "provider_error", "rate limit reset later", ErrorClass::Transient, @@ -2732,7 +2842,7 @@ mod tests { "anthropic/model-c".to_string(), ]; let mut producer = ScriptedProducer::default() - .with_start(Err(HistorianProducerError::tagged_subc( + .with_start(Err(HistorianProducerError::tagged_call( "provider_error", "credential needs re-authentication", ErrorClass::AuthRequired, @@ -2779,7 +2889,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["openai/model-a".to_string(), "openai/model-b".to_string()]; let mut producer = - ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_subc( + ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_call( "provider_error", "credential needs re-authentication", ErrorClass::AuthRequired, @@ -2815,7 +2925,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string(), "other/model-b".to_string()]; let mut producer = - ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_subc( + ScriptedProducer::default().with_start(Err(HistorianProducerError::tagged_call( "provider_error", "context window exceeded", ErrorClass::ContextOverflow, @@ -2872,7 +2982,7 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; let mut producer = ScriptedProducer::default() - .with_start(Err(HistorianProducerError::tagged_subc( + .with_start(Err(HistorianProducerError::tagged_call( "context_overflow", "overflow text would block retry under the deprecated heuristic", ErrorClass::Permanent, @@ -3352,7 +3462,7 @@ mod tests { let models = vec!["prov/model-a".to_string()]; let mut producer = ScriptedProducer::default() .with_start(Ok(run_handle("run-1"))) - .with_output(Err(HistorianProducerError::tagged_subc( + .with_output(Err(HistorianProducerError::tagged_call( "provider_error", "model gone", ErrorClass::Permanent, @@ -3414,13 +3524,13 @@ mod tests { // scripted queue, so completion alone proves the chain stopped. let mut producer = ScriptedProducer::default() .with_start(Ok(run_handle("run-1"))) - .with_output(Err(HistorianProducerError::tagged_subc( + .with_output(Err(HistorianProducerError::tagged_call( "provider_error", "provider overloaded", ErrorClass::Transient, None, ))) - .with_cancel_result(Err(HistorianProducerError::tagged_subc( + .with_cancel_result(Err(HistorianProducerError::tagged_call( "teardown_unconfirmed", "the harness process group could not be confirmed stopped", ErrorClass::Transient, @@ -3457,6 +3567,94 @@ mod tests { assert!(state.failure_backoff_at_ms.is_some()); } + #[tokio::test] + async fn uncertain_cancel_send_outcomes_stop_the_fallback_chain() { + for outcome in [ + HistorianSendOutcome::NotSent, + HistorianSendOutcome::OutcomeUnknown, + ] { + let dir = tempfile::tempdir().unwrap(); + let store = store(dir.path()); + seed_prior_compartment(&store); + let chunk = historian_chunk(); + let prior = prior_ranges(); + let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; + let mut producer = ScriptedProducer::default() + .with_start(Ok(run_handle("run-1"))) + .with_output(Err(HistorianProducerError::tagged_call( + "provider_error", + "provider overloaded", + ErrorClass::Transient, + None, + ))) + .with_cancel_result(Err(HistorianProducerError::Call( + crate::historian_producer::HistorianCallFailure::untagged( + outcome, + "cancel_transport_failure", + "cancel was not confirmed", + ), + ))); + + let err = run_historian_firing( + &mut producer, + fire_request(&store, "placeholder prompt", &models, &chunk, &prior), + ) + .await + .unwrap_err(); + + assert!(matches!(err, HistorianDriveError::Producer(_))); + assert_eq!(producer.cancels, vec!["run-1"]); + assert_eq!( + producer.observed_starts.len(), + 1, + "{outcome:?} cancellation must not start fallback model" + ); + let state = store.load("ses").unwrap().meta.historian; + assert_eq!(state.state, HistorianPhase::Idle); + assert_eq!(state.failure_backoff_at_ms, Some(999)); + } + } + + #[tokio::test] + async fn terminal_cancel_response_allows_fallback() { + let dir = tempfile::tempdir().unwrap(); + let store = store(dir.path()); + seed_prior_compartment(&store); + let chunk = historian_chunk(); + let prior = prior_ranges(); + let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; + let mut producer = ScriptedProducer::default() + .with_start(Ok(run_handle("run-1"))) + .with_output(Err(HistorianProducerError::tagged_call( + "provider_error", + "provider overloaded", + ErrorClass::Transient, + None, + ))) + .with_cancel_result(Err(HistorianProducerError::Call( + crate::historian_producer::HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, + "run_already_terminal", + "run is already stopped", + ), + ))) + .with_start(Ok(run_handle("run-2"))) + .with_output(Ok(producer_output(historian_xml("fallback output")))); + + let outcome = run_historian_firing( + &mut producer, + fire_request(&store, "placeholder prompt", &models, &chunk, &prior), + ) + .await + .expect("terminal cancel response proves fallback is safe"); + + let HistorianDriveOutcome::Completed(success) = outcome else { + panic!("expected fallback completion"); + }; + assert_eq!(success.model, "prov/model-b"); + assert_eq!(producer.observed_starts.len(), 2); + } + #[tokio::test] async fn close_failure_after_publish_keeps_the_completed_outcome() { let dir = tempfile::tempdir().unwrap(); @@ -3598,8 +3796,12 @@ mod tests { let prior = prior_ranges(); let models = vec!["prov/model-a".to_string()]; let mut producer = - ScriptedProducer::default().with_start(Err(HistorianProducerError::Subc( - subc_protocol::ErrorBody::new("route_rejected", "no such module").into(), + ScriptedProducer::default().with_start(Err(HistorianProducerError::Call( + crate::historian_producer::HistorianCallFailure::untagged( + crate::historian_producer::HistorianSendOutcome::Terminal, + "route_rejected", + "no such module", + ), ))); let err = run_historian_firing( @@ -3832,6 +4034,9 @@ mod tests { }; assert_eq!(success.model, "other/model-b"); assert_eq!(producer.observed_starts.len(), 2); + assert_eq!(producer.attempt_closes, 1); + assert_eq!(producer.closes, 1); + assert!(producer.connection_closed); assert_eq!( store .load_historian_assembly_snapshot("ses") diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 564a6971d..49fc7e5d0 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -1,64 +1,34 @@ -//! llm-runner session client used by the historian writer. +//! Broca session client used by historian writer. //! -//! The client intentionally speaks only the JSON session wire over subc routes. It -//! does not depend on llm-runner Rust crates, so Magic Context remains an origin- -//! agnostic consumer module. +//! Transport, authentication, correlation, liveness, and route epochs remain owned by +//! [`mc_host::Client`]. This module interprets only Broca request and stream payloads. use std::{ error::Error, fmt, path::PathBuf, - sync::atomic::{AtomicU64, Ordering}, + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, time::Duration, }; -use serde_json::{json, Value}; -use subc_control::{ClientControlRequest, ClientControlResponse, ConsumerIdentity}; -use subc_protocol::{ - BindIdentity, ErrorBody, Flags, Frame, FrameBuildError, FrameType, Priority, RouteTarget, - SUBC_LAUNCH_NONCE_ENV, SUBC_MODULE_ID_ENV, +use async_trait::async_trait; +use mc_host::{ + CallError, Client, ClientError, RequestOptions, ResponseStream, RouteHandle, RouteIdentity, + RouteTarget, SendOutcome, StreamItem, TargetKind, }; -use subc_transport::{ - authenticate_client, connection_file, read_frame, write_frame, AuthError, ConnectionFileError, - FrameIoError, -}; -use tokio::net::TcpStream; +use serde_json::{json, Value}; +use tokio_util::sync::CancellationToken; -/// The owned-leg runner module the historian producer opens routes to. Renamed -/// llm-runner -> broca in the fleet cut; this binary ships in the same deploy -/// beat as the daemon's module-key rename, so the default flips with it -/// atomically (a full daemon kickstart bounces every module in that window). const DEFAULT_RUNNER_MODULE_ID: &str = "broca"; - -/// Output budget for a historian summarization pass. llm-runner's default (4k) truncated -/// a real 50k-input chunk mid-XML on the rig: a tiered compartment doc for a full chunk -/// legitimately needs five figures. The provider clamps to its own per-model limit, so a -/// generous request costs nothing unless the model actually generates that much. const HISTORIAN_MAX_OUTPUT_TOKENS: u32 = 32_000; - -/// Sampling temperature for historian runs. The prompt and this value were calibrated -/// TOGETHER: at provider-default temperature (1.0) flash-class models drift past the -/// prompt's exclusion rules and copy the format template and rotating-seed reference -/// compartments into their output (observed live on the rig with the calibration model -/// itself), while at 0.1 the same prompt extracts cleanly. Sending the prompt without -/// the temperature is running half the calibration. const HISTORIAN_TEMPERATURE: f64 = 0.1; -const DEFAULT_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(2); const DEFAULT_REQUEST_TIMEOUT: Duration = Duration::from_secs(30); -/// How long to wait for a summarization run to finish. A historian pass legitimately -/// generates 10k+ output tokens and can run several minutes on a flash-class model; a -/// 120s window abandoned a run on the rig WHILE it was still successfully finishing -/// (the terminal arrived moments after the waiter gave up). A fold is a background -/// operation, never latency-sensitive: waiting longer and publishing always beats -/// abandoning a completed run and re-firing the whole 50k-input pass. const DEFAULT_AWAIT_TIMEOUT: Duration = Duration::from_secs(600); -/// How long the timeout recovery path gets to re-drain a run after the main -/// waiter gives up. A completed historian run can land moments after the -/// 600s wait expires; one short replay salvages that durable output without -/// letting the fallback path hang for another full production timeout. const RECOVERY_REDRAIN_TIMEOUT: Duration = Duration::from_secs(60); -/// Fixed error-class strings used by the producer/consumer wire contract. pub const ERROR_CLASS_WIRE_SET: [&str; 4] = [ "transient", "permanent", @@ -77,21 +47,21 @@ pub enum ErrorClass { } impl ErrorClass { - pub fn as_wire_str(self) -> &'static str { + pub const fn as_wire_str(self) -> &'static str { match self { - ErrorClass::Transient => ERROR_CLASS_WIRE_SET[0], - ErrorClass::Permanent => ERROR_CLASS_WIRE_SET[1], - ErrorClass::AuthRequired => ERROR_CLASS_WIRE_SET[2], - ErrorClass::ContextOverflow => ERROR_CLASS_WIRE_SET[3], + Self::Transient => ERROR_CLASS_WIRE_SET[0], + Self::Permanent => ERROR_CLASS_WIRE_SET[1], + Self::AuthRequired => ERROR_CLASS_WIRE_SET[2], + Self::ContextOverflow => ERROR_CLASS_WIRE_SET[3], } } pub fn from_wire(s: &str) -> Option { match s { - "transient" => Some(ErrorClass::Transient), - "permanent" => Some(ErrorClass::Permanent), - "auth_required" | "auth" => Some(ErrorClass::AuthRequired), - "context_overflow" => Some(ErrorClass::ContextOverflow), + "transient" => Some(Self::Transient), + "permanent" => Some(Self::Permanent), + "auth_required" | "auth" => Some(Self::AuthRequired), + "context_overflow" => Some(Self::ContextOverflow), _ => None, } } @@ -103,17 +73,40 @@ pub struct ErrorClassification { pub retry_after_secs: Option, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HistorianSendOutcome { + NotSent, + OutcomeUnknown, + Terminal, +} + +impl From for HistorianSendOutcome { + fn from(value: SendOutcome) -> Self { + match value { + SendOutcome::NotSent => Self::NotSent, + SendOutcome::OutcomeUnknown => Self::OutcomeUnknown, + SendOutcome::Terminal => Self::Terminal, + } + } +} + #[derive(Debug, Clone, PartialEq, Eq)] -pub struct ProducerErrorBody { +pub struct HistorianCallFailure { + pub outcome: HistorianSendOutcome, pub code: String, pub message: String, classification: Option, class_field_present: bool, } -impl ProducerErrorBody { - pub fn untagged(code: impl Into, message: impl Into) -> Self { +impl HistorianCallFailure { + pub fn untagged( + outcome: HistorianSendOutcome, + code: impl Into, + message: impl Into, + ) -> Self { Self { + outcome, code: code.into(), message: message.into(), classification: None, @@ -128,6 +121,7 @@ impl ProducerErrorBody { retry_after_secs: Option, ) -> Self { Self { + outcome: HistorianSendOutcome::Terminal, code: code.into(), message: message.into(), classification: Some(ErrorClassification { @@ -138,38 +132,37 @@ impl ProducerErrorBody { } } - pub fn classification(&self) -> Option { + pub const fn classification(&self) -> Option { self.classification } - pub fn has_class_field(&self) -> bool { + pub const fn has_class_field(&self) -> bool { self.class_field_present } +} - fn from_value(value: Value) -> Self { - let code = value - .get("code") - .and_then(Value::as_str) - .unwrap_or("producer_error") - .to_string(); - let message = value - .get("message") - .and_then(Value::as_str) - .unwrap_or("producer error") - .to_string(); - let (classification, class_field_present) = classification_from_object(&value); - Self { - code, - message, - classification, - class_field_present, - } +impl From for HistorianCallFailure { + fn from(error: CallError) -> Self { + Self::untagged( + error.outcome().into(), + error.code().to_owned(), + error.message().to_owned(), + ) } } -impl From for ProducerErrorBody { - fn from(body: ErrorBody) -> Self { - Self::untagged(body.code, body.message) +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HistorianClientFailure { + pub code: String, + pub message: String, +} + +impl From for HistorianClientFailure { + fn from(error: ClientError) -> Self { + Self { + code: error.code().to_owned(), + message: error.to_string(), + } } } @@ -196,9 +189,6 @@ pub struct RunHandle { #[derive(Debug, Clone, PartialEq, Eq)] pub struct ProducerOutput { pub text: String, - /// True when any unit in the run reported a length-class finish reason. The run - /// terminal can still say completed while a model step hit its output ceiling and - /// cut the text mid-document, so validation failures need this to self-diagnose. pub length_capped: bool, } @@ -215,9 +205,9 @@ pub struct HistorianProducerConfig { pub project_root: PathBuf, pub harness: String, pub module_id: String, - pub handshake_timeout: Duration, pub request_timeout: Duration, pub await_timeout: Duration, + pub cancellation: Option, } impl HistorianProducerConfig { @@ -230,45 +220,28 @@ impl HistorianProducerConfig { connection_file: connection_file.into(), project_root: project_root.into(), harness: harness.into(), - module_id: DEFAULT_RUNNER_MODULE_ID.to_string(), - handshake_timeout: DEFAULT_HANDSHAKE_TIMEOUT, + module_id: DEFAULT_RUNNER_MODULE_ID.to_owned(), request_timeout: DEFAULT_REQUEST_TIMEOUT, await_timeout: DEFAULT_AWAIT_TIMEOUT, + cancellation: None, } } } #[derive(Debug)] pub enum HistorianProducerError { - ConnectionFile { - path: PathBuf, - source: ConnectionFileError, - }, - NoEndpoint { - path: PathBuf, - }, - Connect { - endpoint: String, - source: std::io::Error, - }, - Auth(AuthError), - FrameIo(FrameIoError), - FrameBuild(FrameBuildError), + Client(HistorianClientFailure), + Call(HistorianCallFailure), Json(serde_json::Error), - Subc(ProducerErrorBody), - UnexpectedControlResponse, MissingRunId, MissingSession, UnexpectedStreamEnd, TimedOut, - /// The runner answered with bytes outside the closed wire contract (an - /// undocumented `run.status` state, a status for a different run, ...). - /// Guessing a recovery state for such a reply could silently re-fire a - /// billable run, so the producer fails loud instead. Protocol(String), - /// Neither a primary failure nor a cleanup (cancel/delete/close) failure - /// may mask the other, so both ride one structured error. `primary: None` - /// means the operation itself succeeded and only cleanup failed. + CrossIncarnationUnknown { + daemon_changed: bool, + identity_changed: bool, + }, CleanupFailed { operation: &'static str, primary: Option>, @@ -294,27 +267,36 @@ pub enum HistorianProducerError { impl HistorianProducerError { pub fn retryable_model_failure(message: impl Into) -> Self { - HistorianProducerError::Subc(ProducerErrorBody::untagged( + Self::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, "retryable_model_failure", message, )) } pub fn context_overflow(message: impl Into) -> Self { - HistorianProducerError::Subc(ProducerErrorBody::untagged("context_overflow", message)) + Self::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, + "context_overflow", + message, + )) } pub fn aborted(message: impl Into) -> Self { - HistorianProducerError::Subc(ProducerErrorBody::untagged("aborted", message)) + Self::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, + "aborted", + message, + )) } - pub fn tagged_subc( + pub fn tagged_call( code: impl Into, message: impl Into, class: ErrorClass, retry_after_secs: Option, ) -> Self { - HistorianProducerError::Subc(ProducerErrorBody::tagged( + Self::Call(HistorianCallFailure::tagged( code, message, class, @@ -324,12 +306,11 @@ impl HistorianProducerError { pub fn classification(&self) -> Option { match self { - HistorianProducerError::Subc(body) => body.classification(), - HistorianProducerError::RunFailed { classification, .. } - | HistorianProducerError::RunPaused { classification, .. } => *classification, - // Retry policy must follow the PRIMARY failure; a cleanup failure - // never changes what the run itself did. - HistorianProducerError::CleanupFailed { + Self::Call(failure) => failure.classification(), + Self::RunFailed { classification, .. } | Self::RunPaused { classification, .. } => { + *classification + } + Self::CleanupFailed { primary: Some(primary), .. } => primary.classification(), @@ -339,16 +320,16 @@ impl HistorianProducerError { pub fn has_class_field(&self) -> bool { match self { - HistorianProducerError::Subc(body) => body.has_class_field(), - HistorianProducerError::RunFailed { + Self::Call(failure) => failure.has_class_field(), + Self::RunFailed { class_field_present, .. } - | HistorianProducerError::RunPaused { + | Self::RunPaused { class_field_present, .. } => *class_field_present, - HistorianProducerError::CleanupFailed { + Self::CleanupFailed { primary: Some(primary), .. } => primary.has_class_field(), @@ -356,6 +337,43 @@ impl HistorianProducerError { } } + pub fn code(&self) -> Option<&str> { + match self { + Self::Call(failure) => Some(&failure.code), + Self::CleanupFailed { + primary: Some(primary), + .. + } => primary.code(), + _ => None, + } + } + + pub(crate) fn send_outcome(&self) -> Option { + match self { + Self::Call(failure) => Some(failure.outcome), + Self::CleanupFailed { + primary: Some(primary), + .. + } => primary.send_outcome(), + _ => None, + } + } + + pub fn is_unknown_module(&self) -> bool { + self.code() == Some("unknown_module") + } + + pub fn is_cross_incarnation_unknown(&self) -> bool { + match self { + Self::CrossIncarnationUnknown { .. } => true, + Self::CleanupFailed { + primary: Some(primary), + .. + } => primary.is_cross_incarnation_unknown(), + _ => false, + } + } + pub(crate) fn deprecated_heuristic_decision(&self) -> DeprecatedHeuristicDecision { record_deprecated_heuristic_use(self.heuristic_log_code()); self.heuristic_decision() @@ -381,30 +399,19 @@ impl HistorianProducerError { self.heuristic_decision().abort_or_overflow } - pub fn is_unknown_module(&self) -> bool { - match self { - HistorianProducerError::Subc(body) => body.code == "unknown_module", - HistorianProducerError::CleanupFailed { - primary: Some(primary), - .. - } => primary.is_unknown_module(), - _ => false, - } - } - fn heuristic_decision(&self) -> DeprecatedHeuristicDecision { match self { - HistorianProducerError::CleanupFailed { + Self::CleanupFailed { primary: Some(primary), .. } => primary.heuristic_decision(), - HistorianProducerError::Subc(body) => DeprecatedHeuristicDecision { - retryable_model_failure: retryable_code(&body.code) - || retryable_code(&body.message), - abort_or_overflow: abort_or_overflow(&body.code) - || abort_or_overflow(&body.message), + Self::Call(failure) => DeprecatedHeuristicDecision { + retryable_model_failure: retryable_code(&failure.code) + || retryable_code(&failure.message), + abort_or_overflow: abort_or_overflow(&failure.code) + || abort_or_overflow(&failure.message), }, - HistorianProducerError::RunFailed { detail, .. } => DeprecatedHeuristicDecision { + Self::RunFailed { detail, .. } => DeprecatedHeuristicDecision { retryable_model_failure: retryable_code(detail), abort_or_overflow: abort_or_overflow(detail), }, @@ -417,11 +424,12 @@ impl HistorianProducerError { fn heuristic_log_code(&self) -> &str { match self { - HistorianProducerError::Subc(body) => &body.code, - HistorianProducerError::RunFailed { .. } => "run_failed", - HistorianProducerError::RunPaused { .. } => "run_paused", - HistorianProducerError::TimedOut => "timed_out", - HistorianProducerError::CleanupFailed { .. } => "cleanup_failed", + Self::Call(failure) => &failure.code, + Self::RunFailed { .. } => "run_failed", + Self::RunPaused { .. } => "run_paused", + Self::TimedOut => "timed_out", + Self::CleanupFailed { .. } => "cleanup_failed", + Self::CrossIncarnationUnknown { .. } => "cross_incarnation_unknown", _ => "producer_error", } } @@ -452,40 +460,28 @@ fn abort_or_overflow(s: &str) -> bool { impl fmt::Display for HistorianProducerError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { - HistorianProducerError::ConnectionFile { path, source } => { - write!(f, "read connection file {}: {source}", path.display()) - } - HistorianProducerError::NoEndpoint { path } => { - write!(f, "connection file {} has no endpoint", path.display()) - } - HistorianProducerError::Connect { endpoint, source } => { - write!(f, "connect to {endpoint}: {source}") - } - HistorianProducerError::Auth(e) => write!(f, "authenticate to subc: {e}"), - HistorianProducerError::FrameIo(e) => write!(f, "subc frame I/O: {e}"), - HistorianProducerError::FrameBuild(e) => write!(f, "build subc frame: {e}"), - HistorianProducerError::Json(e) => write!(f, "json: {e}"), - HistorianProducerError::Subc(body) => { - write!(f, "subc error {}: {}", body.code, body.message) - } - HistorianProducerError::UnexpectedControlResponse => { - write!(f, "route.open returned an unexpected control response") - } - HistorianProducerError::MissingRunId => { - write!(f, "session.send did not return an active run_id") - } - HistorianProducerError::MissingSession => { - write!(f, "historian producer has no bound session") - } - HistorianProducerError::UnexpectedStreamEnd => write!( + Self::Client(error) => write!(f, "host client {}: {}", error.code, error.message), + Self::Call(failure) => write!( f, - "subscribe stream ended before the run terminal control unit" + "historian call {} ({:?}): {}", + failure.code, failure.outcome, failure.message ), - HistorianProducerError::TimedOut => write!(f, "historian producer timed out"), - HistorianProducerError::Protocol(detail) => { - write!(f, "runner protocol violation: {detail}") + Self::Json(error) => write!(f, "json: {error}"), + Self::MissingRunId => write!(f, "session.send did not return an active run_id"), + Self::MissingSession => write!(f, "historian producer has no bound session"), + Self::UnexpectedStreamEnd => { + write!(f, "subscribe stream ended before the run terminal control unit") } - HistorianProducerError::CleanupFailed { + Self::TimedOut => write!(f, "historian producer timed out"), + Self::Protocol(detail) => write!(f, "runner protocol violation: {detail}"), + Self::CrossIncarnationUnknown { + daemon_changed, + identity_changed, + } => write!( + f, + "session.send outcome is unknown across replay fence (daemon_changed={daemon_changed}, identity_changed={identity_changed})" + ), + Self::CleanupFailed { operation, primary, cleanup, @@ -495,17 +491,12 @@ impl fmt::Display for HistorianProducerError { } None => write!(f, "{operation} cleanup failed after success: {cleanup}"), }, - HistorianProducerError::RunFailed { run_id, detail, .. } => { - write!(f, "run {run_id} failed: {detail}") - } - HistorianProducerError::TerminalRunMismatch { expected, found } => { - write!( - f, - "run {expected} received terminal control unit after RunStarted {:?}", - found - ) - } - HistorianProducerError::RunPaused { run_id, reason, .. } => { + Self::RunFailed { run_id, detail, .. } => write!(f, "run {run_id} failed: {detail}"), + Self::TerminalRunMismatch { expected, found } => write!( + f, + "run {expected} received terminal control unit after RunStarted {found:?}" + ), + Self::RunPaused { run_id, reason, .. } => { write!(f, "run {run_id} paused")?; if let Some(reason) = reason { write!(f, ": {reason}")?; @@ -519,50 +510,19 @@ impl fmt::Display for HistorianProducerError { impl Error for HistorianProducerError { fn source(&self) -> Option<&(dyn Error + 'static)> { match self { - HistorianProducerError::ConnectionFile { source, .. } => Some(source), - HistorianProducerError::Connect { source, .. } => Some(source), - HistorianProducerError::Auth(e) => Some(e), - HistorianProducerError::FrameIo(e) => Some(e), - HistorianProducerError::FrameBuild(e) => Some(e), - HistorianProducerError::Json(e) => Some(e), - // The cleanup error is the chained cause: the primary, when - // present, already renders inside Display. - HistorianProducerError::CleanupFailed { cleanup, .. } => Some(cleanup.as_ref()), - HistorianProducerError::NoEndpoint { .. } - | HistorianProducerError::Subc(_) - | HistorianProducerError::UnexpectedControlResponse - | HistorianProducerError::MissingRunId - | HistorianProducerError::MissingSession - | HistorianProducerError::UnexpectedStreamEnd - | HistorianProducerError::TimedOut - | HistorianProducerError::Protocol(_) - | HistorianProducerError::RunFailed { .. } - | HistorianProducerError::TerminalRunMismatch { .. } - | HistorianProducerError::RunPaused { .. } => None, + Self::Json(error) => Some(error), + Self::CleanupFailed { cleanup, .. } => Some(cleanup.as_ref()), + _ => None, } } } -impl From for HistorianProducerError { - fn from(e: FrameIoError) -> Self { - HistorianProducerError::FrameIo(e) - } -} - -impl From for HistorianProducerError { - fn from(e: FrameBuildError) -> Self { - HistorianProducerError::FrameBuild(e) - } -} - impl From for HistorianProducerError { - fn from(e: serde_json::Error) -> Self { - HistorianProducerError::Json(e) + fn from(error: serde_json::Error) -> Self { + Self::Json(error) } } -/// Returns `CleanupFailed` whenever cleanup fails, preserving any primary -/// error. pub fn with_cleanup( primary: Result, cleanup: Result<(), HistorianProducerError>, @@ -594,62 +554,187 @@ pub fn attach_cleanup( } } -/// The daemon-assigned identity of an open consumer route. -/// -/// This raw client cannot construct `subc_client_rs::RouteHandle` because that type also -/// fences SDK-managed connections. It retains the serializable wire identity needed to -/// stamp every data-plane frame and reject stale replies. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -struct OpenedRoute { - channel: u16, - epoch: u32, +#[derive(Debug, Clone, PartialEq, Eq)] +struct SemanticIdentity { + project_root: PathBuf, + harness: String, + session: String, +} + +#[async_trait] +trait ProducerStream: Send { + async fn next(&mut self) -> Result, HistorianProducerError>; +} + +struct ManagedStream(ResponseStream); + +#[async_trait] +impl ProducerStream for ManagedStream { + async fn next(&mut self) -> Result, HistorianProducerError> { + self.0.next().await.map_err(map_call_error) + } +} + +#[async_trait] +trait ProducerConnection: Send + Sync { + fn daemon_id(&self) -> [u8; 16]; + async fn open_route( + &self, + target: RouteTarget, + identity: RouteIdentity, + ) -> Result; + async fn request( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result, HistorianProducerError>; + async fn request_stream( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result, HistorianProducerError>; + async fn close_route(&self, route: RouteHandle) -> Result<(), HistorianProducerError>; + async fn close(&self) -> Result<(), HistorianProducerError>; +} + +struct ManagedConnection(Client); + +#[async_trait] +impl ProducerConnection for ManagedConnection { + fn daemon_id(&self) -> [u8; 16] { + self.0.daemon_id() + } + + async fn open_route( + &self, + target: RouteTarget, + identity: RouteIdentity, + ) -> Result { + self.0 + .open_route(target, identity) + .await + .map_err(map_call_error) + } + + async fn request( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result, HistorianProducerError> { + self.0 + .request(route, body, options) + .await + .map(|response| response.body) + .map_err(map_call_error) + } + + async fn request_stream( + &self, + route: RouteHandle, + body: Vec, + options: RequestOptions, + ) -> Result, HistorianProducerError> { + self.0 + .request_stream(route, body, options) + .await + .map(|stream| Box::new(ManagedStream(stream)) as Box) + .map_err(map_call_error) + } + + async fn close_route(&self, route: RouteHandle) -> Result<(), HistorianProducerError> { + self.0.close_route(route).await.map_err(map_client_error) + } + + async fn close(&self) -> Result<(), HistorianProducerError> { + self.0.close().await.map_err(map_client_error) + } +} + +struct Reconnected { + connection: Box, + identity: SemanticIdentity, +} + +#[async_trait] +trait ProducerConnector: Send + Sync { + async fn connect( + &self, + config: &HistorianProducerConfig, + ) -> Result, HistorianProducerError>; + + async fn reconnect( + &self, + config: &HistorianProducerConfig, + identity: &SemanticIdentity, + ) -> Result; +} + +struct ManagedConnector; + +#[async_trait] +impl ProducerConnector for ManagedConnector { + async fn connect( + &self, + config: &HistorianProducerConfig, + ) -> Result, HistorianProducerError> { + Client::connect(&config.connection_file) + .await + .map(|client| Box::new(ManagedConnection(client)) as Box) + .map_err(map_client_error) + } + + async fn reconnect( + &self, + config: &HistorianProducerConfig, + identity: &SemanticIdentity, + ) -> Result { + Ok(Reconnected { + connection: self.connect(config).await?, + identity: identity.clone(), + }) + } +} + +fn map_call_error(error: CallError) -> HistorianProducerError { + HistorianProducerError::Call(error.into()) +} + +fn map_client_error(error: ClientError) -> HistorianProducerError { + HistorianProducerError::Client(error.into()) } pub struct HistorianProducer { config: HistorianProducerConfig, - stream: TcpStream, - next_corr: u64, + connector: Arc, + connection: Box, session_id: Option, - command_route: Option, - subscribe_route: Option, + command_route: Option, + subscribe_route: Option, } impl HistorianProducer { pub async fn connect(config: HistorianProducerConfig) -> Result { - let conn = connection_file::read(&config.connection_file).map_err(|source| { - HistorianProducerError::ConnectionFile { - path: config.connection_file.clone(), - source, - } - })?; - let endpoint = - conn.endpoints - .first() - .ok_or_else(|| HistorianProducerError::NoEndpoint { - path: config.connection_file.clone(), - })?; - let endpoint_label = format!("{}:{}", endpoint.host, endpoint.port); - let mut stream = TcpStream::connect(&endpoint_label) - .await - .map_err(|source| HistorianProducerError::Connect { - endpoint: endpoint_label, - source, - })?; - authenticate_client(&mut stream, &conn, config.handshake_timeout) - .await - .map_err(HistorianProducerError::Auth)?; + Self::connect_with(config, Arc::new(ManagedConnector)).await + } + + async fn connect_with( + config: HistorianProducerConfig, + connector: Arc, + ) -> Result { + let connection = connector.connect(&config).await?; Ok(Self { config, - stream, - next_corr: 1, + connector, + connection, session_id: None, command_route: None, subscribe_route: None, }) } - /// Bind subsequent status/subscribe/cancel calls to an existing session. Reattach - /// probes open the route with the persisted session id and must not call send again. pub fn bind_session(&mut self, session_id: impl Into) { self.session_id = Some(session_id.into()); } @@ -681,24 +766,10 @@ impl HistorianProducer { max_output_tokens: u32, temperature: f64, ) -> Result { - self.bind_session(session_id.to_string()); - // The route identity is the session. Keeping `session` out of this body avoids - // a second, diverging source of truth for the run lineage. - // - // `system` rides the role-scoped SendParams.system field (delivered as a leading - // system message in the run's durable input, byte-exact) — NEVER concatenated - // into the user prompt: the historian's parse/validate contract assumes the - // model saw its role guidance as a system message. Empty means absent, matching - // the wire's empty-as-absent rule, so we omit the field entirely. - // - // The params shape mirrors llm-runner's SendParams (llmr-module-serve wire.rs): - // `model` is a nested {provider, model} object, split from our canonical - // "provider/model" string at the FIRST slash so multi-slash model names keep - // their remainder intact. The server decodes strictly enough that a flat model - // string fails the whole send with invalid_params, which a live rig drive - // surfaced as firings dying before any producer run existed. + self.bind_session(session_id.to_owned()); let (provider, model_name) = model.split_once('/').ok_or_else(|| { - HistorianProducerError::Subc(ProducerErrorBody::untagged( + HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::NotSent, "invalid_model", format!("model '{model}' is not in canonical provider/model form"), )) @@ -720,20 +791,19 @@ impl HistorianProducer { if !system.is_empty() { params.insert("system".into(), json!(system)); } - let body = json!({ + let frozen = serde_json::to_vec(&json!({ "method": "session.send", - "params": params - }); - // Any different byte sequence for this session is an idempotency conflict at the runner, never the original run_id. commentlint: allow(JUDGE) - let frozen = serde_json::to_vec(&body)?; - let response = match self.send_frozen(&frozen).await { + "params": params, + }))?; + let frozen_identity = self.semantic_identity()?; + let frozen_daemon = self.connection.daemon_id(); + let response = match self.send_frozen_once(&frozen).await { Ok(response) => response, - // Resending after an answered (application) error could start a second billable run under a fresh route. commentlint: allow(JUDGE) - Err(err) if send_outcome_unknown(&err) => { - self.reconnect().await?; - self.send_frozen(&frozen).await? + Err(error) if is_outcome_unknown(&error) => { + self.replay_frozen_once(frozen_daemon, frozen_identity, &frozen) + .await? } - Err(err) => return Err(err), + Err(error) => return Err(error), }; let run_id = response .get("run_id") @@ -741,12 +811,12 @@ impl HistorianProducer { .or_else(|| { response .get("result") - .and_then(|r| r.get("run_id")) + .and_then(|result| result.get("run_id")) .and_then(Value::as_str) }) .ok_or(HistorianProducerError::MissingRunId)?; Ok(RunHandle { - run_id: run_id.to_string(), + run_id: run_id.to_owned(), }) } @@ -795,26 +865,16 @@ impl HistorianProducer { pub async fn cancel(&mut self, run_id: &str) -> Result<(), HistorianProducerError> { let route = self.ensure_command_route().await?; - let _ = self - .unary_json( - route, - json!({ "method": "run.cancel", "params": { "run_id": run_id } }), - ) - .await?; + self.unary_json( + route, + json!({ "method": "run.cancel", "params": { "run_id": run_id } }), + ) + .await?; Ok(()) } - /// Delete the bound provider session before releasing its routes. Dreamer - /// sessions contain memory-pool snapshots, so retention settings never apply. pub async fn purge_session(&mut self, session_id: &str) -> Result<(), HistorianProducerError> { - self.bind_session(session_id.to_string()); - // The WHOLE operation is bounded, not just the response wait: - // `unary_json` starts its timer only after the request write - // returns, and the goodbye writes have no timer at all, so a - // backpressured connection could wedge cleanup indefinitely. The - // caller reserves a fixed margin for this after its payload - // deadline; overrunning it lets the caller's cancel land mid-purge - // and leave the already-started run executing to its own timeout. + self.bind_session(session_id.to_owned()); let budget = self.config.request_timeout; let purge = async { let delete = async { @@ -833,92 +893,58 @@ impl HistorianProducer { } } - pub async fn close(&mut self) -> Result<(), HistorianProducerError> { - // Both routes are always released; the first failure is reported so a - // wedged transport cannot silently leak the second route either. - let mut first_error = None; - if let Some(route) = self.subscribe_route.take() { - if let Err(err) = self.send_goodbye(route).await { - first_error.get_or_insert(err); - } - } - if let Some(route) = self.command_route.take() { - if let Err(err) = self.send_goodbye(route).await { - first_error.get_or_insert(err); - } - } - match first_error { - None => Ok(()), - Some(err) => Err(err), - } + pub async fn close_attempt(&mut self) -> Result<(), HistorianProducerError> { + self.close_routes().await } - async fn subscribe_from_start( - &mut self, - run_id: &str, - timeout: Duration, - ) -> Result { - // The whole operation is bounded, not just the drain: opening the - // subscription route and writing the request are themselves - // requests that can stall for their own request timeout, and the - // caller's budget is a deadline for the attempt — overshooting it - // here would eat the transport margin reserved for cleanup and let - // an outer cancel land during `session.delete`. - match tokio::time::timeout(timeout, self.subscribe_and_drain(run_id)).await { - Ok(result) => result, - Err(_) => Err(HistorianProducerError::TimedOut), - } + pub async fn close(&mut self) -> Result<(), HistorianProducerError> { + self.close_routes_and_connection().await } - async fn subscribe_and_drain( - &mut self, - run_id: &str, - ) -> Result { - let route = self.ensure_subscribe_route().await?; - // Subscribe from "start" instead of a cursor. Replay after a cursor is - // exclusive, so persisting an advancing cursor can drop units at or before - // it on reattach. Re-draining from the start is safe because validation - // and compare-and-swap checks during publish are idempotent. - let body = json!({ "method": "session.subscribe", "params": { "from": "start" } }); - let corr = self.send_request(route, body).await?; - self.drain_subscribe(route, corr, run_id).await - } - - /// One transmission of the frozen `session.send` bytes, split from - /// `start_with_generation` so the recovery resend cannot accidentally - /// re-serialize a different body. - async fn send_frozen(&mut self, frozen: &[u8]) -> Result { + async fn send_frozen_once(&mut self, frozen: &[u8]) -> Result { let route = self.ensure_command_route().await?; - let corr = self.next_corr(); - self.write_frame( - FrameType::Request, - route.channel, - route.epoch, - corr, - frozen.to_vec(), - ) - .await?; - let frame = self - .read_terminal_for(route, corr, self.config.request_timeout) + let response = self + .connection + .request( + route, + frozen.to_vec(), + self.request_options(self.config.request_timeout), + ) .await?; - match frame.header.ty { - FrameType::Response => Ok(serde_json::from_slice(&frame.body)?), - FrameType::StreamEnd => Ok(Value::Null), - FrameType::Error => Err(HistorianProducerError::Subc(error_body(&frame.body))), - _ => Err(HistorianProducerError::UnexpectedControlResponse), - } + Ok(serde_json::from_slice(&response)?) } - /// The config is reused unchanged because the runner deduplicates the frozen resend only under the same project/harness/session route identity. commentlint: allow(JUDGE) - async fn reconnect(&mut self) -> Result<(), HistorianProducerError> { - let mut fresh = Self::connect(self.config.clone()).await?; - std::mem::swap(&mut self.stream, &mut fresh.stream); + async fn replay_frozen_once( + &mut self, + frozen_daemon: [u8; 16], + frozen_identity: SemanticIdentity, + frozen: &[u8], + ) -> Result { + let reconnected = self + .connector + .reconnect(&self.config, &frozen_identity) + .await?; + let daemon_changed = reconnected.connection.daemon_id() != frozen_daemon; + let identity_changed = reconnected.identity != frozen_identity; + + let old_cleanup = self.close_routes_and_connection().await; + if let Err(error) = old_cleanup { + eprintln!("mc-module: historian replay cleanup failed: {error}"); + } + self.connection = reconnected.connection; self.command_route = None; self.subscribe_route = None; - Ok(()) + + if daemon_changed || identity_changed { + return Err(HistorianProducerError::CrossIncarnationUnknown { + daemon_changed, + identity_changed, + }); + } + self.send_frozen_once(frozen).await } - async fn ensure_command_route(&mut self) -> Result { + async fn ensure_command_route(&mut self) -> Result { if let Some(route) = self.command_route { return Ok(route); } @@ -927,7 +953,7 @@ impl HistorianProducer { Ok(route) } - async fn ensure_subscribe_route(&mut self) -> Result { + async fn ensure_subscribe_route(&mut self) -> Result { if let Some(route) = self.subscribe_route { return Ok(route); } @@ -936,249 +962,183 @@ impl HistorianProducer { Ok(route) } - async fn open_bound_route(&mut self) -> Result { - let session = self - .session_id - .clone() - .ok_or(HistorianProducerError::MissingSession)?; - let request = ClientControlRequest::RouteOpen { - target: RouteTarget::ManagementSurface { - module_id: self.config.module_id.clone(), - }, - identity: BindIdentity { - project_root: self.config.project_root.clone(), - harness: self.config.harness.clone(), - session, - }, - consumer_identity: consumer_identity_from_env(), - consumer_capabilities: None, - admission_facts: None, - }; - let corr = self.next_corr(); - let body = serde_json::to_vec(&request)?; - self.write_frame(FrameType::Request, 0, 0, corr, body) - .await?; - let frame = self - .read_terminal_for( - OpenedRoute { - channel: 0, - epoch: 0, + async fn open_bound_route(&self) -> Result { + let semantic = self.semantic_identity()?; + self.connection + .open_route( + RouteTarget { + module_id: self.config.module_id.clone(), + kind: TargetKind::ManagementSurface, + }, + RouteIdentity { + project_root: semantic.project_root, + harness: semantic.harness, + session: semantic.session, + consumer_module_id: nonempty_env("SUBC_MODULE_ID"), + consumer_launch_nonce: nonempty_env("SUBC_LAUNCH_NONCE"), + consumer_capabilities: Vec::new(), + admission_facts: None, }, - corr, - self.config.request_timeout, ) - .await?; - match frame.header.ty { - FrameType::Response => { - let response: ClientControlResponse = serde_json::from_slice(&frame.body)?; - if let ClientControlResponse::RouteOpen { - route_channel, - route_epoch, - } = response - { - Ok(OpenedRoute { - channel: route_channel, - epoch: route_epoch, - }) - } else { - Err(HistorianProducerError::UnexpectedControlResponse) - } - } - FrameType::Error => Err(HistorianProducerError::Subc(error_body(&frame.body))), - _ => Err(HistorianProducerError::UnexpectedControlResponse), - } + .await } async fn unary_json( - &mut self, - route: OpenedRoute, + &self, + route: RouteHandle, body: Value, ) -> Result { - let corr = self.send_request(route, body).await?; - let frame = self - .read_terminal_for(route, corr, self.config.request_timeout) + let body = serde_json::to_vec(&body)?; + let response = self + .connection + .request( + route, + body, + self.request_options(self.config.request_timeout), + ) .await?; - match frame.header.ty { - FrameType::Response => Ok(serde_json::from_slice(&frame.body)?), - FrameType::StreamEnd => Ok(Value::Null), - FrameType::Error => Err(HistorianProducerError::Subc(error_body(&frame.body))), - _ => Err(HistorianProducerError::UnexpectedControlResponse), - } + Ok(serde_json::from_slice(&response)?) } - async fn send_request( + async fn subscribe_from_start( &mut self, - route: OpenedRoute, - body: Value, - ) -> Result { - let corr = self.next_corr(); - let bytes = serde_json::to_vec(&body)?; - self.write_frame(FrameType::Request, route.channel, route.epoch, corr, bytes) + run_id: &str, + timeout: Duration, + ) -> Result { + let route = self.ensure_subscribe_route().await?; + let body = serde_json::to_vec(&json!({ + "method": "session.subscribe", + "params": { "from": "start" }, + }))?; + let mut stream = self + .connection + .request_stream(route, body, self.request_options(timeout)) .await?; - Ok(corr) + drain_subscribe(&mut *stream, run_id).await } - async fn drain_subscribe( - &mut self, - route: OpenedRoute, - corr: u64, - run_id: &str, - ) -> Result { - let mut text = String::new(); - let mut last_run_started: Option = None; - let mut length_capped = false; - loop { - let Some(frame) = read_frame(&mut self.stream).await? else { - return Err(HistorianProducerError::UnexpectedStreamEnd); - }; - if frame.header.channel != route.channel - || frame.header.epoch != route.epoch - || frame.header.corr != corr - { - continue; + async fn close_routes(&mut self) -> Result<(), HistorianProducerError> { + let mut first_error = None; + if let Some(route) = self.subscribe_route.take() { + if let Err(error) = self.connection.close_route(route).await { + first_error.get_or_insert(error); } - match frame.header.ty { - FrameType::StreamData => { - let event: Value = serde_json::from_slice(&frame.body)?; - let Some(unit) = control_unit(&event) else { - continue; - }; - if is_run_started_unit(unit) { - last_run_started = unit_run_id(unit).map(ToString::to_string); - } - let terminal = is_terminal_unit(unit); - if !terminal && unit_run_id(unit).is_some_and(|id| id != run_id) { - continue; - } - if is_paused_unit(unit) && unit_run_id(unit) == Some(run_id) { - // A paused run still holds the slot for this historian. Return - // an error so callers stop waiting and retry later instead of - // hanging forever on a run that is paused but not finished. - let info = unit_error_info(unit); - return Err(HistorianProducerError::RunPaused { - run_id: run_id.to_string(), - reason: paused_reason(unit).map(ToString::to_string), - classification: info.classification, - class_field_present: info.class_field_present, - }); - } - if let Some(piece) = unit_text(unit) { - text.push_str(&piece); - } - if unit_is_length_capped(unit) { - length_capped = true; - } - if terminal { - if last_run_started.as_deref() != Some(run_id) { - return Err(HistorianProducerError::TerminalRunMismatch { - expected: run_id.to_string(), - found: last_run_started, - }); - } - if is_error_unit(unit) { - let info = unit_error_info(unit); - return Err(HistorianProducerError::RunFailed { - run_id: run_id.to_string(), - detail: info.detail.unwrap_or_else(|| "run failed".to_string()), - classification: info.classification, - class_field_present: info.class_field_present, - }); - } - // The run terminal control unit is authoritative. StreamEnd is only - // route mechanics and can appear on detach/resubscribe without ending a run. - return Ok(ProducerOutput { - text, - length_capped, - }); - } - } - FrameType::Error => { - return Err(HistorianProducerError::Subc(error_body(&frame.body))) - } - FrameType::StreamEnd => return Err(HistorianProducerError::UnexpectedStreamEnd), - _ => {} + } + if let Some(route) = self.command_route.take() { + if let Err(error) = self.connection.close_route(route).await { + first_error.get_or_insert(error); } } + first_error.map_or(Ok(()), Err) } - async fn read_terminal_for( - &mut self, - route: OpenedRoute, - corr: u64, - timeout: Duration, - ) -> Result { - match tokio::time::timeout(timeout, async { - loop { - let Some(frame) = read_frame(&mut self.stream).await? else { - return Err(HistorianProducerError::UnexpectedStreamEnd); - }; - if frame.header.channel == route.channel - && frame.header.epoch == route.epoch - && frame.header.corr == corr - { - return Ok(frame); - } + async fn close_routes_and_connection(&mut self) -> Result<(), HistorianProducerError> { + let mut result = self.close_routes().await; + if let Err(error) = self.connection.close().await { + if result.is_ok() { + result = Err(error); } - }) - .await - { - Ok(result) => result, - Err(_) => Err(HistorianProducerError::TimedOut), } + result } - async fn write_frame( - &mut self, - ty: FrameType, - channel: u16, - epoch: u32, - corr: u64, - body: Vec, - ) -> Result<(), HistorianProducerError> { - let frame = Frame::build( - ty, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - corr, - body, - )?; - write_frame(&mut self.stream, &frame).await?; - Ok(()) + fn semantic_identity(&self) -> Result { + Ok(SemanticIdentity { + project_root: self.config.project_root.clone(), + harness: self.config.harness.clone(), + session: self + .session_id + .clone() + .ok_or(HistorianProducerError::MissingSession)?, + }) } - async fn send_goodbye(&mut self, route: OpenedRoute) -> Result<(), HistorianProducerError> { - let frame = Frame::build( - FrameType::Goodbye, - Flags::new(false, Priority::Interactive, false), - route.channel, - route.epoch, - 0, - Vec::new(), - )?; - write_frame(&mut self.stream, &frame).await?; - Ok(()) + fn request_options(&self, timeout: Duration) -> RequestOptions { + RequestOptions { + timeout, + cancellation: self.config.cancellation.clone(), + } } +} - fn next_corr(&mut self) -> u64 { - let corr = self.next_corr; - self.next_corr = self.next_corr.saturating_add(1).max(1); - corr +async fn drain_subscribe( + stream: &mut dyn ProducerStream, + run_id: &str, +) -> Result { + let mut text = String::new(); + let mut last_run_started: Option = None; + let mut length_capped = false; + while let Some(item) = stream.next().await? { + let event: Value = serde_json::from_slice(&item.body)?; + let Some(unit) = control_unit(&event) else { + continue; + }; + if is_run_started_unit(unit) { + last_run_started = unit_run_id(unit).map(ToOwned::to_owned); + } + let terminal = is_terminal_unit(unit); + if !terminal && unit_run_id(unit).is_some_and(|id| id != run_id) { + continue; + } + if is_paused_unit(unit) && unit_run_id(unit) == Some(run_id) { + let info = unit_error_info(unit); + return Err(HistorianProducerError::RunPaused { + run_id: run_id.to_owned(), + reason: paused_reason(unit).map(ToOwned::to_owned), + classification: info.classification, + class_field_present: info.class_field_present, + }); + } + if let Some(piece) = unit_text(unit) { + text.push_str(&piece); + } + if unit_is_length_capped(unit) { + length_capped = true; + } + if terminal { + if last_run_started.as_deref() != Some(run_id) { + return Err(HistorianProducerError::TerminalRunMismatch { + expected: run_id.to_owned(), + found: last_run_started, + }); + } + if is_error_unit(unit) { + let info = unit_error_info(unit); + return Err(HistorianProducerError::RunFailed { + run_id: run_id.to_owned(), + detail: info.detail.unwrap_or_else(|| "run failed".to_owned()), + classification: info.classification, + class_field_present: info.class_field_present, + }); + } + return Ok(ProducerOutput { + text, + length_capped, + }); + } } + Err(HistorianProducerError::UnexpectedStreamEnd) } -impl Drop for HistorianProducer { - fn drop(&mut self) { - // Async close is preferred by callers; drop only releases the TCP socket. - } +fn nonempty_env(name: &str) -> Option { + std::env::var(name) + .ok() + .filter(|value| !value.trim().is_empty()) +} + +fn is_outcome_unknown(error: &HistorianProducerError) -> bool { + matches!( + error, + HistorianProducerError::Call(HistorianCallFailure { + outcome: HistorianSendOutcome::OutcomeUnknown, + .. + }) + ) } fn classify_run_state(run_id: &str, value: &Value) -> Result { let value = value.get("result").unwrap_or(value); let response_run_id = value.get("run_id").and_then(Value::as_str); - // Absence is as disqualifying as a mismatch: a response that does not - // name the run has not been proven to describe it, and misreading it as - // `missing` would authorize a second billable run. if response_run_id != Some(run_id) { return Err(HistorianProducerError::Protocol(format!( "run.status answered for run {response_run_id:?}, not {run_id}" @@ -1186,7 +1146,7 @@ fn classify_run_state(run_id: &str, value: &Value) -> Result Result Err(HistorianProducerError::Protocol(format!( "undocumented run state {other:?}" @@ -1205,24 +1165,11 @@ fn classify_run_state(run_id: &str, value: &Value) -> Result bool { - matches!( - err, - HistorianProducerError::FrameIo(_) - | HistorianProducerError::TimedOut - | HistorianProducerError::UnexpectedStreamEnd - ) -} - fn control_unit(event: &Value) -> Option<&Value> { let kind = event.get("kind").and_then(Value::as_str); if kind == Some("display") { return None; } - if kind == Some("control") { - let unit = event.get("unit").unwrap_or(event); - return Some(unit); - } Some(event.get("unit").unwrap_or(event)) } @@ -1238,69 +1185,60 @@ fn unit_run_id(unit: &Value) -> Option<&str> { .and_then(Value::as_str) } -/// Extract the assistant TEXT from a control unit. llm-runner's assistant_message -/// unit nests an assembled message with a content-block array; only `text` blocks are -/// the historian's output. `reasoning` blocks are deliberately EXCLUDED: a reasoning -/// model's thinking legitimately restates the prompt's format template and walks the -/// seed examples, so folding it into the output would corrupt the parse with -/// template/seed prose. Flat `text`/`content` fields are kept as a fallback for -/// simpler unit shapes. fn unit_text(unit: &Value) -> Option { - if !unit_type(unit).is_some_and(|ty| ty.eq_ignore_ascii_case("assistant_message")) { + if !unit_type(unit).is_some_and(|kind| kind.eq_ignore_ascii_case("assistant_message")) { return None; } if let Some(blocks) = unit .get("message") - .and_then(|m| m.get("content")) + .and_then(|message| message.get("content")) .and_then(Value::as_array) { let text: String = blocks .iter() - .filter(|b| b.get("type").and_then(Value::as_str) == Some("text")) - .filter_map(|b| b.get("text").and_then(Value::as_str)) + .filter(|block| block.get("type").and_then(Value::as_str) == Some("text")) + .filter_map(|block| block.get("text").and_then(Value::as_str)) .collect(); return (!text.is_empty()).then_some(text); } unit.get("text") .or_else(|| unit.get("content")) - .or_else(|| unit.get("message").and_then(|m| m.get("text"))) + .or_else(|| unit.get("message").and_then(|message| message.get("text"))) .and_then(Value::as_str) - .map(ToString::to_string) + .map(ToOwned::to_owned) } fn is_terminal_unit(unit: &Value) -> bool { - let Some(ty) = unit_type(unit).map(str::to_ascii_lowercase) else { - return false; - }; - ty == "run_finished" - || ty == "terminal" - || ty == "run_terminal" - || ty == "finished" - || ty == "error" + unit_type(unit) + .map(str::to_ascii_lowercase) + .is_some_and(|kind| { + matches!( + kind.as_str(), + "run_finished" | "terminal" | "run_terminal" | "finished" | "error" + ) + }) } fn is_run_started_unit(unit: &Value) -> bool { unit_type(unit) .map(str::to_ascii_lowercase) - .is_some_and(|ty| ty == "run_started" || ty == "runstarted") + .is_some_and(|kind| kind == "run_started" || kind == "runstarted") } -/// A length-class finish reason on ANY unit (step or terminal): providers spell it -/// "length", "max_tokens", or "max_output_tokens" depending on the wire family. fn unit_is_length_capped(unit: &Value) -> bool { unit.get("finish_reason") .or_else(|| unit.get("finishReason")) .and_then(Value::as_str) .is_some_and(|reason| { - let reason = reason.to_ascii_lowercase(); - reason == "length" || reason == "max_tokens" || reason == "max_output_tokens" + matches!( + reason.to_ascii_lowercase().as_str(), + "length" | "max_tokens" | "max_output_tokens" + ) }) } fn is_paused_unit(unit: &Value) -> bool { - unit_type(unit) - .map(str::to_ascii_lowercase) - .is_some_and(|ty| ty == "paused") + unit_type(unit).is_some_and(|kind| kind.eq_ignore_ascii_case("paused")) } fn paused_reason(unit: &Value) -> Option<&str> { @@ -1312,7 +1250,7 @@ fn paused_reason(unit: &Value) -> Option<&str> { fn is_error_unit(unit: &Value) -> bool { unit_type(unit) .map(str::to_ascii_lowercase) - .is_some_and(|ty| ty == "error" || ty == "run_error") + .is_some_and(|kind| kind == "error" || kind == "run_error") } #[derive(Debug, Default)] @@ -1330,8 +1268,8 @@ fn unit_error_info(unit: &Value) -> UnitErrorInfo { .and_then(Value::as_str) .or_else(|| unit.get("detail").and_then(Value::as_str)) .or_else(|| unit.get("message").and_then(Value::as_str)) - .map(ToString::to_string) - .or_else(|| error.as_str().map(ToString::to_string)) + .map(ToOwned::to_owned) + .or_else(|| error.as_str().map(ToOwned::to_owned)) .or_else(|| Some(error.to_string())); return UnitErrorInfo { detail, @@ -1344,7 +1282,7 @@ fn unit_error_info(unit: &Value) -> UnitErrorInfo { .get("detail") .or_else(|| unit.get("message")) .and_then(Value::as_str) - .map(ToString::to_string); + .map(ToOwned::to_owned); let (classification, class_field_present) = detail .as_deref() .and_then(classification_from_json_text) @@ -1356,22 +1294,6 @@ fn unit_error_info(unit: &Value) -> UnitErrorInfo { } } -fn consumer_identity_from_env() -> Option { - let module_id = std::env::var(SUBC_MODULE_ID_ENV).ok()?; - let launch_nonce = std::env::var(SUBC_LAUNCH_NONCE_ENV).ok()?; - (!module_id.is_empty() && !launch_nonce.is_empty()).then_some(ConsumerIdentity { - module_id, - launch_nonce, - }) -} - -fn error_body(body: &[u8]) -> ProducerErrorBody { - match serde_json::from_slice::(body) { - Ok(value) => ProducerErrorBody::from_value(value), - Err(e) => ProducerErrorBody::untagged("invalid_error_body", e.to_string()), - } -} - fn classification_from_json_text(s: &str) -> Option<(Option, bool)> { serde_json::from_str::(s) .ok() @@ -1397,980 +1319,453 @@ fn classification_from_object(value: &Value) -> (Option, bo } fn retry_after_secs_from_value(value: &Value) -> Option { - if let Some(secs) = value.as_u64() { - return Some(secs); - } - let secs = value.as_f64()?; - if secs.is_finite() && secs >= 0.0 { - Some(secs.ceil() as u64) - } else { - None - } + value.as_u64().or_else(|| { + value + .as_f64() + .filter(|value| value.is_finite() && *value >= 0.0) + .map(|value| value.ceil() as u64) + }) } #[cfg(test)] mod tests { use super::*; - use std::{collections::VecDeque, net::SocketAddr, sync::Arc}; + use std::collections::{HashSet, VecDeque}; + use std::sync::Mutex; + + #[derive(Clone)] + struct FakeConnection { + daemon_id: [u8; 16], + state: Arc>, + } + + #[derive(Default)] + struct FakeState { + requests: Vec>, + identities: Vec, + responses: VecDeque, HistorianProducerError>>, + opened_routes: Vec, + closed_routes: Vec, + close_route_errors: VecDeque, + close_calls: usize, + next_channel: u16, + } + + #[async_trait] + impl ProducerConnection for FakeConnection { + fn daemon_id(&self) -> [u8; 16] { + self.daemon_id + } - use serde_json::json; - use subc_transport::{ - authenticate_server, generate_daemon_id, generate_key, write_atomic, ConnectionInfo, - Endpoint, SCHEMA_VERSION, - }; - use tempfile::TempDir; - use tokio::{net::TcpListener, sync::Mutex}; + async fn open_route( + &self, + _target: RouteTarget, + identity: RouteIdentity, + ) -> Result { + let mut state = self.state.lock().unwrap(); + state.identities.push(identity); + state.next_channel += 1; + let route = RouteHandle { + channel: state.next_channel, + epoch: u32::from(state.next_channel), + }; + state.opened_routes.push(route); + Ok(route) + } - #[test] - fn error_class_wire_strings_match_pinned_contract_set() { - assert_eq!( - ERROR_CLASS_WIRE_SET, - [ - "transient", - "permanent", - "auth_required", - "context_overflow" - ] - ); - assert_eq!(ErrorClass::Transient.as_wire_str(), "transient"); - assert_eq!(ErrorClass::Permanent.as_wire_str(), "permanent"); - assert_eq!(ErrorClass::AuthRequired.as_wire_str(), "auth_required"); - assert_eq!( - ErrorClass::ContextOverflow.as_wire_str(), - "context_overflow" - ); - assert_eq!( - ErrorClass::from_wire("auth"), - Some(ErrorClass::AuthRequired) - ); - } + async fn request( + &self, + _route: RouteHandle, + body: Vec, + _options: RequestOptions, + ) -> Result, HistorianProducerError> { + let mut state = self.state.lock().unwrap(); + state.requests.push(body); + state + .responses + .pop_front() + .unwrap_or_else(|| Ok(br#"{"run_id":"run-default"}"#.to_vec())) + } - #[test] - fn parses_tagged_subc_error_body_from_contract_shape() { - let body = serde_json::to_vec(&json!({ - "code": "provider_error", - "message": "rate limit window", - "class": "transient", - "retry_after_secs": 120, - "provider_code": "rate_limit_exceeded" - })) - .unwrap(); - - let parsed = error_body(&body); - assert_eq!(parsed.code, "provider_error"); - assert_eq!( - parsed.classification(), - Some(ErrorClassification { - class: ErrorClass::Transient, - retry_after_secs: Some(120), - }) - ); - } + async fn request_stream( + &self, + _route: RouteHandle, + _body: Vec, + _options: RequestOptions, + ) -> Result, HistorianProducerError> { + Ok(Box::new(FakeStream(VecDeque::new()))) + } - #[test] - fn parses_current_llm_runner_control_error_shape() { - let unit = json!({ - "type": "error", - "error": { - "class": "permanent", - "message": "model id does not exist", - "status": 404, - "provider_code": "model_not_found" + async fn close_route(&self, route: RouteHandle) -> Result<(), HistorianProducerError> { + let mut state = self.state.lock().unwrap(); + state.closed_routes.push(route); + match state.close_route_errors.pop_front() { + Some(error) => Err(error), + None => Ok(()), } - }); + } - let info = unit_error_info(&unit); - assert_eq!(info.detail.as_deref(), Some("model id does not exist")); - assert_eq!( - info.classification, - Some(ErrorClassification { - class: ErrorClass::Permanent, - retry_after_secs: None, - }) - ); + async fn close(&self) -> Result<(), HistorianProducerError> { + self.state.lock().unwrap().close_calls += 1; + Ok(()) + } } - #[derive(Debug, Default)] - struct ServerLog { - route_sessions: Vec, - route_harnesses: Vec, - sends: Vec, - send_bodies: Vec>, - subscribes: Vec, - goodbyes: Vec, - } - - struct FakeServer { - connection_file: PathBuf, - log: Arc>, - _temp: TempDir, - } - - async fn fake_server(send_response: Value, stream_events: Vec) -> FakeServer { - let temp = tempfile::tempdir().unwrap(); - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr: SocketAddr = listener.local_addr().unwrap(); - let key = generate_key().unwrap(); - let daemon_id = generate_daemon_id().unwrap(); - let connection_file = temp.path().join("subc-connection.json"); - write_atomic( - &connection_file, - &ConnectionInfo { - schema: SCHEMA_VERSION, - wire_version: Some(subc_protocol::PROTOCOL_VERSION), - endpoints: vec![Endpoint { - host: addr.ip().to_string(), - port: addr.port(), - }], - key: key.clone(), - daemon_id, - pid: std::process::id(), - daemon_ver: "fake".to_string(), - }, - ) - .unwrap(); - let log = Arc::new(Mutex::new(ServerLog::default())); - let log_task = Arc::clone(&log); - tokio::spawn(async move { - let (mut stream, _) = listener.accept().await.unwrap(); - authenticate_server( - &mut stream, - &key, - &daemon_id, - "fake", - Duration::from_secs(2), - ) - .await - .unwrap(); - let mut next_route = 10u16; - let mut route_sessions = std::collections::HashMap::::new(); - let mut stream_events: VecDeque = stream_events.into(); - loop { - let Some(frame) = read_frame(&mut stream).await.unwrap() else { - break; - }; - match frame.header.ty { - FrameType::Goodbye => { - log_task.lock().await.goodbyes.push(frame.header.channel); - } - FrameType::Request if frame.header.channel == 0 => { - let req: ClientControlRequest = - serde_json::from_slice(&frame.body).unwrap(); - if let ClientControlRequest::RouteOpen { identity, .. } = req { - let route = next_route; - next_route += 1; - route_sessions.insert(route, identity.session.clone()); - log_task.lock().await.route_sessions.push(identity.session); - log_task.lock().await.route_harnesses.push(identity.harness); - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec(&ClientControlResponse::RouteOpen { - route_channel: route, - route_epoch: 1, - }) - .unwrap(), - ) - .await; - } - } - FrameType::Request => { - let req: Value = serde_json::from_slice(&frame.body).unwrap(); - match req.get("method").and_then(Value::as_str) { - Some("session.send") => { - log_task.lock().await.sends.push(req["params"].clone()); - log_task.lock().await.send_bodies.push(frame.body.clone()); - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec(&send_response).unwrap(), - ) - .await; - } - Some("session.subscribe") => { - log_task.lock().await.subscribes.push(req["params"].clone()); - while let Some(event) = stream_events.pop_front() { - send_stream_data( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - event, - ) - .await; - } - send_stream_end( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - ) - .await; - } - Some("run.status") => { - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec( - &json!({"state":"completed","run_id":"run-1"}), - ) - .unwrap(), - ) - .await; - } - Some("run.cancel") => { - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec(&json!({"ack":true})).unwrap(), - ) - .await; - } - other => panic!( - "unexpected request {other:?} on route {:?}", - route_sessions.get(&frame.header.channel) - ), - } - } - _ => {} - } - } - }); - FakeServer { - connection_file, - log, - _temp: temp, + struct FakeStream(VecDeque, HistorianProducerError>>); + + #[async_trait] + impl ProducerStream for FakeStream { + async fn next(&mut self) -> Result, HistorianProducerError> { + self.0.pop_front().unwrap_or(Ok(None)) } } - async fn send_response_frame( - stream: &mut TcpStream, - channel: u16, - epoch: u32, - corr: u64, - body: Vec, - ) { - let frame = Frame::build( - FrameType::Response, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - corr, - body, - ) - .unwrap(); - write_frame(stream, &frame).await.unwrap(); - } - - async fn send_stream_data( - stream: &mut TcpStream, - channel: u16, - epoch: u32, - corr: u64, - event: Value, - ) { - let frame = Frame::build( - FrameType::StreamData, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - corr, - serde_json::to_vec(&event).unwrap(), - ) - .unwrap(); - write_frame(stream, &frame).await.unwrap(); - } - - async fn send_stream_end(stream: &mut TcpStream, channel: u16, epoch: u32, corr: u64) { - let frame = Frame::build( - FrameType::StreamEnd, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - corr, - Vec::new(), - ) - .unwrap(); - write_frame(stream, &frame).await.unwrap(); - } - - async fn client(server: &FakeServer) -> HistorianProducer { - HistorianProducer::connect(HistorianProducerConfig { - connection_file: server.connection_file.clone(), - project_root: std::env::current_dir().unwrap(), - harness: "mc-test".to_string(), - module_id: "llm-runner".to_string(), - handshake_timeout: Duration::from_secs(2), - request_timeout: Duration::from_secs(2), - await_timeout: Duration::from_secs(2), - }) - .await - .unwrap() + struct FakeConnector { + initial: FakeConnection, + reconnects: Mutex)>>, + reconnect_calls: AtomicU64, } - /// A fixed post-close sleep races TCP delivery of the second goodbye - /// frame; polling for the expected count does not. commentlint: allow(JUDGE) - async fn wait_for_goodbyes(server: &FakeServer, expected: usize) { - for _ in 0..400 { - if server.log.lock().await.goodbyes.len() >= expected { - return; - } - tokio::time::sleep(Duration::from_millis(5)).await; + #[async_trait] + impl ProducerConnector for FakeConnector { + async fn connect( + &self, + _config: &HistorianProducerConfig, + ) -> Result, HistorianProducerError> { + Ok(Box::new(self.initial.clone())) } - panic!("server never observed {expected} goodbye frames"); - } - #[tokio::test] - async fn start_binds_session_at_route_open_and_omits_session_param() { - let server = fake_server(json!({"state":"active","run_id":"run-1"}), Vec::new()).await; - let mut client = client(&server).await; - let handle = client - .start( - "mc-historian:proj:1", - "role guidance", - "prompt", - "prov/model-a", - ) - .await - .unwrap(); - assert_eq!(handle.run_id, "run-1"); - client.close().await.unwrap(); - wait_for_goodbyes(&server, 1).await; - - let log = server.log.lock().await; - assert_eq!(log.route_sessions, vec!["mc-historian:proj:1"]); - assert_eq!(log.sends.len(), 1); - assert!( - log.sends[0].get("session").is_none(), - "session id lives in BindIdentity, not params" - ); - assert_eq!( - log.sends[0]["model"], - json!({ "provider": "prov", "model": "model-a" }), - "model is llm-runner's nested ModelParams object, split at the FIRST slash" - ); - assert_eq!(log.sends[0]["tools"], json!([])); - assert_eq!( - log.sends[0]["generation"]["max_output_tokens"], - json!(HISTORIAN_MAX_OUTPUT_TOKENS), - "an explicit output budget rides every send: llm-runner's default truncated a real summarization pass" - ); - assert_eq!( - log.sends[0]["generation"]["temperature"], - json!(HISTORIAN_TEMPERATURE), - "the calibrated temperature rides every send: prompt and sampling were calibrated together" - ); - assert_eq!( - log.sends[0]["system"], - json!("role guidance"), - "system rides the role-scoped SendParams field, byte-exact" - ); - assert_eq!(log.goodbyes, vec![10]); + async fn reconnect( + &self, + _config: &HistorianProducerConfig, + identity: &SemanticIdentity, + ) -> Result { + self.reconnect_calls.fetch_add(1, Ordering::SeqCst); + let (connection, override_identity) = + self.reconnects.lock().unwrap().pop_front().unwrap(); + Ok(Reconnected { + connection: Box::new(connection), + identity: override_identity.unwrap_or_else(|| identity.clone()), + }) + } } - #[tokio::test] - async fn start_omits_system_param_when_empty() { - // Empty means absent on the wire (the field's empty-as-absent rule); omitting it - // entirely keeps the send byte-shape identical to pre-system clients. - let server = fake_server(json!({"state":"active","run_id":"run-9"}), Vec::new()).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:9", "", "prompt", "prov/model-a") - .await - .unwrap(); - client.close().await.unwrap(); - tokio::time::sleep(Duration::from_millis(20)).await; - let log = server.log.lock().await; - assert_eq!(log.sends.len(), 1); - assert!( - log.sends[0].get("system").is_none(), - "empty system must be omitted, not sent as \"\"" - ); + fn unknown() -> HistorianProducerError { + HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::OutcomeUnknown, + "connection_retired", + "outcome unknown", + )) } - #[test] - fn unit_text_collects_only_assistant_message_units() { - let leaked = json!({ - "type": "example_replay", - "message": {"content": [{"type": "text", "text": "seed output"}]}, - "text": "flat seed output" - }); - let assistant = json!({ - "type": "assistant_message", - "message": {"content": [{"type": "text", "text": "real output"}]} - }); + fn terminal(code: &str) -> HistorianProducerError { + HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, + code, + "terminal", + )) + } - assert_eq!(unit_text(&leaked), None); - assert_eq!(unit_text(&assistant).as_deref(), Some("real output")); + fn connection( + daemon: u8, + responses: impl IntoIterator, HistorianProducerError>>, + ) -> FakeConnection { + FakeConnection { + daemon_id: [daemon; 16], + state: Arc::new(Mutex::new(FakeState { + responses: responses.into_iter().collect(), + ..FakeState::default() + })), + } } - #[tokio::test] - async fn await_output_uses_control_terminal_not_stream_end() { - let terminal_text = r#"x1-1"#; - let events = vec![ - json!({"kind":"display","event":{"type":"text_delta","text":"ignored"}}), - json!({"kind":"control","unit":{"type":"run_started","run_id":"run-1"}}), - json!({"kind":"control","unit":{"type":"assistant_message","message":{"message_id":"m-1","content":[ - {"type":"reasoning","text":"planning prose that restates start=\"FIRST\" and walks the seed examples"}, - {"type":"text","text":terminal_text} - ]}}}), - json!({"kind":"control","unit":{"type":"run_finished","finish_reason":"completed"}}), - ]; - let server = fake_server(json!({"state":"active","run_id":"run-1"}), events).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:2", "", "prompt", "prov/model-a") + async fn producer( + initial: FakeConnection, + reconnect: Option<(FakeConnection, Option)>, + ) -> (HistorianProducer, Arc) { + let connector = Arc::new(FakeConnector { + initial, + reconnects: Mutex::new(reconnect.into_iter().collect()), + reconnect_calls: AtomicU64::new(0), + }); + let config = HistorianProducerConfig { + request_timeout: Duration::from_secs(1), + await_timeout: Duration::from_secs(1), + ..HistorianProducerConfig::new("/unused", "/project", "opencode") + }; + let connected = HistorianProducer::connect_with(config, connector.clone()) .await .unwrap(); - let output = client.await_output("run-1").await.unwrap(); - client.close().await.unwrap(); - wait_for_goodbyes(&server, 2).await; - - assert_eq!(output.text, terminal_text); - let log = server.log.lock().await; - assert_eq!( - log.route_sessions, - vec!["mc-historian:proj:2", "mc-historian:proj:2"] - ); - assert_eq!(log.subscribes, vec![json!({"from":"start"})]); - assert_eq!( - log.goodbyes, - vec![11, 10], - "close releases subscribe and command routes" - ); + (connected, connector) } #[tokio::test] - async fn terminal_without_matching_run_started_fails_loud() { - let events = vec![ - json!({"kind":"control","unit":{"type":"run_started","run_id":"other-run"}}), - json!({"kind":"control","unit":{"type":"run_finished","finish_reason":"completed"}}), - ]; - let server = fake_server(json!({"state":"active","run_id":"run-1"}), events).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:3", "", "prompt", "prov/model-a") + async fn start_opens_expected_identity_and_sends_once() { + let first = connection(1, [Ok(br#"{"run_id":"run-1"}"#.to_vec())]); + let state = Arc::clone(&first.state); + let (mut producer, connector) = producer(first, None).await; + let handle = producer + .start("session-1", "system", "prompt", "provider/model") .await .unwrap(); - - let err = client.await_output("run-1").await.unwrap_err(); - assert!(matches!( - err, - HistorianProducerError::TerminalRunMismatch { - expected, - found: Some(found), - } if expected == "run-1" && found == "other-run" - )); + assert_eq!(handle.run_id, "run-1"); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 0); + let state = state.lock().unwrap(); + assert_eq!(state.requests.len(), 1); + assert_eq!(state.identities.len(), 1); + assert_eq!(state.identities[0].project_root, PathBuf::from("/project")); + assert_eq!(state.identities[0].harness, "opencode"); + assert_eq!(state.identities[0].session, "session-1"); + let body: Value = serde_json::from_slice(&state.requests[0]).unwrap(); + assert_eq!(body["method"], "session.send"); + assert_eq!(body["params"]["prompt"], "prompt"); } #[tokio::test] - async fn paused_unit_returns_run_paused() { - let events = vec![ - json!({"kind":"control","unit":{"type":"run_started","run_id":"run-1"}}), - json!({"kind":"control","unit":{"type":"paused","run_id":"run-1","reason":"auth_required"}}), - ]; - let server = fake_server(json!({"state":"active","run_id":"run-1"}), events).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:4", "", "prompt", "prov/model-a") + async fn same_daemon_and_identity_resends_exact_bytes_once() { + let first = connection(7, [Err(unknown())]); + let second = connection(7, [Ok(br#"{"run_id":"run-2"}"#.to_vec())]); + let first_state = Arc::clone(&first.state); + let second_state = Arc::clone(&second.state); + let (mut producer, connector) = producer(first, Some((second, None))).await; + let handle = producer + .start("session-2", "system", "prompt", "provider/model") .await .unwrap(); - - let err = client.await_output("run-1").await.unwrap_err(); - assert!(matches!( - err, - HistorianProducerError::RunPaused { - run_id, - reason: Some(reason), - .. - } if run_id == "run-1" && reason == "auth_required" - )); - } - #[test] - fn run_state_mapping_is_closed_over_the_exact_wire_vocabulary() { - let state = |state: &str| { - classify_run_state("run-1", &json!({ "run_id": "run-1", "state": state })) - }; - assert!(matches!(state("queued"), Ok(RunState::Active))); - assert!(matches!(state("running"), Ok(RunState::Active))); - assert!(matches!(state("completed"), Ok(RunState::Terminal))); - assert!(matches!(state("failed"), Ok(RunState::Terminal))); - assert!(matches!(state("cancelled"), Ok(RunState::Terminal))); - assert!(matches!(state("missing"), Ok(RunState::Missing { .. }))); - for undocumented in [ - "terminal", - "active", - "finished", - "interrupted", - "paused", - "pending", - "COMPLETED", - "Running", - "cancel", - "error", - "", - ] { - assert!( - matches!( - state(undocumented), - Err(HistorianProducerError::Protocol(_)) - ), - "state {undocumented:?} must be a protocol error, not a guessed recovery state" - ); - } - assert!(matches!( - classify_run_state("run-1", &json!({ "run_id": "run-1" })), - Err(HistorianProducerError::Protocol(_)) - )); - assert!(matches!( - classify_run_state("run-1", &json!({ "run_id": "other", "state": "completed" })), - Err(HistorianProducerError::Protocol(_)) - )); - // A response that names no run has not been proven to describe this - // one; reading it as `missing` would authorize a refire. - assert!(matches!( - classify_run_state("run-1", &json!({ "state": "missing" })), - Err(HistorianProducerError::Protocol(_)) - )); - } - - #[test] - fn cleanup_helper_preserves_primary_and_cleanup_diagnostics() { - assert_eq!( - with_cleanup(Ok(7), Ok(()), "session.delete").unwrap(), - 7, - "clean success stays untouched" - ); - - let cleanup_only = with_cleanup( - Ok(7), - Err(HistorianProducerError::TimedOut), - "session.delete", - ) - .unwrap_err(); - assert!(matches!( - &cleanup_only, - HistorianProducerError::CleanupFailed { primary: None, .. } - )); - assert!(cleanup_only - .to_string() - .contains("session.delete cleanup failed after success")); - - let both = with_cleanup::<()>( - Err(HistorianProducerError::tagged_subc( - "provider_error", - "rate limit window", - ErrorClass::Transient, - Some(9), - )), - Err(HistorianProducerError::TimedOut), - "session.delete", - ) - .unwrap_err(); - assert_eq!( - both.classification(), - Some(ErrorClassification { - class: ErrorClass::Transient, - retry_after_secs: Some(9), - }), - "retry policy must keep following the primary failure" - ); - assert!(both.has_class_field()); - let rendered = both.to_string(); - assert!(rendered.contains("rate limit window")); - assert!(rendered.contains("cleanup also failed")); - - assert!(matches!( - attach_cleanup(HistorianProducerError::TimedOut, Ok(()), "session.delete"), - HistorianProducerError::TimedOut - )); + assert_eq!(handle.run_id, "run-2"); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 1); + let first = first_state.lock().unwrap(); + let second = second_state.lock().unwrap(); + assert_eq!(first.requests.len(), 1); + assert_eq!(second.requests.len(), 1); + assert_eq!(first.requests[0], second.requests[0]); } #[tokio::test] - async fn replay_preserves_length_finish_reason_on_assistant_steps() { - let events = vec![ - json!({"kind":"control","unit":{"type":"run_started","run_id":"run-1"}}), - json!({"kind":"control","unit":{"type":"assistant_message","run_id":"run-1","finish_reason":"length","message":{"role":"assistant","content":[{"type":"text","text":"partial output"}]}}}), - json!({"kind":"control","unit":{"type":"run_finished","run_id":"run-1","finish_reason":"completed"}}), - ]; - let server = fake_server(json!({"run_id":"run-1"}), events).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:5", "", "prompt", "prov/model-a") + async fn second_unknown_outcome_stops_without_third_attempt() { + let first = connection(7, [Err(unknown())]); + let second = connection(7, [Err(unknown())]); + let first_state = Arc::clone(&first.state); + let second_state = Arc::clone(&second.state); + let (mut producer, connector) = producer(first, Some((second, None))).await; + + assert!(producer + .start("session", "", "prompt", "provider/model") .await - .unwrap(); + .is_err()); - let output = client.await_output("run-1").await.unwrap(); - assert_eq!(output.text, "partial output"); - assert!( - output.length_capped, - "a length-class step finish reason must survive a completed terminal" - ); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 1); + assert_eq!(first_state.lock().unwrap().requests.len(), 1); + assert_eq!(second_state.lock().unwrap().requests.len(), 1); } #[tokio::test] - async fn replay_retains_error_class_and_retry_delay_from_the_terminal() { - let events = vec![ - json!({"kind":"control","unit":{"type":"run_started","run_id":"run-1"}}), - json!({"kind":"control","unit":{"type":"error","run_id":"run-1","error":{"class":"transient","message":"rate limited","retry_after_secs":42,"provider_code":"rate_limit"}}}), - ]; - let server = fake_server(json!({"run_id":"run-1"}), events).await; - let mut client = client(&server).await; - client - .start("mc-historian:proj:6", "", "prompt", "prov/model-a") + async fn changed_daemon_returns_typed_unknown_without_resend() { + let first = connection(1, [Err(unknown())]); + let second = connection(2, []); + let second_state = Arc::clone(&second.state); + let (mut producer, connector) = producer(first, Some((second, None))).await; + let error = producer + .start("session", "", "prompt", "provider/model") .await - .unwrap(); - - let err = client.await_output("run-1").await.unwrap_err(); - let HistorianProducerError::RunFailed { - run_id, - detail, - classification, - .. - } = err - else { - panic!("expected RunFailed, got {err:?}"); - }; - assert_eq!(run_id, "run-1"); - assert_eq!(detail, "rate limited"); - assert_eq!( - classification, - Some(ErrorClassification { - class: ErrorClass::Transient, - retry_after_secs: Some(42), - }) - ); + .unwrap_err(); + assert!(matches!( + error, + HistorianProducerError::CrossIncarnationUnknown { + daemon_changed: true, + identity_changed: false + } + )); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 1); + assert!(second_state.lock().unwrap().requests.is_empty()); } - enum SendAction { - Respond(Value), - DropConnection, - Error(Value), - } - - /// Unlike [`fake_server`], keeps accepting connections so a client that - /// reconnects after a dropped send reaches the same scripted state. - async fn scripted_server( - actions: Vec, - reject_harness: Option<&'static str>, - ) -> FakeServer { - let temp = tempfile::tempdir().unwrap(); - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr: SocketAddr = listener.local_addr().unwrap(); - let key = generate_key().unwrap(); - let daemon_id = generate_daemon_id().unwrap(); - let connection_file = temp.path().join("subc-connection.json"); - write_atomic( - &connection_file, - &ConnectionInfo { - schema: SCHEMA_VERSION, - wire_version: Some(subc_protocol::PROTOCOL_VERSION), - endpoints: vec![Endpoint { - host: addr.ip().to_string(), - port: addr.port(), - }], - key: key.clone(), - daemon_id, - pid: std::process::id(), - daemon_ver: "fake".to_string(), - }, - ) - .unwrap(); - let log = Arc::new(Mutex::new(ServerLog::default())); - let log_task = Arc::clone(&log); - tokio::spawn(async move { - let mut actions: VecDeque = actions.into(); - loop { - let Ok((mut stream, _)) = listener.accept().await else { - break; - }; - if authenticate_server( - &mut stream, - &key, - &daemon_id, - "fake", - Duration::from_secs(2), - ) + #[tokio::test] + async fn any_semantic_identity_change_prevents_resend() { + for field in ["project", "harness", "session"] { + let first = connection(3, [Err(unknown())]); + let second = connection(3, []); + let second_state = Arc::clone(&second.state); + let mut changed = SemanticIdentity { + project_root: PathBuf::from("/project"), + harness: "opencode".to_owned(), + session: "session".to_owned(), + }; + match field { + "project" => changed.project_root = PathBuf::from("/other"), + "harness" => changed.harness = "claude-code".to_owned(), + "session" => changed.session = "other-session".to_owned(), + _ => unreachable!(), + } + let (mut producer, _) = producer(first, Some((second, Some(changed)))).await; + let error = producer + .start("session", "", "prompt", "provider/model") .await - .is_err() - { - continue; - } - let mut next_route = 10u16; - 'conn: loop { - let frame = match read_frame(&mut stream).await { - Ok(Some(frame)) => frame, - _ => break 'conn, - }; - match frame.header.ty { - FrameType::Request if frame.header.channel == 0 => { - let req: ClientControlRequest = - serde_json::from_slice(&frame.body).unwrap(); - let ClientControlRequest::RouteOpen { identity, .. } = req else { - continue; - }; - log_task - .lock() - .await - .route_sessions - .push(identity.session.clone()); - log_task - .lock() - .await - .route_harnesses - .push(identity.harness.clone()); - if reject_harness == Some(identity.harness.as_str()) { - send_error_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - json!({ - "code": "unsupported_harness", - "message": "harness must be opencode or pi", - }), - ) - .await; - continue; - } - let route = next_route; - next_route += 1; - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec(&ClientControlResponse::RouteOpen { - route_channel: route, - route_epoch: 1, - }) - .unwrap(), - ) - .await; - } - FrameType::Request => { - let req: Value = serde_json::from_slice(&frame.body).unwrap(); - if req.get("method").and_then(Value::as_str) != Some("session.send") { - continue; - } - log_task.lock().await.sends.push(req["params"].clone()); - log_task.lock().await.send_bodies.push(frame.body.clone()); - match actions.pop_front() { - Some(SendAction::Respond(body)) => { - send_response_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - serde_json::to_vec(&body).unwrap(), - ) - .await; - } - Some(SendAction::Error(body)) => { - send_error_frame( - &mut stream, - frame.header.channel, - frame.header.epoch, - frame.header.corr, - body, - ) - .await; - } - // Dropping the connection leaves the send - // outcome unknown to the client. - Some(SendAction::DropConnection) | None => break 'conn, - } - } - _ => {} - } + .unwrap_err(); + assert!(matches!( + error, + HistorianProducerError::CrossIncarnationUnknown { + daemon_changed: false, + identity_changed: true } - } - }); - FakeServer { - connection_file, - log, - _temp: temp, + )); + assert!(second_state.lock().unwrap().requests.is_empty(), "{field}"); } } - async fn send_error_frame( - stream: &mut TcpStream, - channel: u16, - epoch: u32, - corr: u64, - body: Value, - ) { - let frame = Frame::build( - FrameType::Error, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - corr, - serde_json::to_vec(&body).unwrap(), - ) - .unwrap(); - write_frame(stream, &frame).await.unwrap(); - } - - async fn scripted_client(server: &FakeServer, harness: &str) -> HistorianProducer { - HistorianProducer::connect(HistorianProducerConfig { - connection_file: server.connection_file.clone(), - project_root: std::env::current_dir().unwrap(), - harness: harness.to_string(), - module_id: "broca".to_string(), - handshake_timeout: Duration::from_secs(2), - // Short so the DropConnection scripts fail over quickly. - request_timeout: Duration::from_millis(500), - await_timeout: Duration::from_secs(2), - }) - .await - .unwrap() + #[tokio::test] + async fn not_sent_and_terminal_failures_are_never_replayed() { + for failure in [ + HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::NotSent, + "correlation_exhausted", + "not sent", + )), + terminal("idempotency_conflict"), + ] { + let first = connection(1, [Err(failure)]); + let state = Arc::clone(&first.state); + let (mut producer, connector) = producer(first, None).await; + assert!(producer + .start("session", "", "prompt", "provider/model") + .await + .is_err()); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 0); + assert_eq!(state.lock().unwrap().requests.len(), 1); + } } #[tokio::test] - async fn lost_send_response_reconnects_once_with_byte_identical_body() { - let server = scripted_server( - vec![ - SendAction::DropConnection, - SendAction::Respond(json!({"run_id":"run-orig"})), - ], - None, - ) - .await; - let mut client = scripted_client(&server, "opencode").await; - - let handle = client - .start("ses-recover", "sys", "prompt", "prov/model-a") - .await - .unwrap(); - - assert_eq!( - handle.run_id, "run-orig", - "the deduplicated resend must surface the original run id" - ); - let log = server.log.lock().await; - assert_eq!(log.send_bodies.len(), 2); + async fn close_releases_subscription_and_command_routes() { + let first = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); + let state = Arc::clone(&first.state); + let (mut producer, _) = producer(first, None).await; + producer.bind_session("session"); + producer.ensure_command_route().await.unwrap(); + producer.ensure_subscribe_route().await.unwrap(); + producer.close().await.unwrap(); + let state = state.lock().unwrap(); assert_eq!( - log.send_bodies[0], log.send_bodies[1], - "recovery must resend the frozen bytes verbatim" + state.opened_routes.iter().copied().collect::>(), + state.closed_routes.iter().copied().collect::>() ); assert_eq!( - log.route_sessions, - vec!["ses-recover", "ses-recover"], - "the rebind must carry the same route identity" + state.closed_routes, + vec![ + RouteHandle { + channel: 2, + epoch: 2, + }, + RouteHandle { + channel: 1, + epoch: 1, + }, + ] ); - assert_eq!(log.route_harnesses, vec!["opencode", "opencode"]); + assert_eq!(state.close_calls, 1); } #[tokio::test] - async fn application_send_error_is_not_resent() { - let server = scripted_server( - vec![SendAction::Error( - json!({"code":"idempotency_conflict","message":"different bytes"}), - )], - None, - ) - .await; - let mut client = scripted_client(&server, "opencode").await; - - let err = client - .start("ses-conflict", "", "prompt", "prov/model-a") + async fn attempt_cleanup_reopens_exact_routes_without_closing_connection() { + let first = connection( + 1, + [ + Ok(br#"{"run_id":"run-1"}"#.to_vec()), + Ok(br#"{"run_id":"run-2"}"#.to_vec()), + ], + ); + let state = Arc::clone(&first.state); + let (mut producer, _) = producer(first, None).await; + producer + .start("session", "", "prompt-1", "provider/model-a") .await - .unwrap_err(); - - assert!(matches!( - &err, - HistorianProducerError::Subc(body) if body.code == "idempotency_conflict" - )); - tokio::time::sleep(Duration::from_millis(20)).await; - assert_eq!(server.log.lock().await.send_bodies.len(), 1); - } - - #[tokio::test] - async fn second_unknown_send_outcome_fails_without_a_third_send() { - let server = scripted_server( - vec![SendAction::DropConnection, SendAction::DropConnection], - None, - ) - .await; - let mut client = scripted_client(&server, "opencode").await; + .unwrap(); + producer.ensure_subscribe_route().await.unwrap(); + producer.close_attempt().await.unwrap(); + assert_eq!(state.lock().unwrap().close_calls, 0); - let err = client - .start("ses-twice", "", "prompt", "prov/model-a") + producer + .start("session", "", "prompt-2", "provider/model-b") .await - .unwrap_err(); + .unwrap(); + producer.ensure_subscribe_route().await.unwrap(); + producer.close().await.unwrap(); - assert!(send_outcome_unknown(&err), "got {err:?}"); - // A successful `status` call would prove that `start` reconnected - // after the second dropped connection: the scripted server keeps - // accepting, so only a client parked on the dead connection fails. - let followup = client.status("run-x").await; - assert!( - followup.is_err(), - "the producer must not have reconnected after giving up: {followup:?}" + let state = state.lock().unwrap(); + assert_eq!(state.opened_routes.len(), 4); + assert_eq!( + state.opened_routes.iter().copied().collect::>(), + state.closed_routes.iter().copied().collect::>() ); - assert_eq!(server.log.lock().await.send_bodies.len(), 2); + assert_eq!(state.close_calls, 1); } #[tokio::test] - async fn unsupported_harness_reaches_bind_rejection_untranslated() { - let server = scripted_server(Vec::new(), Some("weird")).await; - let mut client = scripted_client(&server, "weird").await; - - let err = client - .start("ses-weird", "", "prompt", "prov/model-a") - .await - .unwrap_err(); - - assert!(matches!( - &err, - HistorianProducerError::Subc(body) if body.code == "unsupported_harness" - )); - let log = server.log.lock().await; + async fn first_route_close_failure_does_not_skip_second_route_or_client_close() { + let first = connection(1, []); + first + .state + .lock() + .unwrap() + .close_route_errors + .push_back(terminal("close_failed")); + let state = Arc::clone(&first.state); + let (mut producer, _) = producer(first, None).await; + producer.bind_session("session"); + producer.ensure_command_route().await.unwrap(); + producer.ensure_subscribe_route().await.unwrap(); + + assert!(producer.close().await.is_err()); + + let state = state.lock().unwrap(); + assert_eq!(state.opened_routes.len(), 2); assert_eq!( - log.route_harnesses, - vec!["weird"], - "the harness value must reach the bind untranslated" + state.opened_routes.iter().copied().collect::>(), + state.closed_routes.iter().copied().collect::>() ); - assert!(log.send_bodies.is_empty(), "a rejected bind must not send"); - } - - #[tokio::test] - async fn real_factory_binds_routes_with_the_route_bound_harness() { - let server = scripted_server( + assert_eq!( + state.closed_routes, vec![ - SendAction::Respond(json!({"run_id":"run-oc"})), - SendAction::Respond(json!({"run_id":"run-pi"})), + RouteHandle { + channel: 2, + epoch: 2, + }, + RouteHandle { + channel: 1, + epoch: 1, + }, ], - None, - ) - .await; - let factory = crate::RealHistorianProducerFactory { - connection_file: server.connection_file.clone(), - }; - for harness in ["opencode", "pi"] { - let mut producer = crate::HistorianProducerFactory::connect( - &factory, - std::path::Path::new("/proj"), - harness, - ) - .await - .unwrap(); - producer - .start(&format!("ses-{harness}"), "", "prompt", "prov/model-a") - .await - .unwrap(); - producer.close().await.unwrap(); + "first route failure must not skip exact second route" + ); + assert_eq!(state.close_calls, 1); + } + + #[test] + fn run_state_mapping_is_closed_over_known_states() { + for state in ["queued", "running"] { + assert_eq!( + classify_run_state("run", &json!({"run_id":"run", "state":state})).unwrap(), + RunState::Active + ); + } + for state in ["completed", "failed", "cancelled"] { + assert_eq!( + classify_run_state("run", &json!({"run_id":"run", "state":state})).unwrap(), + RunState::Terminal + ); } - let log = server.log.lock().await; - assert_eq!(log.route_harnesses, vec!["opencode", "pi"]); + assert!(classify_run_state("run", &json!({"run_id":"run", "state":"paused"})).is_err()); + assert!(classify_run_state("run", &json!({"run_id":"other", "state":"missing"})).is_err()); + } + + #[test] + fn error_class_wire_strings_match_pinned_contract_set() { + assert_eq!(ErrorClass::Transient.as_wire_str(), "transient"); + assert_eq!(ErrorClass::Permanent.as_wire_str(), "permanent"); + assert_eq!(ErrorClass::AuthRequired.as_wire_str(), "auth_required"); + assert_eq!( + ErrorClass::ContextOverflow.as_wire_str(), + "context_overflow" + ); } } diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index 474dcc4e5..414e6f786 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -1,18 +1,7 @@ -//! The Magic Context subc module. +//! Magic Context component for `mc-host`. //! -//! Harness-agnostic cache-stability transform: receives already-decoded CK items, -//! classifies the pass, drives `cortexkit-cache-core` through `mc-core`, and -//! persists per-session state in the single-writer `mc-store`. Served over the subc -//! wire via `subc-client-rs`'s `serve` (provider role). -//! -//! Lifecycle (handled by `serve`): read `--subc `, authenticate, -//! send HELLO{manifest}, await HELLO_ACK. [`McHandler::on_hello_ack`] is the storage -//! seam — it resolves the descriptor (from `ack.storage`, else a local dev path) and -//! opens the store EXACTLY ONCE (single-writer lease held for the module lifetime). -//! -//! Slice-1 scope: the cache-stability spine. `handle` answers `transform` (the -//! CK-in/CK-out pass: classify → cache-core step → conditional commit), `health` -//! (proves the store opened), and echoes otherwise. +//! [`McHandler`] implements the host-owned primary lifecycle, transforms already-decoded +//! CK items, and persists per-session state in the single-writer `mc-store`. #![forbid(unsafe_code)] @@ -24,6 +13,7 @@ pub mod codec; pub mod compartment_coverage; pub mod config; pub mod decay_render; +pub mod dispatch; pub mod divergence; pub mod healing; pub mod historian; @@ -64,10 +54,16 @@ use crate::smart_note_evaluation::{ SmartNoteEvaluationOutcome, SmartNoteLifecycleState, SmartNoteSelectionCycle, SmartNoteSelectionSnapshot, }; +use async_trait::async_trait; use chrono::{Local, TimeZone}; use cortexkit_lease::LeaseError; use cortexkit_store::StoreError; use cortexkit_store_types::{sqlite_store_path, Isolation, StorageBackend, StorageDescriptor}; +use mc_host::{ + BindOutcome, CompositeComponent, HealthReport, HealthStatus, HostInit, InitError, + ManifestSnapshot, PrimaryComponent, RequestCtx, RequestOutcome, ResourceDeclaration, + RouteHandle, RouteIdentity, ShutdownError, +}; #[cfg(test)] use mc_store::TagNumberRow; use mc_store::{ @@ -87,10 +83,9 @@ use mc_store::{ use serde::Deserialize; use serde_json::{json, Map, Value}; use sha2::{Digest, Sha256}; -use subc_client_rs::{ - async_trait, HandlerOutcome, HealthReport, HealthStatus, ModuleHandler, RequestCtx, - RouteBindRequest, RouteHandle, -}; +use tokio_util::{sync::CancellationToken, task::TaskTracker}; + +use crate::dispatch::{PreparedOutcome, PreparedOutput, PreparedSegment}; use boundary::{BoundaryBlock, BoundaryContext, BoundaryMsg, Role, TriggerContext}; use classify::{ @@ -111,18 +106,7 @@ use scheduler::MIN_PLAUSIBLE_CONTEXT_LIMIT; use selection::SelKind; #[cfg(test)] use session_resolver::ResolvedSession; -use session_resolver::{ - MissingSessionResolver, RealSessionResolver, SessionResolveError, SessionResolver, -}; -#[cfg(test)] -use subc_protocol::manifest::ExecutionMode; -use subc_protocol::{ - manifest::{ - Bindings, Concurrency, ConsumerRole, IdentityBinding, IdentityScope, ModuleManifest, - ProviderRole, StorageBinding, StorageKind, StorageScope, TrustTier, - }, - ModuleHelloAckBody, PROTOCOL_VERSION, -}; +use session_resolver::{MissingSessionResolver, SessionResolveError, SessionResolver}; #[cfg(test)] use transform::ReductionDecision; @@ -262,8 +246,6 @@ impl Default for StoreOpenPolicy { struct StoreOpenCoordinator { phase: AtomicU8, wait_started_at_ms: AtomicU64, - cancelled: AtomicBool, - cancel: Notify, active_waiters: AtomicU64, waiter_completed: Notify, waiter_starts: AtomicU64, @@ -275,8 +257,6 @@ impl StoreOpenCoordinator { Self { phase: AtomicU8::new(STORE_OPEN_IDLE), wait_started_at_ms: AtomicU64::new(0), - cancelled: AtomicBool::new(false), - cancel: Notify::new(), active_waiters: AtomicU64::new(0), waiter_completed: Notify::new(), waiter_starts: AtomicU64::new(0), @@ -308,11 +288,6 @@ impl StoreOpenCoordinator { self.active_waiters.fetch_sub(1, Ordering::AcqRel); self.waiter_completed.notify_waiters(); } - - fn cancel(&self) { - self.cancelled.store(true, Ordering::Release); - self.cancel.notify_waiters(); - } } struct StoreOpenWaiterGuard { @@ -796,8 +771,8 @@ struct ModuleStripSeedWire { strip_kind: String, } -fn state_sync_seq_mismatch_error(expected: u64, found: u64) -> HandlerOutcome { - HandlerOutcome::Error { +fn state_sync_seq_mismatch_error(expected: u64, found: u64) -> PreparedOutcome { + PreparedOutcome::Error { code: "authority_seq_mismatch".to_string(), message: json!({ "code": "authority_seq_mismatch", @@ -808,8 +783,8 @@ fn state_sync_seq_mismatch_error(expected: u64, found: u64) -> HandlerOutcome { } } -fn historian_compartment_sync_busy_error(phase: HistorianPhase) -> HandlerOutcome { - HandlerOutcome::Error { +fn historian_compartment_sync_busy_error(phase: HistorianPhase) -> PreparedOutcome { + PreparedOutcome::Error { code: "historian_compartment_sync_busy".to_string(), message: json!({ "code": "historian_compartment_sync_busy", @@ -911,7 +886,7 @@ struct CompletedStateSyncSeed { generation: u64, expected_seq: u64, total: usize, - result: Vec, + result: PreparedOutput, } #[derive(Debug)] @@ -1037,7 +1012,7 @@ struct CompletedTransformPage { transform_id: String, generation: u64, final_digest: String, - result: Vec, + result: PreparedOutput, } #[derive(Debug)] @@ -2996,11 +2971,14 @@ impl ProjectionCache { } } -/// The module handler. Holds the single store handle (opened once in `on_hello_ack`) -/// and the per-route session bindings (route channel → {project, session}). +/// Host primary for Magic Context. Owns one store lease, full-handle route state, +/// and every module task admitted during this host incarnation. pub struct McHandler { - store: Arc>>, + store: Arc>>>, store_open: Arc, + task_admission_open: Mutex, + cancel: CancellationToken, + tasks: TaskTracker, producer_factory: Arc, session_resolver: Arc, config: Mutex, @@ -3044,12 +3022,9 @@ pub struct McHandler { connect_failure_commit_hook: ConnectFailureCommitHook, #[cfg(test)] publication_fence_write_hook: ConnectFailureCommitHook, - /// Route channel → its session binding. Populated at `on_bind`, removed at - /// `on_route_gone`. The SDK validates the route handle's epoch before dispatching a - /// request, so a channel key cannot resolve a stale route. A `Mutex` (not a - /// lock-free map) is appropriate because writes are rare (once per route open/close) - /// and reads are one cheap lookup per transform. - bindings: Mutex>, + /// Full route handle → its session binding. Epoch is part of every lookup and removal, + /// so channel reuse cannot observe or delete state owned by another incarnation. + bindings: Mutex>, /// The host state-sync payload carries this legacy per-project evaluator flag for wire /// compatibility. Conditioned-write gating reads live protocol-v2 registrations instead: /// state sync is not a liveness signal. @@ -3058,9 +3033,9 @@ pub struct McHandler { /// project, so restart exposes zero evaluator capacity until a fresh route registers. note_evaluator_registrations: Mutex>>, note_evaluator_registration_seq: AtomicU64, - /// Validated transform channel → (session, route root). The root is part of provenance; + /// Validated transform route → (session, route root). The root is part of provenance; /// a cache row for the same session cannot authenticate a facade opened on another root. - transform_route_channels: Mutex>, + transform_route_channels: Mutex>, /// Roots previously observed on a validated transform for each session. This survives route /// teardown so durable cache state remains usable only along an authenticated route lineage. transform_session_roots: Mutex>>, @@ -3099,7 +3074,7 @@ struct NoteEvaluatorRegistration { token: String, registration_generation: i64, evaluator_instance: String, - channel: u16, + route: RouteHandle, policy_version: i64, capacity: i64, retina_handoff: bool, @@ -3151,6 +3126,7 @@ pub trait HistorianProducerFactory: Send + Sync { struct RealHistorianProducerFactory { connection_file: PathBuf, + cancellation: CancellationToken, } #[async_trait] @@ -3162,7 +3138,7 @@ impl HistorianProducerFactory for RealHistorianProducerFactory { ) -> Result, HistorianProducerError> { Ok(Box::new( HistorianProducer::connect(HistorianProducerConfig { - handshake_timeout: Duration::from_secs(2), + cancellation: Some(self.cancellation.clone()), ..HistorianProducerConfig::new(self.connection_file.clone(), project_root, harness) }) .await?, @@ -3501,9 +3477,12 @@ impl HistorianProducerFactory for MissingProducerFactory { _project_root: &Path, _harness: &str, ) -> Result, HistorianProducerError> { - Err(HistorianProducerError::NoEndpoint { - path: PathBuf::from(""), - }) + Err(HistorianProducerError::Client( + historian_producer::HistorianClientFailure { + code: "connection_unavailable".to_owned(), + message: "mc-module has no host connection file".to_owned(), + }, + )) } } @@ -3513,21 +3492,22 @@ impl McHandler { } pub fn new_with_connection_file(connection_file: Option) -> Self { - let producer_factory: Arc = match connection_file.clone() { + let cancel = CancellationToken::new(); + let producer_factory: Arc = match connection_file { Some(path) => Arc::new(RealHistorianProducerFactory { connection_file: path, + cancellation: cancel.clone(), }), None => Arc::new(MissingProducerFactory), }; - let session_resolver: Arc = match connection_file { - Some(path) => Arc::new(RealSessionResolver::new(path)), - None => Arc::new(MissingSessionResolver), - }; McHandler { - store: Arc::new(OnceLock::new()), + store: Arc::new(Mutex::new(None)), store_open: Arc::new(StoreOpenCoordinator::new()), + task_admission_open: Mutex::new(true), + cancel, + tasks: TaskTracker::new(), producer_factory, - session_resolver, + session_resolver: Arc::new(MissingSessionResolver), config: Mutex::new(ConfigCache::default()), #[cfg(test)] fixed_config: None, @@ -3583,8 +3563,42 @@ impl McHandler { } } - fn begin_store_open(&self, descriptor: StorageDescriptor) { - if self.store.get().is_some() + fn store(&self) -> Option> { + self.store.lock().expect("store slot mutex").clone() + } + + fn spawn_tracked_task(&self, future: F) -> Option> + where + F: Future + Send + 'static, + T: Send + 'static, + { + let admission = self + .task_admission_open + .lock() + .expect("module task admission mutex"); + if !*admission { + return None; + } + Some(self.tasks.spawn(future)) + } + + fn spawn_module_task(&self, future: F) -> Option> + where + F: Future + Send + 'static, + T: Send + 'static, + { + self.spawn_tracked_task(future) + } + + fn begin_store_open(&self, descriptor: StorageDescriptor) -> Result<(), InitError> { + if !*self + .task_admission_open + .lock() + .expect("module task admission mutex") + { + return Err(InitError("module task admission is closed".to_owned())); + } + if self.store().is_some() || self .store_open .phase @@ -3596,7 +3610,7 @@ impl McHandler { ) .is_err() { - return; + return Ok(()); } self.store_open @@ -3607,27 +3621,38 @@ impl McHandler { .fetch_add(1, Ordering::Relaxed); let store = Arc::clone(&self.store); let coordinator = Arc::clone(&self.store_open); - tokio::spawn(async move { - let _guard = StoreOpenWaiterGuard { - coordinator: Arc::clone(&coordinator), - }; - Self::run_store_open(store, coordinator, descriptor).await; - }); + let task_coordinator = Arc::clone(&coordinator); + let cancel = self.cancel.clone(); + if self + .spawn_tracked_task(async move { + let _guard = StoreOpenWaiterGuard { + coordinator: Arc::clone(&task_coordinator), + }; + Self::run_store_open(store, task_coordinator, descriptor, cancel).await; + }) + .is_none() + { + coordinator.phase.store(STORE_OPEN_IDLE, Ordering::Release); + coordinator.finish_waiter(); + return Err(InitError("module task admission is closed".to_owned())); + } + Ok(()) } async fn run_store_open( - store_slot: Arc>>, + store_slot: Arc>>>, coordinator: Arc, descriptor: StorageDescriptor, + cancel: CancellationToken, ) { let policy = *coordinator.policy.lock().expect("store open policy mutex"); let mut last_lease_error = match Self::open_store_once(&descriptor).await { Ok(opened) => { - if coordinator.cancelled.load(Ordering::Acquire) { + if cancel.is_cancelled() { coordinator.phase.store(STORE_OPEN_IDLE, Ordering::Release); return; } - let _ = store_slot.set(Arc::new(opened)); + *store_slot.lock().expect("store slot mutex") = Some(Arc::new(opened)); coordinator.phase.store(STORE_OPENED, Ordering::Release); return; } @@ -3655,7 +3680,7 @@ impl McHandler { let mut attempt = 0usize; loop { let elapsed = started.elapsed(); - if coordinator.cancelled.load(Ordering::Acquire) { + if cancel.is_cancelled() { eprintln!( "mc-module: storage lease wait cancelled during shutdown after {:.2}s", elapsed.as_secs_f64() @@ -3675,7 +3700,7 @@ impl McHandler { let delay = jittered_store_open_delay(backoff, policy.max_backoff, attempt) .min(policy.wait_window.saturating_sub(elapsed)); tokio::select! { - _ = coordinator.cancel.notified() => { + _ = cancel.cancelled() => { eprintln!( "mc-module: storage lease wait cancelled during shutdown after {:.2}s", started.elapsed().as_secs_f64() @@ -3685,7 +3710,7 @@ impl McHandler { } _ = tokio::time::sleep(delay) => {} } - if coordinator.cancelled.load(Ordering::Acquire) { + if cancel.is_cancelled() { eprintln!( "mc-module: storage lease wait cancelled during shutdown after {:.2}s", started.elapsed().as_secs_f64() @@ -3699,11 +3724,11 @@ impl McHandler { match Self::open_store_once(&descriptor).await { Ok(opened) => { - if coordinator.cancelled.load(Ordering::Acquire) { + if cancel.is_cancelled() { coordinator.phase.store(STORE_OPEN_IDLE, Ordering::Release); return; } - let _ = store_slot.set(Arc::new(opened)); + *store_slot.lock().expect("store slot mutex") = Some(Arc::new(opened)); coordinator.phase.store(STORE_OPENED, Ordering::Release); eprintln!( "mc-module: storage lease released; store opened after {:.2}s", @@ -3791,8 +3816,11 @@ impl McHandler { session_resolver: Arc, ) -> Self { McHandler { - store: Arc::new(OnceLock::new()), + store: Arc::new(Mutex::new(None)), store_open: Arc::new(StoreOpenCoordinator::new()), + task_admission_open: Mutex::new(true), + cancel: CancellationToken::new(), + tasks: TaskTracker::new(), producer_factory: factory, session_resolver, config: Mutex::new(ConfigCache::default()), @@ -3843,7 +3871,7 @@ impl McHandler { /// Record the route's session binding (called from `on_bind`). Last write wins for a /// reused channel — the daemon won't reuse a channel without a `route.gone` first, so /// this only overwrites a stale entry that somehow survived (defensive). - fn bind_route(&self, channel: u16, binding: SessionBinding) { + fn bind_route(&self, channel: RouteHandle, binding: SessionBinding) { self.remove_note_evaluator_registrations_for_channel(channel); self.transform_route_channels .lock() @@ -3937,13 +3965,13 @@ impl McHandler { }); } - fn remove_note_evaluator_registrations_for_channel(&self, channel: u16) { + fn remove_note_evaluator_registrations_for_channel(&self, channel: RouteHandle) { let mut registrations = self .note_evaluator_registrations .lock() .expect("note evaluator registrations mutex"); registrations.retain(|_, entries| { - entries.retain(|entry| entry.channel != channel); + entries.retain(|entry| entry.route != channel); !entries.is_empty() }); } @@ -3978,24 +4006,27 @@ impl McHandler { /// Resolve the notes-authority project scoping this evaluator route. The /// body never chooses the project; only the server-side route binding does. - fn resolve_note_evaluator_project(&self, channel: u16) -> Result { + fn resolve_note_evaluator_project( + &self, + channel: RouteHandle, + ) -> Result { let binding = self .facade_binding(channel) .map_err(|_| session_unresolved_error())?; let route_project_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return Err(store_unavailable_error()); }; let authority = store .authority_project_state_for_route(&route_project_root, "notes") - .map_err(|error| HandlerOutcome::Error { + .map_err(|error| PreparedOutcome::Error { code: "authority_route_lookup_failed".to_string(), message: error.to_string(), })?; match authority { Some((project, state)) if state == "MODULE" => Ok(project), Some((_, state)) if state == "DRAINING" => Err(authority_draining_error("notes")), - _ => Err(HandlerOutcome::Error { + _ => Err(PreparedOutcome::Error { code: "authority_not_module".to_string(), message: "evaluator registration requires MODULE notes authority on this route" .to_string(), @@ -4006,12 +4037,12 @@ impl McHandler { fn validated_note_evaluator_registration( &self, project: &str, - channel: u16, + channel: RouteHandle, token: &str, registration_generation: i64, evaluator_instance: &str, now: i64, - ) -> Result { + ) -> Result { self.purge_expired_note_evaluator_registrations(now); self.note_evaluator_registrations .lock() @@ -4022,7 +4053,7 @@ impl McHandler { entry.token == token && entry.registration_generation == registration_generation && entry.evaluator_instance == evaluator_instance - && entry.channel == channel + && entry.route == channel && entry.expires_at > now }) }) @@ -4117,8 +4148,7 @@ impl McHandler { // snapshot. Load the persisted epoch first, then inspect the bounded projection and native // cores so stale process state cannot select an outdated entry. let current_revert_epoch = self - .store - .get()? + .store()? .load(&parsed.session_id) .ok()? .meta @@ -4219,8 +4249,7 @@ impl McHandler { ) -> Option { let after = request.full_array_fingerprint.as_deref()?; let revert_epoch = self - .store - .get()? + .store()? .load(&request.session_id) .ok()? .meta @@ -4300,7 +4329,7 @@ impl McHandler { } /// Remove a route and evict process-local session state after its final binding closes. - fn unbind_route(&self, channel: u16) { + fn unbind_route(&self, channel: RouteHandle) { self.remove_note_evaluator_registrations_for_channel(channel); self.transform_route_channels .lock() @@ -4374,7 +4403,7 @@ impl McHandler { /// transform output — a correctly-bound request resolves and proceeds identically. fn resolve_binding( &self, - channel: u16, + channel: RouteHandle, request_session: &str, ) -> Result { let map = self.bindings.lock().expect("bindings mutex"); @@ -4387,22 +4416,22 @@ impl McHandler { fn state_sync_binding( &self, - channel: u16, + channel: RouteHandle, request_session: Option<&str>, - ) -> Result { + ) -> Result { let binding = self .bindings .lock() .expect("bindings mutex") .get(&channel) .cloned() - .ok_or_else(|| HandlerOutcome::Error { + .ok_or_else(|| PreparedOutcome::Error { code: "route_unbound".to_string(), message: "state sync on a channel with no session binding".to_string(), })?; if let Some(request_session) = request_session { if binding.session != request_session { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), @@ -4415,7 +4444,7 @@ impl McHandler { /// Return the channel binding without comparing a request session. OpenCode Rust facade /// routes bind a real session id, while Claude Code facade routes bind an instance token; /// `resolve_facade_scope` applies the corresponding identity mode before touching the store. - fn facade_binding(&self, channel: u16) -> Result { + fn facade_binding(&self, channel: RouteHandle) -> Result { self.bindings .lock() .expect("bindings mutex") @@ -4437,7 +4466,7 @@ impl McHandler { .any(|root| canonical_root(root) == canonical_project_root) }); if !root_observed { - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return false; }; let durable_root_observed = canonical_project_root.to_str().is_some_and(|root| { @@ -4460,8 +4489,7 @@ impl McHandler { return true; } if self - .store - .get() + .store() .is_some_and(|store| store.has_cache_state(session_id).unwrap_or(false)) { return true; @@ -4481,7 +4509,7 @@ impl McHandler { fn bind_authority_route( &self, store: &McStore, - channel: u16, + channel: RouteHandle, context_store_uuid: &str, project: &str, ) -> Result<(), McStoreError> { @@ -4793,7 +4821,7 @@ impl McHandler { let fingerprint_items: Vec<_> = chunk.snapshot.iter().map(|item| item.as_item()).collect(); let observed = historian::compute_chunk_fingerprint(&fingerprint_items); - tokio::spawn(async move { + let _ = self.spawn_module_task(async move { let _guard = guard; let result = async { let action = historian::handle_restart_load( @@ -4851,7 +4879,7 @@ impl McHandler { Some("reattaching") } HistorianPhase::Firing | HistorianPhase::Validating | HistorianPhase::Publishing => { - tokio::spawn(async move { + let _ = self.spawn_module_task(async move { let _guard = guard; if let Err(e) = historian::handle_restart_load( &store, @@ -5458,7 +5486,13 @@ impl McHandler { task: HistorianFiringTask, ) -> Result { let factory = Arc::clone(&self.producer_factory); - let handle = tokio::spawn(Self::execute_historian_firing_task(factory, task)); + let Some(handle) = + self.spawn_module_task(Self::execute_historian_firing_task(factory, task)) + else { + return Err(historian::HistorianDriveError::Producer( + HistorianProducerError::TimedOut, + )); + }; match tokio::time::timeout(historian::completion_wait_budget(), handle).await { Ok(Ok(outcome)) => outcome, Ok(Err(join_err)) => Err(historian::HistorianDriveError::Producer( @@ -5529,7 +5563,14 @@ impl McHandler { }; let wait = historian::wrapup_round_wait_budget().min(remaining); let factory = Arc::clone(&self.producer_factory); - let handle = tokio::spawn(Self::execute_historian_firing_task(factory, task)); + let Some(handle) = + self.spawn_module_task(Self::execute_historian_firing_task(factory, task)) + else { + return Err(WrapupFiringError::Retryable( + RetryableWrapupReason::SnapshotUnavailable, + "module task admission closed before historian round".to_owned(), + )); + }; match tokio::time::timeout(wait, handle).await { Ok(Ok(Ok(outcome))) => Ok(outcome), Ok(Ok(Err(error))) => { @@ -5611,7 +5652,7 @@ impl McHandler { fn spawn_historian_firing(&self, task: HistorianFiringTask) { let factory = Arc::clone(&self.producer_factory); - tokio::spawn(async move { + let _ = self.spawn_module_task(async move { let session_id = task.session_id.clone(); let result = Self::execute_historian_firing_task(factory, task).await; match result { @@ -5623,7 +5664,7 @@ impl McHandler { }); } - fn handle_state_import_value(&self, channel: u16, request: Value) -> HandlerOutcome { + fn handle_state_import_value(&self, channel: RouteHandle, request: Value) -> PreparedOutcome { let raw_session_id = request .get("session_id") .and_then(Value::as_str) @@ -5662,7 +5703,7 @@ impl McHandler { }; if parsed.v != 1 { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_import_version".to_string(), message: "state_import requires v=1".to_string(), }; @@ -5679,7 +5720,7 @@ impl McHandler { } if parsed.batch_count == 0 || parsed.batch_seq >= parsed.batch_count { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "batch_seq_mismatch".to_string(), message: "batch_seq must be inside a nonempty batch_count".to_string(), }; @@ -5689,22 +5730,22 @@ impl McHandler { Ok(binding) => binding, Err(BindingError::Unbound) => { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "route_unbound".to_string(), message: "state_import on a channel with no session binding".to_string(), }; } Err(BindingError::SessionMismatch) => { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), }; } }; - let store = match self.store.get() { - Some(store) => Arc::clone(store), + let store = match self.store() { + Some(store) => Arc::clone(&store), None => { discard(self); return store_unavailable_error(); @@ -5722,7 +5763,7 @@ impl McHandler { Ok(StateImportPreflight::Ready) => {} Err(StateImportError::SessionNotEmpty) => { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_not_empty".to_string(), message: "state_import only accepts a session with no durable state" .to_string(), @@ -5730,7 +5771,7 @@ impl McHandler { } Err(error) => { discard(self); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -5786,7 +5827,7 @@ impl McHandler { "imported": result.imported, "duplicate": result.duplicate, })), - Err(StateImportError::SessionNotEmpty) => HandlerOutcome::Error { + Err(StateImportError::SessionNotEmpty) => PreparedOutcome::Error { code: "session_not_empty".to_string(), message: "state_import only accepts a session with no durable state" .to_string(), @@ -5794,23 +5835,23 @@ impl McHandler { Err(StateImportError::Validation(error)) => { state_import_validation_error(error) } - Err(StateImportError::Store(error)) => HandlerOutcome::Error { + Err(StateImportError::Store(error)) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, } } Err(StateImportStageError::Validation(error)) => state_import_validation_error(error), - Err(StateImportStageError::Protocol { code, message }) => HandlerOutcome::Error { + Err(StateImportStageError::Protocol { code, message }) => PreparedOutcome::Error { code: code.to_string(), message: message.to_string(), }, } } - fn handle_agent_drops_value(&self, channel: u16, request: Value) -> HandlerOutcome { + fn handle_agent_drops_value(&self, channel: RouteHandle, request: Value) -> PreparedOutcome { let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "agent_drops.append requires session_id".to_string(), }; @@ -5818,20 +5859,20 @@ impl McHandler { let command_id = match command_id_from_agent_drops_request(&request) { Ok(command_id) => command_id, Err(message) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message, }; } }; let Some(raw_drop) = request.get("drop").and_then(Value::as_str) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "'drop' must be a nonempty string".to_string(), }; }; if raw_drop.trim().is_empty() { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "'drop' must be a nonempty string".to_string(), }; @@ -5839,7 +5880,7 @@ impl McHandler { let numbers = match parse_tag_range_string(raw_drop) { Ok(numbers) => numbers, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: format!("invalid drop range syntax: {error}"), }; @@ -5848,27 +5889,27 @@ impl McHandler { let _binding = match self.resolve_binding(channel, session_id) { Ok(binding) => binding, Err(BindingError::Unbound) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "route_unbound".to_string(), message: "agent_drops.append on a channel with no session binding".to_string(), }; } Err(BindingError::SessionMismatch) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), }; } }; - let store = match self.store.get() { - Some(store) => Arc::clone(store), + let store = match self.store() { + Some(store) => Arc::clone(&store), None => return store_unavailable_error(), }; let tags = match store.load_tags_for_session(session_id) { Ok(tags) => tags, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }; @@ -5891,7 +5932,7 @@ impl McHandler { drop_ids.sort(); drop_ids.dedup(); if drop_ids.is_empty() { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: format!( "ctx_reduce drop request has no valid tags: {} not found", @@ -5917,7 +5958,7 @@ impl McHandler { } respond(resp) } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, @@ -5926,24 +5967,24 @@ impl McHandler { fn management_binding( &self, - channel: u16, + channel: RouteHandle, request: &Value, operation: &str, - ) -> Result<(String, SessionBinding), HandlerOutcome> { + ) -> Result<(String, SessionBinding), PreparedOutcome> { if request.get("v").and_then(Value::as_u64) != Some(1) { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "bad_request".to_string(), message: format!("{operation} requires v=1"), }); } let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "bad_request".to_string(), message: format!("{operation} requires session_id"), }); }; if session_id.trim().is_empty() { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "bad_request".to_string(), message: format!("{operation} requires a nonempty session_id"), }); @@ -5951,13 +5992,13 @@ impl McHandler { let binding = match self.resolve_binding(channel, session_id) { Ok(binding) => binding, Err(BindingError::Unbound) => { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "route_unbound".to_string(), message: format!("{operation} on a channel with no session binding"), }); } Err(BindingError::SessionMismatch) => { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), @@ -5967,7 +6008,11 @@ impl McHandler { Ok((session_id.to_string(), binding)) } - fn handle_todo_state_set_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_todo_state_set_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let (session_id, _binding) = match self.management_binding(channel, request, "todo_state.set") { Ok(scope) => scope, @@ -5989,7 +6034,7 @@ impl McHandler { return invalid_params_error("state_json must be a JSON todo array"); }; let state_hash = sha256_hex(normalized.as_bytes()); - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -5997,33 +6042,37 @@ impl McHandler { Ok(TodoStateSetOutcome::Updated { .. }) | Ok(TodoStateSetOutcome::Noop) => { respond(json!({ "ok": true })) } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, } } - fn handle_session_flush_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_session_flush_value(&self, channel: RouteHandle, request: &Value) -> PreparedOutcome { let (session_id, _binding) = match self.management_binding(channel, request, "session.flush") { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; match store.arm_soft_refresh(&session_id) { Ok(armed) => respond(json!({ "ok": true, "armed": armed })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, } } - fn handle_session_recomp_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_session_recomp_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let (session_id, _binding) = match self.management_binding(channel, request, "session.recomp") { Ok(scope) => scope, @@ -6035,7 +6084,7 @@ impl McHandler { if command_id.is_empty() || command_id.len() > 128 { return invalid_params_error("command_id must contain 1..=128 bytes"); } - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -6048,7 +6097,7 @@ impl McHandler { } Ok(None) => {} Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6067,7 +6116,7 @@ impl McHandler { let loaded = match store.load(&session_id) { Ok(loaded) => loaded, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6076,7 +6125,7 @@ impl McHandler { let has_compartments = match store.has_compartments(&session_id) { Ok(has_compartments) => has_compartments, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6094,7 +6143,7 @@ impl McHandler { "ok": true, "disposition": row.disposition, })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, @@ -6107,13 +6156,13 @@ impl McHandler { // A transform may have committed between the status reads and the reset. // The recomp latch remains held; ask the caller to retry rather than // claiming a reset that did not use the observed cache version. - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_conflict".to_string(), message: error.to_string(), }; } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }; @@ -6143,20 +6192,24 @@ impl McHandler { "ok": true, "disposition": row.disposition, })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, } } - fn handle_session_delete_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_session_delete_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let (session_id, binding) = match self.management_binding(channel, request, "session.delete") { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -6176,20 +6229,24 @@ impl McHandler { .remove(&session_id); respond(json!({ "ok": true, "deleted_rows": deleted_rows })) } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }, } } - fn handle_session_status_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_session_status_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let (session_id, binding) = match self.management_binding(channel, request, "session.status") { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -6216,7 +6273,7 @@ impl McHandler { match store.load_session_status_snapshot(&session_id, include_compartments_after_seq) { Ok(snapshot) => snapshot, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6240,7 +6297,7 @@ impl McHandler { { Ok(snapshot) => snapshot, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6310,7 +6367,7 @@ impl McHandler { // completion without parsing the summary or issuing another operation: a retained // delivered-command record includes its coverage, row version, and current wrapup state. let m1_signal = match crate::m1_compose::m1_revision_signal_parts_for_pass( - store, + &store, &binding.project_root.to_string_lossy(), &binding.project_root.to_string_lossy(), &session_id, @@ -6320,7 +6377,7 @@ impl McHandler { ) { Ok(signal) => Some(signal), Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6467,7 +6524,7 @@ impl McHandler { fn retryable_wrapup_response( reason: RetryableWrapupReason, summary: impl Into, - ) -> HandlerOutcome { + ) -> PreparedOutcome { respond(json!({ "ok": false, "disposition": "retryable", @@ -6484,7 +6541,7 @@ impl McHandler { expected_generation: u64, expected_revert_epoch: u64, response: TerminalWrapupResponse, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let TerminalWrapupResponse { disposition, rounds, @@ -6521,7 +6578,7 @@ impl McHandler { ); } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6560,7 +6617,7 @@ impl McHandler { ); } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_commit_failed".to_string(), message: format!("could not record terminal wrapup result: {error}"), }; @@ -6586,7 +6643,7 @@ impl McHandler { respond(payload) } - fn replayed_wrapup_response(row: mc_store::WrapupCommandRow) -> HandlerOutcome { + fn replayed_wrapup_response(row: mc_store::WrapupCommandRow) -> PreparedOutcome { if row.disposition == "failed" { if let Some((reason, summary, detail)) = terminal_wrapup_failure_fields(&row.summary) { return respond(json!({ @@ -6609,7 +6666,11 @@ impl McHandler { })) } - async fn handle_session_wrapup_value(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_session_wrapup_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let deadline = Instant::now() .checked_add(self.wrapup_operation_budget()) .unwrap_or_else(Instant::now); @@ -6626,7 +6687,7 @@ impl McHandler { Some(command_id) if !command_id.is_empty() && command_id.len() <= 128 => { Some(command_id) } - _ => return HandlerOutcome::Error { + _ => return PreparedOutcome::Error { code: "bad_request".to_string(), message: "session.wrapup command_id must be a nonempty string of at most 128 bytes" @@ -6634,8 +6695,8 @@ impl McHandler { }, }, }; - let store = match self.store.get() { - Some(store) => Arc::clone(store), + let store = match self.store() { + Some(store) => Arc::clone(&store), None => return store_unavailable_error(), }; let route_project_root = binding.project_root.to_string_lossy().to_string(); @@ -6644,7 +6705,7 @@ impl McHandler { Ok(Some(project)) => project, Ok(None) => route_project_root, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_project_resolution_failed".to_string(), message: error.to_string(), }; @@ -6660,7 +6721,7 @@ impl McHandler { Ok(Some(row)) => return Self::replayed_wrapup_response(row), Ok(None) => {} Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6678,7 +6739,7 @@ impl McHandler { Some(value) => match value.as_i64() { Some(value) => usize::try_from(value.max(0)).unwrap_or(usize::MAX), None => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "session.wrapup keep must be an integer".to_string(), }; @@ -6704,7 +6765,7 @@ impl McHandler { let entry_state = match store.load(&session_id) { Ok(loaded) => loaded, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6747,7 +6808,7 @@ impl McHandler { let initial_snapshot = match store.load_historian_assembly_snapshot(&session_id) { Ok(snapshot) => snapshot, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6810,7 +6871,7 @@ impl McHandler { ); } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6871,7 +6932,7 @@ impl McHandler { let current_state = match store.load(&session_id) { Ok(loaded) => loaded, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -6903,7 +6964,7 @@ impl McHandler { let current_end = match store.max_compartment_end_ordinal(&session_id) { Ok(ordinal) => (ordinal > 0).then_some(ordinal as u64), Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -7052,7 +7113,7 @@ impl McHandler { let final_compartments = match store.load_compartments(&session_id) { Ok(compartments) => compartments, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -7145,8 +7206,12 @@ impl McHandler { } } - fn handle_authority_status_value(&self, channel: u16, request: &Value) -> HandlerOutcome { - let Some(store) = self.store.get() else { + fn handle_authority_status_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some((context_store_uuid, project, domain)) = authority_request_key(request) else { @@ -7158,9 +7223,9 @@ impl McHandler { Ok(Some(row)) => { if row.state == "MODULE" { if let Err(error) = - self.bind_authority_route(store, channel, context_store_uuid, project) + self.bind_authority_route(&store, channel, context_store_uuid, project) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_route_binding_failed".to_string(), message: error.to_string(), }; @@ -7169,15 +7234,19 @@ impl McHandler { respond(json!({ "ok": true, "authority": row })) } Ok(None) => respond(json!({ "ok": true, "authority": null })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "authority_status_failed".to_string(), message: error.to_string(), }, } } - fn handle_authority_prepare_value(&self, channel: u16, request: &Value) -> HandlerOutcome { - let Some(store) = self.store.get() else { + fn handle_authority_prepare_value( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some((context_store_uuid, project, domain)) = authority_request_key(request) else { @@ -7204,7 +7273,7 @@ impl McHandler { match store.authority_seed_checksum(context_store_uuid, project, domain) { Ok(checksum) => checksum, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_checksum_failed".to_string(), message: error.to_string(), }; @@ -7253,9 +7322,9 @@ impl McHandler { Ok(row) => { if row.state == "MODULE" { if let Err(error) = - self.bind_authority_route(store, channel, context_store_uuid, project) + self.bind_authority_route(&store, channel, context_store_uuid, project) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_route_binding_failed".to_string(), message: error.to_string(), }; @@ -7263,15 +7332,15 @@ impl McHandler { } respond(json!({ "ok": true, "authority": row })) } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "authority_prepare_failed".to_string(), message: error.to_string(), }, } } - fn handle_authority_seed_value(&self, request: &Value) -> HandlerOutcome { - let Some(store) = self.store.get() else { + fn handle_authority_seed_value(&self, request: &Value) -> PreparedOutcome { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some((context_store_uuid, project, domain)) = authority_request_key(request) else { @@ -7297,7 +7366,7 @@ impl McHandler { }; let snapshot = row.get("snapshot").unwrap_or(row); if snapshot.get("project_path").and_then(Value::as_str) != Some(project) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_seed_project_mismatch".to_string(), message: "seed snapshot project_path did not match the authority project" .to_string(), @@ -7312,7 +7381,7 @@ impl McHandler { match store.seed_authority_rows(context_store_uuid, project, domain, &seed_rows) { Ok(ids) => ids, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_seed_failed".to_string(), message: error.to_string(), }; @@ -7323,8 +7392,8 @@ impl McHandler { ) } - fn handle_authority_drain_value(&self, request: &Value, method: &str) -> HandlerOutcome { - let Some(store) = self.store.get() else { + fn handle_authority_drain_value(&self, request: &Value, method: &str) -> PreparedOutcome { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some((context_store_uuid, project, domain)) = authority_request_key(request) else { @@ -7418,22 +7487,22 @@ impl McHandler { match result { Ok(row) => respond(json!({ "ok": true, "authority": row })), Err(McStoreError::AuthorityFeedHeadAdvanced { captured, found }) => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "authority_feed_head_advanced".to_string(), message: format!( "authority_feed_head_advanced: captured {captured}, found {found}" ), } } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "authority_drain_failed".to_string(), message: error.to_string(), }, } } - fn handle_mirror_pull_value(&self, request: &Value) -> HandlerOutcome { - let Some(store) = self.store.get() else { + fn handle_mirror_pull_value(&self, request: &Value) -> PreparedOutcome { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some(domain) = request.get("domain").and_then(Value::as_str) else { @@ -7455,7 +7524,7 @@ impl McHandler { }; match page { Ok(page) => respond(json!({ "ok": true, "page": page })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "mirror_pull_failed".to_string(), message: error.to_string(), }, @@ -7486,7 +7555,7 @@ impl McHandler { fn prompt_surface_selection_from_value( &self, request: &Value, - ) -> Result { + ) -> Result { let preset = match request .get("preset") .or_else(|| request.get("prompt_surface_preset")) @@ -7571,9 +7640,9 @@ impl McHandler { fn handle_prompt_surface_manifest_value( &self, - channel: u16, + channel: RouteHandle, request: &Value, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { return invalid_params_error("manifest.get requires session_id"); }; @@ -7581,11 +7650,11 @@ impl McHandler { Ok(binding) => binding, Err(error) => { return match error { - BindingError::Unbound => HandlerOutcome::Error { + BindingError::Unbound => PreparedOutcome::Error { code: "route_unbound".to_string(), message: "manifest.get on a channel with no session binding".to_string(), }, - BindingError::SessionMismatch => HandlerOutcome::Error { + BindingError::SessionMismatch => PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), @@ -7618,13 +7687,13 @@ impl McHandler { })) } - fn handle_guidance_value(&self, channel: u16, request: &Value) -> HandlerOutcome { - let store = match self.store.get() { - Some(store) => Arc::clone(store), + fn handle_guidance_value(&self, channel: RouteHandle, request: &Value) -> PreparedOutcome { + let store = match self.store() { + Some(store) => Arc::clone(&store), None => return store_unavailable_error(), }; let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "guidance.get requires session_id".to_string(), }; @@ -7633,11 +7702,11 @@ impl McHandler { Ok(binding) => binding, Err(error) => { return match error { - BindingError::Unbound => HandlerOutcome::Error { + BindingError::Unbound => PreparedOutcome::Error { code: "route_unbound".to_string(), message: "guidance.get on a channel with no session binding".to_string(), }, - BindingError::SessionMismatch => HandlerOutcome::Error { + BindingError::SessionMismatch => PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), @@ -7651,7 +7720,7 @@ impl McHandler { Some(value) => match value.as_bool() { Some(value) => value, None => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "guidance.get tool_present must be a boolean".to_string(), }; @@ -7677,7 +7746,7 @@ impl McHandler { let expected_variant = if active { "full" } else { "no_reduce" }; if let Some(variant) = request.get("variant").and_then(Value::as_str) { if variant != expected_variant { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: format!( "guidance variant {variant:?} contradicts tool_present={tool_present}" @@ -7688,7 +7757,7 @@ impl McHandler { let date_line = match self.guidance_date_for_session(&store, session_id) { Ok(date) => date, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_write_failed".to_string(), message: error.to_string(), }; @@ -7844,7 +7913,9 @@ impl McHandler { pages.pending_transform_count, completed .iter() - .map(|completed| completed.result.len()) + .filter_map(|completed| { + completed.result.measure().ok().map(|output| output.len()) + }) .sum::(), completed.len(), pages @@ -7897,9 +7968,9 @@ impl McHandler { }) } - fn handle_status_value(&self, request: &Value) -> HandlerOutcome { - let store = match self.store.get() { - Some(store) => Arc::clone(store), + fn handle_status_value(&self, request: &Value) -> PreparedOutcome { + let store = match self.store() { + Some(store) => Arc::clone(&store), None => return store_unavailable_error(), }; let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { @@ -7919,7 +7990,7 @@ impl McHandler { "storage_versions": storage_versions_block(&store), "memory_holders": self.memory_holder_metrics(), })), - Err(e) => HandlerOutcome::Error { + Err(e) => PreparedOutcome::Error { code: "store_load_failed".to_string(), message: e.to_string(), }, @@ -7928,7 +7999,7 @@ impl McHandler { let loaded = match store.load(session_id) { Ok(loaded) => loaded, Err(e) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: e.to_string(), }; @@ -7937,7 +8008,7 @@ impl McHandler { let pass_trace = match store.load_pass_trace(session_id) { Ok(pass_trace) => pass_trace, Err(e) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: e.to_string(), }; @@ -7946,7 +8017,7 @@ impl McHandler { let side_channel_status = match store.historian_side_channel_status(session_id) { Ok(status) => status, Err(e) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: e.to_string(), }; @@ -7987,10 +8058,10 @@ impl McHandler { async fn handle_transform_dispatch( &self, - channel: u16, + channel: RouteHandle, request: Value, inbound_bytes: Option, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let ticket = TransformDispatchTicket::new(&DISPATCH_HEALTH); let outcome = if has_transform_page_fields(&request) { self.handle_transform_page_value(channel, request, TransformLane::Authority, &ticket) @@ -7999,29 +8070,29 @@ impl McHandler { self.handle_transform_value(channel, request, inbound_bytes, &ticket) .await }; - ticket.finish(matches!(outcome, HandlerOutcome::Error { .. })); + ticket.finish(matches!(outcome, PreparedOutcome::Error { .. })); outcome } async fn handle_transform_value( &self, - channel: u16, + channel: RouteHandle, request: Value, inbound_bytes: Option, ticket: &TransformDispatchTicket<'_>, - ) -> HandlerOutcome { + ) -> PreparedOutcome { self.handle_transform_unpaged_value(channel, request, false, inbound_bytes, ticket) .await } async fn handle_transform_unpaged_value( &self, - channel: u16, + channel: RouteHandle, request: Value, from_page_apply: bool, _inbound_bytes: Option, ticket: &TransformDispatchTicket<'_>, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let handler_started_at = Instant::now(); let mut delta_expand_ms = 0.0; const REQUEST_OBSERVED_KEY: &str = "request_observed_at_ms"; @@ -8034,7 +8105,7 @@ impl McHandler { let mut parsed: TransformRequest = match serde_json::from_value(request) { Ok(req) => req, Err(e) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: e.to_string(), }; @@ -8076,13 +8147,13 @@ impl McHandler { return passthrough_transform_response(&parsed); } Err(BindingError::Unbound) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "route_unbound".to_string(), message: "registered dreamer session has no bound route".to_string(), }; } Err(BindingError::SessionMismatch) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "registered dreamer session does not match the bound route" .to_string(), @@ -8090,10 +8161,10 @@ impl McHandler { } } } - let store = match self.store.get() { - Some(store) => Arc::clone(store), + let store = match self.store() { + Some(store) => Arc::clone(&store), None => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_unavailable".to_string(), message: "store not opened (no HELLO_ACK storage seam)".to_string(), }; @@ -8102,13 +8173,13 @@ impl McHandler { let binding = match self.resolve_binding(channel, &parsed.session_id) { Ok(b) => b, Err(BindingError::Unbound) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "route_unbound".to_string(), message: "transform on a channel with no session binding".to_string(), }; } Err(BindingError::SessionMismatch) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "request session_id does not match the channel's bound session" .to_string(), @@ -8174,7 +8245,7 @@ impl McHandler { None }; if !from_page_apply && self.transform_page_in_progress(&binding.session) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_transform_page_in_progress".to_string(), message: "transform is blocked until all transform pages arrive".to_string(), }; @@ -8196,7 +8267,7 @@ impl McHandler { Ok(Some(project)) => project, Ok(None) => route_project_root.clone(), Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_project_resolution_failed".to_string(), message: error.to_string(), }; @@ -8207,7 +8278,7 @@ impl McHandler { Ok(Some(project)) => project, Ok(None) => route_project_root.clone(), Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_project_resolution_failed".to_string(), message: error.to_string(), }; @@ -8223,7 +8294,7 @@ impl McHandler { &content_hash, pass_now, ) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "mural_artifact_store_failed".to_string(), message: error.to_string(), }; @@ -8239,7 +8310,7 @@ impl McHandler { parsed.mural = mural; } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "mural_artifact_store_failed".to_string(), message: error.to_string(), }; @@ -8339,7 +8410,7 @@ impl McHandler { let reject_transform = |e: crate::transform::TransformError| { let message = e.to_string(); let _ = store.trace_pass_rejected(&parsed.session_id, &message, now_ms()); - HandlerOutcome::Error { + PreparedOutcome::Error { code: "transform_failed".to_string(), message, } @@ -8636,26 +8707,30 @@ impl McHandler { } #[cfg(test)] - async fn handle_transform_for_test(&self, channel: u16, request: Value) -> HandlerOutcome { + async fn handle_transform_for_test( + &self, + route: RouteHandle, + request: Value, + ) -> PreparedOutcome { let inbound_bytes = serde_json::to_vec(&request) .map(|bytes| bytes.len()) .unwrap_or(MAX_TRANSFORM_FRAME_BYTES); - self.handle_transform_dispatch(channel, request, Some(inbound_bytes)) + self.handle_transform_dispatch(route, request, Some(inbound_bytes)) .await } #[cfg(test)] async fn handle_transform_for_test_with_body_size( &self, - channel: u16, + route: RouteHandle, request: Value, inbound_bytes: usize, - ) -> HandlerOutcome { - self.handle_transform_dispatch(channel, request, Some(inbound_bytes)) + ) -> PreparedOutcome { + self.handle_transform_dispatch(route, request, Some(inbound_bytes)) .await } - fn handle_state_sync_value(&self, channel: u16, request: Value) -> HandlerOutcome { + fn handle_state_sync_value(&self, channel: RouteHandle, request: Value) -> PreparedOutcome { const ENVELOPE_FIELDS: [&str; 5] = [ "seed_id", "seed_generation", @@ -8682,8 +8757,8 @@ impl McHandler { Ok(binding) => binding, Err(outcome) => return outcome, }; - let store = match self.store.get() { - Some(store) => Arc::clone(store), + let store = match self.store() { + Some(store) => Arc::clone(&store), None => return store_unavailable_error(), }; @@ -8698,7 +8773,7 @@ impl McHandler { Some( StateSyncSeedPhase::Collecting(_) | StateSyncSeedPhase::Applying { .. }, ) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_in_progress".to_string(), message: "a paged state-sync seed is already in progress".to_string(), }; @@ -8755,9 +8830,9 @@ impl McHandler { .filter(|completed| completed.seed_id == seed_id) { if completed.final_digest == digest { - return HandlerOutcome::Response(completed.result.clone()); + return PreparedOutcome::Response(completed.result.clone()); } - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_digest_mismatch".to_string(), message: format!( "completed seed content changed (generation={}, seq={}, total={})", @@ -8768,21 +8843,21 @@ impl McHandler { } if parsed.shadow_generation != seed_generation { self.discard_state_sync_seed(&binding.session); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_attempt_mismatch".to_string(), message: "shadow_generation must match seed_generation".to_string(), }; } if batch_total == 0 || batch_index >= batch_total { self.discard_state_sync_seed(&binding.session); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_protocol_mismatch".to_string(), message: "seed batch index/total is invalid".to_string(), }; } if seed_complete != (batch_index + 1 == batch_total) { self.discard_state_sync_seed(&binding.session); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_protocol_mismatch".to_string(), message: "seed_complete disagrees with the final batch index".to_string(), }; @@ -8815,7 +8890,7 @@ impl McHandler { || seed_complete && scalar_tail_fields < 4 { self.discard_state_sync_seed(&binding.session); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_protocol_mismatch".to_string(), message: "seed scalar tail must appear only and completely on the final batch" .to_string(), @@ -8835,14 +8910,14 @@ impl McHandler { let loaded = match store.load(&binding.session) { Ok(loaded) => loaded, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; } }; if loaded.meta.shadow_generation != seed_generation { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_generation_mismatch".to_string(), message: format!( "seed_generation {seed_generation} did not match durable generation {}", @@ -8912,14 +8987,14 @@ impl McHandler { match phase { StateSyncSeedPhase::Idle => { seeds.set_phase(&binding.session, StateSyncSeedPhase::Idle); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_not_armed".to_string(), message: "paged state-sync batches must start at index zero".to_string(), }; } applying @ StateSyncSeedPhase::Applying { .. } => { seeds.set_phase(&binding.session, applying); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_in_progress".to_string(), message: "the final state-sync seed batch is being applied".to_string(), }; @@ -8933,7 +9008,7 @@ impl McHandler { || expected_seq != parsed.expected_shadow_seq { seeds.release_phase(&awaiting); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_attempt_mismatch".to_string(), message: "seed batch does not match the active state-sync attempt" .to_string(), @@ -8946,7 +9021,7 @@ impl McHandler { .is_none_or(|bytes| bytes > seeds.max_staged_bytes) { seeds.release_phase(&awaiting); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_buffer_overflow".to_string(), message: "state-sync seed staging exceeded the handler-wide byte cap" .to_string(), @@ -8995,7 +9070,7 @@ impl McHandler { { let discarded = StateSyncSeedPhase::Collecting(pending); seeds.release_phase(&discarded); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_attempt_mismatch".to_string(), message: "seed envelope changed during collection".to_string(), }; @@ -9016,7 +9091,7 @@ impl McHandler { } else { let discarded = StateSyncSeedPhase::Collecting(pending); seeds.release_phase(&discarded); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_digest_mismatch".to_string(), message: "redriven seed batch content changed".to_string(), }; @@ -9024,7 +9099,7 @@ impl McHandler { } else if batch_index > pending.next_index { let discarded = StateSyncSeedPhase::Collecting(pending); seeds.release_phase(&discarded); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_order_mismatch".to_string(), message: "seed batches must arrive in strict index order".to_string(), }; @@ -9036,7 +9111,7 @@ impl McHandler { { let discarded = StateSyncSeedPhase::Collecting(pending); seeds.release_phase(&discarded); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "state_sync_seed_buffer_overflow".to_string(), message: "state-sync seed staging exceeded the handler-wide byte cap" @@ -9101,10 +9176,8 @@ impl McHandler { let assembled = assemble_state_sync_seed(batches, generation, expected_seq); let outcome = self.apply_state_sync_wire(&binding, &store, assembled); let completed_result = match &outcome { - HandlerOutcome::Response(bytes) => Some(bytes.clone()), - HandlerOutcome::Error { .. } - | HandlerOutcome::ErrorWithDetail { .. } - | HandlerOutcome::Streamed => None, + PreparedOutcome::Response(bytes) => Some(bytes.clone()), + PreparedOutcome::Error { .. } | PreparedOutcome::Streamed => None, }; let mut seeds = self.state_sync_seeds.lock().expect("state sync seed mutex"); let phase = { @@ -9147,7 +9220,7 @@ impl McHandler { binding: &SessionBinding, store: &McStore, mut parsed: ModuleStateSyncWire, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let note_evaluation_available = parsed.note_evaluation_available.unwrap_or(false); let user_profile_present = parsed.user_profile.is_some(); let user_profile = parsed.user_profile.take().unwrap_or_default(); @@ -9213,7 +9286,7 @@ impl McHandler { let historian_phase = match store.load(&binding.session) { Ok(loaded) => loaded.meta.historian.state, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "store_load_failed".to_string(), message: error.to_string(), }; @@ -9231,7 +9304,7 @@ impl McHandler { let authority_project = match store.authority_project_for_route(&root_path, "memories") { Ok(project) => project, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_project_resolution_failed".to_string(), message: error.to_string(), }; @@ -9392,7 +9465,7 @@ impl McHandler { })) } Err(ModuleStateSyncError::GenerationMismatch { expected, found }) => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "state_sync_generation_mismatch".to_string(), message: format!( "shadow_generation {expected} is stale; current generation is {found}" @@ -9406,12 +9479,12 @@ impl McHandler { historian_compartment_sync_busy_error(phase) } Err(ModuleStateSyncError::InvalidSeedBoundary { declared, detail }) => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "state_sync_seed_boundary_mismatch".to_string(), message: format!("seed boundary {declared:?} rejected: {detail}"), } } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "state_sync_failed".to_string(), message: error.to_string(), }, @@ -9420,11 +9493,11 @@ impl McHandler { async fn handle_transform_page_value( &self, - channel: u16, + channel: RouteHandle, request: Value, lane: TransformLane, ticket: &TransformDispatchTicket<'_>, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let present = TRANSFORM_PAGE_FIELDS .iter() .filter(|field| request.get(**field).is_some()) @@ -9536,7 +9609,7 @@ impl McHandler { && page_complete && completed.final_digest == page_digest { - return HandlerOutcome::Response(completed.result.clone()); + return PreparedOutcome::Response(completed.result.clone()); } return transform_page_error( lane, @@ -9621,10 +9694,8 @@ impl McHandler { ) .await; let completed_result = match &outcome { - HandlerOutcome::Response(bytes) => Some(bytes.clone()), - HandlerOutcome::Error { .. } - | HandlerOutcome::ErrorWithDetail { .. } - | HandlerOutcome::Streamed => None, + PreparedOutcome::Response(bytes) => Some(bytes.clone()), + PreparedOutcome::Error { .. } | PreparedOutcome::Streamed => None, }; let mut transforms = self.transform_pages.lock().expect("transform page mutex"); let phase = { @@ -9690,13 +9761,17 @@ impl McHandler { .contains(session_id) } - async fn handle_dreamer_run_task(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_dreamer_run_task( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let (ledger_session, binding) = match self.management_binding(channel, request, "dreamer.run_task") { Ok(value) => value, Err(outcome) => return outcome, }; - let Some(store) = self.store.get().cloned() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let Some(task) = request.get("task").and_then(Value::as_str) else { @@ -9722,13 +9797,13 @@ impl McHandler { let Some(project) = (match store.authority_project_for_route(&route_root, "memories") { Ok(project) => project, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_lookup_failed".to_string(), message: error.to_string(), }; } }) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_not_module".to_string(), message: "memories authority for this route is not MODULE".to_string(), }; @@ -9737,14 +9812,14 @@ impl McHandler { (match store.module_authority_for_project(&project, "memories") { Ok(authority) => authority, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_lookup_failed".to_string(), message: error.to_string(), }; } }) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_not_module".to_string(), message: "memories authority for this route is not MODULE".to_string(), }; @@ -9753,26 +9828,26 @@ impl McHandler { match store.authority_status(&context_store_uuid, &authority_project, "memories") { Ok(Some(authority)) => authority, Ok(None) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_not_module".to_string(), message: "memories authority row is missing".to_string(), }; } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_lookup_failed".to_string(), message: error.to_string(), }; } }; if authority.state != "MODULE" { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_not_module".to_string(), message: format!("memories authority is {}", authority.state), }; } if authority.generation != authority_generation { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_generation_mismatch".to_string(), message: format!( "authority generation is {}, request used {authority_generation}", @@ -9787,7 +9862,7 @@ impl McHandler { return invalid_params_error("classify payload requires prompt_body"); }; if prompt_body.len() > MAX_CLASSIFY_PROMPT_BYTES { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "payload_too_large".to_string(), message: format!("classify prompt_body exceeds {MAX_CLASSIFY_PROMPT_BYTES} bytes"), }; @@ -9867,7 +9942,7 @@ impl McHandler { .lock() .expect("dream command registry mutex"); if !inflight.insert(command_key.clone()) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "dreamer_run_failed".to_string(), message: "this command is already executing; retry replays its recorded outcome" @@ -9887,7 +9962,7 @@ impl McHandler { Ok(Some(recorded)) => return replay_dream_task_response(&recorded.response_json), Ok(None) => {} Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "dreamer_ledger_failed".to_string(), message: error.to_string(), } @@ -10025,9 +10100,9 @@ impl McHandler { // response. if matches!( &primary, - HistorianProducerError::Subc(body) if body.code == "idempotency_conflict" + primary if primary.code() == Some("idempotency_conflict") ) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "dreamer_run_failed".to_string(), message: primary.to_string(), }; @@ -10055,7 +10130,7 @@ impl McHandler { &response.to_string(), now_ms(), ); - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "dreamer_run_failed".to_string(), message: last_error, }; @@ -10095,7 +10170,7 @@ impl McHandler { // ledger row, a retry derives the same child session and can // recover the completed run instead of hitting a deletion // tombstone. - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "dreamer_ledger_failed".to_string(), message: error.to_string(), }, @@ -10104,9 +10179,9 @@ impl McHandler { async fn handle_memory_set_classification( &self, - channel: u16, + channel: RouteHandle, request: &Value, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["rows"]) else { return invalid_params_error("memory.set_classification requires arguments"); }; @@ -10121,7 +10196,7 @@ impl McHandler { Err(_) => return session_unresolved_error(), }; let route_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let authority_project = @@ -10129,20 +10204,20 @@ impl McHandler { Ok(Some((project, state))) if state == "MODULE" => project, Ok(Some(_)) => return authority_draining_error("memories"), Ok(None) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_not_module".to_string(), message: "classification requires MODULE memories authority".to_string(), }; } Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "authority_lookup_failed".to_string(), message: error.to_string(), }; } }; if authority_project != memory_project { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "facade_project_vocabulary_mismatch".to_string(), message: format!( "classification route is owned by {authority_project}, not {memory_project}" @@ -10212,7 +10287,7 @@ impl McHandler { "rejected": result.rejected.iter().map(|row| json!({ "memory_id": row.memory_id, "reason": row.reason })).collect::>(), })), Err(McStoreError::AuthorityGenerationMismatch { expected, found }) => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "authority_generation_mismatch".to_string(), message: format!("authority generation is {found}, request used {expected}"), } @@ -10221,7 +10296,7 @@ impl McHandler { authority_draining_error("memories") } Err(McStoreError::AuthorityStateMismatch { expected, found }) => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "authority_state_mismatch".to_string(), message: format!("authority state is {found}, expected {expected}"), } @@ -10229,14 +10304,18 @@ impl McHandler { Err(error) if store_error_is_authority_draining(&error) => { authority_draining_error("memories") } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "classification_apply_failed".to_string(), message: error.to_string(), }, } } - async fn handle_memory_set_mural_cue(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_memory_set_mural_cue( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["rows"]) else { return invalid_params_error("memory.set_mural_cue requires arguments"); }; @@ -10251,7 +10330,7 @@ impl McHandler { Err(_) => return session_unresolved_error(), }; let route_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let command_id = args @@ -10260,7 +10339,7 @@ impl McHandler { .filter(|id| !id.is_empty()); if let Some(command_id) = command_id { if let Some(replayed) = - replayed_memory_apply_command(store, &binding.session, "set_mural_cue", command_id) + replayed_memory_apply_command(&store, &binding.session, "set_mural_cue", command_id) { return replayed; } @@ -10339,9 +10418,9 @@ impl McHandler { async fn handle_memory_set_verification( &self, - channel: u16, + channel: RouteHandle, request: &Value, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["rows"]) else { return invalid_params_error("memory.set_verification requires arguments"); }; @@ -10356,7 +10435,7 @@ impl McHandler { Err(_) => return session_unresolved_error(), }; let route_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let command_id = args @@ -10365,7 +10444,7 @@ impl McHandler { .filter(|id| !id.is_empty()); if let Some(command_id) = command_id { if let Some(replayed) = replayed_memory_apply_command( - store, + &store, &binding.session, "set_verification", command_id, @@ -10449,7 +10528,11 @@ impl McHandler { ) } - async fn handle_memory_set_mapping(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_memory_set_mapping( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["rows"]) else { return invalid_params_error("memory.set_mapping requires arguments"); }; @@ -10464,7 +10547,7 @@ impl McHandler { Err(_) => return session_unresolved_error(), }; let route_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let command_id = args @@ -10473,7 +10556,7 @@ impl McHandler { .filter(|id| !id.is_empty()); if let Some(command_id) = command_id { if let Some(replayed) = - replayed_memory_apply_command(store, &binding.session, "set_mapping", command_id) + replayed_memory_apply_command(&store, &binding.session, "set_mapping", command_id) { return replayed; } @@ -10554,7 +10637,7 @@ impl McHandler { ) } - async fn handle_facade_value(&self, channel: u16, request: Value) -> HandlerOutcome { + async fn handle_facade_value(&self, channel: RouteHandle, request: Value) -> PreparedOutcome { let Some(name) = request.get("name").and_then(Value::as_str) else { return unrecognized_request_error(&request); }; @@ -10591,10 +10674,10 @@ impl McHandler { fn bind_facade_route_for_write( &self, - channel: u16, + channel: RouteHandle, arguments: &Map, authority_domain: &str, - ) -> Result<(), HandlerOutcome> { + ) -> Result<(), PreparedOutcome> { let Some(requested_project) = non_empty_string_arg(arguments, "memory_project") else { return Ok(()); }; @@ -10602,12 +10685,12 @@ impl McHandler { .facade_binding(channel) .map_err(|_| session_unresolved_error())?; let route_project_root = binding.project_root.to_string_lossy().to_string(); - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return Err(store_unavailable_error()); }; let authority = store .facade_authority_for_project(requested_project, authority_domain) - .map_err(|error| HandlerOutcome::Error { + .map_err(|error| PreparedOutcome::Error { code: "authority_route_lookup_failed".to_string(), message: error.to_string(), })?; @@ -10617,7 +10700,7 @@ impl McHandler { } store .bind_authority_route(&context_store_uuid, &project, &route_project_root) - .map_err(|error| HandlerOutcome::Error { + .map_err(|error| PreparedOutcome::Error { code: "authority_route_bind_failed".to_string(), message: error.to_string(), })?; @@ -10627,11 +10710,11 @@ impl McHandler { async fn resolve_facade_scope( &self, - channel: u16, + channel: RouteHandle, arguments: Option<&Map>, authority_domain: &str, bind_authority_for_write: bool, - ) -> Result { + ) -> Result { let binding = self .facade_binding(channel) .map_err(|_| session_unresolved_error())?; @@ -10657,13 +10740,13 @@ impl McHandler { Ok(Some(resolved)) => resolved.session_id, Ok(None) => return Err(session_unresolved_error()), Err(SessionResolveError::Timeout) => { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "session_resolve_timeout".to_string(), message: "session.resolve timed out after 2s".to_string(), }); } Err(error) => { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "session_resolve_failed".to_string(), message: error.to_string(), }); @@ -10679,13 +10762,13 @@ impl McHandler { } let requested_project = arguments.and_then(|arguments| non_empty_string_arg(arguments, "memory_project")); - let memory_project_path = match self.store.get() { + let memory_project_path = match self.store() { Some(store) => match store .authority_project_state_for_route(&route_project_root, authority_domain) { Ok(Some((authority_project, authority_state))) => { if requested_project.is_some_and(|requested| requested != authority_project) { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "facade_project_vocabulary_mismatch".to_string(), message: format!( "{authority_domain} facade route {route_project_root} is authority-managed as {authority_project}, but the request supplied {}", @@ -10704,7 +10787,7 @@ impl McHandler { // retryable errors: silently using the route could read or write the wrong owner. Ok(None) => route_project_root.clone(), Err(error) => { - return Err(HandlerOutcome::Error { + return Err(PreparedOutcome::Error { code: "authority_project_resolution_failed".to_string(), message: error.to_string(), }); @@ -10720,7 +10803,11 @@ impl McHandler { }) } - async fn handle_ctx_reduce_facade(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_ctx_reduce_facade( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["drop"]) else { return invalid_params_error("ctx_reduce arguments must be an object"); }; @@ -10742,7 +10829,7 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -10824,7 +10911,11 @@ impl McHandler { mcp_text_result(format!("Queued: {}.", details.join("; ")), false) } - async fn handle_ctx_memory_facade(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_ctx_memory_facade( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["action"]) else { return invalid_params_error("ctx_memory arguments must be an object"); }; @@ -10849,7 +10940,7 @@ impl McHandler { if !facade_scope.memory_enabled { return tool_error_result("Error: memory is disabled for this project.".to_string()); } - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -11051,7 +11142,7 @@ impl McHandler { } "get" => { let ids = memory_ids(args, "get"); - match memory_tool::get_memories(store, memory_project, &ids) { + match memory_tool::get_memories(&store, memory_project, &ids) { Ok(memories) => { let by_id = memories .into_iter() @@ -11079,7 +11170,11 @@ impl McHandler { } } - async fn handle_ctx_search_facade(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_ctx_search_facade( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["query"]) else { return invalid_params_error("ctx_search arguments must be an object"); }; @@ -11098,14 +11193,14 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; let memory_project = facade_scope.memory_project_path.as_str(); let conversation_key = facade_scope.conversation_key.as_str(); match memory_tool::search_memories_and_compartments_for_session( - store, + &store, memory_project, conversation_key, query, @@ -11139,7 +11234,11 @@ impl McHandler { } } - async fn handle_ctx_expand_facade(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_ctx_expand_facade( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["message", "start"]) else { return invalid_params_error("ctx_expand arguments must be an object"); }; @@ -11151,7 +11250,7 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -11254,7 +11353,11 @@ impl McHandler { ) } - fn handle_note_evaluation_register(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_register( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11279,7 +11382,7 @@ impl McHandler { Err(outcome) => return outcome, }; if protocol_version != NOTE_EVALUATOR_PROTOCOL_VERSION { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "protocol_unsupported".to_string(), message: format!( "evaluator protocol {protocol_version} is unsupported; this module accepts {NOTE_EVALUATOR_PROTOCOL_VERSION}" @@ -11319,7 +11422,7 @@ impl McHandler { .expect("note evaluator registrations mutex"); let entries = registrations.entry(project).or_default(); entries.retain(|entry| { - !(entry.evaluator_instance == evaluator_instance && entry.channel == channel) + !(entry.evaluator_instance == evaluator_instance && entry.route == channel) }); // `evaluator_instance` is caller-chosen, so without a cap a single // bound channel could retain unbounded live entries for a full lease @@ -11333,7 +11436,7 @@ impl McHandler { token: token.clone(), registration_generation, evaluator_instance, - channel, + route: channel, policy_version, capacity, retina_handoff, @@ -11352,7 +11455,11 @@ impl McHandler { })) } - fn handle_note_evaluation_heartbeat(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_heartbeat( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11394,7 +11501,7 @@ impl McHandler { entry.token == identity.token && entry.registration_generation == identity.registration_generation && entry.evaluator_instance == identity.evaluator_instance - && entry.channel == channel + && entry.route == channel && entry.expires_at > now }) }) else { @@ -11420,7 +11527,11 @@ impl McHandler { })) } - fn handle_note_evaluation_unregister(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_unregister( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11450,7 +11561,7 @@ impl McHandler { !(entry.token == identity.token && entry.registration_generation == identity.registration_generation && entry.evaluator_instance == identity.evaluator_instance - && entry.channel == channel) + && entry.route == channel) }); if entries.is_empty() { registrations.remove(&project); @@ -11459,7 +11570,11 @@ impl McHandler { respond(json!({ "ok": true })) } - fn handle_note_evaluation_next(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_next( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11500,7 +11615,7 @@ impl McHandler { Err(outcome) => return outcome, }; if wait_ms != 0 { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "positive_wait_unsupported".to_string(), message: "protocol v2.0 accepts only wait_ms=0".to_string(), }; @@ -11530,7 +11645,7 @@ impl McHandler { // the store and leaves no replayable acquisition decision behind. return respond(json!({ "result": "no_work", "wake_owned": true })); } - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; // Fair-cycle ownership (KTD2/KTD3): hold only this slot's lock across @@ -11629,14 +11744,18 @@ impl McHandler { } note_evaluation_acquire_response(outcome) } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "note_store_failed".to_string(), message: error.to_string(), }, } } - fn handle_note_evaluation_renew(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_renew( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11657,7 +11776,7 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; match store.renew_note_evaluation_claim( @@ -11681,14 +11800,18 @@ impl McHandler { "result": kind, "response": note_evaluation_response_value(response), })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "note_store_failed".to_string(), message: error.to_string(), }, } } - fn handle_note_evaluation_complete(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_complete( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11722,7 +11845,7 @@ impl McHandler { Ok(scope) => scope, Err(handler_outcome) => return handler_outcome, }; - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let project_for_apply = project.clone(); @@ -11752,14 +11875,18 @@ impl McHandler { "response": note_evaluation_response_value(Some(response_json)), })), Ok(NoteEvalCompleteOutcome::Conflict { kind }) => respond(json!({ "result": kind })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "note_store_failed".to_string(), message: error.to_string(), }, } } - fn handle_note_evaluation_abandon(&self, channel: u16, request: &Value) -> HandlerOutcome { + fn handle_note_evaluation_abandon( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let body = match note_evaluation_body( request, &[ @@ -11780,7 +11907,7 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; match store.abandon_note_evaluation_claim( @@ -11796,7 +11923,7 @@ impl McHandler { respond(json!({ "result": "unknown_claim" })) } Ok(NoteEvalAbandonOutcome::Invalid) => respond(json!({ "result": "invalid" })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "note_store_failed".to_string(), message: error.to_string(), }, @@ -11805,10 +11932,10 @@ impl McHandler { fn note_evaluation_claim_scope<'a>( &self, - channel: u16, + channel: RouteHandle, body: &'a Map, now: i64, - ) -> Result<(NoteEvaluationIdentity<'a>, i64, &'a str, String), HandlerOutcome> { + ) -> Result<(NoteEvaluationIdentity<'a>, i64, &'a str, String), PreparedOutcome> { let identity = note_evaluation_identity_fields(body)?; let evaluator_slot = note_evaluation_i64_field(body, "evaluator_slot")?; let claim_id = note_evaluation_id_field(body, "claim_id")?; @@ -11831,12 +11958,12 @@ impl McHandler { async fn handle_note_delivery_value( &self, - channel: u16, + channel: RouteHandle, request: &Value, ack: bool, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let Some(session_id) = request.get("session_id").and_then(Value::as_str) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "transform delivery acknowledgement requires session_id".to_string(), }; @@ -11846,7 +11973,7 @@ impl McHandler { .and_then(Value::as_str) .or_else(|| request.get("pass_id").and_then(Value::as_str)); let Some(pass_id) = pass_id.filter(|id| !id.trim().is_empty()) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "bad_request".to_string(), message: "transform delivery acknowledgement requires transform_pass_id" .to_string(), @@ -11860,13 +11987,13 @@ impl McHandler { Err(outcome) => return outcome, }; if scope.conversation_key != session_id { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "session_mismatch".to_string(), message: "delivery acknowledgement session_id does not match the channel binding" .to_string(), }; } - let Some(store) = self.store.get() else { + let Some(store) = self.store() else { return store_unavailable_error(); }; let result = if ack { @@ -11886,14 +12013,18 @@ impl McHandler { }; match result { Ok(changed) => respond(json!({ "ok": true, "updated": changed })), - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: "note_store_failed".to_string(), message: error.to_string(), }, } } - async fn handle_ctx_note_facade(&self, channel: u16, request: &Value) -> HandlerOutcome { + async fn handle_ctx_note_facade( + &self, + channel: RouteHandle, + request: &Value, + ) -> PreparedOutcome { let Some(args) = facade_arguments(request, &["action", "content"]) else { return invalid_params_error("ctx_note arguments must be an object"); }; @@ -11917,7 +12048,7 @@ impl McHandler { Ok(scope) => scope, Err(outcome) => return outcome, }; - let store = match self.store.get() { + let store = match self.store() { Some(store) => store, None => return store_unavailable_error(), }; @@ -11962,7 +12093,7 @@ impl McHandler { .filter(|value| !value.is_empty()); if condition.is_some() && !self.has_live_note_evaluator(project, now) { return refuse_conditioned_note_without_evaluator( - store, + &store, session, action, command_id.as_deref(), @@ -12173,7 +12304,7 @@ impl McHandler { .is_some_and(|value| current.surface_condition.as_deref() != Some(value)); if condition_changed && !self.has_live_note_evaluator(project, now) { return refuse_conditioned_note_without_evaluator( - store, + &store, session, action, command_id.as_deref(), @@ -12263,7 +12394,11 @@ impl McHandler { impl Drop for McHandler { fn drop(&mut self) { - self.store_open.cancel(); + if let Ok(admission) = self.task_admission_open.get_mut() { + *admission = false; + } + self.tasks.close(); + self.cancel.cancel(); } } @@ -12273,63 +12408,212 @@ impl Default for McHandler { } } -#[async_trait] -impl ModuleHandler for McHandler { - /// The storage seam: HELLO_ACK carries the resolved descriptor (or none in - /// standalone dev). Start the store open ONCE here — never at construction, because - /// the path isn't known until the ACK lands. Opening runs off the request lane so a - /// predecessor's live single-writer lease cannot block transform dispatch. - async fn on_hello_ack(&self, ack: &ModuleHelloAckBody) { - self.begin_store_open(resolve_descriptor(ack.storage.as_ref())); +impl CompositeComponent for McHandler { + fn manifest(&self) -> ManifestSnapshot { + manifest(DEFAULT_MODULE_ID) } - /// Return an atomics-only liveness snapshot. The SDK invokes this on its separate - /// channel-0 health task, so neither the store nor a handler lock is touched here. - async fn health(&self) -> HealthReport { - let now = now_ms().max(0) as u64; - self.store_open - .waiting_report(now) - .unwrap_or_else(|| DISPATCH_HEALTH.report(now)) + fn resources(&self) -> ResourceDeclaration { + ResourceDeclaration::default() } - /// Record the route's {project_root, session} so the transform path can resolve the - /// project from the daemon-controlled channel (never a per-pass request field). Accept - /// every route — project resolution, not authorization, is the concern here. - async fn on_bind(&self, req: &RouteBindRequest) -> subc_client_rs::BindDecision { - let config = self.effective_config(&req.identity.project_root); + async fn bind(&self, route: RouteHandle, identity: RouteIdentity) -> BindOutcome { + let config = self.effective_config(&identity.project_root); self.bind_route( - req.handle.channel, + route, SessionBinding { - project_root: req.identity.project_root.clone(), - harness: req.identity.harness.clone(), - session: req.identity.session.clone(), + project_root: identity.project_root, + harness: identity.harness, + session: identity.session, model_key: None, config, - // Older callers may omit the per-pass budget. Keep a safe fallback on the - // route, while authority requests carry the harness-resolved value. history_budget_tokens: memory_render::DEFAULT_HISTORY_BUDGET_TOKENS, }, ); - subc_client_rs::BindDecision::accept() + BindOutcome::Accept + } + + async fn handle(&self, ctx: RequestCtx) -> RequestOutcome { + if let Err(outcome) = enforce_request_byte_cap(ctx.body.as_slice()) { + return settle_prepared(&ctx, outcome).await; + } + let request = serde_json::from_slice::(ctx.body.as_slice()).unwrap_or(Value::Null); + let inbound_bytes = ctx.body.len(); + let outcome = self + .dispatch_value_with_inbound_bytes(ctx.route, request, Some(inbound_bytes)) + .await; + settle_prepared(&ctx, outcome).await } - /// Drop the route's binding on teardown so a reused channel can't resolve a stale - /// project and the map doesn't leak. - async fn on_route_gone(&self, handle: &RouteHandle) { - self.unbind_route(handle.channel); + async fn route_gone(&self, route: RouteHandle) { + self.unbind_route(route); } - async fn handle(&self, ctx: RequestCtx, body: Vec) -> HandlerOutcome { - if let Err(outcome) = enforce_request_byte_cap(&body) { - return outcome; + async fn health(&self) -> HealthReport { + let now = now_ms().max(0) as u64; + self.store_open + .waiting_report(now) + .unwrap_or_else(|| DISPATCH_HEALTH.report(now)) + } + + async fn shutdown(&self) -> Result<(), ShutdownError> { + { + let mut admission = self + .task_admission_open + .lock() + .expect("module task admission mutex"); + *admission = false; + self.tasks.close(); } - let request = serde_json::from_slice::(&body).unwrap_or(Value::Null); - self.dispatch_value_with_inbound_bytes( - ctx.route_handle().channel, - request, - Some(body.len()), - ) - .await + self.cancel.cancel(); + self.tasks.wait().await; + + self.bindings.lock().expect("bindings mutex").clear(); + self.transform_route_channels + .lock() + .expect("transform route channels mutex") + .clear(); + self.note_evaluator_registrations + .lock() + .expect("note evaluator registrations mutex") + .clear(); + self.note_evaluation_capabilities + .lock() + .expect("note evaluation capability mutex") + .clear(); + self.transform_session_roots + .lock() + .expect("transform session roots mutex") + .clear(); + *self + .transform_snapshots + .lock() + .expect("transform snapshots mutex") = + TransformSnapshotCache::new(TRANSFORM_SNAPSHOT_BUDGET_BYTES); + *self + .serialized_outputs + .lock() + .expect("serialized output cache mutex") = SerializedOutputCache::default(); + *self + .native_attachments + .lock() + .expect("native attachment cache mutex") = NativeAttachmentCache::default(); + *self.projections.lock().expect("projection cache mutex") = ProjectionCache::default(); + *self + .boundary_tokens + .lock() + .expect("boundary token cache mutex") = + BoundaryTokenCache::new(BOUNDARY_TOKEN_CACHE_BUDGET_BYTES); + *self.state_sync_seeds.lock().expect("state sync seed mutex") = + StateSyncSeedCoordinator::default(); + *self.transform_pages.lock().expect("transform page mutex") = + TransformPageCoordinator::default(); + *self.state_imports.lock().expect("state import mutex") = StateImportCoordinator::default(); + self.scheduler_observations + .lock() + .expect("scheduler observations mutex") + .clear(); + self.guidance_dates + .lock() + .expect("guidance dates mutex") + .clear(); + self.prompt_surface_epochs + .lock() + .expect("prompt surface epoch mutex") + .clear(); + *self.store.lock().expect("store slot mutex") = None; + Ok(()) + } +} + +impl PrimaryComponent for McHandler { + async fn initialize(&self, init: HostInit) -> Result<(), InitError> { + let descriptor = match init.storage { + Some(storage) => serde_json::from_value(storage) + .map_err(|_| InitError("invalid Magic Context storage descriptor".to_owned()))?, + None => dev_descriptor(), + }; + self.begin_store_open(descriptor) + } +} + +enum PreparedSettlement { + Response(W), + Error { code: String, message: String }, + Streamed, +} + +async fn settle_prepared_with( + outcome: PreparedOutcome, + mut cancelled: Cancelled, + reserve: Reserve, +) -> PreparedSettlement +where + W: std::io::Write, + Reserve: FnOnce(usize) -> Reserved, + Reserved: std::future::Future>, + Cancelled: FnMut() -> bool, +{ + let PreparedOutcome::Response(output) = outcome else { + return match outcome { + PreparedOutcome::Error { code, message } => PreparedSettlement::Error { code, message }, + PreparedOutcome::Streamed => PreparedSettlement::Streamed, + PreparedOutcome::Response(_) => unreachable!(), + }; + }; + let measured = match output.measure() { + Ok(measured) => measured, + Err(error) => { + return PreparedSettlement::Error { + code: "encode_failed".to_owned(), + message: error.to_string(), + }; + } + }; + if cancelled() { + return PreparedSettlement::Error { + code: "request_cancelled".to_owned(), + message: "request was cancelled before output reservation".to_owned(), + }; + } + let mut body = match reserve(measured.len()).await { + Ok(body) => body, + Err(()) => { + return PreparedSettlement::Error { + code: "output_unavailable".to_owned(), + message: "response output reservation is unavailable".to_owned(), + }; + } + }; + if cancelled() { + return PreparedSettlement::Error { + code: "request_cancelled".to_owned(), + message: "request was cancelled before output encoding".to_owned(), + }; + } + if let Err(error) = measured.write_to(&mut body) { + return PreparedSettlement::Error { + code: "encode_failed".to_owned(), + message: error.to_string(), + }; + } + PreparedSettlement::Response(body) +} + +async fn settle_prepared(ctx: &RequestCtx, outcome: PreparedOutcome) -> RequestOutcome { + match settle_prepared_with( + outcome, + || ctx.is_cancelled(), + |len| async move { ctx.reserve_output(len).await.map_err(|_| ()) }, + ) + .await + { + PreparedSettlement::Response(body) => RequestOutcome::Response { + body, + binary: false, + }, + PreparedSettlement::Error { code, message } => RequestOutcome::Error { code, message }, + PreparedSettlement::Streamed => RequestOutcome::Streamed, } } @@ -12338,57 +12622,22 @@ impl McHandler { /// routing arms are unit-testable (`RequestCtx` cannot be constructed /// outside the transport). #[cfg(test)] - async fn dispatch_value(&self, channel: u16, request: Value) -> HandlerOutcome { - self.dispatch_value_with_inbound_bytes(channel, request, None) - .await - } - - /// Cross-crate test seam: dispatch one parsed request, bypassing the transport. `RequestCtx` is transport-owned, so the authenticated round-trip test cannot call `handle()`. commentlint: allow(JUDGE) - #[doc(hidden)] - pub async fn dispatch_value_for_integration( - &self, - channel: u16, - request: Value, - ) -> HandlerOutcome { - self.dispatch_value_with_inbound_bytes(channel, request, None) + async fn dispatch_value(&self, route: RouteHandle, request: Value) -> PreparedOutcome { + self.dispatch_value_with_inbound_bytes(route, request, None) .await } - /// Cross-crate test seam mirroring `on_bind`: `RouteBindRequest` needs a `RouteHandle`, whose constructor is private to the transport crate. commentlint: allow(JUDGE) - #[doc(hidden)] - pub fn bind_route_for_integration( - &self, - channel: u16, - project_root: &Path, - harness: &str, - session: &str, - ) { - let config = self.effective_config(project_root); - self.bind_route( - channel, - SessionBinding { - project_root: project_root.to_path_buf(), - harness: harness.to_owned(), - session: session.to_owned(), - model_key: None, - config, - history_budget_tokens: memory_render::DEFAULT_HISTORY_BUDGET_TOKENS, - }, - ); - } - - /// Cross-crate test seam mirroring the `on_hello_ack` store open: sharing one already-open handle lets the test seed and verify rows without fighting the store's single-writer lease. commentlint: allow(JUDGE) - #[doc(hidden)] - pub fn install_store_for_integration(&self, store: Arc) { - let _ = self.store.set(store); + #[cfg(test)] + fn install_store_for_test(&self, store: Arc) { + *self.store.lock().expect("store slot mutex") = Some(store); } async fn dispatch_value_with_inbound_bytes( &self, - channel: u16, + channel: RouteHandle, request: Value, inbound_bytes: Option, - ) -> HandlerOutcome { + ) -> PreparedOutcome { let method = request .get("method") .and_then(Value::as_str) @@ -12480,9 +12729,9 @@ fn transform_page_error( lane: TransformLane, suffix: &str, message: impl Into, -) -> HandlerOutcome { +) -> PreparedOutcome { let _ = lane; - HandlerOutcome::Error { + PreparedOutcome::Error { code: format!("authority_transform_page_{suffix}"), message: message.into(), } @@ -12496,10 +12745,10 @@ fn transform_page_error( /// mistake is diagnosable from the code alone. /// - Anything else names the discriminator fields we looked for and the /// top-level keys we actually got. -fn unrecognized_request_error(request: &Value) -> HandlerOutcome { +fn unrecognized_request_error(request: &Value) -> PreparedOutcome { let has_mcp_shape = request.get("name").is_some() && request.get("arguments").is_some(); if has_mcp_shape { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "facade_envelope_not_supported".to_string(), message: "MCP tools/call envelope ({name, arguments}) names a tool this module \ does not route on the facade; other module commands use flat bodies \ @@ -12514,7 +12763,7 @@ fn unrecognized_request_error(request: &Value) -> HandlerOutcome { } None => format!("non-object JSON ({})", json_type_name(request)), }; - HandlerOutcome::Error { + PreparedOutcome::Error { code: "unrecognized_request_shape".to_string(), message: format!( "no `method` or `kind` field matched a known request; got top-level keys: [{got_keys}]" @@ -12543,15 +12792,15 @@ fn now_ms() -> i64 { .unwrap_or(0) } -fn unknown_serializer_profile_error() -> HandlerOutcome { - HandlerOutcome::Error { +fn unknown_serializer_profile_error() -> PreparedOutcome { + PreparedOutcome::Error { code: "unknown_serializer_profile".to_string(), message: "missing or unknown serializer_profile".to_string(), } } -fn serve_native_unsupported_profile_error(profile: &str) -> HandlerOutcome { - HandlerOutcome::Error { +fn serve_native_unsupported_profile_error(profile: &str) -> PreparedOutcome { + PreparedOutcome::Error { code: "serve_native_unsupported_profile".to_string(), message: format!("serve_native requires serializer_profile opencode-aisdk, got {profile}"), } @@ -13277,14 +13526,14 @@ fn finalize_native_messages_response( ); } -fn state_import_validation_error(error: StateImportValidationError) -> HandlerOutcome { - HandlerOutcome::Error { +fn state_import_validation_error(error: StateImportValidationError) -> PreparedOutcome { + PreparedOutcome::Error { code: error.code().to_string(), message: error.to_string(), } } -fn passthrough_transform_response(request: &TransformRequest) -> HandlerOutcome { +fn passthrough_transform_response(request: &TransformRequest) -> PreparedOutcome { let mut response = transform::TransformResponse::passthrough( request .messages @@ -13297,7 +13546,7 @@ fn passthrough_transform_response(request: &TransformRequest) -> HandlerOutcome respond_transform(request, response) } -fn need_full_sync_response(request: &TransformRequest) -> HandlerOutcome { +fn need_full_sync_response(request: &TransformRequest) -> PreparedOutcome { respond_transform( request, transform::TransformResponse::need_full_sync(request.full_array_fingerprint.clone()), @@ -13327,15 +13576,15 @@ fn length_capped_or_invalid( validate_classify_manifest(&result.text, expected_ids) } -fn replay_dream_task_response(response_json: &str) -> HandlerOutcome { +fn replay_dream_task_response(response_json: &str) -> PreparedOutcome { let Ok(response) = serde_json::from_str::(response_json) else { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "dreamer_ledger_corrupt".to_string(), message: "recorded dreamer response is not valid JSON".to_string(), }; }; if response.get("ok").and_then(Value::as_bool) == Some(false) { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: response .get("code") .and_then(Value::as_str) @@ -13489,10 +13738,10 @@ fn apply_drive_fault(response: &mut transform::TransformResponse, fault: DriveFa fn respond_transform( request: &TransformRequest, mut response: transform::TransformResponse, -) -> HandlerOutcome { +) -> PreparedOutcome { // drive-fault: corrupt the response before it is serialized (see the SAFETY note // above the fault helpers). No-op unless the feature is compiled in AND MC_DRIVE_FAULT - // selects an arm; must run before ck_messages is taken for the streaming placeholder. + // selects an arm; must run before ck_messages is moved into exact prepared segments. // // The fault fires at most N times (MC_DRIVE_FAULT_COUNT, default 1) then self-disarms // permanently via a fetch_update claim on DRIVE_FAULT_REMAINING. This is critical for the @@ -13514,7 +13763,7 @@ fn respond_transform( } } if response.status == transform::TransformStatus::Ok && request.tail_delta.is_some() { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "transform_delta_unexpanded".to_string(), message: "successful transform response retained an unexpanded tail_delta".to_string(), }; @@ -13524,7 +13773,7 @@ fn respond_transform( && response.native_messages.is_none() && response.native_messages_delta.is_none() { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "transform_native_response_omitted".to_string(), message: "successful serve_native response omitted native content".to_string(), }; @@ -13536,7 +13785,7 @@ fn respond_transform( let mut value = match serde_json::to_value(response) { Ok(value) => value, Err(error) => { - return HandlerOutcome::Error { + return PreparedOutcome::Error { code: "encode_failed".to_string(), message: error.to_string(), }; @@ -13548,71 +13797,46 @@ fn respond_transform( .expect("transform responses serialize as objects") .insert("ck_messages".to_string(), Value::Null); } - let encoded = match serde_json::to_vec(&value) { - Ok(encoded) => encoded, - Err(error) => { - return HandlerOutcome::Error { - code: "encode_failed".to_string(), - message: error.to_string(), + let response_meta_encode_ms = response_encode_started_at.elapsed().as_secs_f64() * 1_000.0; + let output = match messages { + None => PreparedOutput::json(value), + Some(messages) => { + let response_size_account_started_at = Instant::now(); + let segments = messages + .into_iter() + .map(PreparedSegment::served) + .collect::>(); + let response_size_account_ms = + response_size_account_started_at.elapsed().as_secs_f64() * 1_000.0; + let output = match PreparedOutput::transform_segments(value, segments) { + Ok(output) => output, + Err(error) => { + return PreparedOutcome::Error { + code: "encode_failed".to_string(), + message: error.to_string(), + }; + } }; + emit_pass_timing( + session_id, + pass_timings.as_ref(), + response_encode_started_at, + response_meta_encode_ms, + response_size_account_ms, + 0.0, + ); + return PreparedOutcome::Response(output); } }; - let response_meta_encode_ms = response_encode_started_at.elapsed().as_secs_f64() * 1_000.0; - let Some(messages) = messages else { - let outcome = HandlerOutcome::Response(encoded); - emit_pass_timing( - session_id, - pass_timings.as_ref(), - response_encode_started_at, - response_meta_encode_ms, - 0.0, - 0.0, - ); - return outcome; - }; - - const PLACEHOLDER: &[u8] = br#""ck_messages":null"#; - let Some(start) = encoded - .windows(PLACEHOLDER.len()) - .position(|window| window == PLACEHOLDER) - else { - return HandlerOutcome::Error { - code: "encode_failed".to_string(), - message: "transform response lost ck_messages placeholder".to_string(), - }; - }; - let null_start = start + PLACEHOLDER.len() - 4; - let response_size_account_started_at = Instant::now(); - let retained_bytes = messages - .iter() - .map(|message| message.canonical_bytes().len()) - .sum::(); - let response_size_account_ms = - response_size_account_started_at.elapsed().as_secs_f64() * 1_000.0; - let response_splice_started_at = Instant::now(); - let mut output = - Vec::with_capacity(encoded.len() + retained_bytes + messages.len().saturating_sub(1) + 2); - output.extend_from_slice(&encoded[..null_start]); - output.push(b'['); - for (index, message) in messages.iter().enumerate() { - if index > 0 { - output.push(b','); - } - output.extend_from_slice(message.canonical_bytes()); - } - output.push(b']'); - output.extend_from_slice(&encoded[null_start + 4..]); - let response_splice_ms = response_splice_started_at.elapsed().as_secs_f64() * 1_000.0; - let outcome = HandlerOutcome::Response(output); emit_pass_timing( session_id, pass_timings.as_ref(), response_encode_started_at, response_meta_encode_ms, - response_size_account_ms, - response_splice_ms, + 0.0, + 0.0, ); - outcome + PreparedOutcome::Response(output) } fn emit_pass_timing( @@ -13639,14 +13863,8 @@ fn emit_pass_timing( } } -fn respond(value: Value) -> HandlerOutcome { - match serde_json::to_vec(&value) { - Ok(bytes) => HandlerOutcome::Response(bytes), - Err(e) => HandlerOutcome::Error { - code: "encode_failed".to_string(), - message: e.to_string(), - }, - } +fn respond(value: Value) -> PreparedOutcome { + PreparedOutcome::Response(PreparedOutput::json(value)) } fn guidance_bytes_for(text: &str, date_line: &str) -> String { @@ -13996,26 +14214,26 @@ fn sha256_hex(bytes: &[u8]) -> String { format!("{digest:x}") } -fn mcp_text_result(text: String, is_error: bool) -> HandlerOutcome { +fn mcp_text_result(text: String, is_error: bool) -> PreparedOutcome { respond(json!({ "content": [{ "type": "text", "text": text }], "isError": is_error, })) } -fn tool_error_result(message: impl Into) -> HandlerOutcome { +fn tool_error_result(message: impl Into) -> PreparedOutcome { mcp_text_result(message.into(), true) } -fn session_unresolved_error() -> HandlerOutcome { - HandlerOutcome::Error { +fn session_unresolved_error() -> PreparedOutcome { + PreparedOutcome::Error { code: "session_unresolved".to_string(), message: SESSION_UNRESOLVED_MESSAGE.to_string(), } } -fn authority_draining_error(domain: &str) -> HandlerOutcome { - HandlerOutcome::Error { +fn authority_draining_error(domain: &str) -> PreparedOutcome { + PreparedOutcome::Error { code: "authority_draining".to_string(), message: format!("{domain} authority is draining; retry after the ownership transition"), } @@ -14035,36 +14253,36 @@ fn authority_request_key(request: &Value) -> Option<(&str, &str, &str)> { Some((uuid, project, domain)) } -fn invalid_params_error(message: impl Into) -> HandlerOutcome { - HandlerOutcome::Error { +fn invalid_params_error(message: impl Into) -> PreparedOutcome { + PreparedOutcome::Error { code: "invalid_params".to_string(), message: message.into(), } } -fn store_unavailable_error() -> HandlerOutcome { - HandlerOutcome::Error { +fn store_unavailable_error() -> PreparedOutcome { + PreparedOutcome::Error { code: "store_unavailable".to_string(), message: "store not opened (no HELLO_ACK storage seam yet)".to_string(), } } -fn note_evaluation_protocol_retired() -> HandlerOutcome { - HandlerOutcome::Error { +fn note_evaluation_protocol_retired() -> PreparedOutcome { + PreparedOutcome::Error { code: "protocol_retired".to_string(), message: "note.evaluate verdict writes are retired; use the note.evaluation.* evaluator protocol v2".to_string(), } } -fn note_evaluation_bad_request(message: impl Into) -> HandlerOutcome { - HandlerOutcome::Error { +fn note_evaluation_bad_request(message: impl Into) -> PreparedOutcome { + PreparedOutcome::Error { code: "bad_request".to_string(), message: message.into(), } } -fn note_evaluation_registration_unknown() -> HandlerOutcome { - HandlerOutcome::Error { +fn note_evaluation_registration_unknown() -> PreparedOutcome { + PreparedOutcome::Error { code: "registration_unknown".to_string(), message: "no live evaluator registration matches this token, generation, instance, and route" @@ -14078,7 +14296,7 @@ fn note_evaluation_registration_unknown() -> HandlerOutcome { fn note_evaluation_body<'a>( request: &'a Value, allowed: &[&str], -) -> Result<&'a Map, HandlerOutcome> { +) -> Result<&'a Map, PreparedOutcome> { let Some(body) = request.as_object() else { return Err(note_evaluation_bad_request( "request body must be a JSON object", @@ -14105,7 +14323,7 @@ struct NoteEvaluationIdentity<'a> { fn note_evaluation_identity_fields( body: &Map, -) -> Result, HandlerOutcome> { +) -> Result, PreparedOutcome> { Ok(NoteEvaluationIdentity { token: note_evaluation_id_field(body, "token")?, registration_generation: note_evaluation_i64_field(body, "registration_generation")?, @@ -14116,7 +14334,7 @@ fn note_evaluation_identity_fields( fn note_evaluation_id_field<'a>( body: &'a Map, key: &str, -) -> Result<&'a str, HandlerOutcome> { +) -> Result<&'a str, PreparedOutcome> { match body.get(key).and_then(Value::as_str) { Some(value) if !value.is_empty() && value.len() <= NOTE_EVALUATOR_ID_MAX_BYTES => Ok(value), _ => Err(note_evaluation_bad_request(format!( @@ -14125,7 +14343,7 @@ fn note_evaluation_id_field<'a>( } } -fn note_evaluation_i64_field(body: &Map, key: &str) -> Result { +fn note_evaluation_i64_field(body: &Map, key: &str) -> Result { body.get(key) .and_then(Value::as_i64) .ok_or_else(|| note_evaluation_bad_request(format!("'{key}' must be an integer"))) @@ -14134,7 +14352,7 @@ fn note_evaluation_i64_field(body: &Map, key: &str) -> Result, key: &str, -) -> Result { +) -> Result { body.get(key) .and_then(Value::as_bool) .ok_or_else(|| note_evaluation_bad_request(format!("'{key}' must be a boolean"))) @@ -14143,7 +14361,7 @@ fn note_evaluation_bool_field( fn note_evaluation_opt_bool_field( body: &Map, key: &str, -) -> Result, HandlerOutcome> { +) -> Result, PreparedOutcome> { match body.get(key) { None => Ok(None), Some(Value::Bool(value)) => Ok(Some(*value)), @@ -14180,7 +14398,7 @@ fn note_evaluation_response_value(response: Option) -> Value { } } -fn note_evaluation_acquire_response(outcome: NoteEvalAcquireOutcome) -> HandlerOutcome { +fn note_evaluation_acquire_response(outcome: NoteEvalAcquireOutcome) -> PreparedOutcome { match outcome { NoteEvalAcquireOutcome::Claim { claim, @@ -14243,7 +14461,7 @@ fn note_evaluation_acquire_response(outcome: NoteEvalAcquireOutcome) -> HandlerO /// cannot pass because the pairing is matched exactly. fn parse_note_evaluation_wire_outcome( value: &Value, -) -> Result<(String, SmartNoteEvaluationOutcome), HandlerOutcome> { +) -> Result<(String, SmartNoteEvaluationOutcome), PreparedOutcome> { let Some(outcome) = value.as_object() else { return Err(note_evaluation_bad_request("'outcome' must be an object")); }; @@ -14304,7 +14522,7 @@ fn parse_note_evaluation_wire_outcome( fn parse_note_evaluation_wire_artifact( value: &Value, -) -> Result { +) -> Result { let Some(artifact) = value.as_object() else { return Err(note_evaluation_bad_request("'artifact' must be an object")); }; @@ -14501,7 +14719,7 @@ impl RequestMethodProbe { /// legitimately carry a session's full message array (multi-MiB on large /// sessions), so they get the wider cap. Method sniffing on raw bytes avoids /// parsing multi-MiB JSON just to reject it. -fn enforce_request_byte_cap(body: &[u8]) -> Result<(), HandlerOutcome> { +fn enforce_request_byte_cap(body: &[u8]) -> Result<(), PreparedOutcome> { if body.len() <= MAX_FACADE_FRAME_BYTES { return Ok(()); } @@ -15521,11 +15739,13 @@ fn replayed_memory_apply_command( session_id: &str, action: &str, command_id: &str, -) -> Option { +) -> Option { if let Ok(Some(response)) = store.facade_mutation_ledger_response(session_id, "memory", action, command_id) { - return Some(HandlerOutcome::Response(response)); + return Some(PreparedOutcome::Response(PreparedOutput::cached_bytes( + response, + ))); } store .load_dream_task_command(session_id, command_id) @@ -15537,28 +15757,28 @@ fn replayed_memory_apply_command( fn dream_apply_command_outcome( result: Result, failure_code: &str, -) -> HandlerOutcome { +) -> PreparedOutcome { match result { Ok(FacadeMutationOutcome::Applied(bytes) | FacadeMutationOutcome::Duplicate(bytes)) => { - HandlerOutcome::Response(bytes) + PreparedOutcome::Response(PreparedOutput::cached_bytes(bytes)) } Err(error) if store_error_is_authority_draining(&error) => { authority_draining_error("memories") } Err(error) if error.to_string().contains("authority_generation_mismatch:") => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "authority_generation_mismatch".to_string(), message: "memory authority generation changed while applying the command" .to_string(), } } Err(error) if error.to_string().contains("authority_state_mismatch:") => { - HandlerOutcome::Error { + PreparedOutcome::Error { code: "authority_state_mismatch".to_string(), message: "memory authority state changed while applying the command".to_string(), } } - Err(error) => HandlerOutcome::Error { + Err(error) => PreparedOutcome::Error { code: failure_code.to_string(), message: error.to_string(), }, @@ -15568,18 +15788,20 @@ fn dream_apply_command_outcome( fn facade_command_outcome( result: Result, domain: &str, -) -> HandlerOutcome { +) -> PreparedOutcome { match result { - Ok(FacadeMutationOutcome::Applied(bytes)) => HandlerOutcome::Response(bytes), + Ok(FacadeMutationOutcome::Applied(bytes)) => { + PreparedOutcome::Response(PreparedOutput::cached_bytes(bytes)) + } Ok(FacadeMutationOutcome::Duplicate(bytes)) => { let Ok(mut envelope) = serde_json::from_slice::(&bytes) else { - return HandlerOutcome::Response(bytes); + return PreparedOutcome::Response(PreparedOutput::cached_bytes(bytes)); }; if let Some(object) = envelope.as_object_mut() { object.insert("replayed".to_string(), Value::Bool(true)); return respond(envelope); } - HandlerOutcome::Response(bytes) + PreparedOutcome::Response(PreparedOutput::cached_bytes(bytes)) } Err(error) if store_error_is_authority_draining(&error) => authority_draining_error(domain), Err(error) => tool_error_result(format!("Error: {error}")), @@ -15597,7 +15819,7 @@ fn refuse_conditioned_note_without_evaluator( action: &str, command_id: Option<&str>, refusal: &str, -) -> HandlerOutcome { +) -> PreparedOutcome { if let Some(command_id) = command_id { match store.facade_mutation_ledger_response(identity_scope, "ctx_note", action, command_id) { @@ -16259,36 +16481,27 @@ fn ctx_note_schema() -> Value { }) } -/// The module manifest registered at HELLO. The startup manifest owns stable tool IDs and schemas; -/// bound sessions obtain preset-selected description text through `manifest.get`. -pub fn manifest(module_id: &str) -> ModuleManifest { - ModuleManifest { - module_id: module_id.to_string(), - module_version: env!("CARGO_PKG_VERSION").to_string(), - protocol_ver: PROTOCOL_VERSION, - trust_tier: TrustTier::FirstParty, - provides: vec![ProviderRole::ToolProvider { - tools: prompt_surface::module_tools(&PromptSurfaceSelection::default()), - identity_scope: vec![IdentityScope::Project, IdentityScope::Session], - concurrency: Concurrency::ModuleManaged, - emits_push: false, - sub_supervises: false, - }], - consumes: vec![ConsumerRole::ServiceClient { - of: vec!["thalamus".to_string()], - }], - bindings: Bindings { - storage: StorageBinding { - kind: StorageKind::Sqlite, - scope: StorageScope::Project, - owns_schema: true, - }, - vault_grants: Vec::new(), - identity: IdentityBinding { - requires: vec![IdentityScope::Project], - optional: vec![IdentityScope::Session], - }, - }, +pub fn manifest(module_id: &str) -> ManifestSnapshot { + ManifestSnapshot { + module_id: module_id.to_owned(), + module_version: env!("CARGO_PKG_VERSION").to_owned(), + provides: vec![json!({ + "role": "tool_provider", + "tools": prompt_surface::module_tools(&PromptSurfaceSelection::default()), + "identity_scope": ["project", "session"], + "concurrency": "module_managed", + "emits_push": false, + "sub_supervises": false, + })], + control_ops: Vec::new(), + } +} + +#[cfg(test)] +fn test_route(channel_id: u16) -> RouteHandle { + RouteHandle { + channel: channel_id, + epoch: 1, } } @@ -16297,7 +16510,7 @@ mod tests { use super::*; use std::collections::{HashMap, VecDeque}; use std::sync::{ - atomic::{AtomicUsize, Ordering}, + atomic::{AtomicBool, AtomicUsize, Ordering}, Arc, }; @@ -16314,6 +16527,124 @@ mod tests { }; use tokio::sync::Notify; + struct SettlementWriter { + events: Arc>>, + bytes: Vec, + } + + impl std::io::Write for SettlementWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.events.lock().unwrap().push("write"); + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + #[tokio::test] + async fn production_settlement_reserves_before_write_and_returns_exact_body() { + let events = Arc::new(Mutex::new(Vec::new())); + let reserve_events = Arc::clone(&events); + let settlement = settle_prepared_with( + PreparedOutcome::Response(PreparedOutput::cached_bytes(b"exact-body".to_vec())), + || false, + move |len| { + reserve_events.lock().unwrap().push("reserve"); + let events = Arc::clone(&reserve_events); + async move { + Ok::<_, ()>(SettlementWriter { + events, + bytes: Vec::with_capacity(len), + }) + } + }, + ) + .await; + let PreparedSettlement::Response(writer) = settlement else { + panic!("successful settlement must return a response"); + }; + assert_eq!(writer.bytes, b"exact-body"); + let events = events.lock().unwrap(); + assert_eq!(events.first(), Some(&"reserve")); + assert!(events[1..].iter().all(|event| *event == "write")); + } + + #[tokio::test] + async fn production_settlement_error_and_stream_skip_reservation() { + let reservations = Arc::new(AtomicUsize::new(0)); + for outcome in [ + PreparedOutcome::Error { + code: "invalid".to_owned(), + message: "bad".to_owned(), + }, + PreparedOutcome::Streamed, + ] { + let reservations = Arc::clone(&reservations); + let settlement: PreparedSettlement> = settle_prepared_with( + outcome, + || false, + move |_| { + reservations.fetch_add(1, Ordering::SeqCst); + std::future::ready(Ok(Vec::new())) + }, + ) + .await; + assert!(!matches!(settlement, PreparedSettlement::Response(_))); + } + assert_eq!(reservations.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn production_settlement_cancellation_and_denial_emit_no_body() { + let reservations = Arc::new(AtomicUsize::new(0)); + let reserve_count = Arc::clone(&reservations); + let before: PreparedSettlement> = settle_prepared_with( + PreparedOutcome::Response(PreparedOutput::cached_bytes(b"body".to_vec())), + || true, + move |_| { + reserve_count.fetch_add(1, Ordering::SeqCst); + std::future::ready(Ok(Vec::new())) + }, + ) + .await; + assert!(matches!( + before, + PreparedSettlement::Error { ref code, .. } if code == "request_cancelled" + )); + assert_eq!(reservations.load(Ordering::SeqCst), 0); + + let checks = AtomicUsize::new(0); + let reserve_count = Arc::clone(&reservations); + let between: PreparedSettlement> = settle_prepared_with( + PreparedOutcome::Response(PreparedOutput::cached_bytes(b"body".to_vec())), + || checks.fetch_add(1, Ordering::SeqCst) == 1, + move |_| { + reserve_count.fetch_add(1, Ordering::SeqCst); + std::future::ready(Ok(Vec::new())) + }, + ) + .await; + assert!(matches!( + between, + PreparedSettlement::Error { ref code, .. } if code == "request_cancelled" + )); + assert_eq!(reservations.load(Ordering::SeqCst), 1); + + let denied: PreparedSettlement> = settle_prepared_with( + PreparedOutcome::Response(PreparedOutput::cached_bytes(b"body".to_vec())), + || false, + |_| std::future::ready(Err(())), + ) + .await; + assert!(matches!( + denied, + PreparedSettlement::Error { ref code, .. } if code == "output_unavailable" + )); + } + #[test] fn usage_numbers_rejects_implausible_context_limit() { let tiny = ModuleUsage { @@ -16956,7 +17287,7 @@ mod tests { async fn wait_for_store_open(handler: &McHandler) { tokio::time::timeout(Duration::from_secs(10), async { - while handler.store.get().is_none() { + while handler.store().is_none() { tokio::time::sleep(Duration::from_millis(5)).await; } }) @@ -16976,7 +17307,7 @@ mod tests { // wait. handler.set_store_open_policy_for_test(short_store_open_policy(Duration::from_secs(30))); - handler.begin_store_open(descriptor); + handler.begin_store_open(descriptor).unwrap(); wait_for_store_open_phase(&handler, STORE_OPEN_WAITING).await; let before = error_frame(call_transform_outcome(&handler, request(big_messages())).await); assert_eq!(before.0, "store_unavailable"); @@ -16998,7 +17329,7 @@ mod tests { let handler = McHandler::new(); handler.set_store_open_policy_for_test(short_store_open_policy(Duration::from_millis(60))); - handler.begin_store_open(descriptor); + handler.begin_store_open(descriptor).unwrap(); wait_for_store_open_phase(&handler, STORE_OPEN_WAITING).await; let before = error_frame(call_transform_outcome(&handler, request(big_messages())).await); wait_for_store_open_phase(&handler, STORE_OPEN_IDLE).await; @@ -17016,9 +17347,9 @@ mod tests { let handler = McHandler::new(); handler.set_store_open_policy_for_test(short_store_open_policy(Duration::from_millis(500))); - handler.begin_store_open(descriptor.clone()); + handler.begin_store_open(descriptor.clone()).unwrap(); wait_for_store_open_phase(&handler, STORE_OPEN_WAITING).await; - handler.begin_store_open(descriptor); + handler.begin_store_open(descriptor).unwrap(); tokio::time::sleep(Duration::from_millis(30)).await; assert_eq!(handler.store_open.waiter_starts.load(Ordering::Relaxed), 1); @@ -17036,7 +17367,7 @@ mod tests { handler.set_store_open_policy_for_test(short_store_open_policy(Duration::from_secs(5))); let coordinator = Arc::clone(&handler.store_open); - handler.begin_store_open(descriptor); + handler.begin_store_open(descriptor).unwrap(); wait_for_store_open_phase(&handler, STORE_OPEN_WAITING).await; drop(handler); tokio::time::timeout(Duration::from_millis(200), async { @@ -17048,6 +17379,105 @@ mod tests { .expect("store lease waiter should stop on handler shutdown"); } + #[tokio::test] + async fn shutdown_cancels_and_joins_tracked_historian_worker() { + let handler = McHandler::new(); + let observed = Arc::new(AtomicBool::new(false)); + let worker_observed = Arc::clone(&observed); + let cancel = handler.cancel.clone(); + handler + .spawn_tracked_task(async move { + cancel.cancelled().await; + worker_observed.store(true, Ordering::SeqCst); + }) + .expect("historian worker admitted"); + + ::shutdown(&handler) + .await + .unwrap(); + + assert!(observed.load(Ordering::SeqCst)); + assert!(handler.tasks.is_empty()); + } + + fn handler_with_blocking_lifecycle( + call: BlockingLifecycleCall, + ) -> ( + McHandler, + Arc, + Arc, + tempfile::TempDir, + ) { + let state = BlockingLifecycleState::new(call); + let factory = Arc::new(BlockingLifecycleFactory { + state: Arc::clone(&state), + }); + let handler = McHandler::with_producer_factory_and_config(factory, default_test_config()); + *state.cancel.lock().unwrap() = Some(handler.cancel.clone()); + + let dir = tempfile::tempdir().unwrap(); + let data_home = dir.path().join("data"); + std::fs::create_dir_all(&data_home).unwrap(); + let store = + Arc::new(McStore::open(&dev_descriptor_at(data_home.to_str().unwrap())).unwrap()); + handler.install_store_for_test(Arc::clone(&store)); + let project = dir.path().join("project"); + std::fs::create_dir_all(&project).unwrap(); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), "ses")); + (handler, store, state, dir) + } + + async fn wait_for_blocking_lifecycle(state: &BlockingLifecycleState) { + tokio::time::timeout(Duration::from_secs(2), async { + while !state.entered.load(Ordering::SeqCst) { + tokio::task::yield_now().await; + } + }) + .await + .expect("historian call site entered injected blocking future"); + } + + async fn assert_shutdown_joined_lifecycle_task( + handler: &McHandler, + state: &BlockingLifecycleState, + ) { + tokio::time::timeout( + Duration::from_secs(1), + ::shutdown(handler), + ) + .await + .expect("shutdown must join blocked historian call site") + .unwrap(); + assert!(state.exited.load(Ordering::SeqCst)); + assert!(handler.cancel.is_cancelled()); + assert!(handler.tasks.is_empty()); + assert!(!*handler.task_admission_open.lock().unwrap()); + assert!(handler.spawn_module_task(async {}).is_none()); + } + + #[tokio::test] + async fn shutdown_cancels_actual_spawned_historian_start_call_site() { + let (handler, _store, state, _dir) = + handler_with_blocking_lifecycle(BlockingLifecycleCall::Start); + let response = call_transform(&handler, big_messages()).await; + assert_eq!(response["historian"]["fired"], true); + wait_for_blocking_lifecycle(&state).await; + assert_shutdown_joined_lifecycle_task(&handler, &state).await; + } + + #[tokio::test] + async fn shutdown_cancels_actual_spawned_historian_reattach_call_site() { + let (handler, store, state, _dir) = + handler_with_blocking_lifecycle(BlockingLifecycleCall::Reattach); + let messages = big_messages(); + seed_awaiting(&store, &messages); + let response = call_transform(&handler, messages).await; + assert_eq!(response["historian"]["no_fire"], "reattaching"); + wait_for_blocking_lifecycle(&state).await; + assert_shutdown_joined_lifecycle_task(&handler, &state).await; + assert!(handler.reattaching_sessions.lock().unwrap().is_empty()); + } + #[tokio::test] async fn lease_wait_health_detail_advances_elapsed_time() { let dir = tempfile::tempdir().unwrap(); @@ -17058,11 +17488,11 @@ mod tests { let handler = McHandler::new(); handler.set_store_open_policy_for_test(short_store_open_policy(Duration::from_millis(500))); - handler.begin_store_open(descriptor); + handler.begin_store_open(descriptor).unwrap(); wait_for_store_open_phase(&handler, STORE_OPEN_WAITING).await; - let first = ::health(&handler).await; + let first = ::health(&handler).await; tokio::time::sleep(Duration::from_millis(35)).await; - let second = ::health(&handler).await; + let second = ::health(&handler).await; assert_eq!(first.status, HealthStatus::Degraded); assert_eq!(second.status, HealthStatus::Degraded); @@ -17086,19 +17516,13 @@ mod tests { } #[test] - fn manifest_declares_module_id_and_storage() { - let m = manifest("magic-context"); - assert_eq!(m.module_id, "magic-context"); - assert_eq!(m.protocol_ver, PROTOCOL_VERSION); - assert_eq!( - m.consumes, - vec![ConsumerRole::ServiceClient { - of: vec!["thalamus".to_string()] - }] - ); - let ProviderRole::ToolProvider { tools, .. } = &m.provides[0] else { - panic!("magic-context must expose a tool provider role"); - }; + fn manifest_declares_module_id_and_tools_without_resolver_consumer() { + let manifest = manifest("magic-context"); + assert_eq!(manifest.module_id, "magic-context"); + assert_eq!(manifest.provides[0]["role"], "tool_provider"); + assert!(manifest.provides[0].get("consumes").is_none()); + let tools: Vec = + serde_json::from_value(manifest.provides[0]["tools"].clone()).unwrap(); assert_eq!( tools .iter() @@ -17111,8 +17535,7 @@ mod tests { "ctx_expand", "ctx_search", "ctx_note", - ], - "the default startup manifest keeps its legacy byte order" + ] ); let by_name = tools .iter() @@ -17122,10 +17545,7 @@ mod tests { let tool = by_name .get(name) .unwrap_or_else(|| panic!("missing tool {name}")); - assert_eq!( - tool.name, name, - "mcp.jsonc overrides use the bare tool name" - ); + assert_eq!(tool.name, name); assert_eq!(tool.schema["type"], "object"); assert!(tool.schema["properties"].is_object()); assert!(tool.description.as_deref().is_some_and(|text| { @@ -17134,9 +17554,12 @@ mod tests { } assert_eq!( by_name["ctx_memory"].execution_mode, - ExecutionMode::Mutating + prompt_surface::ExecutionMode::Mutating + ); + assert_eq!( + by_name["ctx_search"].execution_mode, + prompt_surface::ExecutionMode::Pure ); - assert_eq!(by_name["ctx_search"].execution_mode, ExecutionMode::Pure); } fn binding(root: &str, session: &str) -> SessionBinding { @@ -17207,7 +17630,7 @@ mod tests { default_test_config(), Arc::new(MissingSessionResolver), ); - handler.store.set(Arc::clone(&store)).ok().unwrap(); + handler.install_store_for_test(Arc::clone(&store)); let project = dir.path().join("project"); std::fs::create_dir_all(&project).unwrap(); let _dir = dir; @@ -17226,14 +17649,17 @@ mod tests { let channel = *channels.entry(session.clone()).or_insert_with(|| { let ch = next_channel; next_channel += 1; - handler.bind_route(ch, binding(project.to_str().unwrap(), &session)); + handler + .bind_route(test_route(ch), binding(project.to_str().unwrap(), &session)); ch }); let started = std::time::Instant::now(); - let outcome = handler.dispatch_value(channel, value.clone()).await; + let outcome = handler + .dispatch_value(test_route(channel), value.clone()) + .await; let ms = started.elapsed().as_millis(); match outcome { - HandlerOutcome::Response(bytes) => { + PreparedOutcome::Response(bytes) => { // Optional: dump the raw TransformResponse bytes per pass so a // consumer (e.g. the gateway plan_outcome harness) can consume the // module's EXACT returned bytes (MC_REPLAY_OUT_DIR=

). @@ -17253,7 +17679,7 @@ mod tests { .to_string(); outcomes.push((name, action, Some(parsed), ms)); } - HandlerOutcome::Error { code, message } => { + PreparedOutcome::Error { code, message } => { println!("[replay] {name} ERROR code={code} ms={ms} message={message}"); outcomes.push((name, format!("ERROR:{code}"), None, ms)); } @@ -17423,13 +17849,14 @@ mod tests { /// Resolve just the project_root (the binding's identity) for the resolve assertions. fn resolved_root(h: &McHandler, channel: u16, session: &str) -> Result { - h.resolve_binding(channel, session).map(|b| b.project_root) + h.resolve_binding(test_route(channel), session) + .map(|b| b.project_root) } #[test] fn route_binding_bind_resolve_unbind() { let h = McHandler::new(); - h.bind_route(7, binding("/repo/proj", "ses_a")); + h.bind_route(test_route(7), binding("/repo/proj", "ses_a")); // resolve succeeds when the channel is bound AND the session matches assert_eq!( @@ -17438,7 +17865,7 @@ mod tests { ); // a teardown removes the binding → a later resolve fails loud (no stale project) - h.unbind_route(7); + h.unbind_route(test_route(7)); assert_eq!(resolved_root(&h, 7, "ses_a"), Err(BindingError::Unbound)); } @@ -17448,7 +17875,7 @@ mod tests { // never bound → Unbound (NEVER a default project, which would be a cross-project read) assert_eq!(resolved_root(&h, 3, "ses_x"), Err(BindingError::Unbound)); - h.bind_route(3, binding("/repo/own", "ses_own")); + h.bind_route(test_route(3), binding("/repo/own", "ses_own")); // bound, but a request claiming a DIFFERENT session on this channel → SessionMismatch assert_eq!( resolved_root(&h, 3, "ses_other"), @@ -17464,9 +17891,9 @@ mod tests { #[test] fn rebind_overwrites_stale_channel_entry() { let h = McHandler::new(); - h.bind_route(5, binding("/a", "s1")); + h.bind_route(test_route(5), binding("/a", "s1")); // a reused channel re-binds to a new session → last write wins (no stale leak) - h.bind_route(5, binding("/b", "s2")); + h.bind_route(test_route(5), binding("/b", "s2")); assert_eq!(resolved_root(&h, 5, "s2").unwrap(), PathBuf::from("/b")); assert_eq!( resolved_root(&h, 5, "s1"), @@ -17474,6 +17901,48 @@ mod tests { ); } + #[tokio::test] + async fn old_epoch_route_gone_preserves_new_epoch_binding_and_dispatch_state() { + let state = Arc::new(ProducerState::default()); + let (handler, _store, _dir, project) = handler_with_store(state, McModuleConfig::default()); + let old = RouteHandle { + channel: 7, + epoch: 1, + }; + let new = RouteHandle { + channel: 7, + epoch: 2, + }; + handler.bind_route(new, binding(project.to_str().unwrap(), "ses")); + let expected_transform_state = ("ses".to_owned(), canonical_root(&project)); + handler + .transform_route_channels + .lock() + .unwrap() + .insert(new, expected_transform_state.clone()); + + CompositeComponent::route_gone(&handler, old).await; + + assert_eq!( + handler.resolve_binding(new, "ses").unwrap().project_root, + project + ); + assert_eq!( + handler.transform_route_channels.lock().unwrap().get(&new), + Some(&expected_transform_state) + ); + let outcome = handler + .dispatch_value( + new, + json!({"method": "session.status", "v": 1, "session_id": "ses"}), + ) + .await; + assert!( + matches!(outcome, PreparedOutcome::Response(_)), + "new epoch must remain usable through real dispatch: {outcome:?}" + ); + } + #[test] fn in_flight_snapshot_entries_are_count_bounded_and_cannot_resurrect() { let mut cache = TransformSnapshotCache::new(1024); @@ -17689,6 +18158,108 @@ mod tests { )); } + #[derive(Clone, Copy, PartialEq, Eq)] + enum BlockingLifecycleCall { + Start, + Reattach, + } + + struct BlockingLifecycleState { + call: BlockingLifecycleCall, + cancel: Mutex>, + entered: AtomicBool, + exited: AtomicBool, + } + + impl BlockingLifecycleState { + fn new(call: BlockingLifecycleCall) -> Arc { + Arc::new(Self { + call, + cancel: Mutex::new(None), + entered: AtomicBool::new(false), + exited: AtomicBool::new(false), + }) + } + + async fn block(&self) -> HistorianProducerError { + self.entered.store(true, Ordering::SeqCst); + let cancel = self + .cancel + .lock() + .unwrap() + .clone() + .expect("test cancellation installed"); + cancel.cancelled().await; + self.exited.store(true, Ordering::SeqCst); + HistorianProducerError::TimedOut + } + } + + struct BlockingLifecycleFactory { + state: Arc, + } + + #[async_trait] + impl HistorianProducerFactory for BlockingLifecycleFactory { + async fn connect( + &self, + _project_root: &Path, + _harness: &str, + ) -> Result, HistorianProducerError> { + Ok(Box::new(BlockingLifecycleProducer { + state: Arc::clone(&self.state), + })) + } + } + + struct BlockingLifecycleProducer { + state: Arc, + } + + #[async_trait] + impl HistorianProducerDriver for BlockingLifecycleProducer { + async fn bind_session(&mut self, _session_id: &str) -> Result<(), HistorianProducerError> { + Ok(()) + } + + async fn start( + &mut self, + _session_id: &str, + _system: &str, + _prompt: &str, + _model: &str, + ) -> Result { + if self.state.call == BlockingLifecycleCall::Start { + return Err(self.state.block().await); + } + Ok(RunHandle { + run_id: "unused".to_owned(), + }) + } + + async fn await_output( + &mut self, + _run_id: &str, + ) -> Result { + Err(HistorianProducerError::TimedOut) + } + + async fn status(&mut self, _run_id: &str) -> Result { + if self.state.call == BlockingLifecycleCall::Reattach { + return Err(self.state.block().await); + } + Ok(RunState::Active) + } + + async fn cancel(&mut self, _run_id: &str) -> Result<(), HistorianProducerError> { + Ok(()) + } + + async fn close(&mut self) -> Result<(), HistorianProducerError> { + Ok(()) + } + } + #[derive(Default)] struct ProducerState { connects: AtomicUsize, @@ -17917,10 +18488,10 @@ mod tests { config, resolver, ); - handler.store.set(Arc::clone(&store)).ok().unwrap(); + handler.install_store_for_test(Arc::clone(&store)); let project = dir.path().join("project"); std::fs::create_dir_all(&project).unwrap(); - handler.bind_route(7, binding(project.to_str().unwrap(), "ses")); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), "ses")); (handler, store, dir, project) } @@ -18278,14 +18849,19 @@ mod tests { channel: u16, request: Value, ) -> Value { - match handler.handle_transform_for_test(channel, request).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + match handler + .handle_transform_for_test(test_route(channel), request) + .await + { + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), } } - async fn call_transform_outcome(handler: &McHandler, request: Value) -> HandlerOutcome { - handler.handle_transform_for_test(7, request).await + async fn call_transform_outcome(handler: &McHandler, request: Value) -> PreparedOutcome { + handler + .handle_transform_for_test(test_route(7), request) + .await } async fn call_dispatch_request(handler: &McHandler, request: Value) -> Value { @@ -18297,24 +18873,24 @@ mod tests { channel: u16, request: Value, ) -> Value { - match handler.dispatch_value(channel, request).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + match handler.dispatch_value(test_route(channel), request).await { + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), } } - fn error_frame(outcome: HandlerOutcome) -> (String, String) { + fn error_frame(outcome: PreparedOutcome) -> (String, String) { match outcome { - HandlerOutcome::Error { code, message } => (code, message), + PreparedOutcome::Error { code, message } => (code, message), other => panic!("expected error outcome, got {other:?}"), } } - fn error_code(outcome: HandlerOutcome) -> String { + fn error_code(outcome: PreparedOutcome) -> String { error_frame(outcome).0 } - async fn call_facade(handler: &McHandler, name: &str, arguments: Value) -> HandlerOutcome { + async fn call_facade(handler: &McHandler, name: &str, arguments: Value) -> PreparedOutcome { call_facade_on_channel(handler, 7, name, arguments).await } @@ -18323,31 +18899,34 @@ mod tests { channel: u16, name: &str, arguments: Value, - ) -> HandlerOutcome { + ) -> PreparedOutcome { handler - .dispatch_value(channel, json!({ "name": name, "arguments": arguments })) + .dispatch_value( + test_route(channel), + json!({ "name": name, "arguments": arguments }), + ) .await } - fn tool_body(outcome: HandlerOutcome) -> Value { + fn tool_body(outcome: PreparedOutcome) -> Value { match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("expected tool response, got {other:?}"), } } - fn tool_is_error(outcome: HandlerOutcome) -> bool { + fn tool_is_error(outcome: PreparedOutcome) -> bool { tool_body(outcome)["isError"].as_bool().unwrap_or(false) } - fn tool_text(outcome: HandlerOutcome) -> String { + fn tool_text(outcome: PreparedOutcome) -> String { tool_body(outcome)["content"][0]["text"] .as_str() .unwrap() .to_string() } - fn tool_json_array(outcome: HandlerOutcome) -> Vec { + fn tool_json_array(outcome: PreparedOutcome) -> Vec { let body = tool_body(outcome); let text = body["content"][0]["text"] .as_str() @@ -18439,7 +19018,10 @@ mod tests { async fn cc_inherits_oc_project_mural_on_a_natural_hard_without_defer_first_apply() { let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(8, binding(project.to_str().unwrap(), "cc-mural")); + handler.bind_route( + test_route(8), + binding(project.to_str().unwrap(), "cc-mural"), + ); let messages = vec![ck("tail", 1, "raw")]; let mural_a = json!({ "enabled": true, @@ -18529,7 +19111,7 @@ mod tests { handler_with_store(Arc::new(ProducerState::default()), config); let mut route = binding(project.to_str().unwrap(), "ses"); route.config = route_config; - handler.bind_route(7, route); + handler.bind_route(test_route(7), route); let mut transform_request = request(vec![ck("a", 1, "alpha")]); transform_request["serializer_profile"] = json!("claude-code-anthropic"); transform_request["model_key"] = json!("anthropic/claude-opus-4-1"); @@ -18548,7 +19130,7 @@ mod tests { handler_with_store(Arc::new(ProducerState::default()), config); let mut route = binding(project.to_str().unwrap(), "ses"); route.config = route_config; - handler.bind_route(7, route); + handler.bind_route(test_route(7), route); let mut transform_request = request(vec![ck("a", 1, "alpha")]); transform_request["serializer_profile"] = json!("claude-code-anthropic"); @@ -18565,7 +19147,7 @@ mod tests { let expected = serde_json::to_vec(&serde_json::to_value(response.clone()).unwrap()).unwrap(); let request = transform_request(vec![ck("wire-byte-cache", 1, "hello")], 1, 100); - let HandlerOutcome::Response(actual) = respond_transform(&request, response) else { + let PreparedOutcome::Response(actual) = respond_transform(&request, response) else { panic!("cached transform response failed to encode"); }; assert_eq!(actual, expected); @@ -18619,12 +19201,12 @@ mod tests { transform_id: "completed".to_string(), generation: 1, final_digest: "digest-final".to_string(), - result: vec![0; 17], + result: PreparedOutput::cached_bytes(vec![0; 17]), }); } let outcome = handler.handle_status_value(&json!({"method": "status"})); - let HandlerOutcome::Response(bytes) = outcome else { + let PreparedOutcome::Response(bytes) = outcome else { panic!("module status did not respond: {outcome:?}"); }; let status: Value = serde_json::from_slice(&bytes).unwrap(); @@ -19589,7 +20171,10 @@ mod tests { let request = Arc::new(request); let producer = Arc::new(ProducerState::default()); let (handler, _store, _dir, project) = handler_with_store(producer, default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), SESSION_ID)); + handler.bind_route( + test_route(7), + binding(project.to_str().unwrap(), SESSION_ID), + ); let projection = Arc::new( crate::ck_wire::project_messages(&request.messages).expect("giant projection"), @@ -20638,7 +21223,7 @@ mod tests { let (handler, _store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); let session = "native-delta-fingerprint-mismatch"; - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); let initial = native_cache_request( session, @@ -20730,7 +21315,7 @@ mod tests { let (handler, _store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); let session = "native-delta-eviction-heal"; - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); let initial = native_cache_request( session, @@ -20819,9 +21404,9 @@ mod tests { let session_a = "projection-lru-a"; let session_b = "projection-lru-b"; let session_c = "projection-lru-oversized"; - handler.bind_route(7, binding(project.to_str().unwrap(), session_a)); - handler.bind_route(8, binding(project.to_str().unwrap(), session_b)); - handler.bind_route(9, binding(project.to_str().unwrap(), session_c)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session_a)); + handler.bind_route(test_route(8), binding(project.to_str().unwrap(), session_b)); + handler.bind_route(test_route(9), binding(project.to_str().unwrap(), session_c)); let initial_a = native_cache_request( session_a, @@ -21209,7 +21794,7 @@ mod tests { let session = "projection-revert-epoch"; let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); let initial = native_cache_request( session, vec![ @@ -21255,7 +21840,7 @@ mod tests { let session = "projection-reconcile-recut"; let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); store .replace_compartments( session, @@ -21342,7 +21927,7 @@ mod tests { let session = "projection-boundary-recut-cas"; let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); let mut previous = crate::transform::tests::seed_astro_divergence(&store, session, 2_442); previous.serializer_profile = "opencode-aisdk".to_string(); previous.serve_native = true; @@ -21414,7 +21999,7 @@ mod tests { let session = "projection-tail-readopt"; let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); store .replace_compartments(session, &[stored_comp(1, 1, 1, "covered", "SUMMARY")]) .unwrap(); @@ -21471,8 +22056,8 @@ mod tests { let source = "projection-lineage-source"; let (handler, store, _dir, project) = handler_with_store(Arc::new(ProducerState::default()), default_test_config()); - handler.bind_route(7, binding(project.to_str().unwrap(), target)); - handler.bind_route(8, binding(project.to_str().unwrap(), source)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), target)); + handler.bind_route(test_route(8), binding(project.to_str().unwrap(), source)); let source_messages = (1..=10) .map(|ordinal| { ck( @@ -21948,7 +22533,7 @@ mod tests { let producer = Arc::new(ProducerState::default()); let (handler, _store, _dir, project) = handler_with_store(producer, default_test_config()); let session = "native-dreamer"; - handler.bind_route(7, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(7), binding(project.to_str().unwrap(), session)); let _registration = handler.register_dreamer_run(session); let first = native_cache_request( @@ -22031,7 +22616,7 @@ mod tests { let not_due_response = call_transform_request(&handler, not_due).await; assert!(not_due_response.get("host_directives").is_none()); - handler.bind_route(8, binding("/tmp/cc", "cc-ses")); + handler.bind_route(test_route(8), binding("/tmp/cc", "cc-ses")); let cc_response = call_transform_request_on_channel( &handler, 8, @@ -22058,7 +22643,7 @@ mod tests { assert!(cc_directive["armed_at_ms"].as_i64().unwrap() > 0); assert!(cc_response.get("host_directives").is_none()); - handler.bind_route(9, binding("/tmp/pi", "pi-ses")); + handler.bind_route(test_route(9), binding("/tmp/pi", "pi-ses")); let pi_response = call_transform_request_on_channel( &handler, 9, @@ -22356,14 +22941,14 @@ mod tests { assert_eq!(date, trimmed_bytes.lines().last().unwrap()); let unknown = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "guidance.get", "session_id": "ses", "variant": "bogus" }), ) .await; assert_eq!(error_code(unknown), "bad_request"); let contradictory = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "guidance.get", "session_id": "ses", @@ -22399,7 +22984,7 @@ mod tests { let (handler, _store, _dir, project) = handler_with_store(producer, config.clone()); let mut route = binding(project.to_str().unwrap(), "ses"); route.config = config; - handler.bind_route(7, route); + handler.bind_route(test_route(7), route); handler.guidance_dates.lock().unwrap().insert( "ses".to_string(), "Today's date: Fri Jan 01 2016".to_string(), @@ -22442,7 +23027,7 @@ mod tests { ]; for (index, session) in sessions.iter().enumerate() { handler.bind_route( - base_channel + index as u16, + test_route(base_channel + index as u16), binding("/tmp/project", session), ); handler @@ -22553,7 +23138,7 @@ mod tests { { let producer = Arc::new(ProducerState::default()); let (handler, _store, _dir, _project) = handler_with_store(producer, default_test_config()); - handler.bind_route(30, binding("/tmp/project", "manifest-ses")); + handler.bind_route(test_route(30), binding("/tmp/project", "manifest-ses")); let first = call_dispatch_request_on_channel( &handler, 30, @@ -22611,10 +23196,9 @@ mod tests { let expected_tools = prompt_surface::session_tools(&PromptSurfaceSelection::default()); for response in [&first, &transitioned] { - let response_tools = serde_json::from_value::>( - response["tools"].clone(), - ) - .unwrap(); + let response_tools = + serde_json::from_value::>(response["tools"].clone()) + .unwrap(); assert_eq!(response_tools.len(), expected_tools.len()); for (actual, expected) in response_tools.iter().zip(&expected_tools) { assert_eq!(actual.name, expected.name); @@ -22629,7 +23213,7 @@ mod tests { let producer = Arc::new(ProducerState::default()); let (handler, _store, _dir, _project) = handler_with_store(producer, default_test_config()); let session = "prompt-config-reload"; - handler.bind_route(31, binding("/tmp/project", session)); + handler.bind_route(test_route(31), binding("/tmp/project", session)); handler.guidance_dates.lock().unwrap().insert( session.to_string(), "Today's date: Fri Jan 01 2016".to_string(), @@ -22831,19 +23415,19 @@ mod tests { assert_ne!(advanced["hash"], first["hash"]); assert_eq!(advanced["content_hash"], first["content_hash"]); - handler.bind_route(8, binding("/tmp/other", "other")); + handler.bind_route(test_route(8), binding("/tmp/other", "other")); handler.guidance_dates.lock().unwrap().insert( "other".to_string(), "Today's date: Sun Jan 03 2016".to_string(), ); let other = match handler .dispatch_value( - 8, + test_route(8), json!({ "kind": "guidance.get", "session_id": "other", "tool_present": true }), ) .await { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected outcome: {other:?}"), }; assert_ne!(other["hash"], advanced["hash"]); @@ -22979,7 +23563,7 @@ mod tests { let state_sync = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -22990,7 +23574,7 @@ mod tests { }), ) .await; - assert!(matches!(state_sync, HandlerOutcome::Response(_))); + assert!(matches!(state_sync, PreparedOutcome::Response(_))); let still_refused = call_facade(&handler, "ctx_note", conditioned.clone()).await; assert!( tool_text(still_refused).contains("Smart-note evaluation is unavailable"), @@ -23069,7 +23653,7 @@ mod tests { assert!(tool_text(ttl_refused).contains("Smart-note evaluation is unavailable")); let stale_heartbeat = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "note.evaluation.heartbeat", "v": 2, @@ -23148,7 +23732,7 @@ mod tests { let v1 = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "note.evaluation.register", "v": 2, @@ -23171,21 +23755,21 @@ mod tests { let mut positive_wait = note_evaluation_next_body(&token, generation, "eval-a", "acq-1"); positive_wait["wait_ms"] = json!(50); - let positive = handler.dispatch_value(7, positive_wait).await; + let positive = handler.dispatch_value(test_route(7), positive_wait).await; assert_eq!(error_code(positive), "positive_wait_unsupported"); let stale = handler .dispatch_value( - 7, + test_route(7), note_evaluation_next_body(&stale_token, stale_generation, "eval-a", "acq-1"), ) .await; assert_eq!(error_code(stale), "registration_unknown"); - handler.bind_route(9, binding(&route_root, "ses")); + handler.bind_route(test_route(9), binding(&route_root, "ses")); let wrong_channel = handler .dispatch_value( - 9, + test_route(9), note_evaluation_next_body(&token, generation, "eval-a", "acq-1"), ) .await; @@ -23193,7 +23777,7 @@ mod tests { let unknown_field = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "note.evaluation.next", "v": 2, @@ -23234,13 +23818,13 @@ mod tests { register_note_evaluator(&handler, 7, "eval-a", false, false).await; assert!(handler.has_live_note_evaluator(&identity, now_ms())); - handler.unbind_route(7); + handler.unbind_route(test_route(7)); assert!(!handler.has_live_note_evaluator(&identity, now_ms())); - handler.bind_route(7, binding(&route_root, "ses")); + handler.bind_route(test_route(7), binding(&route_root, "ses")); let stale = handler .dispatch_value( - 7, + test_route(7), note_evaluation_next_body(&token, generation, "eval-a", "acq-1"), ) .await; @@ -23399,7 +23983,7 @@ mod tests { let oversized = handler .dispatch_value( - 7, + test_route(7), complete_with_outcome(json!({ "phase": "compile", "kind": "compiled_met", @@ -23416,7 +24000,7 @@ mod tests { let smuggled = handler .dispatch_value( - 7, + test_route(7), complete_with_outcome(json!({ "phase": "fallback", "kind": "logic_failed" })), ) .await; @@ -24117,7 +24701,7 @@ mod tests { // Draining authority rejects the poll, leaving the cycle unchanged. let handover = handler .dispatch_value( - 7, + test_route(7), note_evaluation_next_body(&token, generation, "eval-a", "acq-2"), ) .await; @@ -24252,7 +24836,7 @@ mod tests { let state_sync = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -24264,7 +24848,7 @@ mod tests { ) .await; assert!( - matches!(state_sync, HandlerOutcome::Response(_)), + matches!(state_sync, PreparedOutcome::Response(_)), "{state_sync:?}" ); @@ -24427,11 +25011,11 @@ mod tests { ); let project_root = project.to_str().unwrap(); handler.bind_route( - 7, + test_route(7), binding_with_harness(project_root, OPENCODE_HARNESS, "opencode-session"), ); handler.transform_route_channels.lock().unwrap().insert( - 7, + test_route(7), ("opencode-session".to_string(), canonical_root(project_root)), ); handler @@ -24493,12 +25077,15 @@ mod tests { // The transform lane binds through the symlink spelling, while the facade lane binds to // the canonical target. Both route bindings identify the same filesystem lineage. - handler.bind_route(7, binding_with_harness(link_text, OPENCODE_HARNESS, "ses")); + handler.bind_route( + test_route(7), + binding_with_harness(link_text, OPENCODE_HARNESS, "ses"), + ); let transformed = call_transform_request_on_channel(&handler, 7, request(vec![ck("m0", 0, "a")])).await; assert_eq!(transformed["action"], "HARD"); handler.bind_route( - 8, + test_route(8), binding_with_harness(target_text, OPENCODE_HARNESS, "ses"), ); @@ -24534,13 +25121,16 @@ mod tests { // Reverse the lane spellings: transform uses the target and facade uses the symlink. handler.bind_route( - 7, + test_route(7), binding_with_harness(target_text, OPENCODE_HARNESS, "ses"), ); let transformed = call_transform_request_on_channel(&handler, 7, request(vec![ck("m0", 0, "a")])).await; assert_eq!(transformed["action"], "HARD"); - handler.bind_route(8, binding_with_harness(link_text, OPENCODE_HARNESS, "ses")); + handler.bind_route( + test_route(8), + binding_with_harness(link_text, OPENCODE_HARNESS, "ses"), + ); let outcome = call_facade_on_channel( &handler, @@ -24566,7 +25156,7 @@ mod tests { resolver.clone(), ); handler.bind_route( - 7, + test_route(7), binding_with_harness( project.to_str().unwrap(), OPENCODE_HARNESS, @@ -24597,7 +25187,10 @@ mod tests { resolver.clone(), ); let root_a = project.to_str().unwrap(); - handler.bind_route(7, binding_with_harness(root_a, OPENCODE_HARNESS, "ses")); + handler.bind_route( + test_route(7), + binding_with_harness(root_a, OPENCODE_HARNESS, "ses"), + ); let transformed = call_transform_request_on_channel(&handler, 7, request(vec![ck("m0", 0, "a")])).await; assert_eq!(transformed["action"], "HARD"); @@ -24606,7 +25199,10 @@ mod tests { let root_b = project.join("other-root"); std::fs::create_dir_all(&root_b).unwrap(); let root_b = root_b.to_str().unwrap(); - handler.bind_route(8, binding_with_harness(root_b, OPENCODE_HARNESS, "ses")); + handler.bind_route( + test_route(8), + binding_with_harness(root_b, OPENCODE_HARNESS, "ses"), + ); let outcome = call_facade_on_channel( &handler, 8, @@ -24651,9 +25247,9 @@ mod tests { default_test_config(), Arc::new(MissingSessionResolver), ); - handler.store.set(Arc::clone(&store)).ok().unwrap(); + handler.install_store_for_test(Arc::clone(&store)); handler.bind_route( - 7, + test_route(7), binding_with_harness(root_a_text, OPENCODE_HARNESS, "ses"), ); let transformed = @@ -24677,9 +25273,9 @@ mod tests { default_test_config(), resolver.clone(), ); - handler.store.set(Arc::clone(&store)).ok().unwrap(); + handler.install_store_for_test(Arc::clone(&store)); handler.bind_route( - 7, + test_route(7), binding_with_harness(root_a_text, OPENCODE_HARNESS, "ses"), ); @@ -24709,7 +25305,7 @@ mod tests { assert!(resolver.calls().is_empty()); handler.bind_route( - 8, + test_route(8), binding_with_harness(root_b.to_str().unwrap(), OPENCODE_HARNESS, "ses"), ); let cross_root = call_facade_on_channel( @@ -24740,7 +25336,7 @@ mod tests { resolver.clone(), ); handler.bind_route( - 7, + test_route(7), binding_with_harness("/repo", "claude-code", "claude-instance-token"), ); @@ -24771,7 +25367,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); let note = store .insert_project_note(NoteWriteInput { project_path: "/repo", @@ -24791,7 +25387,7 @@ mod tests { let evaluated = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "note.evaluate", "session_id": "session", @@ -24818,8 +25414,11 @@ mod tests { default_test_config(), resolver, ); - handler.bind_route(7, binding("/repo", "token")); - handler.bind_route(8, binding_with_harness("/repo", OPENCODE_HARNESS, "ses")); + handler.bind_route(test_route(7), binding("/repo", "token")); + handler.bind_route( + test_route(8), + binding_with_harness("/repo", OPENCODE_HARNESS, "ses"), + ); activate_module_authority(&store, "context", "git:identity", "/repo", "notes"); let note = store .insert_project_note(NoteWriteInput { @@ -24865,7 +25464,7 @@ mod tests { .unwrap(); let ack = handler .dispatch_value( - 8, + test_route(8), json!({ "method": "transform.ack", "session_id": "ses", @@ -24873,7 +25472,7 @@ mod tests { }), ) .await; - assert!(matches!(ack, HandlerOutcome::Response(_))); + assert!(matches!(ack, PreparedOutcome::Response(_))); assert_eq!( store .get_note_by_id("git:identity", "ses", note.id) @@ -24895,7 +25494,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); store .seed_authority_row( "context-db", @@ -24928,7 +25527,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); for index in 0..105 { let note = store .insert_project_note(NoteWriteInput { @@ -25010,9 +25609,12 @@ mod tests { let key_a = "conversation:root|agent:alpha"; let key_b = "conversation:root|agent:beta"; let suffix_key = "conversation:root|scope:mc-historian:child"; - handler.bind_route(8, binding(project.to_str().unwrap(), key_a)); - handler.bind_route(9, binding(project.to_str().unwrap(), key_b)); - handler.bind_route(10, binding(project.to_str().unwrap(), suffix_key)); + handler.bind_route(test_route(8), binding(project.to_str().unwrap(), key_a)); + handler.bind_route(test_route(9), binding(project.to_str().unwrap(), key_b)); + handler.bind_route( + test_route(10), + binding(project.to_str().unwrap(), suffix_key), + ); store .replace_compartments(key_a, &[stored_comp(1, 1, 1, "a1", "A")]) @@ -25080,20 +25682,20 @@ mod tests { // A flat body with kind="transform" routes to the transform handler. let transform = handler - .dispatch_value(7, request(vec![ck("m1", 1, "hello")])) + .dispatch_value(test_route(7), request(vec![ck("m1", 1, "hello")])) .await; let transform_body: Value = match transform { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("transform should respond, got {other:?}"), }; assert_eq!(transform_body["status"], "ok"); // Explicit echo: opt-in debugging arm still works when asked for by name. let echo = handler - .dispatch_value(7, json!({ "kind": "echo", "probe": 42 })) + .dispatch_value(test_route(7), json!({ "kind": "echo", "probe": 42 })) .await; let echo_body: Value = match echo { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("echo should respond, got {other:?}"), }; assert_eq!(echo_body["ok"], json!(true)); @@ -25103,7 +25705,7 @@ mod tests { // DISTINCT error so a facade misroute is diagnosable from the code. let facade = handler .dispatch_value( - 7, + test_route(7), json!({ "name": "ctx_unknown", "arguments": { "drop": "1-3" } }), ) .await; @@ -25112,10 +25714,10 @@ mod tests { // Anything else: fail loud, never a silent echo. The message names the // keys that were present so a misrouted request is diagnosable. let garbage = handler - .dispatch_value(7, json!({ "foo": 1, "bar": 2 })) + .dispatch_value(test_route(7), json!({ "foo": 1, "bar": 2 })) .await; match garbage { - HandlerOutcome::Error { code, message } => { + PreparedOutcome::Error { code, message } => { assert_eq!(code, "unrecognized_request_shape"); assert!(message.contains("foo"), "message names got keys: {message}"); } @@ -25123,9 +25725,11 @@ mod tests { } // Non-object bodies get the same loud failure with the JSON type named. - let non_object = handler.dispatch_value(7, json!("just a string")).await; + let non_object = handler + .dispatch_value(test_route(7), json!("just a string")) + .await; match non_object { - HandlerOutcome::Error { code, message } => { + PreparedOutcome::Error { code, message } => { assert_eq!(code, "unrecognized_request_shape"); assert!(message.contains("string"), "message names type: {message}"); } @@ -25143,7 +25747,10 @@ mod tests { let (handler, _store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver.clone()); - handler.bind_route(7, binding_with_harness("/repo", "claude-code", "")); + handler.bind_route( + test_route(7), + binding_with_harness("/repo", "claude-code", ""), + ); let no_token = call_facade( &handler, "ctx_search", @@ -25151,7 +25758,7 @@ mod tests { ) .await; match no_token { - HandlerOutcome::Error { code, message } => { + PreparedOutcome::Error { code, message } => { assert_eq!(code, "session_unresolved"); assert_eq!(message, SESSION_UNRESOLVED_MESSAGE); } @@ -25160,7 +25767,7 @@ mod tests { assert_eq!(resolver.calls(), Vec::::new()); handler.bind_route( - 7, + test_route(7), binding_with_harness("/repo", "claude-code", "missing-map"), ); let none = call_facade( @@ -25171,7 +25778,10 @@ mod tests { .await; assert_eq!(error_code(none), "session_unresolved"); - handler.bind_route(7, binding_with_harness("/repo", "claude-code", "slow-map")); + handler.bind_route( + test_route(7), + binding_with_harness("/repo", "claude-code", "slow-map"), + ); let timeout = call_facade( &handler, "ctx_search", @@ -25191,7 +25801,7 @@ mod tests { resolver, ); let project_root = project.to_str().unwrap(); - handler.bind_route(7, binding(project_root, "token")); + handler.bind_route(test_route(7), binding(project_root, "token")); store.fail_next_authority_project_resolution_for_test(); let arguments = json!({ "action": "write", @@ -25229,8 +25839,8 @@ mod tests { ]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding(project_root, "token-a")); - handler.bind_route(8, binding(project_root, "token-b")); + handler.bind_route(test_route(7), binding(project_root, "token-a")); + handler.bind_route(test_route(8), binding(project_root, "token-b")); store .replace_compartments( key_a, @@ -25332,11 +25942,11 @@ mod tests { )]); let (handler, _store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); let echo = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "echo", "name": "ctx_memory", "arguments": { "action": "write" } }), ) .await; @@ -25454,14 +26064,17 @@ mod tests { default_test_config(), resolver.clone(), ); - handler.bind_route(8, binding("/path/that/does/not/exist", "unresolvable")); + handler.bind_route( + test_route(8), + binding("/path/that/does/not/exist", "unresolvable"), + ); let response = call_facade_on_channel(&handler, 8, "ctx_reduce", json!({ "drop": "1" })).await; assert_eq!(error_code(response), "session_unresolved"); assert_eq!(resolver.calls(), vec!["unresolvable"]); assert!( - handler.store.get().is_none(), + handler.store().is_none(), "an unresolved session must not open storage to validate tags" ); } @@ -25501,7 +26114,7 @@ mod tests { // The response observer delivers the same mixed request later. It queues only // known tags, leaving acknowledgement validation side-effect free. let delivered = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -25512,7 +26125,7 @@ mod tests { assert_eq!(tool_body(delivered), json!({ "ok": true, "queued": 2 })); let pending_after_delivery = store.load_pending_agent_drops("ses").unwrap(); let retry = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -25555,9 +26168,8 @@ mod tests { #[test] fn ctx_manifest_schemas_accept_unknown_args_without_advertising_reduced_fields() { let manifest = manifest("magic-context"); - let ProviderRole::ToolProvider { tools, .. } = &manifest.provides[0] else { - panic!("tool provider manifest entry"); - }; + let tools: Vec = + serde_json::from_value(manifest.provides[0]["tools"].clone()).unwrap(); let by_name = tools .iter() .map(|tool| (tool.name.as_str(), tool)) @@ -25726,7 +26338,7 @@ mod tests { .unwrap(); let outcome = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "authority.seed", "context_store_uuid": "store-uuid", @@ -25775,7 +26387,7 @@ mod tests { )]); let (handler, _store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); let malformed = [ json!({ "action": "update", "content": "edited" }), @@ -25822,7 +26434,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); let id = insert_memory(&store, "/repo", "CONSTRAINTS", "Run focused tests.", 1); let plain = tool_text( @@ -25894,7 +26506,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/repo", "token")); + handler.bind_route(test_route(7), binding("/repo", "token")); for arguments in [ json!({"action": "write", "category": "CONSTRAINTS", "content": "first"}), @@ -25930,7 +26542,7 @@ mod tests { FakeSessionResolver::with(&[("token", FakeResolve::Hit("session".to_string()))]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding("/route/facade-ledger", "token")); + handler.bind_route(test_route(7), binding("/route/facade-ledger", "token")); let project = "/route/facade-ledger"; async fn call_and_replay( @@ -25943,7 +26555,7 @@ mod tests { ) { let first = call_facade(handler, name, arguments.clone()).await; let first_bytes = match first { - HandlerOutcome::Response(bytes) => bytes, + PreparedOutcome::Response(bytes) => bytes, other => panic!("first {name}/{action} failed: {other:?}"), }; let mut retry_arguments = arguments; @@ -25955,7 +26567,7 @@ mod tests { store .facade_mutation_ledger_response("session", name, action, command_id) .unwrap(), - Some(first_bytes), + Some(first_bytes.as_ref().to_vec()), "ledger must retain the exact first response for {name}/{action}" ); original_body["replayed"] = Value::Bool(true); @@ -26089,7 +26701,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_project_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_project_root, "token")); + handler.bind_route(test_route(7), binding(route_project_root, "token")); activate_module_authority( &store, "context", @@ -26150,7 +26762,7 @@ mod tests { ) .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("classification facade failed: {other:?}"), }; assert_eq!(response["accepted"], json!([fresh_id])); @@ -26178,7 +26790,7 @@ mod tests { handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_root = project.to_str().unwrap(); let identity = "git:dreamer-applies"; - handler.bind_route(7, binding(route_root, "token")); + handler.bind_route(test_route(7), binding(route_root, "token")); activate_module_authority(&store, "context", identity, route_root, "memories"); let verified_id = insert_memory(&store, identity, "CONSTRAINTS", "verified", 1); let updated_id = insert_memory(&store, identity, "CONSTRAINTS", "updated", 1); @@ -26202,7 +26814,7 @@ mod tests { ] })).await; let verified_body = match verified { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("verification facade failed: {other:?}"), }; assert_eq!(verified_body["accepted"], json!([verified_id])); @@ -26222,7 +26834,7 @@ mod tests { "command_id": "verify-once", "rows": [{"memory_id": verified_id, "content_hash_at_prompt": hash(verified_id), "verification_status": "verified"}] })).await; let replay_body = match replay { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("verification replay failed: {other:?}"), }; assert_eq!( @@ -26245,7 +26857,7 @@ mod tests { "rows": [{"memory_id": updated_id, "content_hash_at_prompt": hash(updated_id), "verification_status": "update", "updated_content": "updated by verifier"}] })).await; let update_body = match update { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("update facade failed: {other:?}"), }; let after_update = @@ -26262,7 +26874,7 @@ mod tests { "memory_project": identity, "context_store_uuid": "context", "authority_generation": generation, "rows": [{"memory_id": archived_id, "content_hash_at_prompt": hash(archived_id), "verification_status": "archive", "archive_reason": "obsolete"}] })).await; - assert!(matches!(archive, HandlerOutcome::Response(_))); + assert!(matches!(archive, PreparedOutcome::Response(_))); let after_archive = crate::m1_compose::m1_revision_signal(&store, identity, "session").unwrap(); assert!( @@ -26274,7 +26886,7 @@ mod tests { "memory_project": identity, "context_store_uuid": "context", "authority_generation": generation, "command_id": "mapping-once", "rows": [{"memory_id": verified_id, "content_hash_at_prompt": hash(verified_id), "mapped_files": ["src/lib.rs", "src/lib.rs"]}] })).await; - assert!(matches!(mapping, HandlerOutcome::Response(_))); + assert!(matches!(mapping, PreparedOutcome::Response(_))); let mapping_feed_head = store .pull_changefeed("memories", 0, 1000) .unwrap() @@ -26284,7 +26896,7 @@ mod tests { "command_id": "mapping-once", "rows": [{"memory_id": verified_id, "content_hash_at_prompt": "stale", "mapped_files": null}] })).await; assert!( - matches!(mapping_replay, HandlerOutcome::Response(_)), + matches!(mapping_replay, PreparedOutcome::Response(_)), "mapping command replay must be idempotent" ); assert_eq!( @@ -26327,7 +26939,7 @@ mod tests { handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_root = project.to_str().unwrap(); let identity = "git:classification-race"; - handler.bind_route(7, binding(route_root, "token")); + handler.bind_route(test_route(7), binding(route_root, "token")); activate_module_authority(&store, "context", identity, route_root, "memories"); let memory_id = insert_memory(&store, identity, "CONSTRAINTS", "classify me", 1); let before = store.get_memory_full(memory_id).unwrap().unwrap(); @@ -26408,7 +27020,7 @@ mod tests { handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_root = project.to_str().unwrap(); let identity = "git:draining"; - handler.bind_route(7, binding(route_root, "token")); + handler.bind_route(test_route(7), binding(route_root, "token")); for domain in ["memories", "notes"] { activate_module_authority(&store, "context", identity, route_root, domain); } @@ -26518,7 +27130,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); let route_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_root, "ses")); + handler.bind_route(test_route(7), binding(route_root, "ses")); activate_module_authority(&store, "context", "git:identity", route_root, "memories"); let generation = store .authority_status("context", "git:identity", "memories") @@ -26528,7 +27140,7 @@ mod tests { let outcome = handler .handle_dreamer_run_task( - 7, + test_route(7), &json!({ "v": 1, "session_id": "ses", @@ -26544,7 +27156,7 @@ mod tests { ) .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("dreamer run failed: {other:?}"), }; @@ -26587,7 +27199,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); let route_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_root, "parent")); + handler.bind_route(test_route(7), binding(route_root, "parent")); activate_module_authority(&store, "context", "git:identity", route_root, "memories"); let generation = store .authority_status("context", "git:identity", "memories") @@ -26601,7 +27213,7 @@ mod tests { let task = tokio::spawn(async move { running_handler .handle_dreamer_run_task( - 7, + test_route(7), &json!({ "v": 1, "session_id": "parent", @@ -26629,7 +27241,7 @@ mod tests { producer: &Arc, payload: Value, command_id: &str, - ) -> (Arc, HandlerOutcome) { + ) -> (Arc, PreparedOutcome) { let (handler, store, _dir, project) = handler_with_store(Arc::clone(producer), default_test_config()); let route_root = project.to_str().unwrap(); @@ -26637,7 +27249,7 @@ mod tests { // route config models. let mut route_binding = binding_with_harness(route_root, "pi", "ses"); route_binding.config.model_chain = vec!["test/route-only-model".to_string()]; - handler.bind_route(7, route_binding); + handler.bind_route(test_route(7), route_binding); activate_module_authority(&store, "context", "git:identity", route_root, "memories"); let generation = store .authority_status("context", "git:identity", "memories") @@ -26646,7 +27258,7 @@ mod tests { .generation; let outcome = handler .handle_dreamer_run_task( - 7, + test_route(7), &json!({ "v": 1, "session_id": "ses", @@ -26692,7 +27304,7 @@ mod tests { ) .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("the chain must recover from a capped attempt: {other:?}"), }; assert_eq!(response["ok"], json!(true)); @@ -26735,7 +27347,7 @@ mod tests { let (producer, outcome) = dreamer_classify_outcome(&producer, payload.clone(), "chain-shape").await; match outcome { - HandlerOutcome::Error { code, .. } => { + PreparedOutcome::Error { code, .. } => { assert_eq!(code, "invalid_params", "payload {payload}") } other => panic!("expected invalid_params for {payload}, got {other:?}"), @@ -26767,7 +27379,7 @@ mod tests { ) .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("dreamer run failed: {other:?}"), }; assert_eq!( @@ -26786,7 +27398,7 @@ mod tests { let producer = Arc::new(ProducerState::default()); producer.await_results.lock().unwrap().extend([ // provider failure - Err(HistorianProducerError::tagged_subc( + Err(HistorianProducerError::tagged_call( "provider_error", "boom", ErrorClass::Transient, @@ -26823,7 +27435,7 @@ mod tests { ) .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("dreamer run failed: {other:?}"), }; assert_eq!(response["diagnostics"]["attempts"], json!(5)); @@ -26878,7 +27490,7 @@ mod tests { // recorded success anyway; the leftover session is bounded by host // terminal retention. let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("a recorded success must survive a cleanup failure: {other:?}"), }; assert_eq!(response["ok"], json!(true)); @@ -26901,7 +27513,7 @@ mod tests { .await_results .lock() .unwrap() - .push_back(Err(HistorianProducerError::tagged_subc( + .push_back(Err(HistorianProducerError::tagged_call( "provider_error", "model gone", ErrorClass::Permanent, @@ -26922,7 +27534,7 @@ mod tests { "cleanup-with-failure", ) .await; - let HandlerOutcome::Error { code, message } = outcome else { + let PreparedOutcome::Error { code, message } = outcome else { panic!("expected dreamer_run_failed"); }; assert_eq!(code, "dreamer_run_failed"); @@ -26936,7 +27548,7 @@ mod tests { async fn dreamer_run_task_backs_off_on_idempotency_conflict_without_purging() { let producer = Arc::new(ProducerState::default()); producer.await_results.lock().unwrap().extend([ - Err(HistorianProducerError::tagged_subc( + Err(HistorianProducerError::tagged_call( "idempotency_conflict", "a byte-different send holds this session", ErrorClass::Permanent, @@ -26947,7 +27559,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); let route_root = project.to_str().unwrap(); - handler.bind_route(7, binding_with_harness(route_root, "pi", "ses")); + handler.bind_route(test_route(7), binding_with_harness(route_root, "pi", "ses")); activate_module_authority(&store, "context", "git:identity", route_root, "memories"); let generation = store .authority_status("context", "git:identity", "memories") @@ -26967,8 +27579,10 @@ mod tests { }, }); - let outcome = handler.handle_dreamer_run_task(7, &request).await; - let HandlerOutcome::Error { code, .. } = outcome else { + let outcome = handler + .handle_dreamer_run_task(test_route(7), &request) + .await; + let PreparedOutcome::Error { code, .. } = outcome else { panic!("an idempotency conflict must fail the command"); }; assert_eq!(code, "dreamer_run_failed"); @@ -26989,9 +27603,11 @@ mod tests { // ledger's INSERT OR IGNORE race and mask the in-flight winner's // outcome. The command's slot stays open, so a later attempt can // still commit success. - let outcome = handler.handle_dreamer_run_task(7, &request).await; + let outcome = handler + .handle_dreamer_run_task(test_route(7), &request) + .await; let response = match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("the command slot must remain open after a conflict: {other:?}"), }; assert_eq!(response["ok"], json!(true)); @@ -27024,7 +27640,7 @@ mod tests { "budget-exhausted", ) .await; - let HandlerOutcome::Error { code, message } = outcome else { + let PreparedOutcome::Error { code, message } = outcome else { panic!("expected dreamer_run_failed"); }; assert_eq!(code, "dreamer_run_failed"); @@ -27051,7 +27667,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); handler.bind_route( - 7, + test_route(7), binding_with_harness(project.to_str().unwrap(), "pi", "ses"), ); @@ -27069,7 +27685,7 @@ mod tests { let (handler, _store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); handler.bind_route( - 7, + test_route(7), binding_with_harness(project.to_str().unwrap(), "opencode", "ses"), ); cache_wrapup_messages(&handler, wrapup_messages(80, 800)); @@ -27077,7 +27693,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -27103,7 +27719,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); handler.bind_route( - 7, + test_route(7), binding_with_harness(project.to_str().unwrap(), "pi", "ses"), ); let messages = big_messages(); @@ -27132,7 +27748,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store(Arc::clone(&producer), default_test_config()); handler.bind_route( - 7, + test_route(7), binding_with_harness(project.to_str().unwrap(), "pi", "ses"), ); let messages = big_messages(); @@ -27154,7 +27770,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_project_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_project_root, "token")); + handler.bind_route(test_route(7), binding(route_project_root, "token")); activate_module_authority( &store, "context", @@ -27382,7 +27998,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_project_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_project_root, "token")); + handler.bind_route(test_route(7), binding(route_project_root, "token")); activate_module_authority( &store, "context", @@ -27417,7 +28033,7 @@ mod tests { let (handler, store, _dir, project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); let route_project_root = project.to_str().unwrap(); - handler.bind_route(7, binding(route_project_root, "token")); + handler.bind_route(test_route(7), binding(route_project_root, "token")); let outcome = call_facade( &handler, @@ -27446,7 +28062,7 @@ mod tests { handler_with_store_and_resolver(producer, config, resolver); let mut disabled_binding = binding("/repo", "token"); disabled_binding.config.memory_enabled = false; - handler.bind_route(7, disabled_binding); + handler.bind_route(test_route(7), disabled_binding); insert_memory(&store, "/repo", "CONSTRAINTS", "hidden needle", 1); assert!(tool_is_error( @@ -27473,7 +28089,7 @@ mod tests { )]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding(own, "token")); + handler.bind_route(test_route(7), binding(own, "token")); seed_workspace(&store, own, foreign); let foreign_private_update = @@ -27575,8 +28191,8 @@ mod tests { ]); let (handler, store, _dir, _project) = handler_with_store_and_resolver(producer, default_test_config(), resolver); - handler.bind_route(7, binding(project_root, "token")); - handler.bind_route(8, binding(project_root, scope)); + handler.bind_route(test_route(7), binding(project_root, "token")); + handler.bind_route(test_route(8), binding(project_root, scope)); store .replace_compartments(scope, &[stored_comp(1, 1, 10, "m10", "SUMMARY")]) .unwrap(); @@ -27612,7 +28228,10 @@ mod tests { assert!(synthetic_text(&update_delta, 1).contains("")); assert!(synthetic_text(&update_delta, 1).contains("updated rule")); - handler.bind_route(9, binding(additive_project_root, additive_scope)); + handler.bind_route( + test_route(9), + binding(additive_project_root, additive_scope), + ); store .replace_compartments( additive_scope, @@ -27626,7 +28245,10 @@ mod tests { let add_before = store .max_memory_mutation_id(&[additive_project_root.to_string()]) .unwrap(); - handler.bind_route(7, binding(additive_project_root, "token-additive")); + handler.bind_route( + test_route(7), + binding(additive_project_root, "token-additive"), + ); let resolver_scope_memory = call_facade( &handler, "ctx_memory", @@ -27774,7 +28396,7 @@ mod tests { "kind": "status", "session_id": "ses", })) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("expected status response, got {other:?}"), }; assert!(current_status["pass_trace"]["first_divergence"].is_string()); @@ -27786,7 +28408,7 @@ mod tests { "kind": "status", "session_id": "ses", })) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("expected status response, got {other:?}"), }; assert!(stable_status["pass_trace"]["first_divergence"].is_null()); @@ -27813,7 +28435,7 @@ mod tests { "kind": "status", "session_id": "ses", })) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("expected status response, got {other:?}"), }; let second_historical: Value = serde_json::from_str( @@ -27839,7 +28461,7 @@ mod tests { assert!(stable_again.get("first_divergence").is_none()); let session_status = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); assert!(session_status["pass_trace"]["first_divergence"].is_null()); @@ -27880,7 +28502,7 @@ mod tests { .unwrap(); let status = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, @@ -27902,8 +28524,8 @@ mod tests { let (handler, store, _dir, _project) = handler_with_store(producer, default_test_config()); let live_version = store.module_store_schema_version().unwrap(); - let decode = |outcome: HandlerOutcome| match outcome { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + let decode = |outcome: PreparedOutcome| match outcome { + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("expected status response, got {other:?}"), }; @@ -28054,7 +28676,7 @@ mod tests { parsed.serializer_profile = SerializerProfile::ClaudeCodeAnthropic.wire_id().to_string(); let retained_bytes = serde_json::to_vec(&parsed).unwrap().len(); let projection = crate::ck_wire::project_messages(&parsed.messages).unwrap(); - let store = handler.store.get().unwrap(); + let store = handler.store().unwrap(); let loaded = store.load(session_id).unwrap(); let mut meta = loaded.meta.clone(); meta.block_identity_by_mid @@ -28122,7 +28744,7 @@ mod tests { fn queue_drop_command_with_id(handler: &McHandler, command_id: &str) -> Value { match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -28130,7 +28752,7 @@ mod tests { "command_id": command_id, }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), } } @@ -28267,7 +28889,7 @@ mod tests { "request budget must reach the HARD decay renderer: {m0}" ); let status = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); let history = decay_render::extract_m0_block(&m0, "session-history").unwrap(); @@ -28290,7 +28912,7 @@ mod tests { .commit("ses", loaded.row_version, &loaded.core, &meta) .unwrap(); let degraded = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); assert_eq!(degraded["historian"]["consecutive_publish_failures"], 3); @@ -28307,7 +28929,7 @@ mod tests { .commit("ses", loaded.row_version, &loaded.core, &meta) .unwrap(); let recovered = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); assert_eq!(recovered["historian"]["consecutive_publish_failures"], 0); @@ -28496,7 +29118,7 @@ mod tests { let response = handler .handle_transform_for_test( - 7, + test_route(7), request(vec![ ck("ccm-0", 0, "first user message"), ck("ccm-1", 1, "assistant reply"), @@ -28532,7 +29154,7 @@ mod tests { let rejected = handler .handle_transform_for_test( - 7, + test_route(7), request(vec![ck("m1", 1, "actual anchor"), ck("m11", 11, "tail")]), ) .await; @@ -28607,7 +29229,7 @@ mod tests { let outcome = handler .dispatch_value( - 7, + test_route(7), state_import_request( "bundle-a", 0, @@ -28650,7 +29272,7 @@ mod tests { let different = handler .dispatch_value( - 7, + test_route(7), state_import_request( "bundle-b", 0, @@ -28684,7 +29306,7 @@ mod tests { assert_eq!(staged["staged"], 1); let gap = handler .dispatch_value( - 7, + test_route(7), state_import_request( "gap", 2, @@ -28770,7 +29392,10 @@ mod tests { ]; for (import_id, compartments, expected_code) in cases { let outcome = handler - .dispatch_value(7, state_import_request(import_id, 0, 1, compartments)) + .dispatch_value( + test_route(7), + state_import_request(import_id, 0, 1, compartments), + ) .await; assert_eq!(error_code(outcome), expected_code, "{import_id}"); assert!(store.load_compartments("ses").unwrap().is_empty()); @@ -28820,7 +29445,7 @@ mod tests { for alias in ["ctx_reduce", "append_agent_drops"] { let outcome = handler .dispatch_value( - 7, + test_route(7), json!({ "method": alias, "session_id": "ses", @@ -28934,7 +29559,7 @@ mod tests { })); let body = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, @@ -28952,7 +29577,7 @@ mod tests { assert!(compartments[0].get("legacy").is_none()); let tail = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, @@ -29057,7 +29682,7 @@ mod tests { let request = json!({ "method": "session.status", "v": 1, "session_id": "ses" }); assert_eq!( - error_code(handler.handle_session_status_value(8, &request)), + error_code(handler.handle_session_status_value(test_route(8), &request)), "route_unbound" ); let mismatch = json!({ @@ -29066,7 +29691,7 @@ mod tests { "session_id": "other", }); assert_eq!( - error_code(handler.handle_session_status_value(7, &mismatch)), + error_code(handler.handle_session_status_value(test_route(7), &mismatch)), "session_mismatch" ); } @@ -29076,7 +29701,10 @@ mod tests { let producer = Arc::new(ProducerState::default()); let (handler, store, _dir, project) = handler_with_store(producer, default_test_config()); let session_id = "ccm-8518e338-extra"; - handler.bind_route(7, binding(project.to_str().unwrap(), session_id)); + handler.bind_route( + test_route(7), + binding(project.to_str().unwrap(), session_id), + ); store .commit_state_import( session_id, @@ -29129,7 +29757,7 @@ mod tests { .unwrap(); let body = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": session_id }), )); let summary = body["summary"].as_str().unwrap(); @@ -29154,7 +29782,10 @@ mod tests { let producer = Arc::new(ProducerState::default()); let (handler, store, _dir, project) = handler_with_store(producer, default_test_config()); let session_id = "ses-delete"; - handler.bind_route(7, binding(project.to_str().unwrap(), session_id)); + handler.bind_route( + test_route(7), + binding(project.to_str().unwrap(), session_id), + ); store .commit( session_id, @@ -29187,7 +29818,7 @@ mod tests { .unwrap(); let deleted = tool_body(handler.handle_session_delete_value( - 7, + test_route(7), &json!({ "method": "session.delete", "v": 1, "session_id": session_id }), )); assert_eq!(deleted["ok"], json!(true)); @@ -29233,7 +29864,7 @@ mod tests { })); let body = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); assert_eq!(body["wrapup_active"], json!(true)); @@ -29254,7 +29885,7 @@ mod tests { .unwrap(); let body = tool_body(handler.handle_session_status_value( - 7, + test_route(7), &json!({ "method": "session.status", "v": 1, "session_id": "ses" }), )); let summary = body["summary"].as_str().unwrap(); @@ -29270,7 +29901,7 @@ mod tests { mint_drop_tag(&store, "a#0"); let outcome = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29323,7 +29954,7 @@ mod tests { assert!(store.load_pending_agent_drops("ses").unwrap().is_empty()); let outcome = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29372,7 +30003,7 @@ mod tests { assert_eq!(store.load_tags_for_session("ses").unwrap().len(), 25); let queued = match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29380,7 +30011,7 @@ mod tests { "command_id": "opencode-drop-range", }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }; assert_eq!(queued, json!({ "ok": true, "queued": 3 })); @@ -29428,7 +30059,7 @@ mod tests { for command_id in [json!(""), json!(" \t "), json!("x".repeat(129))] { let outcome = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29447,7 +30078,7 @@ mod tests { let (handler, store, _dir, _project) = handler_with_store(producer, default_test_config()); for drop in [Value::Null, json!(""), json!(" "), json!(["1"])] { let outcome = handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29488,7 +30119,7 @@ mod tests { // Range syntax plus an unknown tag number: known tags queue, the unknown // number is skipped (the tee replays whatever the model said). let response = match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29496,7 +30127,7 @@ mod tests { "command_id": "raw-1", }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }; assert_eq!(response, json!({ "ok": true, "queued": 2 })); @@ -29505,7 +30136,7 @@ mod tests { // Re-sending the same raw string is idempotent (structural INSERT OR IGNORE). let repeat = match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29513,14 +30144,14 @@ mod tests { "command_id": "raw-2", }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }; assert_eq!(repeat, json!({ "ok": true, "queued": 0 })); // Malformed range syntax is a typed bad_request, nothing partially queued. match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29528,7 +30159,7 @@ mod tests { "command_id": "raw-bad", }), ) { - HandlerOutcome::Error { code, .. } => assert_eq!(code, "bad_request"), + PreparedOutcome::Error { code, .. } => assert_eq!(code, "bad_request"), other => panic!("expected bad_request, got: {other:?}"), } assert_eq!(store.load_pending_agent_drops("ses").unwrap().len(), 2); @@ -29552,7 +30183,7 @@ mod tests { .unwrap(); let first = match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29560,13 +30191,13 @@ mod tests { "command_id": " tool-use-raw ", }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }; assert_eq!(first, json!({ "ok": true, "queued": 1 })); let retry = match handler.handle_agent_drops_value( - 7, + test_route(7), json!({ "method": "agent_drops.append", "session_id": "ses", @@ -29574,7 +30205,7 @@ mod tests { "command_id": "tool-use-raw", }), ) { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + PreparedOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }; assert_eq!(retry, json!({ "ok": true, "queued": 0, "duplicate": true })); @@ -29592,7 +30223,11 @@ mod tests { "session_id": "ses", "command_id": "missing-retry" }); - let missing = tool_body(handler.dispatch_value(7, missing_request.clone()).await); + let missing = tool_body( + handler + .dispatch_value(test_route(7), missing_request.clone()) + .await, + ); assert_eq!(missing["disposition"], json!("retryable")); assert_eq!(missing["reason"], json!("snapshot_unavailable")); assert!(store @@ -29600,7 +30235,7 @@ mod tests { .unwrap() .is_none()); cache_wrapup_messages(&handler, wrapup_messages(20, 40)); - let retry = tool_body(handler.dispatch_value(7, missing_request).await); + let retry = tool_body(handler.dispatch_value(test_route(7), missing_request).await); assert_eq!(retry["disposition"], json!("nothing_to_compact")); assert!(store .load_wrapup_command("ses", "missing-retry") @@ -29626,8 +30261,11 @@ mod tests { "session_id": "ses", "command_id": "malformed-retry" }); - let malformed_response = - tool_body(handler.dispatch_value(7, malformed_request.clone()).await); + let malformed_response = tool_body( + handler + .dispatch_value(test_route(7), malformed_request.clone()) + .await, + ); assert_eq!(malformed_response["disposition"], json!("retryable")); assert_eq!(malformed_response["reason"], json!("snapshot_unavailable")); assert!(store @@ -29635,7 +30273,11 @@ mod tests { .unwrap() .is_none()); cache_wrapup_messages(&handler, wrapup_messages(20, 40)); - let retry = tool_body(handler.dispatch_value(7, malformed_request).await); + let retry = tool_body( + handler + .dispatch_value(test_route(7), malformed_request) + .await, + ); assert_eq!(retry["disposition"], json!("nothing_to_compact")); assert!(store .load_wrapup_command("ses", "malformed-retry") @@ -29657,7 +30299,7 @@ mod tests { "command_id": "no-models" }); - let response = tool_body(handler.dispatch_value(7, request.clone()).await); + let response = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(response["ok"], json!(false), "{response}"); assert_eq!(response["disposition"], json!("failed"), "{response}"); assert_eq!(response["reason"], json!("no_models"), "{response}"); @@ -29677,7 +30319,7 @@ mod tests { assert_eq!(row.rounds, response["rounds"].as_u64().unwrap() as usize); assert_eq!(producer.starts.load(Ordering::SeqCst), 0); - let replay = tool_body(handler.dispatch_value(7, request).await); + let replay = tool_body(handler.dispatch_value(test_route(7), request).await); assert_eq!(replay["ok"], json!(false), "{replay}"); assert_eq!(replay["reason"], json!("no_models"), "{replay}"); assert_eq!(replay["replayed"], json!(true)); @@ -29692,14 +30334,16 @@ mod tests { .lock() .expect("start errors mutex") .extend([ - Err(HistorianProducerError::Subc( - historian_producer::ProducerErrorBody::untagged( + Err(HistorianProducerError::Call( + historian_producer::HistorianCallFailure::untagged( + historian_producer::HistorianSendOutcome::Terminal, "unknown_module", "runner module broca is unavailable", ), )), - Err(HistorianProducerError::Subc( - historian_producer::ProducerErrorBody::untagged( + Err(HistorianProducerError::Call( + historian_producer::HistorianCallFailure::untagged( + historian_producer::HistorianSendOutcome::Terminal, "unknown_module", "runner module broca is unavailable", ), @@ -29719,7 +30363,7 @@ mod tests { "command_id": "runner-missing" }); - let response = tool_body(handler.dispatch_value(7, request.clone()).await); + let response = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(response["ok"], json!(false), "{response}"); assert_eq!(response["disposition"], json!("failed"), "{response}"); assert_eq!( @@ -29745,7 +30389,7 @@ mod tests { "failed" ); - let replay = tool_body(handler.dispatch_value(7, request).await); + let replay = tool_body(handler.dispatch_value(test_route(7), request).await); assert_eq!(replay["replayed"], json!(true)); assert_eq!(replay["reason"], json!("runner_module_unavailable")); assert_eq!(producer.starts.load(Ordering::SeqCst), 2); @@ -29761,7 +30405,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -29790,7 +30434,7 @@ mod tests { let retry = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -29817,7 +30461,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -29855,7 +30499,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -29885,7 +30529,7 @@ mod tests { let large = tool_body( handler_large .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -29929,7 +30573,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -29973,7 +30617,7 @@ mod tests { let response = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -30002,7 +30646,7 @@ mod tests { // one durable ledger key. let empty_id = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30012,7 +30656,7 @@ mod tests { ) .await; match empty_id { - HandlerOutcome::Error { code, message } => { + PreparedOutcome::Error { code, message } => { assert_eq!(code, "bad_request"); assert!(message.contains("nonempty"), "{message}"); } @@ -30027,7 +30671,7 @@ mod tests { let negative_keep = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30065,7 +30709,7 @@ mod tests { let busy = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -30092,7 +30736,7 @@ mod tests { "command_id": "wrapup-one" }); - let first = tool_body(handler.dispatch_value(7, request.clone()).await); + let first = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(first["disposition"], json!("completed"), "{first}"); let starts = producer.starts.load(Ordering::SeqCst); let stored = store @@ -30105,7 +30749,7 @@ mod tests { let mut retry = request; retry["keep"] = json!("ignored-on-replay"); - let replay = tool_body(handler.dispatch_value(7, retry).await); + let replay = tool_body(handler.dispatch_value(test_route(7), retry).await); assert_eq!(replay["replayed"], json!(true)); assert_eq!(replay["disposition"], first["disposition"]); assert_eq!(replay["rounds"], first["rounds"]); @@ -30115,7 +30759,7 @@ mod tests { let different = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30158,7 +30802,7 @@ mod tests { "command_id": "legacy-failed" }); - let first = tool_body(handler.dispatch_value(7, request.clone()).await); + let first = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_ne!(first.get("replayed"), Some(&json!(true)), "{first}"); assert_eq!(first["disposition"], json!("completed"), "{first}"); let starts_after_lost_response = producer.starts.load(Ordering::SeqCst); @@ -30174,7 +30818,7 @@ mod tests { // Simulate losing the first response. The same command id must replay the // durable terminal result without opening another producer run. - let replay = tool_body(handler.dispatch_value(7, request).await); + let replay = tool_body(handler.dispatch_value(test_route(7), request).await); assert_eq!(replay["replayed"], json!(true), "{replay}"); assert_eq!(replay["disposition"], first["disposition"]); assert_eq!(replay["rounds"], first["rounds"]); @@ -30192,10 +30836,12 @@ mod tests { .connect_errors .lock() .expect("connect errors mutex") - .push_back(HistorianProducerError::Connect { - endpoint: "127.0.0.1:1".to_string(), - source: std::io::Error::new(std::io::ErrorKind::ConnectionRefused, "refused"), - }); + .push_back(HistorianProducerError::Client( + historian_producer::HistorianClientFailure { + code: "dial_failed".to_owned(), + message: "daemon dial failed".to_owned(), + }, + )); let (handler, store, _dir, _project) = handler_with_store(Arc::clone(&producer), default_test_config()); cache_wrapup_messages(&handler, wrapup_messages(80, 800)); @@ -30213,7 +30859,7 @@ mod tests { let response = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30246,10 +30892,12 @@ mod tests { .connect_errors .lock() .expect("connect errors mutex") - .push_back(HistorianProducerError::Connect { - endpoint: "127.0.0.1:1".to_string(), - source: std::io::Error::new(std::io::ErrorKind::ConnectionRefused, "refused"), - }); + .push_back(HistorianProducerError::Client( + historian_producer::HistorianClientFailure { + code: "dial_failed".to_owned(), + message: "daemon dial failed".to_owned(), + }, + )); let (handler, store, _dir, _project) = handler_with_store(Arc::clone(&producer), default_test_config()); cache_wrapup_messages(&handler, wrapup_messages(80, 800)); @@ -30271,7 +30919,7 @@ mod tests { let response = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30320,7 +30968,7 @@ mod tests { "command_id": "recut-retry" }); - let stale = tool_body(handler.dispatch_value(7, request.clone()).await); + let stale = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(stale["disposition"], json!("retryable"), "{stale}"); assert_eq!(stale["reason"], json!("snapshot_stale")); assert!(store @@ -30335,7 +30983,7 @@ mod tests { store .commit("ses", loaded.row_version, &loaded.core, &meta) .unwrap(); - let retry = tool_body(handler.dispatch_value(7, request).await); + let retry = tool_body(handler.dispatch_value(test_route(7), request).await); assert!(matches!( retry["disposition"].as_str(), Some("completed" | "nothing_to_compact") @@ -30389,7 +31037,7 @@ mod tests { let response = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30441,7 +31089,7 @@ mod tests { let response = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 5 }), ) .await, @@ -30502,7 +31150,7 @@ mod tests { let stale = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 5 }), ) .await, @@ -30523,7 +31171,7 @@ mod tests { let retry = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 5 }), ) .await, @@ -30572,7 +31220,7 @@ mod tests { let stale = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 5 }), ) .await, @@ -30599,7 +31247,7 @@ mod tests { let retry = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 5 }), ) .await, @@ -30624,7 +31272,7 @@ mod tests { let session_id = format!("lease-{index}"); let channel = 20 + index as u16; handler.bind_route( - channel, + test_route(channel), binding(project.to_str().unwrap(), session_id.as_str()), ); cache_wrapup_messages_for_session( @@ -30648,7 +31296,7 @@ mod tests { blocked.push(tokio::spawn(async move { handler .dispatch_value( - 20 + index as u16, + test_route(20 + index as u16), json!({ "method": "session.wrapup", "v": 1, @@ -30664,7 +31312,7 @@ mod tests { let overflow = tokio::time::timeout( Duration::from_millis(100), handler.dispatch_value( - 20 + ACTIVE_LEASE_LIMIT as u16, + test_route(20 + ACTIVE_LEASE_LIMIT as u16), json!({ "method": "session.wrapup", "v": 1, @@ -30691,7 +31339,7 @@ mod tests { let later = tool_body( handler .dispatch_value( - 20 + ACTIVE_LEASE_LIMIT as u16, + test_route(20 + ACTIVE_LEASE_LIMIT as u16), json!({ "method": "session.wrapup", "v": 1, @@ -30729,7 +31377,7 @@ mod tests { let first = tokio::spawn(async move { first_handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 1 }), ) .await @@ -30739,7 +31387,7 @@ mod tests { let second = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30771,7 +31419,7 @@ mod tests { let later = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30804,7 +31452,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 1 }), ) .await, @@ -30833,7 +31481,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses", "keep": 1 }), ) .await, @@ -30890,7 +31538,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -30919,7 +31567,7 @@ mod tests { let mismatch = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30943,7 +31591,7 @@ mod tests { let retry = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, @@ -30968,7 +31616,7 @@ mod tests { let evicted = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -31004,7 +31652,7 @@ mod tests { "keep": 5, "command_id": "budget-retry" }); - let body = tool_body(handler.dispatch_value(7, request.clone()).await); + let body = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(body["disposition"], json!("retryable"), "{body}"); assert_eq!(body["reason"], json!("budget_exhausted")); assert!(store @@ -31024,7 +31672,7 @@ mod tests { .wrapup_operation_budget .lock() .expect("wrapup operation budget mutex") = None; - let retry = tool_body(handler.dispatch_value(7, request).await); + let retry = tool_body(handler.dispatch_value(test_route(7), request).await); assert!(matches!( retry["disposition"].as_str(), Some("completed" | "nothing_to_compact") @@ -31053,7 +31701,7 @@ mod tests { "session_id": "ses", "command_id": "backoff-retry" }); - let body = tool_body(handler.dispatch_value(7, request.clone()).await); + let body = tool_body(handler.dispatch_value(test_route(7), request.clone()).await); assert_eq!(body["disposition"], json!("retryable")); assert_eq!(body["reason"], json!("backoff_active")); assert!(body["summary"] @@ -31073,7 +31721,7 @@ mod tests { store .commit("ses", loaded.row_version, &loaded.core, &meta) .unwrap(); - let retry = tool_body(handler.dispatch_value(7, request).await); + let retry = tool_body(handler.dispatch_value(test_route(7), request).await); assert_eq!(retry["disposition"], json!("nothing_to_compact")); assert!(store .load_wrapup_command("ses", "backoff-retry") @@ -31101,7 +31749,7 @@ mod tests { let body = tool_body( handler .dispatch_value( - 7, + test_route(7), json!({ "method": "session.wrapup", "v": 1, "session_id": "ses" }), ) .await, @@ -31867,10 +32515,12 @@ mod tests { .connect_errors .lock() .unwrap() - .push_back(HistorianProducerError::Connect { - endpoint: "127.0.0.1:1".to_string(), - source: std::io::Error::new(std::io::ErrorKind::ConnectionRefused, "refused"), - }); + .push_back(HistorianProducerError::Client( + historian_producer::HistorianClientFailure { + code: "dial_failed".to_owned(), + message: "daemon dial failed".to_owned(), + }, + )); let (handler, store, _dir, _project) = handler_with_store(Arc::clone(&producer), default_test_config()); let messages = big_messages(); @@ -31948,8 +32598,8 @@ mod tests { }), config, ); - handler2.store.set(Arc::clone(&store)).ok().unwrap(); - handler2.bind_route(7, binding(_project.to_str().unwrap(), "ses")); + handler2.install_store_for_test(Arc::clone(&store)); + handler2.bind_route(test_route(7), binding(_project.to_str().unwrap(), "ses")); let fired = call_transform(&handler2, messages).await; assert_eq!(fired["historian"]["fired"], true); // The clearing write happens in the spawned firing's persist. Durable state is @@ -31970,7 +32620,7 @@ mod tests { let (handler, store, _dir, _project) = handler_with_store(Arc::clone(&producer), default_test_config()); let session = historian::historian_producer_session_id("proj", "parent-session", 3); - handler.bind_route(9, binding("/tmp/nonexistent-proj", &session)); + handler.bind_route(test_route(9), binding("/tmp/nonexistent-proj", &session)); let messages = [ck("m1", 1, "seed block + new_messages payload")]; let req = serde_json::json!({ "kind": "transform", @@ -31980,8 +32630,10 @@ mod tests { "render_config": "cfg0", "messages": messages.iter().map(|m| serde_json::to_value(m).unwrap()).collect::>(), }); - let out = handler.handle_transform_dispatch(9, req, None).await; - let HandlerOutcome::Response(bytes) = out else { + let out = handler + .handle_transform_dispatch(test_route(9), req, None) + .await; + let PreparedOutcome::Response(bytes) = out else { panic!("pass-through must be a response"); }; let v: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); @@ -32039,7 +32691,10 @@ mod tests { assert!(progress["eligible_chunk_tokens"].is_number()); let req: TransformRequest = serde_json::from_value(request(messages)).unwrap(); - let project_path = handler.resolve_binding(7, "ses").unwrap().project_root; + let project_path = handler + .resolve_binding(test_route(7), "ses") + .unwrap() + .project_root; let project_path_string = project_path.to_string_lossy().to_string(); let response_without_historian = transform::transform( &store, @@ -32398,10 +33053,10 @@ mod tests { full["native_messages"] = json!([]); let full_body_bytes = serde_json::to_vec(&full).unwrap().len(); let outcome = handler - .handle_transform_for_test_with_body_size(7, full, full_body_bytes) + .handle_transform_for_test_with_body_size(test_route(7), full, full_body_bytes) .await; assert!( - matches!(outcome, HandlerOutcome::Response(_)), + matches!(outcome, PreparedOutcome::Response(_)), "{outcome:?}" ); let full_charge = { @@ -32427,10 +33082,10 @@ mod tests { }); let delta_body_bytes = serde_json::to_vec(&first_delta).unwrap().len(); let outcome = handler - .handle_transform_for_test_with_body_size(7, first_delta, delta_body_bytes) + .handle_transform_for_test_with_body_size(test_route(7), first_delta, delta_body_bytes) .await; assert!( - matches!(outcome, HandlerOutcome::Response(_)), + matches!(outcome, PreparedOutcome::Response(_)), "{outcome:?}" ); let expanded_charge = { @@ -32465,10 +33120,14 @@ mod tests { }); let second_delta_body_bytes = serde_json::to_vec(&second_delta).unwrap().len(); let outcome = handler - .handle_transform_for_test_with_body_size(7, second_delta, second_delta_body_bytes) + .handle_transform_for_test_with_body_size( + test_route(7), + second_delta, + second_delta_body_bytes, + ) .await; assert!( - matches!(outcome, HandlerOutcome::Response(_)), + matches!(outcome, PreparedOutcome::Response(_)), "{outcome:?}" ); assert!(matches!( @@ -32498,7 +33157,7 @@ mod tests { let rejected = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32516,7 +33175,7 @@ mod tests { // the rows that could invalidate the producer's compartment snapshot. let metadata_only = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32526,13 +33185,13 @@ mod tests { }), ) .await; - assert!(matches!(metadata_only, HandlerOutcome::Response(_))); + assert!(matches!(metadata_only, PreparedOutcome::Response(_))); assert_eq!(store.load("ses").unwrap().meta.shadow_seq, 1); seed_idle(&store); let retried = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32542,7 +33201,7 @@ mod tests { }), ) .await; - assert!(matches!(retried, HandlerOutcome::Response(_))); + assert!(matches!(retried, PreparedOutcome::Response(_))); assert_eq!(store.load_compartments("ses").unwrap().len(), 1); } @@ -32560,7 +33219,7 @@ mod tests { let rejected = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32596,23 +33255,22 @@ mod tests { }; let (first, second) = tokio::join!( - handler.dispatch_value(7, request()), - handler.dispatch_value(7, request()), + handler.dispatch_value(test_route(7), request()), + handler.dispatch_value(test_route(7), request()), ); let outcomes = [first, second]; assert_eq!( outcomes .iter() - .filter(|outcome| matches!(outcome, HandlerOutcome::Response(_))) + .filter(|outcome| matches!(outcome, PreparedOutcome::Response(_))) .count(), 1 ); let errors = outcomes .into_iter() .filter_map(|outcome| match outcome { - HandlerOutcome::Error { code, .. } - | HandlerOutcome::ErrorWithDetail { code, .. } => Some(code), - HandlerOutcome::Response(_) | HandlerOutcome::Streamed => None, + PreparedOutcome::Error { code, .. } => Some(code), + PreparedOutcome::Response(_) | PreparedOutcome::Streamed => None, }) .collect::>(); assert_eq!(errors, vec!["authority_seq_mismatch"]); @@ -32634,7 +33292,7 @@ mod tests { let mismatched = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32662,7 +33320,7 @@ mod tests { let absent = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32682,12 +33340,10 @@ mod tests { }), ) .await; - assert!(matches!(absent, HandlerOutcome::Response(_)), "{absent:?}"); + assert!(matches!(absent, PreparedOutcome::Response(_)), "{absent:?}"); let absent_response = match &absent { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(bytes).unwrap(), - HandlerOutcome::Error { .. } - | HandlerOutcome::ErrorWithDetail { .. } - | HandlerOutcome::Streamed => Value::Null, + PreparedOutcome::Response(bytes) => serde_json::from_slice::(bytes).unwrap(), + PreparedOutcome::Error { .. } | PreparedOutcome::Streamed => Value::Null, }; assert_eq!(absent_response["memories_skipped"], json!(true)); assert!(store @@ -32709,7 +33365,7 @@ mod tests { let workspace = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32739,12 +33395,10 @@ mod tests { }), ) .await; - assert!(matches!(workspace, HandlerOutcome::Response(_))); + assert!(matches!(workspace, PreparedOutcome::Response(_))); let workspace_response = match &workspace { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(bytes).unwrap(), - HandlerOutcome::Error { .. } - | HandlerOutcome::ErrorWithDetail { .. } - | HandlerOutcome::Streamed => Value::Null, + PreparedOutcome::Response(bytes) => serde_json::from_slice::(bytes).unwrap(), + PreparedOutcome::Error { .. } | PreparedOutcome::Streamed => Value::Null, }; assert_eq!(workspace_response["memories_skipped"], json!(true)); assert!(store @@ -32769,7 +33423,7 @@ mod tests { let present = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -32788,14 +33442,14 @@ mod tests { ) .await; assert!( - matches!(present, HandlerOutcome::Response(_)), + matches!(present, PreparedOutcome::Response(_)), "{present:?}" ); let before_absent = store.workspace_fingerprint(&project_path, 0).unwrap(); let absent = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -32805,7 +33459,7 @@ mod tests { }), ) .await; - assert!(matches!(absent, HandlerOutcome::Response(_)), "{absent:?}"); + assert!(matches!(absent, PreparedOutcome::Response(_)), "{absent:?}"); assert_eq!( store.workspace_fingerprint(&project_path, 0).unwrap(), before_absent, @@ -32820,7 +33474,7 @@ mod tests { let explicit_empty = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -32833,7 +33487,7 @@ mod tests { ) .await; assert!( - matches!(explicit_empty, HandlerOutcome::Response(_)), + matches!(explicit_empty, PreparedOutcome::Response(_)), "{explicit_empty:?}" ); assert_ne!( @@ -32850,7 +33504,7 @@ mod tests { let project_path = project.to_string_lossy().to_string(); let sync = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32875,7 +33529,7 @@ mod tests { }), ) .await; - assert!(matches!(sync, HandlerOutcome::Response(_))); + assert!(matches!(sync, PreparedOutcome::Response(_))); let response = call_transform_request(&handler, request(vec![ck("m0", 0, "live authority input")])) @@ -32901,7 +33555,7 @@ mod tests { let call_id = pair.call_id.clone(); let seeded = handler .dispatch_value( - 7, + test_route(7), json!({ "method": "state_sync", "session_id": "ses", @@ -32917,7 +33571,7 @@ mod tests { }), ) .await; - assert!(matches!(seeded, HandlerOutcome::Response(_)), "{seeded:?}"); + assert!(matches!(seeded, PreparedOutcome::Response(_)), "{seeded:?}"); assert!(store.load("ses").unwrap().meta.synthetic_todo.is_some()); let mut first_request = request(vec![ck("tail", 0, "live tail")]); @@ -32975,7 +33629,7 @@ mod tests { // The seed must add compatibility rows without replacing the module's newer fold cursor. let second_seed = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -32988,7 +33642,7 @@ mod tests { ) .await; assert!( - matches!(second_seed, HandlerOutcome::Response(_)), + matches!(second_seed, PreparedOutcome::Response(_)), "{second_seed:?}" ); @@ -33026,7 +33680,7 @@ mod tests { let first_seed = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -33040,14 +33694,14 @@ mod tests { }), ) .await; - assert!(matches!(first_seed, HandlerOutcome::Response(_))); + assert!(matches!(first_seed, PreparedOutcome::Response(_))); let adopted = store.load("ses").unwrap(); assert!(adopted.meta.initialized); assert_eq!(adopted.core.boundary_id, "m1#0"); let second_seed = handler .dispatch_value( - 7, + test_route(7), json!({ "kind": "state_sync", "session_id": "ses", @@ -33058,7 +33712,7 @@ mod tests { }), ) .await; - assert!(matches!(second_seed, HandlerOutcome::Response(_))); + assert!(matches!(second_seed, PreparedOutcome::Response(_))); let retained = store.load("ses").unwrap(); assert!(retained.meta.initialized); @@ -33111,10 +33765,10 @@ mod tests { let inbound_bytes = serde_json::to_vec(&first).unwrap().len() + serde_json::to_vec(&final_page).unwrap().len(); - let first_ack = handler.dispatch_value(7, first).await; - assert!(matches!(first_ack, HandlerOutcome::Response(_))); - let response = handler.dispatch_value(7, final_page).await; - let HandlerOutcome::Response(bytes) = response else { + let first_ack = handler.dispatch_value(test_route(7), first).await; + assert!(matches!(first_ack, PreparedOutcome::Response(_))); + let response = handler.dispatch_value(test_route(7), final_page).await; + let PreparedOutcome::Response(bytes) = response else { panic!("authority page assembly should execute: {response:?}"); }; let response: Value = serde_json::from_slice(&bytes).unwrap(); @@ -33186,16 +33840,16 @@ mod tests { > TRANSFORM_PAGE_MAX_BYTES ); - let first_ack = handler.dispatch_value(7, first).await; - let HandlerOutcome::Response(first_ack) = first_ack else { + let first_ack = handler.dispatch_value(test_route(7), first).await; + let PreparedOutcome::Response(first_ack) = first_ack else { panic!("first delta page should stage: {first_ack:?}"); }; let first_ack: Value = serde_json::from_slice(&first_ack).unwrap(); assert_eq!(first_ack["staged"], true); assert_eq!(first_ack["next_expected_index"], 1); - let response = handler.dispatch_value(7, final_page).await; - let HandlerOutcome::Response(response) = response else { + let response = handler.dispatch_value(test_route(7), final_page).await; + let PreparedOutcome::Response(response) = response else { panic!("reassembled tail delta should execute: {response:?}"); }; let response: Value = serde_json::from_slice(&response).unwrap(); @@ -33232,8 +33886,8 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(7, first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), first).await, + PreparedOutcome::Response(_) )); let newer = paged_transform_page( "transform", @@ -33250,7 +33904,7 @@ mod tests { })], ); assert_eq!( - error_code(handler.dispatch_value(7, newer).await), + error_code(handler.dispatch_value(test_route(7), newer).await), "authority_transform_page_attempt_mismatch" ); let retry_first = paged_transform_page( @@ -33268,8 +33922,8 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(7, retry_first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), retry_first).await, + PreparedOutcome::Response(_) )); let gap = paged_transform_page( "transform", @@ -33286,7 +33940,7 @@ mod tests { })], ); assert_eq!( - error_code(handler.dispatch_value(7, gap).await), + error_code(handler.dispatch_value(test_route(7), gap).await), "authority_transform_page_order_mismatch" ); @@ -33305,8 +33959,8 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(7, replacement).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), replacement).await, + PreparedOutcome::Response(_) )); let digest_first = paged_transform_page( @@ -33324,8 +33978,8 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(7, digest_first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), digest_first).await, + PreparedOutcome::Response(_) )); let mut changed_final = paged_transform_page( "transform", @@ -33347,7 +34001,7 @@ mod tests { "ck": ck("m1", 1, "changed-final").ck, }]); assert_eq!( - error_code(handler.dispatch_value(7, changed_final).await), + error_code(handler.dispatch_value(test_route(7), changed_final).await), "authority_transform_page_digest_mismatch" ); @@ -33366,8 +34020,10 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(7, generation_first).await, - HandlerOutcome::Response(_) + handler + .dispatch_value(test_route(7), generation_first) + .await, + PreparedOutcome::Response(_) )); let generation_changed = paged_transform_page( "transform", @@ -33384,7 +34040,11 @@ mod tests { })], ); assert_eq!( - error_code(handler.dispatch_value(7, generation_changed).await), + error_code( + handler + .dispatch_value(test_route(7), generation_changed) + .await + ), "authority_transform_page_attempt_mismatch" ); @@ -33394,7 +34054,11 @@ mod tests { "transform_page_id": "partial", }); assert_eq!( - error_code(handler.dispatch_value(7, partial_envelope).await), + error_code( + handler + .dispatch_value(test_route(7), partial_envelope) + .await + ), "invalid_params" ); } @@ -33403,8 +34067,14 @@ mod tests { async fn paged_transform_sessions_are_isolated() { let state = Arc::new(ProducerState::default()); let (handler, _store, _dir, project) = handler_with_store(state, default_test_config()); - handler.bind_route(8, binding(project.to_str().unwrap(), "authority-a")); - handler.bind_route(9, binding(project.to_str().unwrap(), "authority-b")); + handler.bind_route( + test_route(8), + binding(project.to_str().unwrap(), "authority-a"), + ); + handler.bind_route( + test_route(9), + binding(project.to_str().unwrap(), "authority-b"), + ); for (channel, session, mid, text) in [ (8, "authority-a", "a0", "authority a"), @@ -33425,8 +34095,8 @@ mod tests { })], ); assert!(matches!( - handler.dispatch_value(channel, first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(channel), first).await, + PreparedOutcome::Response(_) )); } for (channel, session, mid, text) in [ @@ -33447,8 +34117,10 @@ mod tests { "ck": ck(mid, 1, text).ck, })], ); - let response = handler.dispatch_value(channel, final_page).await; - assert!(matches!(response, HandlerOutcome::Response(_))); + let response = handler + .dispatch_value(test_route(channel), final_page) + .await; + assert!(matches!(response, PreparedOutcome::Response(_))); } } @@ -33468,11 +34140,11 @@ mod tests { vec![serde_json::to_value(ck("discarded-m0", 0, "discarded first")).unwrap()], ); assert!(matches!( - handler.dispatch_value(7, first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), first).await, + PreparedOutcome::Response(_) )); - handler.bind_route(7, binding(project_root, "replacement")); + handler.bind_route(test_route(7), binding(project_root, "replacement")); let discard_logs = handler .transform_page_discard_logs .lock() @@ -33486,7 +34158,7 @@ mod tests { ] ); - handler.bind_route(7, binding(project_root, "ses")); + handler.bind_route(test_route(7), binding(project_root, "ses")); let fresh_first = paged_transform_page( "transform", "ses", @@ -33498,8 +34170,8 @@ mod tests { vec![serde_json::to_value(ck("fresh-m0", 0, "fresh first")).unwrap()], ); assert!(matches!( - handler.dispatch_value(7, fresh_first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), fresh_first).await, + PreparedOutcome::Response(_) )); let fresh_final = paged_transform_page( "transform", @@ -33512,11 +34184,11 @@ mod tests { vec![serde_json::to_value(ck("fresh-m1", 1, "fresh final")).unwrap()], ); assert!(matches!( - handler.dispatch_value(7, fresh_final).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), fresh_final).await, + PreparedOutcome::Response(_) )); - handler.unbind_route(7); + handler.unbind_route(test_route(7)); assert_eq!( handler .transform_page_discard_logs @@ -33527,7 +34199,7 @@ mod tests { "route teardown must not log when no transform pages were staged" ); - handler.bind_route(7, binding(project_root, "ses")); + handler.bind_route(test_route(7), binding(project_root, "ses")); let teardown_first = paged_transform_page( "transform", "ses", @@ -33539,10 +34211,10 @@ mod tests { vec![serde_json::to_value(ck("teardown-m0", 0, "teardown first")).unwrap()], ); assert!(matches!( - handler.dispatch_value(7, teardown_first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(7), teardown_first).await, + PreparedOutcome::Response(_) )); - handler.unbind_route(7); + handler.unbind_route(test_route(7)); assert_eq!( handler .transform_page_discard_logs @@ -33612,13 +34284,14 @@ mod tests { .lock() .expect("state sync seed clock mutex") = Some(initial_now); - handler.bind_route(8, binding(project.to_str().unwrap(), "dead")); - handler.bind_route(9, binding(project.to_str().unwrap(), "live")); + handler.bind_route(test_route(8), binding(project.to_str().unwrap(), "dead")); + handler.bind_route(test_route(9), binding(project.to_str().unwrap(), "live")); let dead_first = paged_seed_batch("dead", "lost", 0, 0, 0, 2, vec![]); let dead_bytes = serde_json::to_vec(&dead_first).unwrap().len(); assert_eq!( - match handler.dispatch_value(8, dead_first).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + match handler.dispatch_value(test_route(8), dead_first).await { + PreparedOutcome::Response(bytes) => + serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }["next_expected_index"], json!(1) @@ -33633,8 +34306,9 @@ mod tests { let live_first = paged_seed_batch("live", "active", 0, 0, 0, 2, vec![]); let live_bytes = serde_json::to_vec(&live_first).unwrap().len(); assert_eq!( - match handler.dispatch_value(9, live_first).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + match handler.dispatch_value(test_route(9), live_first).await { + PreparedOutcome::Response(bytes) => + serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }["next_expected_index"], json!(1) @@ -33643,8 +34317,9 @@ mod tests { let live_final = paged_seed_batch("live", "active", 0, 0, 1, 2, vec![]); assert_eq!( - match handler.dispatch_value(9, live_final).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + match handler.dispatch_value(test_route(9), live_final).await { + PreparedOutcome::Response(bytes) => + serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }["ok"], json!(true) @@ -33654,8 +34329,9 @@ mod tests { let fresh_dead = paged_seed_batch("dead", "fresh", 0, 0, 0, 2, vec![]); let fresh_dead_bytes = serde_json::to_vec(&fresh_dead).unwrap().len(); assert_eq!( - match handler.dispatch_value(8, fresh_dead).await { - HandlerOutcome::Response(bytes) => serde_json::from_slice::(&bytes).unwrap(), + match handler.dispatch_value(test_route(8), fresh_dead).await { + PreparedOutcome::Response(bytes) => + serde_json::from_slice::(&bytes).unwrap(), other => panic!("unexpected handler outcome: {other:?}"), }["next_expected_index"], json!(1) @@ -33712,7 +34388,11 @@ mod tests { let foreign_midstream = paged_seed_batch(session, "foreign", 0, 0, 3, 4, vec![]); assert_eq!( - error_code(handler.dispatch_value(7, foreign_midstream).await), + error_code( + handler + .dispatch_value(test_route(7), foreign_midstream) + .await + ), "state_sync_seed_attempt_mismatch" ); assert_eq!(seed_accounting(&handler), (0, 0)); @@ -33743,7 +34423,7 @@ mod tests { let state = Arc::new(ProducerState::default()); let (handler, store, _dir, project) = handler_with_store(state, default_test_config()); let session = "paged-profile"; - handler.bind_route(8, binding(project.to_str().unwrap(), session)); + handler.bind_route(test_route(8), binding(project.to_str().unwrap(), session)); let mut first = paged_seed_batch(session, "profile-seed", 0, 0, 0, 3, vec![]); first["user_profile"] = json!(["prefers root cause"]); @@ -33760,18 +34440,18 @@ mod tests { let final_batch = paged_seed_batch(session, "profile-seed", 0, 0, 2, 3, vec![]); assert!(matches!( - handler.dispatch_value(8, first).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(8), first).await, + PreparedOutcome::Response(_) )); assert!(store.load_active_user_memories().unwrap().is_empty()); assert!(matches!( - handler.dispatch_value(8, second).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(8), second).await, + PreparedOutcome::Response(_) )); assert!(store.load_active_user_memories().unwrap().is_empty()); assert!(matches!( - handler.dispatch_value(8, final_batch).await, - HandlerOutcome::Response(_) + handler.dispatch_value(test_route(8), final_batch).await, + PreparedOutcome::Response(_) )); let profile = store.load_active_user_memories().unwrap(); diff --git a/crates/mc-module/src/main.rs b/crates/mc-module/src/main.rs deleted file mode 100644 index 4cc5bae79..000000000 --- a/crates/mc-module/src/main.rs +++ /dev/null @@ -1,51 +0,0 @@ -//! mc-module entrypoint: boot on `subc-client-rs`'s `serve` (provider role). -//! -//! `serve` owns the handshake (read `--subc `, authenticate, send -//! HELLO{manifest}, await HELLO_ACK, then dispatch route data requests to the -//! handler). The handler opens the single-writer store in `on_hello_ack`. - -#![forbid(unsafe_code)] - -use std::error::Error; -use std::path::PathBuf; - -use mc_module::{manifest, McHandler, DEFAULT_MODULE_ID}; - -#[tokio::main(flavor = "current_thread")] -async fn main() -> Result<(), Box> { - // Fleet convention: a side-effect-free single-line --version, evaluated before - // any runtime argument so supervisors and test substrates can probe the binary - // without a connection file. - if std::env::args().skip(1).any(|arg| arg == "--version") { - println!("ck-mc {}", env!("CARGO_PKG_VERSION")); - return Ok(()); - } - let module_id = std::env::var(subc_protocol::SUBC_MODULE_ID_ENV) - .ok() - .filter(|value| !value.trim().is_empty()) - .unwrap_or_else(|| DEFAULT_MODULE_ID.to_string()); - - let connection_file = parse_subc_arg(std::env::args_os().skip(1))?; - subc_client_rs::serve_with( - &connection_file, - manifest(&module_id), - McHandler::new_with_connection_file(Some(connection_file.clone())), - ) - .await?; - Ok(()) -} - -fn parse_subc_arg(mut args: I) -> Result> -where - I: Iterator, -{ - while let Some(arg) = args.next() { - if arg == "--subc" { - return args - .next() - .map(PathBuf::from) - .ok_or_else(|| "--subc requires a connection-file path".into()); - } - } - Err("missing --subc ".into()) -} diff --git a/crates/mc-module/src/prompt_surface.rs b/crates/mc-module/src/prompt_surface.rs index 6eedf3d5d..b52489785 100644 --- a/crates/mc-module/src/prompt_surface.rs +++ b/crates/mc-module/src/prompt_surface.rs @@ -1,9 +1,24 @@ use std::collections::BTreeMap; use serde::{Deserialize, Serialize}; -use serde_json::json; +use serde_json::{json, Value}; use sha2::{Digest, Sha256}; -use subc_protocol::manifest::{ExecutionMode, Tool}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ExecutionMode { + Pure, + Mutating, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Tool { + pub name: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub description: Option, + pub execution_mode: ExecutionMode, + pub schema: Value, +} use super::{ ctx_expand_description, ctx_expand_schema, ctx_memory_description, ctx_memory_schema, diff --git a/crates/mc-module/src/session_resolver.rs b/crates/mc-module/src/session_resolver.rs index 6fb5b8f55..77b4af44c 100644 --- a/crates/mc-module/src/session_resolver.rs +++ b/crates/mc-module/src/session_resolver.rs @@ -1,27 +1,9 @@ -//! Resolves Claude Code MCP facade instance tokens to the opaque conversation key. -//! -//! The MCP shim binds its route with a per-launch instance token, while the parent -//! conversation key is what names that Claude Code conversation's compartments. The -//! resolver is the only place that crosses that boundary: handlers pass the token as a -//! lookup argument, then use the returned key for conversation-scoped reads and as the -//! source session recorded on memory writes. +use std::{error::Error, fmt, path::Path}; -use std::{error::Error, fmt, path::Path, path::PathBuf, time::Duration}; - -use serde_json::{json, Value}; -use subc_client_rs::{ - async_trait, CallError, CallOptions, CloseRouteOptions, ConsumerOptions, RetryBackoff, - SubcConsumer, -}; -use subc_protocol::{BindIdentity, RouteTarget}; - -const DEFAULT_THALAMUS_MODULE_ID: &str = "thalamus"; -const SESSION_RESOLVE_DEADLINE: Duration = Duration::from_secs(2); +use async_trait::async_trait; #[derive(Debug, Clone, PartialEq, Eq)] pub struct ResolvedSession { - /// Opaque composite conversation key returned by the thalamus gateway. Callers must not parse it; - /// the instance token is only the lookup input. pub session_id: String, pub last_traffic_ms: i64, } @@ -57,88 +39,6 @@ pub trait SessionResolver: Send + Sync { ) -> Result, SessionResolveError>; } -pub struct RealSessionResolver { - connection_file: PathBuf, - module_id: String, -} - -impl RealSessionResolver { - pub fn new(connection_file: PathBuf) -> Self { - Self { - connection_file, - module_id: DEFAULT_THALAMUS_MODULE_ID.to_string(), - } - } - - /// The management route this resolver opens. Split out so a local test can - /// pin the runtime target's exact module id: the gateway registers as - /// "thalamus", and a stale id here kills every stateful facade tool call - /// at route-open. - fn route_target(&self) -> RouteTarget { - RouteTarget::ManagementSurface { - module_id: self.module_id.clone(), - } - } - - async fn resolve_once( - &self, - project_root: &Path, - harness: &str, - instance_token: &str, - ) -> Result, SessionResolveError> { - let target = self.route_target(); - let identity = BindIdentity { - project_root: project_root.to_path_buf(), - harness: harness.to_string(), - session: instance_token.to_string(), - }; - let consumer = SubcConsumer::connect(&self.connection_file, consumer_options()) - .await - .map_err(|error| SessionResolveError::Transport(error.to_string()))?; - let response = consumer - .call( - target.clone(), - identity.clone(), - serde_json::to_vec(&json!({ - "method": "session.resolve", - "params": { "instance_token": instance_token } - })) - .map_err(|error| SessionResolveError::Transport(error.to_string()))?, - call_options(), - ) - .await; - consumer - .close_route(target, identity, CloseRouteOptions::default()) - .await; - consumer.close().await; - let response = response.map_err(call_error_to_resolve_error)?; - let value: Value = serde_json::from_slice(&response).map_err(|error| { - SessionResolveError::InvalidResponse(format!("response was not JSON: {error}")) - })?; - parse_resolve_response(&value) - } -} - -#[async_trait] -impl SessionResolver for RealSessionResolver { - async fn resolve_session( - &self, - project_root: &Path, - harness: &str, - instance_token: &str, - ) -> Result, SessionResolveError> { - match tokio::time::timeout( - SESSION_RESOLVE_DEADLINE, - self.resolve_once(project_root, harness, instance_token), - ) - .await - { - Ok(result) => result, - Err(_) => Err(SessionResolveError::Timeout), - } - } -} - pub struct MissingSessionResolver; #[async_trait] @@ -149,71 +49,7 @@ impl SessionResolver for MissingSessionResolver { _harness: &str, _instance_token: &str, ) -> Result, SessionResolveError> { - Err(SessionResolveError::Transport( - "mc-module was started without a subc connection file".to_string(), - )) - } -} - -fn consumer_options() -> ConsumerOptions { - ConsumerOptions { - handshake_timeout: SESSION_RESOLVE_DEADLINE, - // Channel-0 calls share the resolve deadline: session.resolve is the only - // call this consumer makes, and a facade tool call must fail fast rather - // than outlive its MCP request. - call_timeout: SESSION_RESOLVE_DEADLINE, - reconnect_backoff: RetryBackoff { - base: Duration::from_millis(25), - cap: Duration::from_millis(50), - max_attempts: 1, - }, - restored_debounce: Duration::from_millis(10), - } -} - -fn call_options() -> CallOptions { - CallOptions { - timeout: SESSION_RESOLVE_DEADLINE, - route_retry: RetryBackoff { - base: Duration::from_millis(25), - cap: Duration::from_millis(50), - max_attempts: 1, - }, - route_retry_deadline: SESSION_RESOLVE_DEADLINE, - ..CallOptions::default() - } -} - -fn call_error_to_resolve_error(error: CallError) -> SessionResolveError { - match error { - CallError::Module(body) if body.code == "session_resolve_timeout" => { - SessionResolveError::Timeout - } - other => SessionResolveError::Transport(other.to_string()), - } -} - -fn parse_resolve_response(value: &Value) -> Result, SessionResolveError> { - let payload = value.get("result").unwrap_or(value); - match payload.get("session_id") { - Some(Value::Null) | None => Ok(None), - Some(Value::String(session_id)) => { - let last_traffic_ms = payload - .get("last_traffic_ms") - .and_then(Value::as_i64) - .ok_or_else(|| { - SessionResolveError::InvalidResponse( - "resolved session omitted integer last_traffic_ms".to_string(), - ) - })?; - Ok(Some(ResolvedSession { - session_id: session_id.clone(), - last_traffic_ms, - })) - } - Some(other) => Err(SessionResolveError::InvalidResponse(format!( - "session_id must be string or null, got {other}" - ))), + Ok(None) } } @@ -221,42 +57,14 @@ fn parse_resolve_response(value: &Value) -> Result, Sess mod tests { use super::*; - #[test] - fn parse_accepts_direct_or_json_rpc_result_and_null_unresolved() { + #[tokio::test] + async fn unsupported_mapping_is_local_absence() { assert_eq!( - parse_resolve_response(&json!({"session_id": "conv:key", "last_traffic_ms": 7})) + MissingSessionResolver + .resolve_session(Path::new("/project"), "claude-code", "instance") + .await .unwrap(), - Some(ResolvedSession { - session_id: "conv:key".to_string(), - last_traffic_ms: 7, - }) - ); - assert_eq!( - parse_resolve_response( - &json!({"result": {"session_id": "conv:2", "last_traffic_ms": 9}}) - ) - .unwrap(), - Some(ResolvedSession { - session_id: "conv:2".to_string(), - last_traffic_ms: 9, - }) - ); - assert_eq!( - parse_resolve_response(&json!({"session_id": null})).unwrap(), None ); } - - #[test] - fn real_resolver_routes_to_the_thalamus_management_surface() { - let resolver = RealSessionResolver::new(PathBuf::from("/nonexistent")); - // Pin the exact runtime module id: the gateway registers as "thalamus"; - // any other id (e.g. the retired "ai-proxy") fails every facade call. - assert_eq!( - resolver.route_target(), - RouteTarget::ManagementSurface { - module_id: "thalamus".to_string(), - } - ); - } } diff --git a/crates/mc-module/tests/boundary_counter_durability.rs b/crates/mc-module/tests/boundary_counter_durability.rs index 7ef68f44f..fa24d1c65 100644 --- a/crates/mc-module/tests/boundary_counter_durability.rs +++ b/crates/mc-module/tests/boundary_counter_durability.rs @@ -1,37 +1,38 @@ -use cortexkit_store_types::{Isolation, StorageBackend, StorageDescriptor}; +#![cfg(unix)] +#![forbid(unsafe_code)] + +mod support; + +use std::time::{Duration, Instant}; + use mc_core::CoreState; +use mc_host::TargetKind; use mc_store::{McStore, McStoreError, ModuleMeta}; +use serde_json::json; +use support::direct_host::{request_json, storage_descriptor, FixtureProcess, BUDGET}; -fn descriptor(path: &std::path::Path) -> StorageDescriptor { - StorageDescriptor { - module_id: "mc-module-boundary-counter-test".to_string(), - storage_namespace: "mc_cache".to_string(), - isolation: Isolation::Module, - backend: StorageBackend::Sqlite { - path: path.join("store.db").to_string_lossy().to_string(), - }, - } -} - -#[test] -fn competing_module_passes_keep_one_increment_and_reopen_keeps_it() { - let directory = tempfile::tempdir().unwrap(); - let store = McStore::open(&descriptor(directory.path())).unwrap(); +#[tokio::test] +async fn competing_pass_counter_survives_direct_primary_lifecycle_and_reopen() { + let root = tempfile::tempdir().expect("state root"); + let descriptor = storage_descriptor(root.path()); + let store = McStore::open(&descriptor).expect("seed store opens"); let session = "module-counter"; let core = CoreState::default(); let initial = ModuleMeta { boundary_divergence_pending_count: 0, ..Default::default() }; - store.commit(session, None, &core, &initial).unwrap(); + store + .commit(session, None, &core, &initial) + .expect("initial state commits"); - let winner = store.load(session).unwrap(); - let loser = store.load(session).unwrap(); + let winner = store.load(session).expect("winner snapshot"); + let loser = store.load(session).expect("loser snapshot"); let mut winner_meta = winner.meta.clone(); winner_meta.boundary_divergence_pending_count = 1; store .commit(session, winner.row_version, &winner.core, &winner_meta) - .unwrap(); + .expect("winner commits"); let mut loser_meta = loser.meta.clone(); loser_meta.boundary_divergence_pending_count = 1; @@ -42,21 +43,39 @@ fn competing_module_passes_keep_one_increment_and_reopen_keeps_it() { found: 2 }) )); - assert_eq!( - store - .load(session) - .unwrap() - .meta - .boundary_divergence_pending_count, - 1 - ); - drop(store); - let reopened = McStore::open(&descriptor(directory.path())).unwrap(); + + let fixture = FixtureProcess::start_at(root.path().to_path_buf()); + let client = fixture.client().await; + let route = fixture + .open_route(&client, "magic-context", TargetKind::ToolProvider, session) + .await; + let deadline = Instant::now() + BUDGET; + loop { + let status = request_json( + &client, + route, + json!({"kind": "status", "session_id": session}), + ) + .await; + if status["store_open"] == true { + assert_eq!(status["session_id"], session); + break; + } + assert!( + Instant::now() < deadline, + "direct primary store did not open" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + client.close().await.expect("managed client closes"); + fixture.shutdown(); + + let reopened = McStore::open(&descriptor).expect("store reopens after fixture drain"); assert_eq!( reopened .load(session) - .unwrap() + .expect("state reloads") .meta .boundary_divergence_pending_count, 1 diff --git a/crates/mc-module/tests/broca_roundtrip.rs b/crates/mc-module/tests/broca_roundtrip.rs index e0d39e468..0d2cb4012 100644 --- a/crates/mc-module/tests/broca_roundtrip.rs +++ b/crates/mc-module/tests/broca_roundtrip.rs @@ -1,1447 +1,198 @@ -//! U6 authenticated round-trip proof (R25, AE15): the REAL `HistorianProducer` -//! and the REAL module classify path complete LLM round trips against a real -//! in-process `mc-host` — three-child static composite, HMAC handshake, wire-v2 -//! framing, and the Broca run supervisor — with a deterministic backend instead -//! of a subprocess (R14). The two round trips collectively select both -//! harnesses: historian under `opencode`, classify under `pi`. -//! -//! Hermetic by construction: loopback only, temp data directories, no provider -//! credentials, no `packages/e2e-tests` involvement (R27). - +#![cfg(unix)] #![forbid(unsafe_code)] -use std::collections::{BTreeMap, HashMap}; -use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicUsize, Ordering}; -use std::sync::{Arc, Mutex}; -use std::time::Duration; - -use serde_json::{json, Value}; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::TcpStream; - -use mc_host::broca::backend::{ - BackendError, BackendEvent, BackendFuture, BackendRequest, BackendTerminal, - ErrorClass as BackendErrorClass, EventSink, FinishReason, LlmExecutionBackend, -}; -use mc_host::broca::supervisor::{SessionKey, Supervisor}; -use mc_host::broca::BrocaComponent; -use mc_host::{ - BindOutcome, CancellationToken, CompositeComponent, HealthReport, HostConfig, HostError, - HostInit, HostLimits, InitError, ManifestSnapshot, PrimaryComponent, RequestCtx, - RequestOutcome, RouteHandle, RouteIdentity, SecondaryComponent, ShutdownError, StaticComposite, -}; - -use mc_module::classify::attempt_child_session_id; -use mc_module::historian::{ - reattach_historian_producer, HistorianReattachOutcome, HistorianReattachRequest, -}; -use mc_module::historian_producer::{ - ErrorClass, HistorianProducer, HistorianProducerConfig, HistorianProducerError, RunState, -}; -use mc_module::historian_validate::{HistorianChunk, ValidateOptions}; -use mc_module::McHandler; - -use mc_store::{ - CompartmentSetGeneration, HistorianChunkRange, HistorianDurableState, HistorianPhase, - InsertMemoryInput, McStore, -}; - -use subc_control::{ClientControlRequest, ClientControlResponse}; -use subc_protocol::{BindIdentity, Flags, Frame, FrameType, Priority, RouteTarget}; -use subc_transport::{authenticate_client, connection_file, read_frame, write_frame}; - -const BUDGET: Duration = Duration::from_secs(10); - -// --------------------------------------------------------------------------- -// Deterministic backend. Reimplemented here rather than shared with mc-host's -// `tests/support` because Cargo compiles test-support modules per crate; the -// public `LlmExecutionBackend` contract (R14) is exactly the seam that makes -// this cheap. -// --------------------------------------------------------------------------- - -type RunFn = dyn Fn(BackendRequest, EventSink, CancellationToken) -> BackendFuture + Send + Sync; - -/// One backend invocation as the supervisor delivered it, retained so tests -/// can prove harness propagation, model order, and per-attempt sessions. -#[derive(Debug, Clone)] -struct SeenRun { - session: String, - provider: String, - model: String, - harness: &'static str, -} - -struct ScriptedBackend { - starts: AtomicUsize, - seen: Mutex>, - run: Box, -} - -impl ScriptedBackend { - fn with_behavior( - run: impl Fn(BackendRequest, EventSink, CancellationToken) -> BackendFuture - + Send - + Sync - + 'static, - ) -> Arc { - Arc::new(Self { - starts: AtomicUsize::new(0), - seen: Mutex::new(Vec::new()), - run: Box::new(run), - }) - } - - fn starts(&self) -> usize { - self.starts.load(Ordering::SeqCst) - } - - fn seen(&self) -> Vec { - self.seen.lock().expect("seen mutex").clone() - } - - /// Emits `manifest` and completes, except for models containing `flaky`, - /// which fail with a classified transient error carrying retry metadata — - /// the AE13 transient fixture the classify fallback tests consume. - fn classify_manifest(manifest: String) -> Arc { - Self::with_behavior(move |request, events, _cancel| { - let manifest = manifest.clone(); - Box::pin(async move { - if request.model.contains("flaky") { - return BackendTerminal::Failed(BackendError { - class: BackendErrorClass::Transient, - message: "provider rate limited".to_owned(), - retry_after_secs: Some(7), - provider_code: Some("rate_limited".to_owned()), - }); - } - events.emit(BackendEvent::AssistantText { - text: manifest, - finish_reason: None, - }); - BackendTerminal::Completed { - finish_reason: FinishReason::Completed, - } - }) - }) - } +mod support; - /// Every run parks on the returned gate and deliberately ignores its - /// cancel token, which is what keeps `run.cancel` settlement blocked for - /// the reserved-capacity saturation test; release the gate before host - /// shutdown so the drain stays bounded. - fn stuck_until_gate() -> (Arc, Arc) { - let gate = Arc::new(tokio::sync::Semaphore::new(0)); - let run_gate = Arc::clone(&gate); - let backend = Self::with_behavior(move |_request, _events, _cancel| { - let gate = Arc::clone(&run_gate); - Box::pin(async move { - gate.acquire().await.expect("gate stays open").forget(); - BackendTerminal::Completed { - finish_reason: FinishReason::Completed, - } - }) - }); - (backend, gate) - } -} - -impl LlmExecutionBackend for ScriptedBackend { - fn execute( - &self, - request: BackendRequest, - events: EventSink, - cancel: CancellationToken, - ) -> BackendFuture { - self.starts.fetch_add(1, Ordering::SeqCst); - self.seen.lock().expect("seen mutex").push(SeenRun { - session: request.session.clone(), - provider: request.provider.clone(), - model: request.model.clone(), - harness: request.harness.as_str(), - }); - (self.run)(request, events, cancel) - } -} - -// --------------------------------------------------------------------------- -// Real-loopback three-child host: a minimal echo `magic-context` primary, a -// stub `synapse` secondary, and the REAL BrocaComponent under test. -// --------------------------------------------------------------------------- - -#[derive(Clone)] -struct FixedComponent { - id: &'static str, - role: &'static str, -} +use std::time::{Duration, Instant}; -impl CompositeComponent for FixedComponent { - fn manifest(&self) -> ManifestSnapshot { - ManifestSnapshot { - module_id: self.id.to_owned(), - module_version: "0.0.1-roundtrip".to_owned(), - provides: vec![json!({"role": self.role})], - control_ops: Vec::new(), - } - } - - async fn bind(&self, _route: RouteHandle, _identity: RouteIdentity) -> BindOutcome { - BindOutcome::Accept - } - - async fn handle(&self, ctx: RequestCtx) -> RequestOutcome { - let body = serde_json::to_vec(&json!({"served_by": self.id})).expect("body serializes"); - let Ok(mut output) = ctx.reserve_output(body.len()).await else { - return RequestOutcome::Error { - code: "internal_error".to_owned(), - message: "echo output reservation failed".to_owned(), - }; - }; - output - .extend_from_slice(&body) - .expect("reservation matches body length"); - RequestOutcome::Response { - body: output, - binary: false, - } - } - - async fn route_gone(&self, _route: RouteHandle) {} - - async fn health(&self) -> HealthReport { - HealthReport::ok() - } - - async fn shutdown(&self) -> Result<(), ShutdownError> { - Ok(()) - } -} - -impl PrimaryComponent for FixedComponent { - async fn initialize(&self, _init: HostInit) -> Result<(), InitError> { - Ok(()) - } -} - -impl SecondaryComponent for FixedComponent { - async fn initialize(&self) -> Result<(), InitError> { - Ok(()) - } -} - -struct RoundTripHost { - connection_file: PathBuf, - supervisor: Arc, - routes: Arc>>, - shutdown: CancellationToken, - join: Option>>, -} - -impl RoundTripHost { - /// Starts a host at `data_root` and waits for its publication. The data - /// root is caller-owned so the restart test can run two incarnations over - /// one directory and prove Broca state never survives an incarnation - /// (R11). - async fn start(backend: Arc, data_root: &Path) -> Self { - let broca = BrocaComponent::new(backend); - let supervisor = broca.supervisor(); - let routes = broca.route_index(); - let composite = StaticComposite::new( - FixedComponent { - id: "magic-context", - role: "tool_provider", - }, - FixedComponent { - id: "synapse", - role: "management_surface", +use mc_host::{RequestOptions, ResponseStream, TargetKind}; +use serde_json::{json, Value}; +use support::direct_host::{request_json, send_body, FixtureProcess, BUDGET}; + +async fn subscribe(client: &mc_host::Client, route: mc_host::RouteHandle) -> ResponseStream { + client + .request_stream( + route, + serde_json::to_vec(&json!({ + "method": "session.subscribe", + "params": {"from": "start"} + })) + .expect("subscribe serializes"), + RequestOptions { + timeout: BUDGET, + cancellation: None, }, - broca, ) - .expect("distinct component ids"); - - let mut config = HostConfig { - data_dir: Some(data_root.to_path_buf()), - daemon_ver: "mc-host/broca-roundtrip".to_owned(), - limits: HostLimits { - // Small but interoperable: one 64 MiB frame must still fit - // beside the production Broca component's declared retained - // reservation. - max_resident_bytes: mc_host::config::MIN_RESIDENT_BYTES * 2 - + mc_host::broca::config::DECLARED_RETAINED_RESIDENT_BYTES, - ..Default::default() - }, - ..Default::default() - }; - config.timing.shutdown_deadline = Duration::from_secs(5); - config.timing.route_close_budget = Duration::from_secs(2); - config.timing.lifecycle_callback_deadline = Duration::from_secs(3); - - let publication = data_root - .join("cortexkit") - .join("run") - .join(mc_host::CONNECTION_FILE_NAME); - let shutdown = CancellationToken::new(); - let run_shutdown = shutdown.clone(); - let join = tokio::spawn(async move { mc_host::run(composite, config, run_shutdown).await }); - - // A restart over the same data_dir republished at the same path, so a - // stale predecessor snapshot must not be mistaken for this host's own. - let existing = std::fs::read(&publication).ok(); - let deadline = tokio::time::Instant::now() + BUDGET; - loop { - if join.is_finished() { - panic!( - "host exited before publishing: {:?}", - join.await.expect("run task joins") - ); - } - match std::fs::read(&publication) { - Ok(bytes) if Some(&bytes) != existing.as_ref() => break, - _ => {} - } - assert!( - tokio::time::Instant::now() < deadline, - "host did not publish in time" - ); - tokio::time::sleep(Duration::from_millis(10)).await; - } - - Self { - connection_file: publication, - supervisor, - routes, - shutdown, - join: Some(join), - } - } - - async fn shutdown( - mut self, - ) -> ( - Arc, - Arc>>, - ) { - self.shutdown.cancel(); - let join = self.join.take().expect("host runs once"); - tokio::time::timeout(Duration::from_secs(20), join) - .await - .expect("host finishes within its shutdown budget") - .expect("run task joins") - .expect("graceful shutdown"); - (Arc::clone(&self.supervisor), Arc::clone(&self.routes)) - } + .await + .expect("subscription starts") } -impl Drop for RoundTripHost { - fn drop(&mut self) { - // A panicking test must not leave a host holding the instance lock. - self.shutdown.cancel(); +async fn drain(stream: &mut ResponseStream) -> Vec { + let mut items = Vec::new(); + while let Some(item) = stream.next().await.expect("subscription settles") { + items.push(serde_json::from_slice(&item.body).expect("stream item JSON")); } + items } -/// The U6 teardown contract: after host shutdown, zero active runs, backend -/// (subprocess-permit) holds, subscribers, retained bytes, sessions, and -/// route mappings remain. -fn assert_supervisor_drained( - supervisor: &Supervisor, - routes: &Arc>>, -) { - let metrics = supervisor.metrics(); - assert_eq!(metrics.live_runs, 0, "no active runs after shutdown"); - assert_eq!(metrics.sessions, 0, "no retained sessions after shutdown"); - assert_eq!(metrics.tombstones, 0, "no tombstones after shutdown"); - assert_eq!(metrics.free_command_permits, 32); - assert_eq!(metrics.free_run_slots, 32); - assert_eq!(metrics.free_subscriber_permits, 64); - assert_eq!(metrics.free_backend_permits, 8, "all backend permits home"); - assert_eq!( - metrics.retained_bytes_available, metrics.retained_bytes_capacity, - "every retained-byte charge released" - ); - assert!( - routes.lock().expect("route index mutex").is_empty(), - "no route mappings after shutdown" - ); +fn unit_type(item: &Value) -> Option<&str> { + item["unit"]["type"].as_str() } -async fn wait_until(mut condition: impl FnMut() -> bool, what: &str) { - let deadline = tokio::time::Instant::now() + BUDGET; - while !condition() { +async fn wait_for_counter(fixture: &FixtureProcess, field: &str, value: u64) { + let deadline = Instant::now() + BUDGET; + loop { + if fixture.counters(500)[field] == value { + return; + } assert!( - tokio::time::Instant::now() < deadline, - "timed out waiting for {what}" + Instant::now() < deadline, + "counter {field} never reached {value}" ); tokio::time::sleep(Duration::from_millis(10)).await; } } -/// Production defaults wait minutes on runs and reconnects; every failure -/// mode this suite exercises must instead surface within the test budget. -fn producer_config( - connection_file: &Path, - project_root: &Path, - harness: &str, -) -> HistorianProducerConfig { - HistorianProducerConfig { - handshake_timeout: Duration::from_secs(5), - request_timeout: BUDGET, - await_timeout: BUDGET, - ..HistorianProducerConfig::new(connection_file, project_root, harness) - } -} - -fn store_descriptor(dir: &Path) -> cortexkit_store_types::StorageDescriptor { - cortexkit_store_types::StorageDescriptor { - module_id: "magic-context-roundtrip".to_owned(), - storage_namespace: "mc_cache".to_owned(), - isolation: cortexkit_store_types::Isolation::Module, - backend: cortexkit_store_types::StorageBackend::Sqlite { - path: dir.join("store.db").to_string_lossy().into_owned(), - }, - } -} - -/// Walks the memories-domain authority to MODULE and binds the route root, -/// mirroring the production drain flow so classify's authority checks see -/// legitimate rows rather than hand-poked state. -fn activate_module_authority(store: &McStore, context: &str, project: &str, route_root: &str) { - let preparing = store - .authority_begin_prepare(context, project, "memories") - .expect("authority prepare"); - let checksum = store - .authority_seed_checksum(context, project, "memories") - .expect("authority checksum"); - store - .authority_verify_prepare( - context, - project, - "memories", - preparing.generation, - &checksum, - &checksum, - ) - .expect("authority verify"); - let module = store - .authority_ack_prepare(context, project, "memories", preparing.generation) - .expect("authority ack"); - assert_eq!(module.state, "MODULE"); - store - .bind_authority_route(context, project, route_root) - .expect("authority route bind"); -} - -fn seed_memory(store: &McStore, project: &str, content: &str) -> (i64, String) { - let id = store - .insert_memory(InsertMemoryInput { - project_path: project, - route_project_root: None, - category: "PROJECT_RULES", - content, - source_session_id: Some(project), - source_type: Some("test"), - importance: Some(50), - expires_at: None, - metadata_json: None, - now_ms: 1, - }) - .expect("memory inserted"); - let hash = store - .get_memory_full(id) - .expect("memory readable") - .expect("memory present") - .normalized_hash; - (id, hash) -} - -/// The attribute scan a real consumer would run over the deterministic -/// manifest — the Rust test plays the TypeScript role here (KTD10), so rows -/// are built from the manifest text instead of from the fixture inputs. -fn parse_classify_manifest(text: &str) -> Vec<(i64, i64, String, bool)> { - let attr = |fragment: &str, name: &str| -> String { - let key = format!("{name}=\""); - let start = fragment.find(&key).expect("attribute present") + key.len(); - let end = fragment[start..].find('"').expect("attribute closed") + start; - fragment[start..end].to_owned() - }; - text.split(" Value { - match outcome { - subc_client_rs::HandlerOutcome::Response(bytes) => { - serde_json::from_slice(&bytes).expect("response is JSON") - } - other => panic!("expected a response, got {other:?}"), - } -} - -// --------------------------------------------------------------------------- -// Raw authenticated wire client for the scenarios `HistorianProducer` cannot -// express (tool-provider echo, unread concurrent frames). Built on the same -// production `subc-transport` client primitives the producer itself uses, so -// it adds no second framing implementation. -// --------------------------------------------------------------------------- - -struct RawWire { - stream: TcpStream, - next_corr: u64, -} - -impl RawWire { - async fn connect(connection_path: &Path) -> Self { - let conn = connection_file::read(connection_path).expect("publication validates"); - let endpoint = conn.endpoints.first().expect("published endpoint"); - let mut stream = TcpStream::connect((endpoint.host.as_str(), endpoint.port)) - .await - .expect("loopback connect"); - authenticate_client(&mut stream, &conn, Duration::from_secs(5)) - .await - .expect("authenticated handshake"); - Self { - stream, - next_corr: 0, - } - } - - async fn send_request(&mut self, channel: u16, epoch: u32, body: Value) -> u64 { - self.next_corr += 1; - let frame = Frame::build( - FrameType::Request, - Flags::new(false, Priority::Interactive, false), - channel, - epoch, - self.next_corr, - serde_json::to_vec(&body).expect("body serializes"), - ) - .expect("frame builds"); - write_frame(&mut self.stream, &frame) - .await - .expect("frame writes"); - self.next_corr - } - - /// Reads frames until one matches, discarding everything else — the - /// discard path is what drains unread stream data during saturation. - async fn frame_for(&mut self, channel: u16, epoch: u32, corr: u64) -> Frame { - tokio::time::timeout(BUDGET, async { - loop { - let frame = read_frame(&mut self.stream) - .await - .expect("frame reads") - .expect("connection stays open"); - if frame.header.channel == channel - && frame.header.epoch == epoch - && frame.header.corr == corr - && frame.header.ty != FrameType::Ping - { - return frame; - } - } - }) - .await - .expect("terminal frame within budget") - } - - async fn route_open( - &mut self, - target: RouteTarget, - project_root: &Path, - harness: &str, - session: &str, - ) -> (u16, u32) { - let request = ClientControlRequest::RouteOpen { - target, - identity: BindIdentity { - project_root: project_root.to_path_buf(), - harness: harness.to_owned(), - session: session.to_owned(), - }, - consumer_identity: None, - consumer_capabilities: None, - admission_facts: None, - }; - let corr = self - .send_request( - 0, - 0, - serde_json::to_value(&request).expect("control encodes"), - ) - .await; - let frame = self.frame_for(0, 0, corr).await; - assert_eq!(frame.header.ty, FrameType::Response, "route.open succeeds"); - let response: ClientControlResponse = - serde_json::from_slice(&frame.body).expect("control response decodes"); - match response { - ClientControlResponse::RouteOpen { - route_channel, - route_epoch, - } => (route_channel, route_epoch), - other => panic!("unexpected control response {other:?}"), - } - } -} - -fn error_code(frame: &Frame) -> String { - let body: Value = serde_json::from_slice(&frame.body).expect("error body is JSON"); - body["code"] - .as_str() - .expect("error body carries a code") - .to_owned() -} - -// --------------------------------------------------------------------------- -// Historian round trip (harness: opencode). -// --------------------------------------------------------------------------- - -/// The authenticated historian round trip: one deterministic run, ordered -/// replay preserved across a redrain, a length-cap unit whose finish metadata -/// survives to `ProducerOutput`, both routes closed, and no retained -/// subscriber (R25, AE15, AE13). #[tokio::test] -async fn historian_round_trip_preserves_replay_and_releases_routes() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - let backend = ScriptedBackend::with_behavior(|_request, events, _cancel| { - Box::pin(async move { - events.emit(BackendEvent::AssistantText { - text: "compartment doc ".to_owned(), - finish_reason: None, - }); - // A step-level length-class reason with a completed terminal is - // the AE13 length-cap fixture: truncation must reach producer - // policy without becoming a provider error. - events.emit(BackendEvent::AssistantText { - text: "cut mid-document".to_owned(), - finish_reason: Some(FinishReason::MaxOutputTokens), - }); - BackendTerminal::Completed { - finish_reason: FinishReason::Completed, - } - }) - }); - let host = RoundTripHost::start(Arc::clone(&backend) as Arc<_>, data_root.path()).await; - let routes = Arc::clone(&host.routes); - let supervisor = Arc::clone(&host.supervisor); - - let mut producer = HistorianProducer::connect(producer_config( - &host.connection_file, - project.path(), - "opencode", - )) - .await - .expect("producer authenticates"); - assert_eq!( - backend.starts(), - 0, - "authentication alone must start nothing" - ); - - let handle = producer - .start( - "mc-historian:roundtrip", - "historian system role", - "summarize this chunk", - "test/historian-model", +async fn real_broca_success_block_release_failure_and_counters() { + let fixture = FixtureProcess::start(); + let client = fixture.client().await; + let success_route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "successful-broca", ) - .await - .expect("session.send admits one run"); - let output = producer - .await_output(&handle.run_id) - .await - .expect("subscription drains to the terminal"); - assert_eq!(output.text, "compartment doc cut mid-document"); - assert!( - output.length_capped, - "the length-class finish reason must survive replay" - ); - assert_eq!(backend.starts(), 1, "exactly one deterministic run started"); - - // Reattach-style redrain from `start`: retained replay is ordered and - // byte-identical, and it must not start a second backend (R8, R9). - let redrained = producer - .redrain_output(&handle.run_id) - .await - .expect("redrain replays the retained run"); - assert_eq!(redrained, output); - assert_eq!(backend.starts(), 1); - - assert_eq!( - producer - .status(&handle.run_id) - .await - .expect("status settles"), - RunState::Terminal - ); - assert_eq!( - backend.seen()[0].harness, - "opencode", - "the route-bound harness reaches the backend" - ); - - producer.close().await.expect("both routes close"); - // Route teardown is asynchronous host-side; the bind-time mappings must - // still drain to zero once the goodbyes are processed. - wait_until( - || routes.lock().expect("route index mutex").is_empty(), - "both producer routes to close", - ) - .await; - assert_eq!( - supervisor.metrics().free_subscriber_permits, - 64, - "no retained subscriber after the round trip" - ); - - let (supervisor, routes) = host.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); -} - -// --------------------------------------------------------------------------- -// Classify round trip (harness: pi). -// --------------------------------------------------------------------------- - -struct ClassifyRig { - store: Arc, - handler: McHandler, - generation: u64, - _store_dir: tempfile::TempDir, -} - -const CLASSIFY_PROJECT: &str = "git:roundtrip"; -const CLASSIFY_CONTEXT: &str = "ctx-roundtrip"; -const CLASSIFY_CHANNEL: u16 = 7; - -/// One module-side classify fixture: a seeded store with MODULE memories -/// authority for `route_root`, and a real `McHandler` whose producer factory -/// points at the live host's connection file. -fn classify_rig(connection: &Path, route_root: &Path) -> ClassifyRig { - let store_dir = tempfile::tempdir().expect("temp store dir"); - let store = Arc::new(McStore::open(&store_descriptor(store_dir.path())).expect("store opens")); - let route_root_key = route_root.to_string_lossy().into_owned(); - activate_module_authority(&store, CLASSIFY_CONTEXT, CLASSIFY_PROJECT, &route_root_key); - let generation = store - .authority_status(CLASSIFY_CONTEXT, CLASSIFY_PROJECT, "memories") - .expect("authority readable") - .expect("authority present") - .generation; - - let handler = McHandler::new_with_connection_file(Some(connection.to_path_buf())); - handler.install_store_for_integration(Arc::clone(&store)); - handler.bind_route_for_integration(CLASSIFY_CHANNEL, route_root, "pi", "dreamer-parent"); - ClassifyRig { - store, - handler, - generation, - _store_dir: store_dir, - } -} - -fn dreamer_request(rig: &ClassifyRig, command_id: &str, items: Value, model_chain: Value) -> Value { - // The wire shape TypeScript sends: the module reads `memory_id` to build - // the accept predicate for each attempt's manifest. - let items: Vec = items - .as_array() - .expect("item ids") - .iter() - .map(|id| json!({ "memory_id": id, "content_hash": "seeded-hash" })) - .collect(); - json!({ - "method": "dreamer.run_task", - "v": 1, - "session_id": "dreamer-parent", - "task": "classify", - "command_id": command_id, - "authority_generation": rig.generation, - "payload": { - "prompt_body": "", - "items": items, - "model_chain": model_chain, - // The live chunk-slice remainder TypeScript now owns (R24); any - // deterministic completion lands far inside it. - "timeout_ms": 60_000, - }, - }) -} - -/// The authenticated classify round trip under the OTHER harness (`pi`): the -/// explicit TypeScript-owned model order reaches the backend, exactly the -/// seeded IDs are validated and applied, and every attempt session is deleted -/// (R25, AE14, AE15). -#[tokio::test] -async fn classify_round_trip_applies_seeded_ids_and_deletes_attempt_sessions() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - - let store_dir = tempfile::tempdir().expect("temp store dir"); - let store = Arc::new(McStore::open(&store_descriptor(store_dir.path())).expect("store opens")); - let route_root_key = project.path().to_string_lossy().into_owned(); - activate_module_authority(&store, CLASSIFY_CONTEXT, CLASSIFY_PROJECT, &route_root_key); - let (rule_id, rule_hash) = seed_memory(&store, CLASSIFY_PROJECT, "always run the linter"); - let (fact_id, fact_hash) = seed_memory(&store, CLASSIFY_PROJECT, "the API speaks JSON"); - let generation = store - .authority_status(CLASSIFY_CONTEXT, CLASSIFY_PROJECT, "memories") - .expect("authority readable") - .expect("authority present") - .generation; - - let manifest = format!( - "\ - " - ); - let backend = ScriptedBackend::classify_manifest(manifest.clone()); - let host = RoundTripHost::start(Arc::clone(&backend) as Arc<_>, data_root.path()).await; - - let handler = McHandler::new_with_connection_file(Some(host.connection_file.clone())); - handler.install_store_for_integration(Arc::clone(&store)); - handler.bind_route_for_integration(CLASSIFY_CHANNEL, project.path(), "pi", "dreamer-parent"); - let rig = ClassifyRig { - store: Arc::clone(&store), - handler, - generation, - _store_dir: store_dir, - }; - - let response = response_json( - rig.handler - .dispatch_value_for_integration( - CLASSIFY_CHANNEL, - dreamer_request( - &rig, - "rt-classify", - json!([rule_id, fact_id]), - json!([ - "prov-one/classifier-primary", - "prov-two/classifier-fallback" - ]), - ), - ) - .await, - ); - assert_eq!(response["ok"], json!(true)); - assert_eq!(response["manifest_text"], json!(manifest)); - assert_eq!(response["diagnostics"]["attempts"], json!(1)); - assert_eq!( - response["diagnostics"]["model"], - json!("prov-one/classifier-primary"), - "the first usable model of the explicit request chain wins" - ); + .await; - // The backend saw exactly the TypeScript-owned order head, the derived - // attempt-0 child session, and the route-bound `pi` harness (KTD9-KTD10). - let seen = backend.seen(); - assert_eq!(seen.len(), 1); - let child_session = attempt_child_session_id( - CLASSIFY_PROJECT, - "dreamer-parent", - "rt-classify", - 0, - "prov-one/classifier-primary", - ); - assert_eq!(seen[0].session, child_session); - assert_eq!(seen[0].provider, "prov-one"); - assert_eq!(seen[0].model, "classifier-primary"); - assert_eq!(seen[0].harness, "pi"); + let success_control = fixture.control(1, "backend-success"); + assert_eq!(success_control["id"], 1); + assert_eq!(success_control["ok"], true); + let success = request_json(&client, success_route, send_body("success request")).await; + assert!(success["run_id"].is_string()); + let mut success_stream = subscribe(&client, success_route).await; + let success_items = drain(&mut success_stream).await; assert_eq!( - response["diagnostics"]["child_session_id"], - json!(child_session) - ); - - // Play the TypeScript role: rows come from the returned manifest, then - // apply through the module's own facade surface. - let rows: Vec = parse_classify_manifest( - response["manifest_text"] - .as_str() - .expect("manifest is text"), - ) - .into_iter() - .map(|(id, importance, scope, shareable)| { - let hash = if id == rule_id { - &rule_hash - } else { - &fact_hash - }; - json!({ - "memory_id": id, - "content_hash_at_prompt": hash, - "importance": importance, - "scope": scope, - "shareable": shareable, - }) - }) - .collect(); - let applied = response_json( - rig.handler - .dispatch_value_for_integration( - CLASSIFY_CHANNEL, - json!({ - "name": "memory.set_classification", - "arguments": { - "memory_project": CLASSIFY_PROJECT, - "context_store_uuid": CLASSIFY_CONTEXT, - "authority_generation": rig.generation, - "rows": rows, - }, - }), - ) - .await, + success_items + .iter() + .filter_map(unit_type) + .collect::>(), + ["run_started", "assistant_message", "run_finished"] ); - assert_eq!(applied["accepted"], json!([rule_id, fact_id])); - assert_eq!(applied["rejected"], json!([])); - let rule = rig - .store - .get_memory_full(rule_id) - .expect("memory readable") - .expect("memory present"); - assert_eq!(rule.importance, Some(81)); - assert_eq!(rule.scope, "project"); - assert_eq!(rule.shareable, 1); - let fact = rig - .store - .get_memory_full(fact_id) - .expect("memory readable") - .expect("memory present"); - assert_eq!(fact.importance, Some(17)); - assert_eq!(fact.scope, "universe"); - assert_eq!(fact.shareable, 0); - - // Attempt-session deletion proof over the wire: the child session is - // already a tombstone, so a repeated delete is a side-effect-free success - // (R10, AE7) rather than a purge of anything live. - let metrics = host.supervisor.metrics(); - assert_eq!(metrics.live_runs, 0, "the attempt run was purged"); - assert_eq!(metrics.tombstones, 1, "delete left its bounded tombstone"); - let mut prober = - HistorianProducer::connect(producer_config(&host.connection_file, project.path(), "pi")) - .await - .expect("prober authenticates"); - prober - .purge_session(&child_session) - .await - .expect("repeated session.delete is idempotent"); - assert_eq!(backend.starts(), 1, "the probe started no new backend"); - - let (supervisor, routes) = host.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); -} - -/// A transient first classify model advances to a DISTINCT second attempt -/// session and its retry metadata survives the real wire: typed on the -/// producer error, bounded provider text in the dreamer failure diagnostics -/// (AE13, R18, R22). -#[tokio::test] -async fn transient_first_model_advances_session_and_keeps_retry_metadata() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - let manifest = "".to_owned(); - let backend = ScriptedBackend::classify_manifest(manifest.clone()); - let host = RoundTripHost::start(Arc::clone(&backend) as Arc<_>, data_root.path()).await; - let rig = classify_rig(&host.connection_file, project.path()); - - let response = response_json( - rig.handler - .dispatch_value_for_integration( - CLASSIFY_CHANNEL, - dreamer_request( - &rig, - "rt-fallback", - json!([]), - json!(["prov/flaky-classifier", "prov/steady-classifier"]), - ), - ) - .await, - ); - assert_eq!(response["ok"], json!(true)); - assert_eq!(response["diagnostics"]["attempts"], json!(2)); assert_eq!( - response["diagnostics"]["model"], - json!("prov/steady-classifier") + success_items[1]["unit"]["message"]["content"][0]["text"], + "fixture-success" ); - let seen = backend.seen(); - assert_eq!(seen.len(), 2, "one backend run per attempt"); - assert_eq!( - seen[0].session, - attempt_child_session_id( - CLASSIFY_PROJECT, - "dreamer-parent", - "rt-fallback", - 0, - "prov/flaky-classifier" - ) - ); - assert_eq!( - seen[1].session, - attempt_child_session_id( - CLASSIFY_PROJECT, - "dreamer-parent", - "rt-fallback", - 1, - "prov/steady-classifier" - ) - ); - assert_ne!(seen[0].session, seen[1].session); - // Exhausting a transient-only chain keeps the bounded provider message in - // the dreamer failure diagnostics instead of swallowing it. - let failure = rig - .handler - .dispatch_value_for_integration( - CLASSIFY_CHANNEL, - dreamer_request( - &rig, - "rt-transient-only", - json!([]), - json!(["prov/flaky-classifier"]), - ), + let blocked_route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "blocked-broca", ) .await; - match failure { - subc_client_rs::HandlerOutcome::Error { code, message } => { - assert_eq!(code, "dreamer_run_failed"); - assert!( - message.contains("provider rate limited"), - "provider diagnostics must survive: {message}" - ); - } - other => panic!("expected dreamer_run_failed, got {other:?}"), - } - - // The typed retry metadata itself is producer-visible over the same wire: - // class and retry delay ride the classified error terminal (R18). - let mut producer = - HistorianProducer::connect(producer_config(&host.connection_file, project.path(), "pi")) - .await - .expect("producer authenticates"); - let handle = producer - .start("retry-metadata-probe", "", "probe", "prov/flaky-classifier") - .await - .expect("send admits the probe run"); - match producer.await_output(&handle.run_id).await { - Err(HistorianProducerError::RunFailed { - classification, - class_field_present, - detail, - .. - }) => { - assert!(class_field_present); - let classification = classification.expect("classified terminal"); - assert_eq!(classification.class, ErrorClass::Transient); - assert_eq!(classification.retry_after_secs, Some(7)); - assert!(detail.contains("provider rate limited")); - } - other => panic!("expected a classified run failure, got {other:?}"), - } - producer.close().await.expect("probe routes close"); - - let (supervisor, routes) = host.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); -} - -// --------------------------------------------------------------------------- -// Host restart: strict missing plus the existing refire transition. -// --------------------------------------------------------------------------- - -fn fixed_completion_now_ms() -> i64 { - 2 -} - -/// Broca run state is process-local (R11, AE8): after a restart over the same -/// data_dir the persisted run ID reports strict `missing`, and the existing -/// durable reattach path turns that into the refire-eligible abandon without -/// any Broca persistence or new backend start. -#[tokio::test] -async fn host_restart_reports_missing_and_reattach_becomes_refire_eligible() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - let producer_session = "mc-historian:restart".to_owned(); - - let first_backend = ScriptedBackend::with_behavior(|_request, events, _cancel| { - Box::pin(async move { - events.emit(BackendEvent::AssistantText { - text: "first incarnation output".to_owned(), - finish_reason: None, - }); - BackendTerminal::Completed { - finish_reason: FinishReason::Completed, - } - }) - }); - let host_a = RoundTripHost::start(Arc::clone(&first_backend) as Arc<_>, data_root.path()).await; - let mut producer = HistorianProducer::connect(producer_config( - &host_a.connection_file, - project.path(), - "opencode", - )) - .await - .expect("producer authenticates"); - let handle = producer - .start(&producer_session, "", "chunk", "test/historian-model") - .await - .expect("send admits the run"); - producer - .await_output(&handle.run_id) - .await - .expect("run completes before the restart"); - producer.close().await.expect("routes close"); - drop(producer); - host_a.shutdown().await; - - // Fresh incarnation, same data_dir: run identity must not carry over. - let second_backend = ScriptedBackend::classify_manifest(String::new()); - let host_b = - RoundTripHost::start(Arc::clone(&second_backend) as Arc<_>, data_root.path()).await; - - let mut prober = HistorianProducer::connect(producer_config( - &host_b.connection_file, - project.path(), - "opencode", - )) - .await - .expect("prober authenticates"); - prober.bind_session(&producer_session); - match prober.status(&handle.run_id).await.expect("status settles") { - RunState::Missing { .. } => {} - other => panic!("a restarted host must report strict missing, got {other:?}"), - } - prober.close().await.expect("prober routes close"); - - // Seed the durable AwaitingProducer row the crashed process would have - // left behind, then run the REAL reattach path against the new host. - let store_dir = tempfile::tempdir().expect("temp store dir"); - let store = McStore::open(&store_descriptor(store_dir.path())).expect("store opens"); - let loaded = store.load("ses").expect("fresh session loads"); - let mut meta = loaded.meta; - meta.historian = HistorianDurableState { - state: HistorianPhase::AwaitingProducer, - firing_seq: 1, - chunk_range: Some(HistorianChunkRange { - from_ordinal: 1, - to_ordinal: 3, - }), - chunk_fingerprint: "restart-fp".to_owned(), - selected_range_identities: Vec::new(), - producer_session_id: Some(producer_session.clone()), - producer_run_id: Some(handle.run_id.clone()), - fired_at_ms: Some(1), - expected_revert_epoch: 0, - compartment_set_generation: CompartmentSetGeneration::default(), - ..HistorianDurableState::default() - }; - store - .commit("ses", loaded.row_version, &loaded.core, &meta) - .expect("seeded state commits"); - - let mut reattach_producer = HistorianProducer::connect(producer_config( - &host_b.connection_file, - project.path(), - "opencode", - )) - .await - .expect("reattach producer authenticates"); - let validation_chunk = HistorianChunk { - start_index: 1, - end_index: 3, - lines: Vec::new(), - present_ordinals: Vec::new(), - tool_only_ranges: Vec::new(), - completed_tool_arcs: Vec::new(), - }; - let boundary_dates = BTreeMap::new(); - let outcome = reattach_historian_producer( - &mut reattach_producer, - HistorianReattachRequest { - store: &store, - session_id: "ses", - project_path: "restart-project", - observed_chunk_fingerprint: "restart-fp", - validation_chunk: &validation_chunk, - chunk_transcript: "", - raw_chunk_messages: "", - boundary_dates: &boundary_dates, - prior_compartments: &[], - validate_options: ValidateOptions::default(), - publication_floor_ordinal: 0, - now_ms: 1, - failure_backoff_at_ms: 60_000, - completion_now_ms: fixed_completion_now_ms, - publication_fence: None, - }, - ) - .await - .expect("reattach settles"); + assert_eq!(fixture.control(2, "block-next-call")["ok"], true); + let blocked = request_json(&client, blocked_route, send_body("blocked request")).await; + assert!(blocked["run_id"].is_string()); + wait_for_counter(&fixture, "blocked", 1).await; + let mut blocked_stream = subscribe(&client, blocked_route).await; + let released = fixture.control(3, "release-blocked-call"); + assert_eq!(released["id"], 3); + assert_eq!(released["result"]["accepted"], true); + let blocked_items = drain(&mut blocked_stream).await; assert_eq!( - outcome, - HistorianReattachOutcome::RefireEligible { firing_seq: 1 }, - "strict missing must drive the existing abandon/refire transition" - ); - let reloaded = store.load("ses").expect("seeded session reloads"); - assert_eq!(reloaded.meta.historian.state, HistorianPhase::Idle); - assert!( - reloaded.meta.historian.failure_backoff_at_ms.is_some(), - "abandon must arm the durable backoff that gates the refire" + blocked_items[1]["unit"]["message"]["content"][0]["text"], + "fixture-released" ); - assert_eq!( - second_backend.starts(), - 0, - "the ledger decides the refire; missing status alone starts nothing (R11)" - ); - - let (supervisor, routes) = host_b.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); -} - -// --------------------------------------------------------------------------- -// Rejected admission: no deterministic backend may ever start. -// --------------------------------------------------------------------------- -/// Length-prefixed auth message, written from the protocol text (§5.2) so the -/// deliberately-failing handshake shares no code with the passing one. -async fn write_auth_message(stream: &mut TcpStream, body: &Value) { - let bytes = serde_json::to_vec(body).expect("auth message serializes"); - stream - .write_all( - &u32::try_from(bytes.len()) - .expect("bounded auth message") - .to_le_bytes(), + let failed_route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "failed-broca", ) - .await - .expect("length writes"); - stream.write_all(&bytes).await.expect("auth body writes"); -} - -async fn read_auth_message(stream: &mut TcpStream) -> Value { - let mut len_bytes = [0u8; 4]; - stream - .read_exact(&mut len_bytes) - .await - .expect("auth length reads"); - let mut body = vec![0u8; u32::from_le_bytes(len_bytes) as usize]; - stream.read_exact(&mut body).await.expect("auth body reads"); - serde_json::from_slice(&body).expect("auth body is JSON") -} - -/// An unauthenticated connection and authenticated binds with an empty root -/// or unsupported harness all fail before admission and start ZERO -/// deterministic backends (R4, AE1). -#[tokio::test] -async fn rejected_connections_and_binds_start_no_backend() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - let (backend, _gate) = ScriptedBackend::stuck_until_gate(); - let host = RoundTripHost::start(Arc::clone(&backend) as Arc<_>, data_root.path()).await; - - // Unauthenticated: a syntactically valid handshake with a forged client - // proof (no key knowledge) must be dropped before any route exists. - let conn = connection_file::read(&host.connection_file).expect("publication validates"); - let endpoint = conn.endpoints.first().expect("published endpoint"); - let mut stream = TcpStream::connect((endpoint.host.as_str(), endpoint.port)) - .await - .expect("loopback connect"); - write_auth_message( - &mut stream, - &json!({"client_nonce": vec![7u8; 32], "role": "client"}), - ) - .await; - let server_message = read_auth_message(&mut stream).await; - assert!(server_message.get("server_nonce").is_some()); - write_auth_message(&mut stream, &json!({"client_auth": vec![0u8; 32]})).await; - let mut probe = [0u8; 1]; - let closed = tokio::time::timeout(BUDGET, stream.read(&mut probe)) - .await - .expect("host reacts to the forged proof"); - assert!( - matches!(closed, Ok(0) | Err(_)), - "a forged client proof must close the connection" - ); - - // Authenticated, empty root: rejected at control validation, before any - // component bind can exist. - let mut empty_root = HistorianProducer::connect(producer_config( - &host.connection_file, - Path::new(""), - "opencode", - )) - .await - .expect("bearer still authenticates"); - empty_root - .start("rejected-root", "", "prompt", "test/model") - .await - .expect_err("an empty project_root cannot bind"); + .await; + assert_eq!(fixture.control(4, "typed-failure")["ok"], true); + let failed = request_json(&client, failed_route, send_body("failure request")).await; + assert!(failed["run_id"].is_string()); + let mut failed_stream = subscribe(&client, failed_route).await; + let failed_items = drain(&mut failed_stream).await; + let error = failed_items + .iter() + .find(|item| unit_type(item) == Some("error")) + .expect("typed failure event"); + assert_eq!(error["unit"]["error"]["class"], "permanent"); + assert_eq!(error["unit"]["error"]["provider_code"], "fixture_terminal"); - // Authenticated, unsupported harness: reaches the Broca bind and is - // rejected there, untranslated (R4). - let mut bad_harness = HistorianProducer::connect(producer_config( - &host.connection_file, - project.path(), - "webstorm", - )) - .await - .expect("bearer still authenticates"); - bad_harness - .start("rejected-harness", "", "prompt", "test/model") - .await - .expect_err("an unsupported harness cannot bind"); + let counters = fixture.counters(5); + assert_eq!(counters["started"], 3); + assert_eq!(counters["completed"], 2); + assert_eq!(counters["blocked"], 1); + assert_eq!(counters["released"], 1); + assert_eq!(counters["failed"], 1); + assert_eq!(counters["cancelled"], 0); - assert_eq!( - backend.starts(), - 0, - "no rejected connection or bind may start a deterministic backend" - ); - let (supervisor, routes) = host.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); + client.close().await.expect("managed client closes"); + fixture.shutdown(); } -// --------------------------------------------------------------------------- -// Reserved-capacity isolation under full Broca saturation. -// --------------------------------------------------------------------------- - -/// Exhausted Broca capacity under blocked settlement: the next Broca request -/// is rejected fast while a Magic Context echo still settles on the general -/// class. #[tokio::test] -async fn saturated_broca_reserves_do_not_block_magic_context_echo() { - let data_root = tempfile::tempdir().expect("temp data root"); - let project = tempfile::tempdir().expect("temp project root"); - let (backend, gate) = ScriptedBackend::stuck_until_gate(); - let host = RoundTripHost::start(Arc::clone(&backend) as Arc<_>, data_root.path()).await; - let supervisor = Arc::clone(&host.supervisor); - - let mut wire = RawWire::connect(&host.connection_file).await; - let (mc_channel, mc_epoch) = wire - .route_open( - RouteTarget::ToolProvider { - module_id: "magic-context".to_owned(), - }, - project.path(), - "opencode", - "echo-session", +async fn real_broca_cancel_shutdown_and_full_route_handle_cleanup() { + let fixture = FixtureProcess::start(); + let client = fixture.client().await; + let route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "cancelled-broca", ) .await; - - // Run 0 is the blocked-cancel target and deliberately gets NO subscribers: `run.cancel` commits the cancellation terminal synchronously before blocking on teardown, which would settle that run's subscribers. commentlint: allow(JUDGE) - // 62 held subscriptions plus 32 blocked cancels is therefore the real protocol's maximum blocked settlement; the exact 96-permit class boundary is pinned by mc-host's dispatch test with a saturating stub. commentlint: allow(JUDGE) - let mut broca_routes = Vec::new(); - let mut run_ids = Vec::new(); - for index in 0..32 { - let session = format!("sat-{index}"); - let route = wire - .route_open( - RouteTarget::ManagementSurface { - module_id: "broca".to_owned(), - }, - project.path(), - "opencode", - &session, - ) - .await; - let corr = wire - .send_request( - route.0, - route.1, - json!({ - "method": "session.send", - "params": { - "prompt": format!("saturation {index}"), - "model": {"provider": "test", "model": "gated"}, - "tools": [], - "generation": {"max_output_tokens": 1000, "temperature": 0.1}, - }, - }), - ) - .await; - let frame = wire.frame_for(route.0, route.1, corr).await; - assert_eq!(frame.header.ty, FrameType::Response); - let body: Value = serde_json::from_slice(&frame.body).expect("send response is JSON"); - run_ids.push( - body["run_id"] - .as_str() - .expect("send returns run_id") - .to_owned(), - ); - broca_routes.push(route); - } - // The blocked cancels below target sat-0's run, which must actually hold - // a backend permit — a queued run's cancel would settle immediately and - // never pin its command permit. The eight permit holders start - // concurrently, so the count is awaited rather than asserted: the gate - // blocks every completion, making eight the settled value. - wait_until( - || backend.starts() == 8 && backend.seen().iter().any(|run| run.session == "sat-0"), - "sat-0 and all eight backend permit holders to start", + assert_eq!(fixture.control(1, "block-next-call")["ok"], true); + let sent = request_json(&client, route, send_body("cancel request")).await; + let run_id = sent["run_id"].as_str().expect("run id").to_owned(); + wait_for_counter(&fixture, "blocked", 1).await; + let cancelled = request_json( + &client, + route, + json!({"method": "run.cancel", "params": {"run_id": run_id}}), ) .await; - - // Both subscription positions on every run except the cancel target. commentlint: allow(JUDGE) - for route in &broca_routes[1..] { - for _ in 0..2 { - wire.send_request( - route.0, - route.1, - json!({"method": "session.subscribe", "params": {"from": "start"}}), - ) - .await; - } - } - wait_until( - || supervisor.metrics().free_subscriber_permits == 2, - "all 62 reachable subscriber permits to be held", + assert_eq!(cancelled["ok"], true); + wait_for_counter(&fixture, "cancelled", 1).await; + let status = request_json( + &client, + route, + json!({"method": "run.status", "params": {"run_id": run_id}}), ) .await; - - // 32 blocked cancels: the backend ignores its cancel token, so each - // cancel waits on run teardown while holding one command permit. - for _ in 0..32 { - wire.send_request( - broca_routes[0].0, - broca_routes[0].1, - json!({"method": "run.cancel", "params": {"run_id": run_ids[0]}}), + assert_eq!(status["state"], "cancelled"); + + client.close_route(route).await.expect("old route closes"); + let replacement = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "replacement-broca", ) .await; - } - wait_until( - || supervisor.metrics().free_command_permits == 0, - "all 32 command permits to be held", - ) - .await; - - // The next Broca request fails fast at its exhausted application cap, never blocking behind the 94 requests stuck in settlement. commentlint: allow(JUDGE) - let overflow_corr = wire - .send_request( - broca_routes[1].0, - broca_routes[1].1, - json!({"method": "run.status", "params": {"run_id": run_ids[1]}}), + assert_ne!(route, replacement, "reused channel must carry a new epoch"); + let old = client + .request( + route, + serde_json::to_vec(&send_body("stale route")).unwrap(), + RequestOptions::default(), ) - .await; - let overflow = wire - .frame_for(broca_routes[1].0, broca_routes[1].1, overflow_corr) - .await; - assert_eq!(overflow.header.ty, FrameType::Error); - assert_eq!(error_code(&overflow), "queue_full"); - - // The general class is untouched: the echo settles concurrently. - let echo_corr = wire - .send_request(mc_channel, mc_epoch, json!({"probe": "echo"})) - .await; - let echo = wire.frame_for(mc_channel, mc_epoch, echo_corr).await; - assert_eq!(echo.header.ty, FrameType::Response); - let echo_body: Value = serde_json::from_slice(&echo.body).expect("echo body is JSON"); - assert_eq!(echo_body["served_by"], json!("magic-context")); + .await + .expect_err("closed full handle is rejected"); + assert_eq!(old.code(), "route_not_live"); + let replacement_response = request_json(&client, replacement, send_body("new route")).await; + assert!(replacement_response["run_id"].is_string()); - // Unblock every gated backend before shutdown so cancels, queued runs, - // and subscriptions all settle inside the drain budget. - gate.add_permits(64); - drop(wire); - let (supervisor, routes) = host.shutdown().await; - assert_supervisor_drained(&supervisor, &routes); + client.close().await.expect("managed client closes"); + fixture.shutdown(); } diff --git a/crates/mc-module/tests/direct_host.rs b/crates/mc-module/tests/direct_host.rs new file mode 100644 index 000000000..5cfdc2320 --- /dev/null +++ b/crates/mc-module/tests/direct_host.rs @@ -0,0 +1,460 @@ +#![cfg(unix)] +#![forbid(unsafe_code)] + +mod support; + +use std::fs; +use std::process::Command; +use std::time::{Duration, Instant}; + +use mc_host::TargetKind; +use mc_store::{McStore, StoredCompartment}; +use serde_json::{json, Value}; +use support::direct_host::{ + mode, request_json, send_body, storage_descriptor, workspace_root, FixtureProcess, BUDGET, + REDACTION_SENTINEL, +}; + +fn base64(bytes: &[u8]) -> String { + const TABLE: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let mut encoded = String::with_capacity(bytes.len().div_ceil(3) * 4); + for chunk in bytes.chunks(3) { + let value = (u32::from(chunk[0]) << 16) + | (u32::from(*chunk.get(1).unwrap_or(&0)) << 8) + | u32::from(*chunk.get(2).unwrap_or(&0)); + encoded.push(TABLE[((value >> 18) & 0x3f) as usize] as char); + encoded.push(TABLE[((value >> 12) & 0x3f) as usize] as char); + encoded.push(if chunk.len() > 1 { + TABLE[((value >> 6) & 0x3f) as usize] as char + } else { + '=' + }); + encoded.push(if chunk.len() > 2 { + TABLE[(value & 0x3f) as usize] as char + } else { + '=' + }); + } + encoded +} + +fn redaction_forms(publication: &str) -> Vec { + let connection: Value = serde_json::from_str(publication).expect("publication JSON"); + let mut forms = vec![publication.to_owned()]; + for field in ["key", "daemon_id"] { + let bytes = connection[field] + .as_array() + .expect("byte array") + .iter() + .map(|value| value.as_u64().unwrap() as u8) + .collect::>(); + forms.push(serde_json::to_string(&bytes).unwrap()); + forms.push(format!( + "[{}]", + bytes + .iter() + .map(u8::to_string) + .collect::>() + .join(", ") + )); + forms.push(bytes.iter().map(|byte| format!("{byte:02x}")).collect()); + forms.push(base64(&bytes)); + } + forms +} + +async fn wait_for_store(client: &mc_host::Client, route: mc_host::RouteHandle, session: &str) { + let deadline = Instant::now() + BUDGET; + loop { + let body = serde_json::to_vec(&json!({"kind": "status", "session_id": session})).unwrap(); + match client + .request( + route, + body, + mc_host::RequestOptions { + timeout: BUDGET, + cancellation: None, + }, + ) + .await + { + Ok(response) => { + let status: Value = serde_json::from_slice(&response.body).unwrap(); + if status["store_open"] == true { + return; + } + } + Err(error) if error.code() == "store_unavailable" => {} + Err(error) => panic!("store readiness request failed: {error}"), + } + assert!(Instant::now() < deadline, "module store did not open"); + tokio::task::yield_now().await; + } +} + +#[tokio::test] +async fn readiness_permissions_catalog_and_real_unary_transform() { + let fixture = FixtureProcess::start(); + let immediate_control = fixture.control(0, "counters"); + assert_eq!(immediate_control["ok"], true); + assert_eq!(mode(fixture.root()), 0o700); + assert_eq!(mode(&fixture.control_path()), 0o600); + assert_eq!(mode(&fixture.connection_file()), 0o600); + + let info = mc_host::read_connection_file(fixture.connection_file()) + .expect("strict connection publication"); + assert_eq!(info.wire_version, 2); + assert_eq!( + fixture.readiness()["catalog"], + json!(["magic-context", "synapse", "broca"]) + ); + + let client = fixture.client().await; + let session = "direct-unary"; + let primary = fixture + .open_route(&client, "magic-context", TargetKind::ToolProvider, session) + .await; + let synapse = fixture + .open_route( + &client, + "synapse", + TargetKind::ManagementSurface, + "synapse-route", + ) + .await; + let broca = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "broca-route", + ) + .await; + wait_for_store(&client, primary, session).await; + + let response = request_json( + &client, + primary, + json!({ + "kind": "transform", + "v": 2, + "session_id": session, + "serializer_profile": "owned-llmrunner", + "render_config": "direct-host-config", + "full_array_fingerprint": "direct-host-fingerprint", + "messages": [{ + "mid": "m1", + "ordinal": 1, + "ck": { + "role": "user", + "content": [{"kind": {"type": "text", "text": "direct host request"}}], + "meta": {"harness_id": "m1"} + } + }] + }), + ) + .await; + assert_eq!(response["status"], "ok"); + assert_eq!(response["served_from"], "transform"); + assert_eq!( + response["full_array_fingerprint"], + "direct-host-fingerprint" + ); + + client + .close_route(primary) + .await + .expect("primary route closes"); + client + .close_route(synapse) + .await + .expect("synapse route closes"); + client.close_route(broca).await.expect("broca route closes"); + client.close().await.expect("managed client closes"); + fixture.shutdown(); +} + +#[tokio::test] +async fn direct_primary_replays_transform_state_across_fixture_restart() { + let root = tempfile::tempdir().expect("persistent fixture root"); + fs::create_dir_all(root.path().join("project")).expect("project root"); + let descriptor = storage_descriptor(root.path()); + let store = McStore::open(&descriptor).expect("seed store opens"); + store + .replace_compartments( + "restart-transform", + &[StoredCompartment { + sequence: 1, + start_message: 1, + end_message: 10, + end_message_id: "m10#0".to_owned(), + title: "Seeded compartment".to_owned(), + content: "RESTART-SUMMARY".to_owned(), + p1: Some("RESTART-SUMMARY".to_owned()), + importance: 50, + ..Default::default() + }], + ) + .expect("compartment seed commits"); + drop(store); + + let request = json!({ + "kind": "transform", + "v": 2, + "session_id": "restart-transform", + "serializer_profile": "owned-llmrunner", + "render_config": "restart-config", + "messages": [ + { + "mid": "m10", + "ordinal": 10, + "ck": { + "role": "user", + "content": [{"kind": {"type": "text", "text": "covered"}}], + "meta": {"harness_id": "m10"} + } + }, + { + "mid": "m11", + "ordinal": 11, + "ck": { + "role": "user", + "content": [{"kind": {"type": "text", "text": "tail"}}], + "meta": {"harness_id": "m11"} + } + } + ] + }); + + let first = FixtureProcess::start_at(root.path().to_path_buf()); + let client = first.client().await; + let route = first + .open_route( + &client, + "magic-context", + TargetKind::ToolProvider, + "restart-transform", + ) + .await; + wait_for_store(&client, route, "restart-transform").await; + let materialized = request_json(&client, route, request.clone()).await; + assert_eq!(materialized["action"], "HARD"); + let first_m0 = materialized["ck_messages"] + .as_array() + .expect("ck messages") + .iter() + .find(|message| message["meta"]["synthetic"] == true) + .expect("synthetic m0")["content"][0]["kind"]["text"] + .as_str() + .expect("m0 text") + .to_owned(); + assert!(first_m0.contains("RESTART-SUMMARY")); + client.close().await.expect("first client closes"); + first.shutdown(); + + let second = FixtureProcess::start_at(root.path().to_path_buf()); + let client = second.client().await; + let route = second + .open_route( + &client, + "magic-context", + TargetKind::ToolProvider, + "restart-transform", + ) + .await; + wait_for_store(&client, route, "restart-transform").await; + let replay = request_json(&client, route, request).await; + assert_eq!(replay["action"], "SOFT+"); + let replay_m0 = replay["ck_messages"] + .as_array() + .expect("ck messages") + .iter() + .find(|message| message["meta"]["synthetic"] == true) + .expect("synthetic m0")["content"][0]["kind"]["text"] + .as_str() + .expect("m0 text"); + assert_eq!(replay_m0, first_m0); + client.close().await.expect("second client closes"); + second.shutdown(); +} + +#[tokio::test] +async fn malformed_unknown_duplicate_and_overcap_controls_do_not_mutate_backend() { + let fixture = FixtureProcess::start(); + let publication = fs::read_to_string(fixture.connection_file()).expect("publication readable"); + let redaction_forms = redaction_forms(&publication); + let sensitive_blob = format!("{}|{}", REDACTION_SENTINEL, redaction_forms.join("|")); + let before = fixture.counters(1); + + let unknown = fixture.control_raw( + format!( + "{}\n", + json!({"id": 2, "command": {"name": format!("unknown-{REDACTION_SENTINEL}")}}) + ) + .as_bytes(), + ); + assert_eq!(unknown["id"], 2); + assert_eq!(unknown["ok"], false); + assert_eq!(unknown["error"]["code"], "unknown_command"); + + let malformed = fixture.control_raw( + format!( + "{{\"id\":3,\"secret\":{},\"command\":\n", + json!(&sensitive_blob) + ) + .as_bytes(), + ); + assert_eq!(malformed["ok"], false); + assert_eq!(malformed["error"]["code"], "malformed_request"); + + let duplicate = fixture.control_raw( + format!( + "{{\"id\":4,\"id\":5,\"secret\":{},\"command\":{{\"name\":\"block-next-call\"}}}}\n", + json!(&sensitive_blob) + ) + .as_bytes(), + ); + assert_eq!(duplicate["ok"], false); + assert_eq!(duplicate["error"]["code"], "malformed_request"); + + let mut oversized = serde_json::to_string(&sensitive_blob).unwrap().into_bytes(); + oversized.resize(64 * 1024 + 1, b'x'); + oversized.push(b'\n'); + let overcap = fixture.control_raw(&oversized); + assert_eq!(overcap["ok"], false); + assert_eq!(overcap["error"]["code"], "request_too_large"); + + assert_eq!(fixture.counters(6), before); + + let client = fixture.client().await; + let route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "malformed-control", + ) + .await; + let sent = request_json(&client, route, send_body("state probe")).await; + assert!(sent["run_id"].is_string()); + let deadline = Instant::now() + BUDGET; + loop { + let counters = fixture.counters(7); + if counters["completed"] == 1 { + assert_eq!(counters["blocked"], 0); + break; + } + assert!( + Instant::now() < deadline, + "default success state was mutated" + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + client.close().await.expect("managed client closes"); + let readiness = fixture.readiness().to_string(); + let output = fixture.shutdown(); + let surfaces = format!( + "{unknown}\n{malformed}\n{duplicate}\n{overcap}\n{before}\n{readiness}\n{}\n{}", + output.stdout, output.stderr + ); + assert!(!surfaces.contains(REDACTION_SENTINEL)); + for form in redaction_forms { + assert!( + !surfaces.contains(&form), + "secret form leaked across fixture surfaces: {form}" + ); + } +} + +#[tokio::test] +async fn control_shutdown_cleans_state_and_redacts_all_fixture_surfaces() { + let fixture = FixtureProcess::start(); + let publication = fs::read_to_string(fixture.connection_file()).expect("publication readable"); + let client = fixture.client().await; + let route = fixture + .open_route(&client, "broca", TargetKind::ManagementSurface, "redaction") + .await; + assert_eq!(fixture.control(1, "block-next-call")["ok"], true); + let sent = request_json(&client, route, send_body(REDACTION_SENTINEL)).await; + assert!(sent["run_id"].is_string()); + let deadline = Instant::now() + BUDGET; + let counters = loop { + let counters = fixture.counters(7); + if counters["blocked"] == 1 { + break counters; + } + assert!(Instant::now() < deadline, "backend call never blocked"); + tokio::time::sleep(Duration::from_millis(10)).await; + }; + let readiness = fixture.readiness().to_string(); + let output = fixture.shutdown(); + drop(client); + let surfaces = format!( + "{readiness}\n{counters}\n{}\n{}", + output.stdout, output.stderr + ); + assert!(!surfaces.contains(REDACTION_SENTINEL)); + for form in redaction_forms(&publication) { + assert!( + !surfaces.contains(&form), + "secret form leaked across fixture surfaces: {form}" + ); + } +} + +#[tokio::test] +async fn sigterm_releases_blocked_backend_and_cleans_runtime_state() { + let mut fixture = FixtureProcess::start(); + let block = fixture.control(1, "block-next-call"); + assert_eq!(block["id"], 1); + assert_eq!(block["ok"], true); + let client = fixture.client().await; + let route = fixture + .open_route( + &client, + "broca", + TargetKind::ManagementSurface, + "sigterm-blocked", + ) + .await; + let response = request_json(&client, route, send_body(REDACTION_SENTINEL)).await; + assert!(response["run_id"].is_string()); + + let deadline = Instant::now() + BUDGET; + while fixture.counters(2)["blocked"] != 1 { + assert!(Instant::now() < deadline, "backend call never blocked"); + tokio::time::sleep(Duration::from_millis(10)).await; + } + fixture.signal_term(); + drop(client); + let output = fixture.wait_for_exit(); + assert!(!output.stdout.contains(REDACTION_SENTINEL)); + assert!(!output.stderr.contains(REDACTION_SENTINEL)); +} + +#[test] +fn cargo_metadata_has_no_ck_mc_binary() { + let output = Command::new("cargo") + .args(["metadata", "--format-version", "1", "--no-deps"]) + .current_dir(workspace_root()) + .output() + .expect("cargo metadata runs"); + assert!(output.status.success(), "cargo metadata failed"); + let metadata: Value = serde_json::from_slice(&output.stdout).expect("metadata JSON"); + let package = metadata["packages"] + .as_array() + .expect("packages") + .iter() + .find(|package| package["name"] == "mc-module") + .expect("mc-module package"); + let targets = package["targets"].as_array().expect("targets"); + let removed_binary = ["ck", "mc"].join("-"); + assert!(targets.iter().all(|target| { + target["name"] != removed_binary + && !target["kind"] + .as_array() + .expect("target kind") + .iter() + .any(|kind| kind == "bin") + })); +} diff --git a/crates/mc-module/tests/host_adapter.rs b/crates/mc-module/tests/host_adapter.rs new file mode 100644 index 000000000..1dd93ec83 --- /dev/null +++ b/crates/mc-module/tests/host_adapter.rs @@ -0,0 +1,157 @@ +use std::path::Path; +use std::time::Duration; + +use cortexkit_store_types::StorageDescriptor; +use mc_host::{ + BindOutcome, CompositeComponent, HealthStatus, HostInit, PrimaryComponent, RouteHandle, + RouteIdentity, +}; +use mc_module::{dev_descriptor_at, McHandler}; +use mc_store::McStore; + +fn assert_primary() {} + +fn identity(root: &Path, session: &str) -> RouteIdentity { + RouteIdentity { + project_root: root.to_path_buf(), + harness: "test".to_owned(), + session: session.to_owned(), + consumer_module_id: None, + consumer_launch_nonce: None, + consumer_capabilities: Vec::new(), + admission_facts: None, + } +} + +fn init(descriptor: &StorageDescriptor) -> HostInit { + HostInit { + subc_capabilities: Vec::new(), + storage: Some(serde_json::to_value(descriptor).expect("storage descriptor serializes")), + } +} + +#[tokio::test] +async fn host_lifecycle_uses_full_route_handles() { + assert_primary::(); + let data = tempfile::tempdir().unwrap(); + let descriptor = dev_descriptor_at(data.path().to_str().unwrap()); + let handler = McHandler::new(); + + let manifest = handler.manifest(); + assert_eq!(manifest.module_id, "magic-context"); + assert_eq!(manifest.provides[0]["role"], "tool_provider"); + assert!(handler.resources().reserved_handler_tasks == 0); + PrimaryComponent::initialize(&handler, init(&descriptor)) + .await + .unwrap(); + + let old = RouteHandle { + channel: 17, + epoch: 4, + }; + let newer = RouteHandle { + channel: 17, + epoch: 5, + }; + assert!(matches!( + handler.bind(old, identity(data.path(), "old")).await, + BindOutcome::Accept + )); + assert!(matches!( + handler.bind(newer, identity(data.path(), "new")).await, + BindOutcome::Accept + )); + handler.route_gone(old).await; + assert_ne!(handler.health().await.status, HealthStatus::Failing); + + handler.route_gone(newer).await; + handler.shutdown().await.unwrap(); + assert!(PrimaryComponent::initialize(&handler, init(&descriptor)) + .await + .is_err()); +} + +#[tokio::test] +async fn invalid_initialization_fails_and_partial_shutdown_is_safe() { + let handler = McHandler::new(); + let error = PrimaryComponent::initialize( + &handler, + HostInit { + subc_capabilities: Vec::new(), + storage: Some(serde_json::json!({"backend": "not-a-storage-descriptor"})), + }, + ) + .await + .expect_err("invalid storage must prevent publication"); + assert!(error + .to_string() + .contains("invalid Magic Context storage descriptor")); + assert_ne!(handler.health().await.status, HealthStatus::Failing); + handler.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn shutdown_cancels_and_joins_blocked_store_open() { + let data = tempfile::tempdir().unwrap(); + let descriptor = dev_descriptor_at(data.path().to_str().unwrap()); + let held = McStore::open(&descriptor).expect("hold single-writer lease"); + let handler = McHandler::new(); + PrimaryComponent::initialize(&handler, init(&descriptor)) + .await + .unwrap(); + + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if handler + .health() + .await + .detail + .as_deref() + .is_some_and(|detail| detail.contains("waiting on storage lease")) + { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("store waiter reached blocked phase"); + + tokio::time::timeout(Duration::from_secs(1), handler.shutdown()) + .await + .expect("shutdown joined blocked waiter") + .unwrap(); + drop(held); + McStore::open(&descriptor).expect("shutdown retained no store lease"); +} + +#[test] +fn adapter_source_has_one_prepared_outcome_and_tracked_spawn_boundary() { + let source = include_str!("../src/lib.rs"); + let production = source + .split("#[cfg(test)]\nmod tests {") + .next() + .expect("production source"); + + assert!(!production.contains("HandlerOutcome")); + assert!(!production.contains("ModuleHandler")); + let removed_client_api = ["subc", "client", "rs"].join("_") + "::"; + assert!(!production.contains(&removed_client_api)); + assert!(!production.contains("tokio::spawn(")); + assert!(production.contains("spawn_module_task")); + assert!(production.contains("task_admission_open")); + assert!(production.contains("self.tasks.close()")); + assert!(production.contains("self.cancel.cancel()")); + assert!(production.contains("self.tasks.wait().await")); + + let transform = production + .split("fn respond_transform") + .nth(1) + .expect("transform preparation") + .split("fn emit_pass_timing") + .next() + .unwrap(); + assert!(transform.contains("PreparedOutput::transform_segments")); + assert!(!transform.contains("serde_json::to_vec")); + assert!(!transform.contains("Vec::with_capacity")); +} diff --git a/crates/mc-module/tests/prepared_output.rs b/crates/mc-module/tests/prepared_output.rs new file mode 100644 index 000000000..62d6212d8 --- /dev/null +++ b/crates/mc-module/tests/prepared_output.rs @@ -0,0 +1,282 @@ +use std::cell::Cell; +use std::io::{self, Write}; +use std::sync::Arc; + +use mc_module::dispatch::{ + PreparedOutcome, PreparedOutput, PreparedOutputError, PreparedSegment, MAX_WIRE_BODY_BYTES, +}; +use serde_json::json; + +fn reserved_vec(output: &PreparedOutput) -> Result, PreparedOutputError> { + let measured = output.measure()?; + let mut destination = Vec::with_capacity(measured.len()); + measured.write_to(&mut destination)?; + Ok(destination) +} + +#[test] +fn json_measurement_matches_small_and_facade_sized_bytes() { + for value in [ + json!({"ok": true, "items": [1, 2, 3]}), + json!({ + "content": [{"type": "text", "text": "x".repeat(900 * 1024)}], + "isError": false, + }), + ] { + let expected = serde_json::to_vec(&value).unwrap(); + let output = PreparedOutput::json(value); + let measured = output.measure().unwrap(); + assert_eq!(measured.len(), expected.len()); + assert_eq!(reserved_vec(&output).unwrap(), expected); + } +} + +#[test] +fn transform_segments_preserve_existing_golden_bytes() { + let messages = vec![ + PreparedSegment::exact(Arc::from( + br#"{"mid":"m1","content":[{"kind":{"text":"hello"}}]}"#.as_slice(), + )), + PreparedSegment::exact(Arc::from( + br#"{"mid":"m2","content":[{"kind":{"text":"cached"}}]}"#.as_slice(), + )), + ]; + let output = PreparedOutput::transform_segments( + json!({"status": "ok", "ck_messages": null, "cache_ttl": "1h"}), + messages, + ) + .unwrap(); + let expected = br#"{"cache_ttl":"1h","ck_messages":[{"mid":"m1","content":[{"kind":{"text":"hello"}}]},{"mid":"m2","content":[{"kind":{"text":"cached"}}]}],"status":"ok"}"#; + + let measured = output.measure().unwrap(); + assert_eq!(measured.len(), expected.len()); + assert_eq!(reserved_vec(&output).unwrap(), expected); +} + +struct ReservationWriter<'a> { + reserved: &'a Cell, + writes: &'a Cell, + bytes: Vec, +} + +impl<'a> ReservationWriter<'a> { + fn reserve(reserved: &'a Cell, writes: &'a Cell, capacity: usize) -> Self { + reserved.set(true); + Self { + reserved, + writes, + bytes: Vec::with_capacity(capacity), + } + } +} + +impl Write for ReservationWriter<'_> { + fn write(&mut self, bytes: &[u8]) -> io::Result { + assert!(self.reserved.get(), "copy occurred before reservation"); + self.writes.set(self.writes.get() + 1); + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +#[test] +fn cached_bytes_copy_only_after_destination_reservation() { + let expected = br#"{"cached":true,"value":"stable"}"#.to_vec(); + let output = PreparedOutput::cached_bytes(expected.clone()); + let reserved = Cell::new(false); + let writes = Cell::new(0); + + let measured = output.measure().unwrap(); + assert!(!reserved.get()); + assert_eq!(writes.get(), 0); + + let mut destination = ReservationWriter::reserve(&reserved, &writes, measured.len()); + measured.write_to(&mut destination).unwrap(); + assert!(writes.get() > 0); + assert_eq!(destination.bytes, expected); +} + +#[test] +fn typed_errors_and_stream_markers_have_no_prepared_body() { + let error = PreparedOutcome::Error { + code: "invalid_params".to_string(), + message: "bad request".to_string(), + }; + let streamed = PreparedOutcome::Streamed; + let response = PreparedOutcome::Response(PreparedOutput::json(json!({"ok": true}))); + + assert!(matches!(error, PreparedOutcome::Error { .. })); + assert!(matches!(streamed, PreparedOutcome::Streamed)); + assert!(matches!(response, PreparedOutcome::Response(_))); +} + +#[derive(Default)] +struct CountingSink { + written: usize, +} + +impl Write for CountingSink { + fn write(&mut self, bytes: &[u8]) -> io::Result { + self.written = self.written.checked_add(bytes.len()).unwrap(); + Ok(bytes.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +#[test] +fn exactly_at_wire_cap_succeeds_without_destination_allocation() { + let output = PreparedOutput::cached_bytes(vec![0x5a; MAX_WIRE_BODY_BYTES]); + let measured = output.measure().unwrap(); + assert_eq!(measured.len(), MAX_WIRE_BODY_BYTES); + + let mut destination = CountingSink::default(); + assert_eq!( + measured.write_to(&mut destination).unwrap(), + MAX_WIRE_BODY_BYTES + ); + assert_eq!(destination.written, MAX_WIRE_BODY_BYTES); +} + +#[test] +fn cap_plus_one_and_arithmetic_overflow_fail_before_write() { + let envelope = json!({"ck_messages": null}); + let fixed_len = br#"{"ck_messages":[]}"#.len(); + let cap_plus_one = PreparedOutput::transform_segments( + envelope.clone(), + vec![PreparedSegment::inconsistent_for_test( + Arc::from([]), + MAX_WIRE_BODY_BYTES + 1 - fixed_len, + )], + ) + .unwrap(); + assert!(matches!( + cap_plus_one.measure(), + Err(PreparedOutputError::BodyTooLarge { + len, + max: MAX_WIRE_BODY_BYTES + }) if len == MAX_WIRE_BODY_BYTES + 1 + )); + + let overflow = PreparedOutput::transform_segments( + envelope, + vec![PreparedSegment::inconsistent_for_test( + Arc::from([]), + usize::MAX, + )], + ) + .unwrap(); + assert!(matches!( + overflow.measure(), + Err(PreparedOutputError::LengthOverflow) + )); +} + +fn settle_with_cancellation( + output: &PreparedOutput, + cancel_before_reserve: bool, + reserve: bool, + cancel_before_write: bool, +) -> Option> { + let measured = output.measure().ok()?; + if cancel_before_reserve || !reserve { + return None; + } + let mut destination = Vec::with_capacity(measured.len()); + if cancel_before_write { + return None; + } + measured.write_to(&mut destination).ok()?; + Some(destination) +} + +#[test] +fn cancellation_before_reservation_or_write_emits_nothing() { + let output = PreparedOutput::json(json!({"ok": true})); + assert_eq!(settle_with_cancellation(&output, true, true, false), None); + assert_eq!(settle_with_cancellation(&output, false, true, true), None); +} + +#[test] +fn reserve_denial_emits_nothing() { + let output = PreparedOutput::cached_bytes(b"cached".to_vec()); + assert_eq!(settle_with_cancellation(&output, false, false, false), None); +} + +struct FailAfter { + remaining: usize, + accepted: usize, +} + +impl Write for FailAfter { + fn write(&mut self, bytes: &[u8]) -> io::Result { + if self.remaining == 0 { + return Err(io::Error::other("injected serializer failure")); + } + let written = bytes.len().min(self.remaining); + self.remaining -= written; + self.accepted += written; + Ok(written) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } +} + +#[test] +fn serializer_failure_retains_no_partial_terminal() { + let output = PreparedOutput::json(json!({"value": "serializer must fail after bytes"})); + let measured = output.measure().unwrap(); + let mut destination = FailAfter { + remaining: 5, + accepted: 0, + }; + let mut terminal = None; + + let result = measured.write_to(&mut destination); + if result.is_ok() { + terminal = Some(destination.accepted); + } + + assert!(matches!(result, Err(PreparedOutputError::Serialize(_)))); + assert!(destination.accepted > 0); + assert_eq!(terminal, None); +} + +#[test] +fn inconsistent_source_reports_length_mismatch_without_emission() { + let output = PreparedOutput::transform_segments( + json!({"ck_messages": null}), + vec![PreparedSegment::inconsistent_for_test( + Arc::from(b"1".as_slice()), + 2, + )], + ) + .unwrap(); + let measured = output.measure().unwrap(); + let mut destination = Vec::with_capacity(measured.len()); + let mut terminal = None; + + let result = measured.write_to(&mut destination); + if result.is_ok() { + terminal = Some(destination.clone()); + } + + let expected = br#"{"ck_messages":[1]}"#; + assert!(matches!( + result, + Err(PreparedOutputError::LengthMismatch { + measured: 20, + written: 19 + }) + )); + assert_eq!(destination, expected); + assert_eq!(terminal, None); +} diff --git a/crates/mc-module/tests/real_daemon.rs b/crates/mc-module/tests/real_daemon.rs deleted file mode 100644 index 3f29b6404..000000000 --- a/crates/mc-module/tests/real_daemon.rs +++ /dev/null @@ -1,657 +0,0 @@ -//! End-to-end acceptance test: the cache-stability transform driven THROUGH a live -//! subc daemon (a real ck-subc spawns mc-module as a provider, and a SubcConsumer -//! calls the `transform` op over the wire). -//! -//! Covered here (the cases drivable through the real production path): the first-pass -//! Hard fold, growing-tail and nonce-only defers (cached prefix byte-stable), an -//! epoch (render-config) Hard, a share-nothing boundary absence that degrades to raw -//! pending-rewrite pass-through, and a process restart replaying byte-identical. The m1 -//! delta SOFT and the deferred-drop drain need a content/reducer producer not yet -//! built, so they are exercised in the library tests with stubbed inputs instead. - -#![forbid(unsafe_code)] - -use std::{ - fs, - path::{Path, PathBuf}, - process::{Child, Command, Stdio}, - sync::atomic::{AtomicU64, Ordering}, - sync::{Mutex, OnceLock}, - time::Duration, -}; - -use serde_json::{json, Value}; -use subc_client_rs::{CallOptions, ConsumerOptions, RetryBackoff, SubcConsumer}; -use subc_protocol::{BindIdentity, RouteTarget}; - -static TEMP_COUNTER: AtomicU64 = AtomicU64::new(0); - -const MODULE_ID: &str = "magic-context"; -// Cold daemon startup takes ~7s on an idle machine (measured); under CI or -// sibling-build load it can exceed 10s, which failed this suite spuriously. -const START_TIMEOUT: Duration = Duration::from_secs(60); - -// ---- process lifecycle ---- - -struct LiveDaemon { - child: Child, - runtime_dir: PathBuf, - config_dir: PathBuf, - connection_file: PathBuf, -} - -impl Drop for LiveDaemon { - fn drop(&mut self) { - let _ = self.child.kill(); - let _ = self.child.wait(); - let _ = fs::remove_dir_all(&self.runtime_dir); - let _ = fs::remove_dir_all(&self.config_dir); - } -} - -struct ModuleProcess { - child: Child, -} - -impl ModuleProcess { - fn kill_and_wait(&mut self) { - let _ = self.child.kill(); - let _ = self.child.wait(); - } -} - -impl Drop for ModuleProcess { - fn drop(&mut self) { - self.kill_and_wait(); - } -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn mc_transform_spine_through_real_daemon() { - // Clear any inherited supervision environment variables so this test opens the - // real daemon as an ordinary client instead of reusing a reserved supervised - // identity. - std::env::remove_var(subc_protocol::SUBC_MODULE_ID_ENV); - std::env::remove_var(subc_protocol::SUBC_LAUNCH_NONCE_ENV); - - let workspace = workspace_root(); - let subconscious = subconscious_root(&workspace); - - // Build the daemon (from the sibling subconscious workspace) and our module. - let daemon_bin = ensure_binary( - &subconscious, - subconscious.join("target/debug/ck-subc"), - &["build", "-p", "subc-core", "--bins"], - ); - let module_bin = ensure_binary( - &workspace, - workspace.join("target/debug/ck-mc"), - &["build", "-p", "mc-module"], - ); - - let temp = unique_temp_dir("mc-module-real-daemon"); - let runtime_dir = temp.join("runtime"); - let config_dir = temp.join("config"); - let data_home = temp.join("data"); // store lands here (dev_descriptor → XDG_DATA_HOME) - fs::create_dir_all(&runtime_dir).unwrap(); - fs::create_dir_all(&data_home).unwrap(); - write_empty_config(&config_dir); - - // Seed the store the spawned module will open. m0/m1 are composed FROM the store, so - // the acceptance vectors need real compartments (a boundary) + memories. We open the - // SAME descriptor the module computes, seed, then DROP the handle to release the - // single-writer lease BEFORE spawning the module (which re-acquires it). This is the - // production reality: the historian/dreamer write the store out of band; the module - // reads it. No test-only wire surface. - seed_store(&data_home); - - let daemon = spawn_daemon(&daemon_bin, &runtime_dir, &config_dir); - wait_for_connection_file(&daemon.connection_file, START_TIMEOUT).await; - - let mut module = spawn_module(&module_bin, &daemon.connection_file, &data_home); - - let consumer = SubcConsumer::connect(&daemon.connection_file, fast_consumer_options()) - .await - .unwrap(); - - // Module registration is asynchronous relative to our first route.open, and the - // daemon's unknown_module is a terminal control-plane reject (route_retry only - // covers transport failures). Under load the module's debug-build boot can lose - // this race by tens of seconds, so poll registration with a bounded probe before - // the first real call instead of relying on call-site retries. - wait_for_module_registration(&consumer, START_TIMEOUT).await; - - // ===== PRODUCTION-PATH cases (session "spine"): m0/m1 composed FROM the seeded store. - // The seed (seed_store) gave "spine" one compartment covering ordinals 1..=10 (end id - // "m10", P1 "SUMMARY-1-10") and no memories. m0 is the compartment SUMMARY, the anchor - // is "m10", and the raw covered message stays in the live array (trimmed from output). ===== - - // bootstrap: the first pass folds Hard. m0 = the decay-rendered compartment summary. - let r = call( - &consumer, - json!({ - "session_id": "spine", "render_config": "cfg0", - "serializer_profile": "owned-llmrunner", - "full_array_fingerprint": "fp-spine-bootstrap", - "messages": [ck("m10", 10, "raw covered"), ck("t11", 11, "tail")] - }), - ) - .await; - assert_eq!(r["status"], "ok"); - assert_eq!(r["served_from"], "transform"); - assert_eq!(r["full_array_fingerprint"], "fp-spine-bootstrap"); - assert_eq!(r["action"], "HARD", "bootstrap must fold Hard"); - assert_eq!( - r["boundary_id"], "m10#0", - "anchor = the compartment's end message id" - ); - assert!( - m0(&r).contains("SUMMARY-1-10"), - "m0 is the summary: {}", - m0(&r) - ); - assert!( - !m0(&r).contains("raw covered"), - "m0 is NOT the raw covered bytes" - ); - assert_eq!(m1(&r), M1_PLACEHOLDER); - assert_eq!( - tail_ids(&r), - vec!["t11"], - "covered raw msg trimmed, tail kept" - ); - assert_eq!(r["committed"], true); - - let status = call_raw( - &consumer, - "spine", - json!({ "kind": "status", "session_id": "spine" }), - ) - .await; - assert_eq!(status["ok"], true); - assert_eq!(status["store_open"], true); - assert_eq!(status["session_id"], "spine"); - assert_eq!(status["pass_trace"]["receive_count"], 1); - assert_eq!(status["pass_trace"]["reject_count"], 0); - assert!( - status["pass_trace"]["last_completed_at_ms"] - .as_i64() - .unwrap() - > 0 - ); - - // growing-tail defers. Send the FULL live array each pass (the module locates the - // boundary "m10" over it). Prefix blocks byte-identical; tail verbatim; no write. - let mut prev_m0: Option = None; - for n in 11..=14u64 { - let mut items = vec![ck("m10", 10, "raw covered")]; - for k in 11..=n { - let bytes = if k == 11 { - "tail".to_string() - } else { - format!("tail{k}") - }; - items.push(ck(&format!("t{k}"), k, &bytes)); - } - let d = call( - &consumer, - json!({ "session_id": "spine", "render_config": "cfg0", "messages": items }), - ) - .await; - assert_eq!(d["action"], "SOFT+", "defer must not bust"); - assert_eq!( - d["committed"], - json!(n > 11), - "only first-seen tail mids persist identity vectors" - ); - if let Some(p) = &prev_m0 { - assert_eq!(&m0(&d), p, "m0 changed on defer over the wire"); - } - let tail: Vec = (11..=n).map(|k| format!("t{k}")).collect(); - assert_eq!(tail_ids(&d), tail, "tail must be verbatim live items"); - prev_m0 = Some(m0(&d)); - } - - // epoch-Hard: a render-config change rematerializes (m0 re-composed from the store). - let e = call( - &consumer, - json!({ - "session_id": "spine", "render_config": "cfg1", - "messages": [ck("m10", 10, "raw covered")] - }), - ) - .await; - assert_eq!(e["action"], "HARD", "epoch change must fold Hard"); - assert!(m0(&e).contains("SUMMARY-1-10")); - - // A share-nothing boundary absence is not a safe re-cut target. It returns the raw - // array, arms the pending-rewrite alarm, and leaves the held lineage intact. - let rev = call( - &consumer, - json!({ "session_id": "spine", "render_config": "cfg1", "messages": [ck("z", 90, "other")] }), - ) - .await; - assert_eq!( - rev["action"], "PASSTHROUGH", - "share-nothing revert degrades raw" - ); - assert_eq!( - rev["reconcile_pending"], false, - "pending raw traffic must not set reconcile" - ); - assert_eq!( - tail_ids(&rev), - vec!["z"], - "raw pass-through returns the live array" - ); - - // The boundary returns (m10 back in the array) → pending clears in a normal defer: it - // writes once to clear the alarm but the prefix stays byte-identical (still SOFT+). - let reconciled = call( - &consumer, - json!({ "session_id": "spine", "render_config": "cfg1", "messages": [ck("m10", 10, "raw covered")] }), - ) - .await; - assert_eq!( - reconciled["action"], "SOFT+", - "reconcile-clear is a defer, not a bust" - ); - assert_eq!(reconciled["reconcile_pending"], false, "flag cleared"); - assert!(m0(&reconciled).contains("SUMMARY-1-10"), "m0 still frozen"); - - // ===== memory folds into m0 from the store (session "soft"): the seed gave it the - // same single compartment AND a memory (id 5, "a durable rule"), so the bootstrap HARD - // composes m0 with that memory in the block. ===== - let boot = call( - &consumer, - json!({ "session_id": "soft", "render_config": "cfg0", "messages": [ck("m10", 10, "raw")] }), - ) - .await; - assert_eq!(boot["action"], "HARD"); - // the memory was seeded before the bootstrap HARD, so it is folded into m0. - assert!( - m0(&boot).contains("a durable rule"), - "memory folded into m0: {}", - m0(&boot) - ); - - // Native serving runs with the module's differential flag below, so both the cold full - // encoder and the incremental path execute and byte-compare inside the real provider process. - let native_request = json!({ - "session_id": "native", - "render_config": "cfg0", - "serializer_profile": "opencode-aisdk", - "serve_native": true, - "full_array_fingerprint": "fp-native", - "messages": [ck("native-1", 1, "native tail")], - "native_messages": [{ - "info": { "id": "native-1", "role": "user", "custom": "preserve" }, - "parts": [{ "type": "text", "text": "native tail" }] - }] - }); - let native_first = call(&consumer, native_request.clone()).await; - assert_eq!(native_first["status"], "ok"); - assert_eq!( - native_first["native_messages"] - .as_array() - .unwrap() - .last() - .unwrap()["info"]["custom"], - "preserve" - ); - let native_replay = call(&consumer, native_request).await; - assert_eq!(native_replay["action"], "SOFT+"); - assert_eq!( - native_replay["timings"]["native_cache_encoded_messages"], 0, - "steady real-daemon native replay must encode no messages" - ); - assert!( - native_replay["timings"]["native_cache_reused_messages"] - .as_u64() - .unwrap_or_default() - > 0 - ); - assert_eq!( - native_replay["native_messages"], native_first["native_messages"], - "real-daemon incremental native replay drifted" - ); - - // ===== restart the module and confirm byte-identical replay (spine session) ===== - module.kill_and_wait(); - drop(module); - tokio::time::sleep(Duration::from_millis(200)).await; // OS releases the single-writer lease - let _module2 = spawn_module(&module_bin, &daemon.connection_file, &data_home); - - // replay the spine at the frozen baseline (boundary "m10" present) → pure defer, no write, - // m0 reproduces byte-identical across the restart (the lineage baseline is durable). - let after = call( - &consumer, - json!({ "session_id": "spine", "render_config": "cfg1", "messages": [ck("m10", 10, "raw covered")] }), - ) - .await; - assert_eq!(after["action"], "SOFT+", "restart must not bust"); - assert_eq!(after["committed"], false, "restart replay writes nothing"); - assert!( - m0(&after).contains("SUMMARY-1-10"), - "lineage m0 reproduces across restart" - ); - - drop(consumer); - drop(daemon); -} - -/// Seed the module's store before it opens (release the single-writer lease before spawn). -/// Mirrors the out-of-band historian/dreamer writers: "spine" gets one compartment (a -/// boundary), "soft" gets the same compartment plus a foldable memory. -fn seed_store(data_home: &Path) { - use mc_store::{McStore, StoredCompartment}; - let descriptor = mc_module::dev_descriptor_at(&data_home.to_string_lossy()); - let store = McStore::open(&descriptor).expect("open store to seed"); - let c = |seq: i64, start: i64, end: i64, end_id: &str, p1: &str| StoredCompartment { - sequence: seq, - start_message: start, - end_message: end, - end_message_id: format!("{end_id}#0"), - title: format!("C{seq}"), - content: p1.to_string(), - p1: Some(p1.to_string()), - importance: 50, - ..Default::default() - }; - store - .replace_compartments("spine", &[c(1, 1, 10, "m10", "SUMMARY-1-10")]) - .unwrap(); - store - .replace_compartments("soft", &[c(1, 1, 10, "m10", "S")]) - .unwrap(); - // A memory under the "soft" session's project identity. The module resolves the project - // from the route binding (the identity's project_root), so seed the memory under the - // SAME deterministic project_root_for("soft") that identity_for() will bind for that - // session — otherwise the module would read a different project's (empty) memory set. - let proj = project_root_for("soft"); - store - .seed_memory(5, &proj, "ARCHITECTURE", "a durable rule", 70) - .unwrap(); - // drop `store` here → release the single-writer lease before the module spawns -} - -const M1_PLACEHOLDER: &str = "(no new content since last materialization)"; - -fn ck(id: &str, ordinal: u64, bytes: &str) -> Value { - json!({ - "mid": id, - "ordinal": ordinal, - "ck": { - "role": "user", - "content": [{ "kind": { "type": "text", "text": bytes } }], - "meta": { "harness_id": id } - } - }) -} - -/// The m0 synthetic message bytes from a response's ck_messages. -fn m0(r: &Value) -> String { - synthetic_bytes(r, 0) -} -fn m1(r: &Value) -> String { - synthetic_bytes(r, 1) -} -fn synthetic_bytes(r: &Value, index: usize) -> String { - let msg = r["ck_messages"] - .as_array() - .unwrap() - .iter() - .filter(|m| m["meta"]["synthetic"] == json!(true)) - .nth(index) - .unwrap_or_else(|| panic!("no synthetic message {index} in ck_messages: {r}")); - msg["content"][0]["kind"]["text"] - .as_str() - .unwrap() - .to_string() -} -/// The non-synthetic tail item ids, in order. -fn tail_ids(r: &Value) -> Vec { - r["ck_messages"] - .as_array() - .unwrap() - .iter() - .filter(|m| m["meta"]["synthetic"] != json!(true)) - .map(|m| m["meta"]["harness_id"].as_str().unwrap_or("").to_string()) - .collect() -} -// ---- helpers (adapted from subc-client-rs/tests/real_daemon.rs) ---- - -async fn call(consumer: &SubcConsumer, mut body: Value) -> Value { - // The handler dispatches on `kind`; tag the envelope as a v2 transform op and - // supply the serializer profile all production transform requests must carry. - if let Value::Object(map) = &mut body { - map.insert("kind".to_string(), Value::String("transform".to_string())); - map.entry("v".to_string()).or_insert_with(|| json!(2)); - map.entry("serializer_profile".to_string()) - .or_insert_with(|| Value::String("owned-llmrunner".to_string())); - } - let session = body - .get("session_id") - .and_then(Value::as_str) - .expect("transform body carries session_id") - .to_string(); - call_raw(consumer, &session, body).await -} - -async fn call_raw(consumer: &SubcConsumer, session: &str, body: Value) -> Value { - // Each logical session uses one stable consumer identity whose `session` matches the - // request body's session_id. That keeps every call for that session on one consistent - // daemon route, and the status/health requests reuse the same route on purpose. - let bytes = consumer - .call( - RouteTarget::ToolProvider { - module_id: MODULE_ID.to_string(), - }, - identity_for(session), - serde_json::to_vec(&body).unwrap(), - fast_call_options(), - ) - .await - .unwrap_or_else(|e| panic!("module call failed: {e:?}")); - serde_json::from_slice(&bytes).unwrap() -} - -fn spawn_daemon(daemon_bin: &Path, runtime_dir: &Path, config_dir: &Path) -> LiveDaemon { - let child = Command::new(daemon_bin) - .env("XDG_RUNTIME_DIR", runtime_dir) - .env("XDG_CONFIG_HOME", config_dir) - .env("SUBC_PORT", "0") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::piped()) - .spawn() - .unwrap_or_else(|e| panic!("failed to spawn daemon {}: {e}", daemon_bin.display())); - LiveDaemon { - child, - runtime_dir: runtime_dir.to_path_buf(), - config_dir: config_dir.to_path_buf(), - connection_file: runtime_dir.join("subc-connection.json"), - } -} - -fn spawn_module(module_bin: &Path, connection_file: &Path, data_home: &Path) -> ModuleProcess { - let mut child = Command::new(module_bin) - .arg("--subc") - .arg(connection_file) - .env(subc_protocol::SUBC_MODULE_ID_ENV, MODULE_ID) - .env("XDG_DATA_HOME", data_home) - .env("MC_NATIVE_ATTACHMENT_DIFFERENTIAL", "1") - .env("MC_PREFIX_PROJECTION_DIFFERENTIAL", "1") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::piped()) - .spawn() - .unwrap_or_else(|e| panic!("failed to spawn module {}: {e}", module_bin.display())); - // The module logs to stderr; an undrained pipe fills its 64KB buffer and the module - // BLOCKS on a stderr write mid-boot, so it never registers (observed as a spurious - // unknown_module reject once boot logging grew past the buffer). Drain continuously - // and forward so failures keep the module's log visible. - if let Some(stderr) = child.stderr.take() { - std::thread::spawn(move || { - use std::io::BufRead as _; - for line in std::io::BufReader::new(stderr).lines() { - let Ok(line) = line else { break }; - eprintln!("mc-module: {line}"); - } - }); - } - ModuleProcess { child } -} - -fn write_empty_config(config_dir: &Path) { - fs::create_dir_all(config_dir.join("cortexkit")).unwrap(); - fs::write( - config_dir.join("cortexkit").join("subc.jsonc"), - serde_json::to_string_pretty(&json!({ "version": 1, "modules": {} })).unwrap(), - ) - .unwrap(); -} - -fn fast_consumer_options() -> ConsumerOptions { - ConsumerOptions { - handshake_timeout: Duration::from_secs(2), - // Debug-build module cold start under parallel cargo load can push the FIRST - // transform (bootstrap HARD) past 10s; this suite gates correctness, not latency. - call_timeout: Duration::from_secs(60), - reconnect_backoff: RetryBackoff { - base: Duration::from_millis(50), - cap: Duration::from_millis(250), - max_attempts: 40, - }, - restored_debounce: Duration::from_millis(10), - } -} - -fn fast_call_options() -> CallOptions { - CallOptions { - // See fast_consumer_options: first-call cold start under load needs headroom. - timeout: Duration::from_secs(60), - route_retry: RetryBackoff { - base: Duration::from_millis(50), - cap: Duration::from_millis(250), - max_attempts: 60, - }, - route_retry_deadline: Duration::from_secs(60), - ..CallOptions::default() - } -} - -/// A DETERMINISTIC project_root per session, shared by `identity_for` (the route binding) -/// and `seed_store` (the memory's project_path) so the module resolves the SAME project a -/// seeded memory was written under. A per-process base keeps runs isolated. -fn project_root_for(session: &str) -> String { - static BASE: OnceLock = OnceLock::new(); - let base = BASE.get_or_init(|| { - let d = unique_temp_dir("mc-module-projects"); - fs::create_dir_all(&d).unwrap(); - // Canonicalize so the seeded project_path matches the binding's project_root after - // any path resolution in the daemon/on_bind (e.g. macOS /var → /private/var). - fs::canonicalize(&d).unwrap_or(d) - }); - let p = base.join(session); - fs::create_dir_all(&p).unwrap(); - p.to_string_lossy().to_string() -} - -/// One stable BindIdentity per logical session: repeated calls for the same session reuse -/// the SAME (target, identity) route (one on_bind), the production "one route per session" -/// shape. The project_root is the deterministic `project_root_for(session)` so seeds match. -fn identity_for(session: &str) -> BindIdentity { - static REG: OnceLock>> = OnceLock::new(); - let reg = REG.get_or_init(|| Mutex::new(std::collections::HashMap::new())); - let mut map = reg.lock().unwrap(); - map.entry(session.to_string()) - .or_insert_with(|| BindIdentity { - project_root: PathBuf::from(project_root_for(session)), - harness: "mc-module-test".to_string(), - session: session.to_string(), - }) - .clone() -} - -async fn wait_for_module_registration(consumer: &SubcConsumer, wait: Duration) { - let deadline = tokio::time::Instant::now() + wait; - loop { - let probe = consumer - .call( - RouteTarget::ToolProvider { - module_id: MODULE_ID.to_string(), - }, - identity_for("registration-probe"), - serde_json::to_vec(&serde_json::json!({ "kind": "status", "v": 1 })).unwrap(), - fast_call_options(), - ) - .await; - match probe { - Ok(_) => return, - Err(err) => { - let text = format!("{err:?}"); - if !text.contains("unknown_module") { - // Registered (or a different failure the real calls will surface) — - // registration itself is no longer the blocker. - return; - } - } - } - if tokio::time::Instant::now() >= deadline { - panic!("module did not register with the daemon within {wait:?}"); - } - tokio::time::sleep(Duration::from_millis(100)).await; - } -} - -async fn wait_for_connection_file(path: &Path, wait: Duration) { - let deadline = tokio::time::Instant::now() + wait; - loop { - if path.exists() { - return; - } - if tokio::time::Instant::now() >= deadline { - panic!("daemon did not write {} within {wait:?}", path.display()); - } - tokio::time::sleep(Duration::from_millis(20)).await; - } -} - -fn ensure_binary(manifest_dir: &Path, path: PathBuf, cargo_args: &[&str]) -> PathBuf { - static BUILD_LOCK: OnceLock> = OnceLock::new(); - let _guard = BUILD_LOCK - .get_or_init(|| Mutex::new(())) - .lock() - .unwrap_or_else(|p| p.into_inner()); - let output = Command::new("cargo") - .args(cargo_args) - .current_dir(manifest_dir) - .output() - .unwrap_or_else(|e| panic!("failed to run cargo {cargo_args:?}: {e}")); - assert!( - output.status.success(), - "cargo {cargo_args:?} failed:\nstdout:\n{}\nstderr:\n{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - assert!(path.exists(), "expected binary at {}", path.display()); - path -} - -fn workspace_root() -> PathBuf { - PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .parent() - .and_then(Path::parent) - .unwrap() - .to_path_buf() -} - -fn subconscious_root(workspace: &Path) -> PathBuf { - workspace.parent().unwrap().join("subconscious") -} - -fn unique_temp_dir(name: &str) -> PathBuf { - let nonce = TEMP_COUNTER.fetch_add(1, Ordering::Relaxed); - std::env::temp_dir().join(format!("{name}-{}-{nonce}", std::process::id())) -} diff --git a/crates/mc-module/tests/support/direct_host.rs b/crates/mc-module/tests/support/direct_host.rs new file mode 100644 index 000000000..00e421508 --- /dev/null +++ b/crates/mc-module/tests/support/direct_host.rs @@ -0,0 +1,386 @@ +use std::fs; +use std::io::{BufRead, BufReader, Read, Write}; +use std::os::unix::fs::PermissionsExt; +use std::os::unix::net::UnixStream; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::sync::{mpsc, Arc, Mutex, OnceLock}; +use std::thread::JoinHandle; +use std::time::{Duration, Instant}; + +use mc_host::{Client, RequestOptions, RouteHandle, RouteIdentity, RouteTarget, TargetKind}; +use serde_json::{json, Value}; + +pub const BUDGET: Duration = Duration::from_secs(20); +pub const CONTROL_FILE: &str = "direct-host-control.sock"; +pub const STORE_FILE: &str = "mc-store.db"; +pub const REDACTION_SENTINEL: &str = "u5-redaction-sentinel-DO-NOT-LOG"; + +static BUILD_LOCK: OnceLock> = OnceLock::new(); + +pub fn workspace_root() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .parent() + .and_then(Path::parent) + .expect("mc-module is under workspace/crates") + .to_path_buf() +} + +fn fixture_binary() -> PathBuf { + let _guard = BUILD_LOCK + .get_or_init(|| Mutex::new(())) + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let workspace = workspace_root(); + let output = Command::new("cargo") + .args([ + "build", + "-p", + "mc-module", + "--example", + "direct_host_fixture", + "--features", + "direct-host-fixture", + ]) + .current_dir(&workspace) + .output() + .expect("cargo builds direct host fixture"); + assert!( + output.status.success(), + "direct host fixture build failed:\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + let target = std::env::var_os("CARGO_TARGET_DIR") + .map(PathBuf::from) + .unwrap_or_else(|| workspace.join("target")); + let binary = target.join("debug/examples/direct_host_fixture"); + assert!(binary.is_file(), "missing fixture at {}", binary.display()); + binary +} + +pub struct FixtureProcess { + child: Option, + root: PathBuf, + _root_owner: Option, + readiness: Value, + stdout: Arc>>, + stderr: Arc>>, + stdout_thread: Option>, + stderr_thread: Option>, +} + +impl FixtureProcess { + pub fn start() -> Self { + Self::start_in(tempfile::tempdir().expect("fixture state root")) + } + + pub fn start_in(root: tempfile::TempDir) -> Self { + let path = root.path().to_path_buf(); + Self::start_at_inner(path, Some(root)) + } + + pub fn start_at(root: PathBuf) -> Self { + Self::start_at_inner(root, None) + } + + fn start_at_inner(root: PathBuf, root_owner: Option) -> Self { + let mut child = Command::new(fixture_binary()) + .arg("--state-root") + .arg(&root) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("start direct host fixture"); + let stdout_pipe = child.stdout.take().expect("fixture stdout"); + let stderr_pipe = child.stderr.take().expect("fixture stderr"); + let stdout = Arc::new(Mutex::new(Vec::new())); + let stderr = Arc::new(Mutex::new(Vec::new())); + let (ready_tx, ready_rx) = mpsc::sync_channel(1); + let stdout_lines = Arc::clone(&stdout); + let stdout_thread = std::thread::spawn(move || { + for line in BufReader::new(stdout_pipe).lines() { + let Ok(line) = line else { break }; + stdout_lines + .lock() + .expect("fixture stdout mutex") + .push(line.clone()); + let _ = ready_tx.try_send(line); + } + }); + let stderr_lines = Arc::clone(&stderr); + let stderr_thread = std::thread::spawn(move || { + for line in BufReader::new(stderr_pipe).lines() { + let Ok(line) = line else { break }; + stderr_lines + .lock() + .expect("fixture stderr mutex") + .push(line); + } + }); + let line = ready_rx.recv_timeout(BUDGET).unwrap_or_else(|error| { + let status = child.try_wait().expect("fixture status"); + let diagnostics = stderr.lock().expect("fixture stderr mutex").join("\n"); + panic!("fixture readiness failed ({error}; status={status:?}): {diagnostics}") + }); + assert!(line.len() <= 64 * 1024, "readiness must be bounded"); + let readiness: Value = serde_json::from_str(&line).expect("readiness JSON"); + assert_eq!(readiness["status"], "ready"); + assert_eq!(readiness["wire_version"], 2); + assert_eq!( + child.try_wait().expect("fixture status after readiness"), + None, + "fixture exited immediately after readiness" + ); + Self { + child: Some(child), + root, + _root_owner: root_owner, + readiness, + stdout, + stderr, + stdout_thread: Some(stdout_thread), + stderr_thread: Some(stderr_thread), + } + } + + pub fn root(&self) -> &Path { + &self.root + } + + pub fn control_path(&self) -> PathBuf { + self.root().join(CONTROL_FILE) + } + + pub fn connection_file(&self) -> PathBuf { + mc_host::runtime_dir_path(Some(self.root())) + .expect("fixture runtime path") + .join(mc_host::CONNECTION_FILE_NAME) + } + + pub fn store_path(&self) -> PathBuf { + self.root().join(STORE_FILE) + } + + pub fn readiness(&self) -> &Value { + &self.readiness + } + + pub fn control(&self, id: u64, name: &str) -> Value { + self.control_raw(format!("{}\n", json!({"id": id, "command": {"name": name}})).as_bytes()) + } + + pub fn control_raw(&self, bytes: &[u8]) -> Value { + let path = self.control_path(); + let mut stream = UnixStream::connect(&path).unwrap_or_else(|error| { + let entries = fs::read_dir(self.root()) + .map(|entries| { + entries + .filter_map(Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .collect::>() + }) + .unwrap_or_default(); + panic!( + "connect control socket {} failed immediately after readiness: {error}; root_exists={}; entries={entries:?}; stderr={}", + path.display(), + self.root().exists(), + self.stderr.lock().expect("fixture stderr mutex").join("\\n") + ); + }); + stream + .set_read_timeout(Some(BUDGET)) + .expect("control read timeout"); + stream.write_all(bytes).expect("write control request"); + let mut response = Vec::new(); + BufReader::new(stream) + .take((64 * 1024 + 1) as u64) + .read_until(b'\n', &mut response) + .expect("read control response"); + assert!(response.len() <= 64 * 1024 + 1, "control response bounded"); + serde_json::from_slice(&response).expect("control response JSON") + } + + pub fn counters(&self, id: u64) -> Value { + let response = self.control(id, "counters"); + assert_eq!(response["id"], id); + assert_eq!(response["ok"], true); + response["result"].clone() + } + + pub async fn client(&self) -> Client { + Client::connect(self.connection_file()) + .await + .expect("managed client connects") + } + + pub async fn open_route( + &self, + client: &Client, + module_id: &str, + kind: TargetKind, + session: &str, + ) -> RouteHandle { + client + .open_route( + RouteTarget { + module_id: module_id.to_owned(), + kind, + }, + identity(self.root(), "opencode", session), + ) + .await + .expect("route opens") + } + + pub fn signal_term(&mut self) { + let pid = self.child.as_ref().expect("fixture child").id().to_string(); + let status = Command::new("kill") + .args(["-TERM", &pid]) + .status() + .expect("send SIGTERM"); + assert!(status.success(), "SIGTERM delivery failed"); + } + + pub fn shutdown(mut self) -> CapturedOutput { + let response = self.control(9_999, "graceful-shutdown"); + assert_eq!(response["ok"], true); + self.wait_for_exit() + } + + pub fn wait_for_exit(&mut self) -> CapturedOutput { + let deadline = Instant::now() + BUDGET; + let status = loop { + if let Some(status) = self + .child + .as_mut() + .expect("fixture child") + .try_wait() + .expect("fixture wait") + { + break status; + } + assert!(Instant::now() < deadline, "fixture exceeded exit budget"); + std::thread::sleep(Duration::from_millis(10)); + }; + assert!(status.success(), "fixture exited with {status}"); + self.child.take(); + if let Some(thread) = self.stdout_thread.take() { + thread.join().expect("stdout reader joins"); + } + if let Some(thread) = self.stderr_thread.take() { + thread.join().expect("stderr reader joins"); + } + let output = CapturedOutput { + stdout: self.stdout.lock().expect("fixture stdout mutex").join("\n"), + stderr: self.stderr.lock().expect("fixture stderr mutex").join("\n"), + }; + assert!(!self.control_path().exists(), "control socket cleaned up"); + assert!( + !self.connection_file().exists(), + "connection publication cleaned up" + ); + let lifecycle = mc_host::lifecycle_dir_path(Some(self.root())) + .expect("lifecycle path") + .join(mc_host::LIFECYCLE_RECORD_NAME); + assert!(!lifecycle.exists(), "lifecycle record cleaned up"); + output + } +} + +impl Drop for FixtureProcess { + fn drop(&mut self) { + if self.child.is_none() { + return; + } + if self + .child + .as_mut() + .and_then(|child| child.try_wait().ok().flatten()) + .is_some() + { + return; + } + let _ = self.control(9_998, "graceful-shutdown"); + let deadline = Instant::now() + BUDGET; + while Instant::now() < deadline { + if self + .child + .as_mut() + .and_then(|child| child.try_wait().ok().flatten()) + .is_some() + { + return; + } + std::thread::sleep(Duration::from_millis(10)); + } + if let Some(child) = self.child.as_mut() { + let _ = child.kill(); + let _ = child.wait(); + } + } +} + +pub struct CapturedOutput { + pub stdout: String, + pub stderr: String, +} + +pub fn identity(root: &Path, harness: &str, session: &str) -> RouteIdentity { + RouteIdentity { + project_root: root.join("project"), + harness: harness.to_owned(), + session: session.to_owned(), + consumer_module_id: None, + consumer_launch_nonce: None, + consumer_capabilities: Vec::new(), + admission_facts: None, + } +} + +pub async fn request_json(client: &Client, route: RouteHandle, body: Value) -> Value { + let response = client + .request( + route, + serde_json::to_vec(&body).expect("request serializes"), + RequestOptions { + timeout: BUDGET, + cancellation: None, + }, + ) + .await + .expect("request succeeds"); + serde_json::from_slice(&response.body).expect("response JSON") +} + +pub fn send_body(prompt: &str) -> Value { + json!({ + "method": "session.send", + "params": { + "prompt": prompt, + "model": {"provider": "fixture", "model": "deterministic"}, + "tools": [], + "generation": {"max_output_tokens": 1_024, "temperature": 0.1} + } + }) +} + +pub fn mode(path: &Path) -> u32 { + fs::symlink_metadata(path) + .expect("path metadata") + .permissions() + .mode() + & 0o777 +} + +pub fn storage_descriptor(root: &Path) -> cortexkit_store_types::StorageDescriptor { + cortexkit_store_types::StorageDescriptor { + module_id: "magic-context".to_owned(), + storage_namespace: "mc_cache".to_owned(), + isolation: cortexkit_store_types::Isolation::Module, + backend: cortexkit_store_types::StorageBackend::Sqlite { + path: root.join(STORE_FILE).to_string_lossy().into_owned(), + }, + } +} diff --git a/crates/mc-module/tests/support/mod.rs b/crates/mc-module/tests/support/mod.rs new file mode 100644 index 000000000..59ae4615b --- /dev/null +++ b/crates/mc-module/tests/support/mod.rs @@ -0,0 +1,4 @@ +#![allow(dead_code)] + +#[cfg(unix)] +pub mod direct_host; diff --git a/docs/evidence/claims-backfill/v84-process-crash.json b/docs/evidence/claims-backfill/v84-process-crash.json index 4f860df58..fce8db839 100644 --- a/docs/evidence/claims-backfill/v84-process-crash.json +++ b/docs/evidence/claims-backfill/v84-process-crash.json @@ -1,8 +1,8 @@ { "schemaVersion": "claims-process-crash-evidence/v1", - "commitUnderTest": "a19337ded6888b0fe650bb143498bf66834af4da", + "commitUnderTest": "bd00ffffbdd026a87c3536d013d3d9c0f7ea8bf7", "dirtyDiffDigestPolicy": "sha256(sorted U6 implementation path + NUL + full file bytes + NUL); evidence file excluded", - "dirtyDiffDigest": "dcda69132df97bb45f92f484bb1dbc242a81ed26a2fcea493a26796409a6d5c3", + "dirtyDiffDigest": "d4665b9db71c39dcc511c10a6e365b01916bebedf9a8ed488709d46e8bc977f3", "implementationFiles": [ "ARCHITECTURE.md", "STRUCTURE.md", diff --git a/docs/mc-host-wire-protocol.md b/docs/mc-host-wire-protocol.md index 3c919f39e..fde7e371c 100644 --- a/docs/mc-host-wire-protocol.md +++ b/docs/mc-host-wire-protocol.md @@ -9,17 +9,9 @@ Task: `magic-context-c50.2`; two-target revision and Synapse application protoco The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** are normative as defined by RFC 2119 and RFC 8174. -When sources disagree, implementations MUST use this authority order: +This document is the direct-only wire authority. `mc-host` owns the Rust wire, authentication, discovery, control, routing, and managed-client contracts. Repository implementations and conformance tests provide executable evidence; historical published-package behavior is provenance only and cannot enable a compatibility path. -1. this direct-profile contract and its explicitly settled decisions; -2. exact published `@cortexkit/subc-client` 0.4.1 behavior; -3. behavior used from published `subc-protocol` 0.10.0, `subc-transport` 0.5.0, `subc-control` 0.1.1, and `subc-client-rs` 0.3.0; -4. current repository consumers and tests; -5. private-version observations, only where committed evidence states them. - -Unobserved private behavior is not authority. The locked private versions (`subc-protocol` 0.12.0, `subc-transport` 0.5.1, `subc-control` 0.1.2, and `subc-client-rs` 0.3.1) remain compatibility risks owned by downstream compiler-closure work. - -This document is sufficient to implement a compatible host and client without private source. Executable fixtures and host/client code are intentionally outside this task. +Canonical version-2 literals remain part of this contract even when they retain `subc` spelling. In particular, `subc_ops`, `subc-connection.json`, `SUBC_MODULE_ID`, `SUBC_LAUNCH_NONCE`, `subc-server-v1`, and `subc-client-v1` MUST NOT be renamed without a separately versioned wire or lifecycle migration. ## 2. Profile, actors, and trust boundary @@ -27,22 +19,21 @@ This document is sufficient to implement a compatible host and client without pr Actors: -- **Host:** `mc-host`; owns credentials, connection generations, channels, epochs, correlations accepted from each peer, and handler lifecycle. -- **TypeScript clients:** `SubcModuleTransport`, the Synapse embedding provider, and the wake-plane catalog probe. -- **Raw Rust client:** `HistorianProducer`; authenticates directly, opens command and subscription routes, sends unary and streaming requests, reads errors, and sends route `Goodbye`. -- **Managed Rust client:** published `SubcConsumer`; owns bounded reconnect and route-open policy around fresh control RPCs. -- **Handler:** the linked static composite; receives synthetic initialization, target-aware bind, request, route-gone, internal health, and shutdown callbacks, and dispatches each to the owning component (`magic-context`, `synapse`, or `broca`). +- **Host:** `mc-host`; owns credentials, connection generations, negotiation, channels, epochs, correlations, and component lifecycle. +- **Managed TypeScript client:** `McHostClient` and `McHostModuleTransport`; own secure discovery, authentication, mandatory negotiation, route epochs, deadlines, cancellation, Ping/Pong, and cleanup for plugin, Synapse, wake-plane, CLI, and fixture callers. +- **Managed Rust client:** `mc_host::Client`; owns the same boundary for `HistorianProducer` and Rust fixtures, including typed send outcomes, streaming, checked correlation allocation, reserved control admission, and deterministic close. +- **Handler:** the directly linked static composite; receives initialization, target-aware bind, request, route-gone, internal health, and shutdown callbacks and dispatches each to `magic-context`, `synapse`, or `broca`. The 32-byte connection key is a bearer capability. Possession grants every direct-profile operation — including host-global `host.shutdown` (Section 7.6) — and permits any `BindIdentity`. Client `role`, `consumer_identity`, `project_root`, `harness`, and `session` are claims or scoping metadata; none grants authority. A key reader MUST therefore be trusted as the same local security principal as the host, and every key reader is stop-capable: a diagnostic or proxy principal that holds the bearer is not read-only, whatever its role label or mount permissions claim. Production transport is unencrypted TCP on numeric IPv4 loopback only. It provides no secrecy or per-frame MAC after authentication. The drive rig's read-only credential mount plus authenticated loopback proxy is a trusted diagnostic exception; it does not make remote transport supported. -Initial secure publication support is Unix-like systems. Windows support is deferred until atomic replacement, ACL validation, instance locking, link handling, and ownership-fenced cleanup have a reviewed contract. A Windows build MUST NOT claim conformance merely because published `subc-transport` can create a file there. +Initial secure publication support is Unix-like systems. Windows support is deferred until atomic replacement, ACL validation, instance locking, link handling, and ownership-fenced cleanup have a reviewed contract. ```mermaid flowchart TB CF[Owner-only connection file] -->|endpoint, key, daemon ID| TS[TypeScript clients] - CF --> RC[Raw and managed Rust clients] + CF --> RC[Managed Rust clients] TS <-->|authenticated v2 frames| H[mc-host] RC <-->|authenticated v2 frames| H H -->|initialize, bind, handle, route-gone, health| M[Linked McHandler] @@ -88,18 +79,21 @@ Clients MUST read `${dataDir}/cortexkit/run/subc-connection.json`. Host and clie Example bytes are deterministic and non-secret. Real key and daemon-ID bytes MUST come from the OS CSPRNG. -Current writers MUST include `wire_version: 2`. Published TypeScript 0.4.1 ignores this additive field. A legacy omission means fixed v2; it never negotiates a downgrade. Any present value other than 2 MUST fail before TCP connect. +Writers MUST include numeric `wire_version: 2`. Clients MUST reject an absent, null, string, fractional, or non-2 value before endpoint selection or TCP dial. There is no omission default and no version downgrade. A client MUST: -1. take one snapshot of the resolved regular file, capped at 65,536 bytes; +1. open the parent and connection file without following links, then take one descriptor-anchored regular-file snapshot capped at 65,536 bytes; 2. reject a larger file before JSON parsing; -3. require schema 1, at least one endpoint, exactly 32 key bytes for this profile, exactly 16 daemon-ID bytes, a numeric PID, and a nonempty daemon version; -4. select `endpoints[0]` only; -5. require host exactly `127.0.0.1` and port `1..=65535`; -6. reject wildcard addresses, IPv6, hostnames, malformed arrays, and insecure ownership or permissions. +3. require schema 1, numeric wire version 2, at least one endpoint, exactly 32 key bytes, exactly 16 daemon-ID bytes, a numeric PID, and a nonempty daemon version; +4. verify owner-only regular-file metadata before and after the read and verify the directory entry still names the same file; +5. select `endpoints[0]` only; +6. require host exactly `127.0.0.1` and port `1..=65535`; +7. reject wildcard addresses, IPv6, hostnames, malformed arrays, replacement, and insecure ownership or permissions. -Published Rust accepts key lengths of at least 32; this profile narrows publication and acceptance to exactly 32 so mixed implementations have one credential shape. +The validated descriptor snapshot is the sole source of credentials and endpoint authority. A client MUST NOT validate by pathname and then reopen that pathname for the key, daemon ID, or endpoint. + +Publication and acceptance require exactly 32 key bytes so all direct implementations use one credential shape. ### 4.2 File and link safety @@ -113,7 +107,7 @@ The host MUST acquire a single-instance lock before minting credentials, binding Stale publication temporaries matching the host's private naming pattern MAY be removed after ten minutes. Cleanup is best effort and MUST NOT delay publication. The host MUST never publish through a symlink. -A trusted container MAY read an explicitly configured read-only symlink or bind mount. Client validation applies to the resolved target: one regular file, owner-controlled source, 64 KiB snapshot cap. Link replacement during validation MUST fail closed. This exception does not permit host publication or cleanup through links. +Host publication, client discovery, and cleanup MUST reject symbolic links and unsafe ancestors. Deployment bind mounts are outside pathname traversal; the mounted file must still pass regular-file, owner, mode, bounded-read, replacement, and descriptor-identity checks. Shutdown removal MUST occur while the instance lock is held. Before unlinking, the host MUST reread metadata without following links and confirm that the file is its own publication, including matching daemon ID. An old process MUST NOT remove a replacement host's credential. The handler is dropped before lock release. @@ -166,7 +160,7 @@ Each authentication message is `u32` little-endian byte length followed by that ### 5.2 Messages and proofs -Fixed parameters from published `subc-transport` 0.5.0: +Host-owned authentication uses these fixed parameters: | Parameter | Value | | --- | --- | @@ -252,18 +246,18 @@ Flags: | ---: | --- | --- | --- | | 0 | `Request` | required | consumer to host; channel-0 control or routed opaque request | | 1 | `Response` | required | host to consumer; unary or control success terminal | -| 2 | `Push` | compatibility-only | decoded and fenced; host does not emit it in this profile | +| 2 | `Push` | reserved | decoded and fenced; host does not emit it in this profile | | 3 | `StreamData` | required | host to consumer; zero or more nonterminal stream items | | 4 | `StreamEnd` | required | host to consumer; terminal, usually zero body | | 5 | `Error` | required | host to consumer; terminal canonical `ErrorBody` | | 6 | `Cancel` | required | consumer to host; best-effort cancellation of matching request | | 7 | `Ping` | required | host to consumer liveness probe | | 8 | `Pong` | required | consumer to host; echoes Ping control identity and flags | -| 9 | `Hello` | compatibility-only, role-invalid | external provider registration; never valid from a consumer | -| 10 | `HelloAck` | compatibility-only, role-invalid | external provider registration; never valid on a consumer connection | +| 9 | `Hello` | reserved, role-invalid | external provider registration is unsupported; never valid from a consumer | +| 10 | `HelloAck` | reserved, role-invalid | external provider registration is unsupported; never valid on a consumer connection | | 11 | `Goodbye` | required | either direction; route or connection teardown | -`Cancel`, `Ping`, `Pong`, and `Goodbye` are pure-header frames and MUST declare `len = 0`. A nonzero body is malformed. `Hello` and `HelloAck` values remain decodable so numeric compatibility is preserved, but receiving either on an authenticated consumer connection is a role violation. The host MUST close that generation; it MUST NOT reinterpret the peer as a provider. +`Cancel`, `Ping`, `Pong`, and `Goodbye` are pure-header frames and MUST declare `len = 0`. A nonzero body is malformed. `Hello` and `HelloAck` numeric assignments remain reserved, but receiving either on an authenticated consumer connection is a role violation. The host MUST close that generation; it MUST NOT reinterpret the peer as a provider. A consumer-originated `Response`, `Push`, `StreamData`, `StreamEnd`, or `Error`, and every host-originated `Request`, are role-invalid. The receiver MUST close the generation rather than extend this profile implicitly. @@ -365,7 +359,7 @@ The direct profile routes exactly three static target pairs. Classification runs A successful classification carries the validated typed target into the handler bind, so the composite dispatches on host-validated data and never re-parses the client body. The Synapse target stays in this matrix even when its model bundle is missing or invalid: classification still succeeds, the bind is invoked, and the component rejects it with terminal `artifact_invalid` (Section 7.5.1). The Broca component serves the five-operation LLM-run management protocol consumed by `HistorianProducer` (`session.send`, `session.subscribe`, `run.status`, `run.cancel`, `session.delete`); its application protocol is specified by the Broca revision that implements it, not this section. That protocol's load-bearing properties for this profile are: run lifetime is detached from transport lifetime (waiter loss never stops a run; only `run.cancel`, `session.delete`, or host shutdown does, and each terminates and reaps the complete harness subprocess group before settling), run state is process-local and bounded (a restarted host reports old run IDs as strict `missing`), and subprocess execution is confined to the hardened OpenCode/Pi adapter trust boundary (no shell, private prompt delivery, daemon-owned environment snapshot, bounded redacted output). A Broca bind additionally requires harness `opencode` or `pi`; any other harness is rejected at bind with `invalid_identity` and no run state. Dynamic routing, provider discovery, and `internal_service` routing remain outside this document. -This exclusion has one known in-repo casualty: `RealSessionResolver` (`crates/mc-module/src/session_resolver.rs`), constructed whenever `McHandler` receives a connection file, unconditionally opens a `management_surface` route to module `thalamus`, and the linked manifest consumes that service. Against a conforming direct host that `route.open` receives terminal `unknown_module` (the kind is recognized but no static module named `thalamus` exists), so stateful facade calls that resolve sessions fail at route-open. This contract deliberately does not add a thalamus-compatible route; `magic-context-c50.4` owns replacing or disabling that resolver path (for example, a host-served session-resolve equivalent or the existing `MissingSessionResolver` fallback) before mc-module runs against this profile. +The direct component exposes no `thalamus` resolver route. A facade request without an explicit or route-bound session returns the existing typed `session_unresolved` result locally and opens no resolver transport route. Bound OpenCode sessions retain their proven direct path. ### 7.3 `catalog.list` @@ -547,9 +541,9 @@ Commit runs inside retained host work (the connection writer task), so cancellin ### 7.7 Transport negotiation and candidate activation -Task: `magic-context-ymc.3`. Negotiation selects one connection-scoped frame transport for the authenticated generation. It is versioned independently of the base wire: this section defines **negotiation version 1**. The base v2 envelope, authentication, frame layout, channel-0 body cap, and loopback-only endpoint rules are unchanged; negotiation is ordinary authenticated channel-0 control traffic. +Negotiation selects one connection-scoped frame transport for the authenticated generation. It is mandatory and versioned independently of the base wire: this section defines **negotiation version 1**. Authentication creates a setup-only generation. Its first post-authentication request MUST be a valid channel-0 `transport.negotiate` request using negotiation version 1. No application or other control traffic is permitted until a valid selection commits the transport. -Negotiation is optional. A client that omits it continues on TCP: the first consumer `Request` whose operation is not `transport.negotiate` — on channel 0 or any routed channel — commits the generation to TCP. Raw and managed legacy clients therefore observe no new handshake or error. +A first application request, routed request, other control operation, `Cancel`, `Pong`, or `Goodbye` retires the setup generation without dispatch or same-generation TCP continuation. A client that receives `unsupported_operation`, `connection_in_use`, a malformed response, a mismatched version, or an unoffered selection while negotiating MUST retire the generation and MUST NOT continue on TCP. #### 7.7.1 Bounds and strict decoding @@ -590,25 +584,23 @@ The host MUST select exactly one offered `(transport, capability_version)` entry A non-TCP grant adds a one-use activation token and a bounded opaque provider descriptor. Both fields are required together on a non-TCP selection, and both are malformed on a TCP selection. `reason` is malformed on a grant: ```json -{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff","descriptor":{}} +{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00000000000000000000000000000000","descriptor":{}} ``` -The example token is a fixed synthetic vector. A real token MUST be freshly generated from the OS CSPRNG for each grant, is bound to the granting generation, and is consumed by exactly one activation. Descriptors and activation tokens are binding data, not authority: the authenticated bootstrap remains the sole authorization boundary, and every non-TCP provider MUST enforce owner-only endpoint access, exclusive peer attachment, provider-incarnation fencing, and stale-descriptor rejection before it yields a candidate channel. +The all-zero example token is a documentation-only placeholder. A real token MUST be freshly generated from the OS CSPRNG for each grant, is bound to the granting generation, and is consumed by exactly one activation. Descriptors and activation tokens are binding data, not authority: the authenticated bootstrap remains the sole authorization boundary, and every non-TCP provider MUST enforce owner-only endpoint access, exclusive peer attachment, provider-incarnation fencing, and stale-descriptor rejection before it yields a candidate channel. A response whose `negotiation_version` differs from the request's grammar version, or whose `selected` entry was not offered, is malformed on the client side and retires the setup; the client MUST NOT treat it as fallback evidence. Tokens, descriptors, and offer parameters MUST stay absent from events, errors, cause chains, stacks, `Debug`/`Display` formatting, and panic output on both sides. #### 7.7.3 Fallback reasons -The fallback vocabulary is closed. Fallback always selects the offered `tcp` entry: +The fallback vocabulary is closed. Fallback always selects the exact offered `tcp` entry: | `reason` | Meaning | | --- | --- | -| `unavailable` | the host has no installed provider for any non-TCP offer | -| `negotiation_version_mismatch` | the host does not speak the requested `negotiation_version`; the request still parsed under version-1 grammar | +| `unavailable` | no installed provider serves an offered non-TCP transport | | `capability_version_mismatch` | an offered transport is installed but no offered `capability_version` intersects the host's | -| `connection_in_use` | the first negotiation arrived after the generation already committed to TCP | -A valid TCP selection carrying one of these reasons, a direct TCP selection, or an exact legacy terminal `unsupported_operation` for `transport.negotiate` is the complete set of TCP-continuation evidence. Timeout, malformed content, an unoffered selection, token mismatch, provider attachment failure, activation or commit failure, channel failure, and base-wire or authentication failure are **not** fallback evidence and MUST fail closed without same-generation TCP fallback. +A direct TCP selection or a valid TCP selection carrying one of these reasons commits TCP. This preserves explicit negotiated fallback when an optional candidate is unavailable. Negotiation-version mismatch, `unsupported_operation`, `connection_in_use`, timeout, malformed content, an unoffered selection, token mismatch, provider attachment failure, activation or commit failure, channel failure, and base-wire or authentication failure are not fallback evidence and MUST fail closed without same-generation TCP continuation. #### 7.7.4 Candidate activation and commit @@ -617,7 +609,7 @@ A non-TCP grant creates a setup-only, non-routable candidate channel. The candid Candidate `Request` correlation 1 and its `Response`: ```json -{"op":"transport.activate","negotiation_version":1,"activation_token":"00112233445566778899aabbccddeeff"} +{"op":"transport.activate","negotiation_version":1,"activation_token":"00000000000000000000000000000000"} ``` ```json @@ -638,25 +630,24 @@ Both responses are tagged and carry no provider data; under Section 7.7.1 strict #### 7.7.5 Setup states -Selection is sticky for the generation: a direct TCP selection, a TCP fallback, or a committed non-TCP grant remains fixed until retirement. Reconnect rereads discovery data, reauthenticates, and negotiates from a fresh state instance. At most one candidate may be prepared per setup. On a TCP-committed generation only the **first** late negotiation returns the offered TCP entry with `connection_in_use`; a second negotiation, or any application-bearing operation while a candidate is being set up, is a protocol failure that retires the generation. +Selection is sticky for the generation: a direct TCP selection, a negotiated TCP fallback, or a committed non-TCP grant remains fixed until retirement. Reconnect rereads discovery data, reauthenticates, and negotiates from a fresh setup-only state. At most one candidate may be prepared. Any late or repeated negotiation, or any application-bearing operation while a candidate is being set up, retires the generation. ```mermaid stateDiagram-v2 [*] --> Authenticating - Authenticating --> BootstrapTcp: auth succeeds + Authenticating --> SetupOnly: auth succeeds Authenticating --> Retired: auth or base-wire failure - BootstrapTcp --> TcpCommitted: TCP selection or legacy traffic - BootstrapTcp --> CandidatePrepared: one candidate prepared - BootstrapTcp --> Retired: malformed or timeout - CandidatePrepared --> Activating: bootstrap liveness joined and grant sent + SetupOnly --> TcpCommitted: valid direct or fallback TCP selection + SetupOnly --> CandidatePrepared: valid optional candidate grant + SetupOnly --> Retired: application traffic, wrong version, malformed exchange, or timeout + CandidatePrepared --> Activating: grant sent CandidatePrepared --> Retired: provider failure Activating --> AwaitingCommit: activation corr 1 acknowledged Activating --> Retired: token, timeout, or channel failure AwaitingCommit --> ProviderActive: commit corr 2 complete and valid AwaitingCommit --> Retired: commit or promotion failure - TcpCommitted --> TcpCommitted: first late negotiation returns connection_in_use - TcpCommitted --> Retired: repeated negotiation, protocol failure, or retirement - ProviderActive --> Retired: channel retirement + TcpCommitted --> Retired: late negotiation or retirement + ProviderActive --> Retired: late negotiation or channel retirement Retired --> [*] ``` @@ -668,17 +659,17 @@ stateDiagram-v2 Host startup order is normative: -1. acquire single-instance lock; -2. mint fresh key and daemon ID; -3. construct one host-owned `ModuleHelloAckBody` from storage and capability configuration; -4. invoke linked handler initialization exactly once and wait for that callback to return; +1. acquire the single-instance lock; +2. mint a fresh key and daemon ID; +3. construct trusted `HostInit` storage and capability configuration; +4. invoke linked component initialization exactly once and wait for it to return; 5. bind `127.0.0.1` on a nonzero port; -6. atomically publish connection file; +6. atomically publish the connection file; 7. accept clients and routes. -The synthetic acknowledgment preserves external registration lifecycle effects without a provider socket. `Hello`/`HelloAck` never appear on a consumer connection. +There is no provider registration socket or synthetic compatibility handshake. `Hello` and `HelloAck` remain role-invalid on consumer connections. -`McHandler::on_hello_ack` begins asynchronous store opening. Publication therefore means transport-ready, not storage-ready. Discovery, authentication, catalog, and route bind MUST work while storage opens. A storage-dependent request during that window receives terminal application error `store_unavailable`; client MUST NOT classify it as transport disconnect. +`McHandler::initialize` may begin asynchronous store opening. Publication therefore means transport-ready, not storage-ready. Discovery, authentication, negotiation, catalog, and route bind MUST work while storage opens. A storage-dependent request during that window receives terminal application error `store_unavailable`; the client MUST NOT classify it as a transport disconnect. ```mermaid sequenceDiagram @@ -687,12 +678,14 @@ sequenceDiagram participant F as Connection file participant C as Client H->>H: lock, fresh key and daemon ID - H->>M: synthetic HelloAck; initialize once + H->>M: HostInit; initialize once H->>H: bind numeric loopback H->>F: atomic owner-only publish C->>F: bounded validate/read C->>H: TCP + three-message authentication - C->>H: v2 envelope traffic + C->>H: first Request transport.negotiate v1 + H-->>C: valid transport selection + C->>H: application v2 envelope traffic ``` ### 8.2 Route allocation and bind @@ -706,7 +699,7 @@ For every valid `route.open`, host MUST: 5. on acceptance, install route then return tagged `route.open` response; 6. on rejection, call route-gone exactly once because handler observed the handle, release it after callback completes, then return terminal bind error. -The channel namespace is process-global because linked `McHandler` keys bindings by `u16` channel alone. Two simultaneous connections, two roots for one session, and two sessions MUST never hold the same live channel. Channel reuse is neither unconditional nor forbidden: it is permitted only after all prior work is settled/cancelled, route-gone completes exactly once, and epoch advances strictly. At `u32::MAX`, that channel is permanently retired for the host incarnation. If all channels are live or retired, host returns terminal `target_unavailable` without calling bind. +The channel namespace is process-global, and `McHandler` keys bindings and cleanup by the complete `(channel, epoch)` handle. Two simultaneous connections, two roots for one session, and two sessions MUST never hold the same live channel. Channel reuse is permitted only after all prior work is settled or cancelled, route-gone completes exactly once, and the epoch advances strictly. Late frames and callbacks for an old epoch cannot observe, mutate, or remove new route state. At `u32::MAX`, that channel is permanently retired for the host incarnation. If all channels are live or retired, the host returns terminal `target_unavailable` without calling bind. A bind stores `project_root`, `harness`, and `session` as handler scope. Multiple routes for one session are valid. Host MUST NOT merge them by session alone. @@ -749,7 +742,7 @@ A routed `Request` carries exactly one correlation. Host may produce: - unary: one `Response` or `Error` terminal; - streaming: zero or more `StreamData` frames, then exactly one `StreamEnd` or `Error` terminal. -All response frames MUST echo channel, epoch, and correlation. `StreamData` is nonterminal. `StreamEnd` is transport terminal; application protocols may define an earlier in-band terminal event. The raw Rust historian intentionally treats its in-band run terminal as authoritative and treats premature `StreamEnd` as failure. +All response frames MUST echo channel, epoch, and correlation. `StreamData` is nonterminal. `StreamEnd` is transport terminal; application protocols may define an earlier in-band terminal event. The managed Rust historian treats its in-band run terminal as authoritative and treats premature `StreamEnd` as failure. Transport never parses routed application bodies. Handler `Response(Vec)` becomes `Response`; handler `Error` becomes canonical `ErrorBody`; handler `Streamed` ends with `StreamEnd` after emitted stream items. @@ -761,7 +754,7 @@ Client sends pure-header `Cancel` with target route and correlation. If request Consumer liveness uses pure-header `Ping`/`Pong`, not a channel-0 JSON health operation. Host sends Ping on channel 0; client returns Pong with identical version, flags, channel, epoch, and correlation. A missed Pong invalidates the connection only under host's bounded liveness policy. -Compatibility note: current `HistorianProducer` (`crates/mc-module/src/historian_producer.rs`) implements no Ping handling — both receive loops discard every frame that does not match the awaited route and correlation, so it can never answer a host Ping. `magic-context-c50.4` owns adding the Pong echo to that client. Until it lands, a host deployment MUST NOT enable missed-Pong connection invalidation, or it will terminate healthy long-running historian awaits (up to 600 s) as dead peers. +Managed Rust and TypeScript readers own Ping/Pong independently of application waits and stream consumption. A Ping during a unary or streaming request MUST produce Pong without settling, cancelling, or delaying the application request. Stream queue saturation MUST NOT block this liveness path. Handler health is host-internal. Host invokes `McHandler::health` on a dedicated control task, never as an ordinary routed request and never while holding handler/store locks. Current handler health is atomics-only and returns `ok`, `degraded`, or `failing` plus optional detail and metrics. Waiting for a predecessor's storage lease reports `degraded` without making transport unready. @@ -786,7 +779,7 @@ Connection close is Goodbye on channel 0, epoch 0, correlation 0, followed by so A completed socket write is not proof of handler dispatch, but absence of proof is insufficient for replay. Partial and uncertain writes are always `outcome_unknown`. -Current `SubcModuleTransport.call()` retries broad request-side connection failures on a fresh generation. That can execute a request twice and is nonconforming unless failure is proven `not_sent` or the operation explicitly owns idempotent replay. `magic-context-c50.5` MUST narrow it: retry only proven pre-send failures and the host's no-dispatch stale-route response; otherwise return `outcome_unknown`. Plugin outer retry MUST NOT silently multiply SDK retries. +Managed Rust and TypeScript clients retry only proven pre-send failures and the host's no-dispatch stale-route response unless an application contract explicitly owns idempotent replay. Any request that may have reached the socket without a terminal returns `outcome_unknown`. Outer caller policy MUST NOT silently multiply client retries. ### 10.2 Error and retry matrix @@ -805,29 +798,25 @@ Current `SubcModuleTransport.call()` retries broad request-side connection failu | malformed framing / EOF | no terminal possible | classify pending writes from byte evidence; invalidate generation | | request deadline after possible write | no | `outcome_unknown`, no generic retry | -A retry is always a new RPC with a new correlation. Terminality of one `unknown_module` response does not prohibit published managed-client policy from issuing a later `route.open`. Private 0.3.1 behavior is unverified drift and does not change this contract. +A retry is always a new RPC with a new correlation. Terminality of one `unknown_module` response does not prohibit managed-client policy from issuing a later `route.open` within its owning deadline. ## 11. Deadlines and backoff layers Every operation owns one absolute deadline; per-stage timers MUST NOT multiply it. Separate domains are authentication, frame body read, route-open policy, request/response, shutdown, SDK reconnect, and plugin reconnect. -Current repository guidance, not wire constants: +Managed Rust and TypeScript client defaults: -| Layer | Current value / bound | +| Owner | Default / bound | | --- | --- | -| TypeScript handshake | 2 s | -| TypeScript transform attempt | 5 s | -| TypeScript ordinary attempt | 15 s | -| TypeScript wrapup attempt | 3,800 s | -| TypeScript connection probe backoff | starts 1 s, caps 30 s | -| Raw Rust handshake | 2 s | -| Raw Rust ordinary request | 30 s | -| Raw Rust historian await | 600 s; application redrain 60 s | -| Session-resolver whole call | 2 s, one configured route attempt | - -Current TypeScript loop allows queue wait plus two full attempts: up to about 15 s for transform, 45 s for ordinary calls, and 11,400 s for wrapup when every phase consumes its bound. These are compatibility observations, not desired retry permission. After `magic-context-c50.5` enforces send outcomes, a second body send is allowed only for `not_sent` or explicit idempotent application policy. +| discovery, dial, authentication, and mandatory negotiation | one 2 s absolute handshake deadline | +| frame completion after first header byte | one 30 s absolute deadline; idle first-header wait is unbounded | +| route open, including retries and backoff | one 30 s absolute deadline | +| request | one caller-overridable 30 s absolute deadline | +| client shutdown and cleanup | one 5 s absolute deadline | +| ordinary queued data frames | 256 slots | +| reserved pure-header `Pong`, `Cancel`, and `Goodbye` frames | 32 slots | -Backoff budget counts the first attempt. Route-open retry and reconnect policy MUST stop at their owning deadline. Retry delay does not reset request deadline. +Data and reserved-control frames share one queued-byte budget; reserved admission is not a byte-budget bypass. Data traffic cannot consume control slots. Exhausting control reserve retires the generation and deterministically settles pending work. Backoff counts the first attempt, and retry delay or a later stage never resets the owning deadline. ## 12. Reconnect, restart, and shutdown @@ -869,26 +858,29 @@ An authenticated `host.shutdown` (Section 7.6) initiates this same graceful orde ### 13.1 Startup, route, call, close -1. Host locks runtime state, creates fresh credentials, invokes synthetic handler initialization, binds loopback, and publishes schema 1. -2. Client validates one bounded file snapshot and completes all three auth messages. -3. Client sends channel-0 `route.open` correlation 1. -4. Host allocates global channel 7, epoch 77, binds handler, and returns tagged response. -5. Client sends opaque request on `(7,77,2)`. -6. Host dispatches once and returns one terminal on `(7,77,2)`. -7. Client sends route Goodbye `(7,77,0)`. -8. Host blocks reuse until task settlement and exactly one route-gone callback complete; any reuse of channel 7 has epoch greater than 77. +1. Host locks runtime state, creates fresh credentials, initializes directly linked components, binds loopback, and publishes schema 1 with `wire_version: 2`. +2. Client validates one descriptor-anchored snapshot and completes all three auth messages. +3. Client sends `transport.negotiate` version 1 as correlation 1 and validates the selected transport. +4. Client sends channel-0 `route.open` correlation 2. +5. Host allocates global channel 7, epoch 77, binds the component, and returns the tagged response. +6. Client sends an opaque request on `(7,77,3)`. +7. Host dispatches once and returns one terminal on `(7,77,3)`. +8. Client sends route Goodbye `(7,77,0)`. +9. Host blocks reuse until task settlement and exactly one route-gone callback complete; any reuse of channel 7 has epoch greater than 77. ### 13.2 Storage opening -Host may publish after `on_hello_ack` starts asynchronous storage acquisition. A client can authenticate, list catalog, and bind a route. A storage-dependent request returns terminal `store_unavailable`. Later fresh request succeeds after storage opens; no reconnect is required. +Host may publish after `McHandler::initialize` starts asynchronous storage acquisition. A client can authenticate, negotiate, list catalog, and bind a route. A storage-dependent request returns terminal `store_unavailable`. A later fresh request succeeds after storage opens; no reconnect is required. ### 13.3 Temporary target absence A `route.open` correlation receiving `unknown_module` is complete. Managed SDK may wait within its route-open deadline and send a new `route.open` with a new correlation. It never reuses the terminal correlation and never sends application body before route success. -### 13.4 Raw Rust streaming client +### 13.4 Managed Rust streaming client -`HistorianProducer` authenticates from the same connection file, opens separate command and subscription routes for one identity, sends unary commands, and consumes matching `StreamData` until its application run terminal. Transport `StreamEnd` before that event is failure. On close it sends Goodbye for both routes. On connection loss it does not replay a possibly sent command; caller creates a fresh producer and application durable replay uses its own run ID/cursor semantics. Model-runner target availability is owned by `magic-context-c50.11`. +`HistorianProducer` uses `mc_host::Client` to discover, authenticate, negotiate, open separate command and subscription routes for one identity, send unary commands, and consume matching `StreamData` until its application run terminal. Transport `StreamEnd` before that event is failure. The managed reader answers Ping while the stream is pending. Every terminal path closes both route handles. + +For `session.send`, the producer freezes exact request bytes, authenticated daemon ID, and `(project, harness, session)` identity. After `outcome_unknown`, it may reconnect and resend those bytes once only if daemon ID and identity are unchanged. Daemon or identity change preserves the typed unknown outcome and performs no resend; the durable driver applies backoff and stops that firing before another model. ### 13.5 Timeout after write @@ -904,8 +896,8 @@ Every scenario has one required outcome. These are review vectors; executable fi | ID | Scenario | Expected result | | --- | --- | --- | -| AE1 | Fresh authenticated call | Valid file, three-message auth, tagged route response, and matching terminal succeed | -| AE2 | Malformed envelope | Unsupported version, type, flags, oversize, or truncation closes generation; no resync/Error | +| AE1 | Fresh authenticated call | Valid version-2 file, three-message auth, mandatory negotiation, tagged route response, and matching terminal succeed | +| AE2 | Malformed envelope or setup | Unsupported frame version, type, flags, oversize, truncation, application-before-negotiation, or invalid negotiation closes the generation; no application dispatch or TCP continuation | | AE3 | Caller-supplied identity | Key holder may select identity; fields scope handler state and add no authority | | AE4 | Temporarily unavailable module | Each `unknown_module` terminates one correlation; policy retry uses a new correlation and never sends body early | | AE5 | Unknown routed channel | Host dispatch count stays zero; client may reopen and retry body exactly once on fresh route | @@ -921,7 +913,7 @@ Every scenario has one required outcome. These are review vectors; executable fi | V2 | File 65,537 bytes | Reject before JSON parsing or key logging | | V3 | Connection file mode `0644` | Reject as insecure; do not connect | | V4 | Hostname, wildcard, IPv6, or port zero | Reject endpoint before connect | -| V5 | Trusted read-only link | Resolve one regular owner-controlled target and authenticate | +| V5 | Symlinked connection file or unsafe ancestor | Reject before dialing; do not follow the link | | V6 | Link target swapped during read | Fail closed; do not combine snapshots | | V7 | Old cleanup after new publish | Daemon-ID mismatch prevents unlink | | V8 | Valid auth JSON padded with whitespace to exactly 4,096 bytes | Pass size/JSON validation and advance to next handshake stage | @@ -941,8 +933,8 @@ Every scenario has one required outcome. These are review vectors; executable fi | V22 | Correlation reaches `u64::MAX` | Use once, then retire/reconnect generation before another request | | V23 | Unauthenticated slow reader | Absolute deadline closes and releases handshake slot | | V24 | Sensitive diagnostics | Key/proof/body/identity secrets redacted; bounded counters remain observable | -| V25 | Raw Rust unary Error | Matching Error becomes terminal `ProducerErrorBody`; no hidden replay | -| V26 | Raw Rust stream disconnect | No complete stream terminal; outcome/recovery handled by caller's durable semantics | +| V25 | Managed Rust unary Error | Matching Error becomes typed terminal `CallError`; no hidden replay | +| V26 | Managed Rust stream disconnect | No complete stream terminal; outcome classification and fenced replay remain caller-owned | | V27 | Two roots for same session | Separate routes and bindings; no session-only aliasing | | V28 | Aggregate resource pressure | Reject new admission/work finitely; one admitted valid max frame remains interoperable | | V29 | Retryable then non-retryable route errors | Allowlisted error may create one fresh correlation; `invalid_control_request` creates none | @@ -967,65 +959,51 @@ Every scenario has one required outcome. These are review vectors; executable fi | V48 | Reserved-class saturation | Every reserved pending/task permit held through blocked settlement rejects the next reserved-class request `server_busy` while a general request still dispatches and settles; saturating the general class never consumes a reserved permit | | V49 | Declarations exceed configured limits | A reservation that leaves zero general pending slots, zero general task slots, or less than one maximum ingress body fails startup before publication | | V50 | Child shutdown failure | A Broca shutdown panic or returned error still drains Synapse and Magic Context; the incarnation reports one deterministic redacted non-graceful failure | +| V51 | Missing, null, string, fractional, or non-2 `wire_version` | Client rejects before endpoint dial | +| V52 | First post-auth request is application traffic or another control operation | Host retires setup generation; zero application dispatch and no TCP continuation | +| V53 | Negotiation receives `unsupported_operation`, `connection_in_use`, malformed response, version mismatch, or unoffered selection | Client retires generation; no application request continues on TCP | +| V54 | Optional candidate unavailable or capability version mismatched | Valid negotiation explicitly selects offered TCP with the matching fallback reason; generation may continue | -### 14.1 Downstream fixture oracle +### 14.1 Fixture oracle -Fixtures MUST use committed literal bytes and an independent decoder/oracle; importing production proof, header, or frame helpers to generate expected values proves only self-consistency. V1-V50 define the deterministic cases. A green suite establishes only that checked implementations, vectors, schedules, platforms, and bounds passed. +Fixtures MUST use committed literal bytes and an independent decoder/oracle; importing production proof, header, or frame helpers to generate expected values proves only self-consistency. V1-V54 define deterministic cases. A green suite establishes only that checked implementations, vectors, schedules, platforms, and bounds passed. Broad Rust E2E, mutation, performance, and release qualification remain owned by `magic-context-c50.9`. ## 15. Consumer traceability | Consumer | Required contract | Verification owner | | --- | --- | --- | -| `packages/plugin/src/hooks/magic-context/module-transport.ts` | v2 auth/frame, route cache by generation, opaque bodies, close race, outcome-safe retry | `magic-context-c50.5` | -| `packages/plugin/src/features/magic-context/memory/embedding-synapse.ts` | managed call and `not_sent` / `outcome_unknown` / `terminal` distinction; its synapse `management_surface` route binds under the Section 7.2 matrix and speaks the Section 7.5 application protocol | `magic-context-c50.5`, synapse protocol in `magic-context-c50.6` | -| `packages/plugin/src/features/magic-context/smart-notes/wake-plane.ts` | tagged truthful catalog; absent `wake.create` fails open (final direct-host posture, `magic-context-c50.7`) | `magic-context-c50.5`, `magic-context-c50.7` | -| `crates/mc-module/src/historian_producer.rs` | raw auth, first endpoint, route open, monotonic correlation, streaming, Error, Goodbye; Ping/Pong echo required (Section 9.3, currently missing) | `magic-context-c50.4`, route target in `magic-context-c50.11` | -| `crates/mc-module/src/session_resolver.rs` | managed Rust route-open deadline and terminal module errors; its thalamus `management_surface` target is unsupported by this profile and MUST be replaced or disabled (Section 7.2) | `magic-context-c50.4` | -| `crates/mc-module/src/lib.rs` | initialize once, bind before response, route-gone once, atomics-only health, store readiness | `magic-context-c50.3` / `.4` | -| `packages/e2e-tests/tests/rust-park-self-heal.test.ts` | existing module-restart/park-heal evidence only; whole-host credential-rotation case still required | `magic-context-c50.9`, after `.11` | -| `scripts/drive-rig/*` | trusted credential mount and loopback proxy exception | drive-rig validation in downstream E2E | - -## 16. Requirement traceability - -| Requirement | Normative sections | Scenarios | +| `McHostModuleTransport` | strict version-2 discovery, mandatory negotiation, generation and epoch route cache, opaque bodies, close races, outcome-safe retry | direct host-client tests | +| Synapse and wake-plane callers | managed calls, typed send outcomes, truthful catalog, and absent `wake.create` fail-open behavior | direct TypeScript caller tests | +| `HistorianProducer` | `mc_host::Client`, mandatory negotiation, full route handles, streaming, Ping/Pong, same-incarnation exact-byte replay fence, and both-route cleanup | module historian and managed-client tests | +| session resolver | local typed `session_unresolved` absence with zero resolver route attempts when no session is proven | module resolver tests | +| `McHandler` | direct `PrimaryComponent`, initialize once, bind before response, full-handle route-gone, atomics-only health, and tracked shutdown | module adapter tests | +| direct-host fixtures | owner-only bounded Unix controls and host-owned clients; no provider process or sibling workspace | focused fixture tests; broad qualification in `magic-context-c50.9` | + +## 16. Direct-boundary traceability + +| Contract | Normative sections | Scenarios | | --- | --- | --- | -| R1 single normative owner | 1, 17 | AE1-AE13 | -| R2 terms, authority, verified/private distinction | 1-3, 18 | AE1, AE4, AE7 | -| R3 byte/JSON examples and scenario tables | 4-7, 13-14 | AE1-AE13, V1-V47 | -| R4 no executable fixtures/implementation | 1, 17 | AE1-AE13 | -| R5 discovery schema/endpoints | 4.1 | AE1, AE7, V1-V4 | -| R6 publication/cleanup/redaction | 4.2, 12 | AE7, AE13, V5-V7, V24 | -| R7 authentication | 5 | AE1, V8-V11, V23, V39 | -| R8 envelope/caps/resources | 6 | AE2, V12-V15, V28, V35 | -| R9 frame/control classification/catalog | 6.2, 7 | AE9, AE10, V17, V40-V42 | -| R10 tagged control and opaque application bodies | 7, 9.1 | AE1, V16, V38, V41 | -| R11 route/bind/direct lifecycle | 8.1-8.2 | AE3, AE8, AE11-AE13, V20, V27, V31-V36 | -| R12 request identity/counters | 8.3 | AE5, AE7, AE11, V18-V22, V31-V32, V43-V44 | -| R13 terminal/cancel/close/health | 9 | AE8, AE9, AE12, AE13, V19, V25-V26, V33-V35, V38 | -| R14 deadlines/outcomes | 10-11 | AE2, AE4-AE6, V23, V29-V30, V37 | -| R15 reconnect/rotation/stale state | 12 | AE7, AE13, V7, V22 | -| R16 terminal route error vs fresh retry | 10.2, 13.3 | AE4, AE5, V29-V30 | +| one host-owned wire and client authority | 1-3, 15, 17-18 | AE1-AE13 | +| strict descriptor version and secure snapshot | 4 | V1-V7, V51 | +| authentication and secret handling | 5 | V8-V11, V23-V24, V39 | +| framing, control, and canonical literals | 6-7 | V12-V17, V40-V42 | +| mandatory first negotiation and fail-closed setup | 7.7 | AE1-AE2, V52-V54 | +| full route handles, correlation, and terminal ownership | 8-10 | V18-V22, V27-V38, V43-V44 | +| managed-client deadlines, control reserve, cancellation, and liveness | 9-11 | AE4-AE9, V23, V29-V30, V33-V37 | +| restart and shutdown cleanup | 12-13 | AE7, AE13, V45-V50 | ## 17. Scope boundaries -In scope: discovery and secure publication, pre-envelope authentication, v2 framing, `route.open`, `catalog.list`, `host.shutdown`, lifecycle evidence and native state probing (Section 4.3), the fixed three-module static composition and its reserved pending/task/resident capacity classes, the Synapse application protocol (Section 7.5), routing/correlation/streaming/cancel/close, internal health, send outcomes, and generation recovery. - -The Broca application protocol (the five run-management operations, run lifecycle, replay, and subprocess trust boundary) is normative for the Broca revision that implements it; this section fixes only Broca's catalog identity, route classification, capacity class, and shutdown ordering. +This direct-boundary migration owns host/client secure connection-file primitives, version-2 wire and authentication, mandatory negotiation, host-owned Rust and TypeScript API names, static composition, route epochs, managed-client behavior, and focused direct-host fixture proof. -Deferred: executable cross-language golden fixtures; host, shim, and client code; private dependency compiler closure; model-runner routing; test-only TypeScript provider API; deployment-specific numeric quotas beyond required finite bounds. +`magic-context-c50.8` owns the production host executable and launcher, production connection-file orchestration during startup and teardown, user-facing configuration and doctor behavior, packaging, and distribution. This contract does not claim those lifecycle flows are delivered here. -Outside: flow-credit protocol, dynamic multi-module supervision, behavioral admission policy, production remote transport, new plugin/tool APIs, storage semantics, and handler business operations. +`magic-context-c50.9` owns broad Rust E2E, mutation campaigns, performance qualification, and release evidence. Focused protocol, component, and fixture gates do not constitute broad release qualification. -## 18. Source parity ledger +The Broca application protocol remains normative in its owning revision. Flow credit, dynamic module supervision, remote transport, new plugin/tool APIs, storage semantics, and handler business semantics remain outside this wire contract. -Numeric wire values and published behavior above come from: +## 18. Provenance ledger -- `subc-protocol` 0.10.0 `src/lib.rs`: version 2, 21-byte layout, 64 MiB cap, frame values, flags, validation, route/identity shapes. -- `subc-transport` 0.5.0 `src/auth.rs`, `connection_file.rs`, and `frame_io.rs`: domains, proof order, nonce/proof lengths, 4,096-byte auth cap, schema 1, owner-only file, atomic write, and complete-frame I/O. -- `subc-control` 0.1.1 `src/lib.rs`: tagged control request/response and catalog entry shapes. -- `subc-client-rs` 0.3.0 `src/consumer.rs`: fresh route-open retries, one no-dispatch `unknown_channel` retry, generation fencing, streaming, and send-outcome classification. -- exact `@cortexkit/subc-client` 0.4.1: TypeScript handshake/frame compatibility. Repository fake-peer tests exercise the flow but import package encoders/constants and are not an independent byte oracle. -- `docs/subc-api-surface-inventory-2026-08-17.md`: checksums, source-version provenance, and used-surface inventory. -- `docs/rust-mode-transport-overhead-2026-08-10.md`: measured framing cost only; its historical global-FIFO prose is not queue-topology authority. +`mc-host` source and conformance tests are current authority. Historical package sources established the frozen numeric values, version-2 frame layout, authentication domains and proof order, schema-1 fields, and control JSON shapes. `docs/subc-api-surface-inventory-2026-08-17.md` preserves checksums and compiler-closure history only; it does not recommend a compatibility dependency or shim. -Private version disagreement MUST be recorded as drift and routed to its downstream owner, never guessed into this wire contract. +`docs/rust-mode-transport-overhead-2026-08-10.md` remains measured framing-cost evidence only. Its historical queue prose is not topology authority. Any disagreement with old published or private behavior is migration history, not permission to add a compatibility branch. diff --git a/docs/subc-api-surface-inventory-2026-08-17.md b/docs/subc-api-surface-inventory-2026-08-17.md index ef7c81941..c7493e4fa 100644 --- a/docs/subc-api-surface-inventory-2026-08-17.md +++ b/docs/subc-api-surface-inventory-2026-08-17.md @@ -1,32 +1,19 @@ -# `subc` API Surface Inventory and Shim-vs-Rewrite Decision +# Historical `subc` API Surface Inventory Task: `magic-context-c50.1` (epic `magic-context-c50`, hand-rolled Rust module host) Date: 2026-08-17 Plan: `2026-08-17-0505-subc-api-surface-spike-plan.md` +Final disposition: direct migration completed by the direct `mc-host` boundary plan dated 2026-08-24. -## Decision +## Final decision -> **Superseded (2026-08-22).** The shim-adoption decision below is the original -> c50.1 analysis, preserved as the record of that spike. `magic-context-c50.1` -> has since decided the boundary is ported directly to the mc-host SDK with no -> `subc-*` compatibility shims (executed by `magic-context-c50.4`). The -> inventory and its compiler-closure proof -> (`docs/evidence/subc-compiler-closure/`) remain the authoritative enumeration -> of the surface that port must cover. +`mc-host` is the single production authority for wire, authentication, discovery, control, routing, component lifecycle, and managed Rust clients. `mc-module` implements the host-owned component contract directly. Production Rust and TypeScript callers use host-owned APIs. No `subc-*` compatibility crate, alias, shim, published dependency, provider process, or fallback API is recommended or supported. -**Option (a): compatible local `subc-*` crates, so `mc-module` compiles unmodified.** -Option (b), rewriting the `mc-module` boundary against a repo-owned SDK, is rejected. +The inventories below and `docs/evidence/subc-compiler-closure/` remain historical proof of the surface that the migration had to close. `docs/evidence/subc-surface-probe/` remains an excluded historical crate and is not an implementation dependency. Old option analysis is retained only to explain the spike's evidence and must not be read as current architecture guidance. -The decisive fact is that four of the five crates are **MIT-licensed and published on -crates.io**, and their published source is shape-compatible with 83 of the 86 enumerated -`mc-module` requirements. The compatible-crate path therefore does not require *writing* -a compatible client SDK; it requires **adopting** one (vendored into this repo, or -depended on from crates.io) and patching three compiler-confirmed deltas. Only -`subc-core` — the daemon — is unpublished, and replacing that daemon is the epic's actual -work regardless of which option is chosen. +## Historical spike decision (superseded) -Rewriting the boundary would buy nothing that adoption does not already give, and would -cost a 350-site edit inside `crates/mc-module/src/lib.rs` alone. +On 2026-08-17, the spike favored adopting compatible local crates because published MIT sources matched 83 of 86 enumerated `mc-module` requirements and isolated three deltas. The later direct-boundary decision superseded that recommendation after the compiler-closure inventory made a complete direct port tractable. No production compatibility layer was retained. ## Evidence Provenance @@ -72,7 +59,7 @@ Reproduction artifacts in this repository: - `docs/evidence/subc-surface-probe/delta-ledger.txt` — the compiler output when the probe is flipped to `mc-module`'s exact forms, isolating the deltas to exactly three errors. -## Compiler Closure: Complete (2026-08-24) +## Historical compiler closure: complete (2026-08-24) The disposable-stub compiler pass (plan step 4) ran on 2026-08-24 with the `../commons` and `../subconscious` siblings restored, at revision @@ -97,7 +84,7 @@ therefore mechanically proven **complete** for the current code, not just correc per-row: with those reconciliations the inventory *is* the compile footprint. The completeness gate on `magic-context-c50.4` is cleared. -## Rust Inventory +## Historical Rust inventory 86 API rows across four crates, plus one declared-only dependency edge. Every row's build target is the `mc_module` lib unless noted. Weight legend: **T** type-only, **P** @@ -221,7 +208,7 @@ compatible-crate path cannot satisfy by adoption, and it is exactly what `mc-hos (`subc_reversibility_*`). No import, no call. Classified as non-API so the sweep is closed. -## TypeScript Inventory +## Historical TypeScript inventory The installed version **is** the latest published version, so every row below is verified against the exact 0.4.1 source and declarations; all 34 are `exact`. @@ -278,7 +265,7 @@ Sweep closure: 8 files across `packages/` import `@cortexkit/subc-client` direct above). The other 28 files matching `subc` reference only env vars, connection-file paths, config keys, or prose. -## Compatibility Matrix Summary +## Historical compatibility matrix | | Rows | exact | changed | absent | private/unknown | | --- | ---: | ---: | ---: | ---: | ---: | @@ -302,7 +289,9 @@ The three deltas, each isolated to one `rustc` error in 3. `error[E0599]: no associated function 'new' found for struct 'ErrorBody'` — a 2-arg constructor added after 0.10.0, used once, in a test. Fix: add three lines. -## Why Shims, Criterion by Criterion +## Historical shim rationale (superseded) + +This table records why the 2026-08-17 spike initially favored shims. It is not a current recommendation; the direct migration is final. | Criterion | Finding | Favors | | --- | --- | --- | @@ -319,7 +308,7 @@ the manifest types through 350+ sites of unrelated transform logic, and discardi already 96% shape-identical to freely licensed source. No third option is introduced: the crates.io-vs-vendored choice is an implementation detail *inside* the shim path. -## Downstream Constraints +## Historical downstream constraints ### `magic-context-c50.2` (minimal wire protocol + handshake) diff --git a/packages/cli/package.json b/packages/cli/package.json index 124f58df6..4b3cef9f6 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -50,6 +50,7 @@ "@biomejs/biome": "^2.5.1", "@types/better-sqlite3": "^7.6.13", "@types/node": "^22.20.0", + "bun-types": "^1.3.11", "typescript": "^5.8.0" }, "engines": { diff --git a/packages/cli/src/commands/doctor-authority.ts b/packages/cli/src/commands/doctor-authority.ts index 1d46eca69..5298bb42c 100644 --- a/packages/cli/src/commands/doctor-authority.ts +++ b/packages/cli/src/commands/doctor-authority.ts @@ -10,7 +10,7 @@ import { } from "@magic-context/core/features/magic-context/context-authority"; import { resolveProjectIdentity } from "@magic-context/core/features/magic-context/memory/project-identity"; import { bumpProjectMemoryEpoch } from "@magic-context/core/features/magic-context/storage-project-state"; -import { SubcModuleTransport } from "@magic-context/core/hooks/magic-context/module-transport"; +import { McHostModuleTransport } from "@magic-context/core/hooks/magic-context/module-transport"; import type { Database } from "@magic-context/core/shared/sqlite"; import { openExistingContextDatabaseForMutation } from "../lib/database-access"; @@ -93,7 +93,7 @@ export async function assertProjectsUseTsAuthority(args: { } function authorityClient( - transport: SubcModuleTransport, + transport: McHostModuleTransport, projectRoot: string, ): AuthorityModuleClient { return { @@ -135,7 +135,7 @@ export async function reportAuthorityMarkers(args: { } catch { // A doctor run must still report the durable fences when cwd identity fails. } - const transport = new SubcModuleTransport(); + const transport = new McHostModuleTransport(); for (const marker of markers) { if (marker.project_path !== currentIdentity) { args.warn( @@ -185,7 +185,7 @@ export async function runDoctorDrainAuthority( console.log(`No authority_managed marker exists for ${projectPath}.`); return 0; } - const module = authorityClient(new SubcModuleTransport(), projectRoot); + const module = authorityClient(new McHostModuleTransport(), projectRoot); let drainedAny = false; for (const domain of AUTHORITY_DOMAINS) { const status = await module.authorityStatus({ diff --git a/packages/cli/src/commands/migrate-session.ts b/packages/cli/src/commands/migrate-session.ts index e0515f846..d0c97bd36 100644 --- a/packages/cli/src/commands/migrate-session.ts +++ b/packages/cli/src/commands/migrate-session.ts @@ -31,7 +31,7 @@ import { selectRelocatableMemoryIds, } from "@magic-context/core/features/magic-context/memory/relocate-memory"; import { bumpProjectMemoryEpoch } from "@magic-context/core/features/magic-context/storage-project-state"; -import { SubcModuleTransport } from "@magic-context/core/hooks/magic-context/module-transport"; +import { McHostModuleTransport } from "@magic-context/core/hooks/magic-context/module-transport"; import { getMagicContextStorageDir } from "@magic-context/core/shared/data-path"; import type { Database as DatabaseType } from "@magic-context/core/shared/sqlite"; @@ -649,7 +649,7 @@ export async function runMigrateSessionCli(args: string[]): Promise { } const deps = realDeps(opencodeDb, contextDb); const plan = planMigrateSession(sessionId, expandedTo, deps); - const transport = new SubcModuleTransport(); + const transport = new McHostModuleTransport(); const safety = await assertMigrateSessionIsSafeToRehome({ plan, contextDb: contextDb as DatabaseType, diff --git a/packages/cli/src/lib/logs-opencode.test.ts b/packages/cli/src/lib/logs-opencode.test.ts index f900bb53d..6fdcad2d8 100644 --- a/packages/cli/src/lib/logs-opencode.test.ts +++ b/packages/cli/src/lib/logs-opencode.test.ts @@ -116,7 +116,8 @@ describe("sanitizeLogContent — secret token redaction (council finding #9)", ( }); it("redacts AWS secret access keys in assignment context", () => { - const log = "aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"; + const syntheticKey = "wJalrXUtnFEMI/" + "K7MDENG/bPxRfiCYEXAMPLEKEY"; // gitleaks:allow redaction-test fixture + const log = `aws_secret_access_key=${syntheticKey}`; const sanitized = sanitizeLogContent(log); expect(sanitized).toContain(""); expect(sanitized).not.toContain("wJalrXUtnFEMI"); @@ -127,7 +128,7 @@ describe("sanitizeLogContent — secret token redaction (council finding #9)", ( describe("Slack tokens", () => { it("redacts xoxb (bot) tokens", () => { - const log = "SLACK_BOT_TOKEN=xoxb-1234567890-abcdefghij-ABCDEFG12345"; + const log = "SLACK_BOT_TOKEN=xoxb-1234567890-abcdefghij-ABCDEFG12345"; // gitleaks:allow redaction-test fixture const sanitized = sanitizeLogContent(log); // env-var wins expect(sanitized).toBe("SLACK_BOT_TOKEN="); @@ -239,8 +240,12 @@ describe("sanitizeLogContent — secret token redaction (council finding #9)", ( describe("JWT tokens", () => { it("redacts a three-segment JWT", () => { - const log = - "Got JWT: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1c2VyIn0.dozjgNryP4J3jVmNHl0w5N_XgL1JxXYbXvpvYTByA in response"; + const syntheticJwt = [ + "eyJhbGciOiJIUzI1NiJ9", + "eyJzdWIiOiJ1c2VyIn0", + "dozjgNryP4J3jVmNHl0w5N_XgL1JxXYbXvpvYTByA", + ].join("."); + const log = `Got JWT: ${syntheticJwt} in response`; const sanitized = sanitizeLogContent(log); expect(sanitized).toContain(""); expect(sanitized).not.toContain("eyJhbGciOiJIUzI1NiJ9"); diff --git a/packages/e2e-tests/README.md b/packages/e2e-tests/README.md index 153973008..c705537cc 100644 --- a/packages/e2e-tests/README.md +++ b/packages/e2e-tests/README.md @@ -64,37 +64,34 @@ current JSON protocol was introduced in `0.16.0`. `pi-runner`. Each test creates a session, sends prompts, and asserts against SQLite state, log output, and captured mock requests. -- **`tests/rust-*.test.ts`** — Rust-mode (ck-mc over subc) lane. Drives the FULL - production path opencode → plugin → subc daemon → ck-mc module through a - hermetic stack (`src/rust-harness.ts` + `src/rust-runner/hermetic-subc.ts`), - reusing the mock provider and session-driving machinery unchanged. Run it - separately (it is NOT part of the default `test` run or the CI host suite): +- **`tests/rust-*.test.ts`** — Rust-mode lane. Starts the repository-local + `direct_host_fixture`, then drives OpenCode → plugin → `McHostModuleTransport` + → real `McHandler`. `src/rust-runner/hermetic-mc-host.ts` owns fixture build, + managed-client readiness, bounded backend controls, and teardown. Run the broad + lane separately: ```bash # From this package (or `bun run test:rust-e2e` from the repo root) bun run test:rust-e2e ``` - Runtime: ~1-2 minutes locally once the binaries are warm (the first run builds - `ck-mc` release + reuses a prebuilt `ck-subc`). Each scenario keeps its session - small (tens of turns, tiny context limits) so the suite stays fast. + U7 qualifies only the focused fixture, smoke, and historian scenarios. Broad + Rust-mode matrix, mutation, performance, and release qualification remain + downstream work. ### Rust-mode lane: how it works -The lane spawns a real `ck-subc` daemon (from the sibling `subconscious` -workspace, the same binary `crates/mc-module/tests/real_daemon.rs` uses) and the -`ck-mc` module (this workspace) connected to it, then boots `opencode serve` in -Rust transform mode against them. The wiring uses no product change: the plugin's -Rust module client reads the default connection file at -`${XDG_DATA_HOME}/cortexkit/run/subc-connection.json`, and the harness points the -daemon's `XDG_RUNTIME_DIR` there so its connection file lands exactly where the -plugin looks. The module opens its own store under the same data dir (the -production shared-cortexkit layout). - -Environment honesty: `RustTestHarness.detectPrereqs()` preflights the stack -(cargo present, sibling `subconscious` workspace present, supported platform) and -the suite SKIPs with a printed reason when any is missing — never green-washing, -never hanging. +The harness builds the `mc-module` `direct_host_fixture` example and starts it +under each test's owner-only data directory. Fixture directly composes real Magic +Context, Synapse, and Broca components. Readiness requires its private control +socket, version-2 connection publication, and a successful managed-client catalog +probe. Closed JSONL controls select backend success, blocking, release, typed +failure, counters, or graceful shutdown. Host crash, restart, pause, and resume +controls act on the whole direct host. + +`RustTestHarness.detectPrereqs()` checks Unix socket support, Cargo, and current +repository metadata. Missing sibling checkouts and removed binaries do not affect +this lane. **Pressure technique (load-bearing apparatus rule):** scenarios reach high fill by SHRINKING the context limit against REAL message bytes, never by inflating @@ -109,42 +106,19 @@ pinned at exactly 0.0 the whole time.) If a scenario needs a shortcut, shrink the window; if you must inflate, document which asserted conditions become unreachable. -Gated scenarios (skip with a printed reason until their dependency lands): - -- **Fold-dependent** (`fold-under-pressure`, `ctx-reduce-roundtrip`) — the Rust - module runs its own historian, which drives an LLM through a separate `broca` - runner module the current hermetic stack does not spawn. Without it no - compartment is published, so no fold (or drop-on-fold) can land. Set - `MC_RUST_E2E_FOLD=1` once a hermetic broca runner is wired. -- **Removal reconcile** (`removal-self-heal`) — a mid-session `session.revert` - still wedges the Rust ordinal resolver (a distinct gap from the merged - tail-readopt / park-self-heal fix). Set `MC_RUST_E2E_REMOVAL=1` once the removal - ordinal-reconcile self-heal lands. -- **Duplicate tool-use IDs** (`duplicate-tool-use-id`) — consuming a queued drop on - a selection bust needs the hermetic `broca` runner. Set - `MC_RUST_E2E_DUPLICATE_IDS=1` once that runner is provisioned; the test body - always walks every served message array, even while gated. - ### CI -The Rust-mode lane is intentionally NOT wired into `.github/workflows/ci.yml`. -Beyond the Rust toolchain (which GitHub-hosted runners can install), it needs the -sibling **`subconscious`** workspace checked out beside this repo to build the -`ck-subc` daemon — the CI checkout does not provision that separate repo, so the -lane cannot build there without extra wiring. When a runner (or a container image) -provisions the subconscious sibling, add a job that runs -`bun run --cwd packages/e2e-tests test:rust-e2e`. Until then the lane runs -locally / on a suitably provisioned host only, and the CI host suite explicitly -excludes `rust-*.test.ts`. +The broad Rust-mode lane is not part of the default host suite. Focused direct-host +coverage runs with Cargo and this repository; broader matrix qualification remains +separate from U7. ## Requirements - `opencode` CLI available on PATH for OpenCode suites (`which opencode`). - Pi CLI installed for Pi suites (see `packages/pi-plugin/README.md`). - Bun. -- For the Rust-mode lane (`tests/rust-*.test.ts`): `cargo` on PATH and the sibling - `subconscious` workspace checked out beside this repo (to build `ck-subc`). The - lane skips with a printed reason when these are absent. +- For the Rust-mode lane (`tests/rust-*.test.ts`): Unix sockets, `cargo` on PATH, + and the current repository checkout. Fixture builds on demand. - No `OPENCODE_SERVER_PASSWORD` required — the spawner explicitly strips it so the test server runs unsecured on a random localhost port. diff --git a/packages/e2e-tests/mode-manifest.json b/packages/e2e-tests/mode-manifest.json index cce763b48..8ffec7451 100644 --- a/packages/e2e-tests/mode-manifest.json +++ b/packages/e2e-tests/mode-manifest.json @@ -1,6 +1,6 @@ { "schema": 1, - "header": "CI intentionally runs only the manifest-derived TS invocation; the Rust invocation requires private sibling path-deps (../commons and ../subconscious) and remains a mandatory local release-gate leg.", + "header": "CI runs the manifest-derived TS invocation. Focused direct-host Rust tests require Cargo and this repository; broad Rust-mode qualification remains a separate gate.", "entries": [ { "path": "tests/cache-invariants.test.ts", @@ -546,7 +546,7 @@ "ts": false, "rust": true }, - "rationale": "hermetic Broca producer publishes deterministic v2 compartments and proves outage/strict-validation failures", + "rationale": "direct Broca backend controls prove historian routing and typed backend-failure handling", "contract_refs": [ "RUST-E2E", "HISTORIAN-PRODUCER" diff --git a/packages/e2e-tests/mutations/fm-oc-5.json b/packages/e2e-tests/mutations/fm-oc-5.json index 099426b71..289d879fc 100644 --- a/packages/e2e-tests/mutations/fm-oc-5.json +++ b/packages/e2e-tests/mutations/fm-oc-5.json @@ -6,8 +6,8 @@ "name": "FM_OC_5_RUNG_SWAP", "applied_diff": { "path": "packages/e2e-tests/tests/rust-fm-oc-5.test.ts", - "before": "h.subc.stopModule();\n await h.sendPrompt", - "after": "h.subc.continueModule();\n await h.sendPrompt", + "before": "h.mcHost.pauseHost();\n await h.sendPrompt", + "after": "h.mcHost.resumeHost();\n await h.sendPrompt", "changed": true }, "observed_failure": { diff --git a/packages/e2e-tests/mutations/rust-ctx-reduce-roundtrip.json b/packages/e2e-tests/mutations/rust-ctx-reduce-roundtrip.json index f4808abe6..2119dcc71 100644 --- a/packages/e2e-tests/mutations/rust-ctx-reduce-roundtrip.json +++ b/packages/e2e-tests/mutations/rust-ctx-reduce-roundtrip.json @@ -12,7 +12,7 @@ }, "observed_failure": { "exit_status": 1, - "output": "bun test v1.3.14 (0d9b296a)\n\ntests/rust-ctx-reduce-roundtrip.test.ts:\n100 | const queued = (await h.subc.moduleStatus(\n101 | sessionId,\n102 | h.env.workdir,\n103 | \"session.status\",\n104 | )) as ModuleStatus;\n105 | expect(queued.pending_drop_count ?? 0).toBeGreaterThan(0);\n ^\nerror: expect(received).toBeGreaterThan(expected)\n\nExpected: > 0\nReceived: 0\n\n at (/Users/ufukaltinok/.local/share/cortexkit/alfonso/worktrees/8f93aad09f2535d0/bg_a2abda75/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts:105:52)\n(fail) rust invariant: ctx_reduce round-trip > consumes an agent ctx_reduce drop on the next producer-backed bust [8540.69ms]\n\n 0 pass\n 1 fail\n 2 expect() calls\nRan 1 test across 1 file. [8.79s]\n" + "output": "bun test v1.3.14 (0d9b296a)\n\ntests/rust-ctx-reduce-roundtrip.test.ts:\n100 | const queued = (await h.mcHost.primaryStatus(\n101 | sessionId,\n102 | h.env.workdir,\n103 | \"session.status\",\n104 | )) as ModuleStatus;\n105 | expect(queued.pending_drop_count ?? 0).toBeGreaterThan(0);\n ^\nerror: expect(received).toBeGreaterThan(expected)\n\nExpected: > 0\nReceived: 0\n\n at (/Users/ufukaltinok/.local/share/cortexkit/alfonso/worktrees/8f93aad09f2535d0/bg_a2abda75/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts:105:52)\n(fail) rust invariant: ctx_reduce round-trip > consumes an agent ctx_reduce drop on the next producer-backed bust [8540.69ms]\n\n 0 pass\n 1 fail\n 2 expect() calls\nRan 1 test across 1 file. [8.79s]\n" }, "reverted_rerun": { "exit_status": 0, diff --git a/packages/e2e-tests/mutations/rust-historian-producer.json b/packages/e2e-tests/mutations/rust-historian-producer.json index 8036affad..74a794a95 100644 --- a/packages/e2e-tests/mutations/rust-historian-producer.json +++ b/packages/e2e-tests/mutations/rust-historian-producer.json @@ -1,29 +1,18 @@ { - "drill": "RUST-HISTORIAN-PRODUCER", + "drill": "RUST-HISTORIAN-DIRECT-BACKEND", "command": "MC_E2E_MODE=rust bun test --timeout 600000 --max-concurrency=1 tests/rust-historian-producer.test.ts", "mutations": [ { - "name": "RUST_HISTORIAN_BAD_TIER", + "name": "RUST_HISTORIAN_TYPED_FAILURE", "applied_diff": { - "path": "packages/e2e-tests/src/rust-runner/fake-broca.ts", - "before": "${title}\nDeterministic historian coverage ${start}-${end}.\nPublished by the hermetic Broca producer.\nReplay is stable for this chunk.", - "after": "", + "path": "packages/e2e-tests/tests/rust-historian-producer.test.ts", + "before": "await h.mcHost.failNextBackendCall();", + "after": "await h.mcHost.backendSuccess();", "changed": true }, - "observed_failure": { - "exit_status": 1, - "output": "bun test v1.3.14 (0d9b296a)\n\ntests/rust-historian-producer.test.ts:\n123 | while (Date.now() < publishDeadline) {\n124 | published = await sessionStatus(sessionId);\n125 | if ((published.compartment_count ?? 0) >= 1) break;\n126 | await Bun.sleep(100);\n127 | }\n128 | expect(published.compartment_count ?? 0).toBeGreaterThanOrEqual(1);\n ^\nerror: expect(received).toBeGreaterThanOrEqual(expected)\n\nExpected: >= 1\nReceived: 0\n\n at (/Users/ufukaltinok/.local/share/cortexkit/alfonso/worktrees/8f93aad09f2535d0/bg_a2abda75/packages/e2e-tests/tests/rust-historian-producer.test.ts:128:54)\n(fail) rust historian: hermetic Broca producer > publishes deterministic tiered output and records producer contact [125519.38ms]\n(pass) rust historian: hermetic Broca producer > takes the loud historian failure path when Broca goes down mid-run [732.46ms]\n\n 1 pass\n 1 fail\n 6 expect() calls\nRan 2 tests across 1 file. [129.88s]\n" - }, - "validation_probe": { - "exit_status": 0, - "output": "bun test v1.3.14 (0d9b296a)\n\ntests/rust-historian-producer.test.ts:\n(pass) rust historian: hermetic Broca producer > publishes deterministic tiered output and records producer contact [7431.93ms]\n(pass) rust historian: hermetic Broca producer > takes the loud historian failure path when Broca goes down mid-run [693.68ms]\n\n 2 pass\n 0 fail\n 8 expect() calls\nRan 2 tests across 1 file. [11.96s]\n" - }, - "reverted_rerun": { - "exit_status": 0, - "output": "bun test v1.3.14 (0d9b296a)\n\ntests/rust-historian-producer.test.ts:\n(pass) rust historian: hermetic Broca producer > publishes deterministic tiered output and records producer contact [6731.53ms]\n(pass) rust historian: hermetic Broca producer > takes the loud historian failure path when Broca goes down mid-run [635.79ms]\n\n 2 pass\n 0 fail\n 8 expect() calls\nRan 2 tests across 1 file. [10.94s]\n", - "status": "pass" - }, - "adequacy_finding": null + "observed_failure": null, + "reverted_rerun": null, + "adequacy_finding": "Direct-host mutation evidence must be regenerated by mutation:rust-historian." } ] } diff --git a/packages/e2e-tests/package.json b/packages/e2e-tests/package.json index 3caca38da..014966aec 100644 --- a/packages/e2e-tests/package.json +++ b/packages/e2e-tests/package.json @@ -5,6 +5,7 @@ "type": "module", "scripts": { "test": "bun test --timeout 120000", + "typecheck": "tsc --noEmit -p tsconfig.json", "test:validate-manifest": "bun test scripts/validate-mode-manifest.test.ts scripts/check-rust-prerequisites.test.ts", "validate-mode-manifest": "bun scripts/validate-mode-manifest.ts --mode ts", "adjudicate:thinking-block": "bash scripts/adjudicate-thinking-block.sh", @@ -13,7 +14,6 @@ "mutation:rust-ctx-reduce": "bun scripts/run-rust-ctx-reduce-mutation.ts" }, "dependencies": { - "@cortexkit/subc-client": "0.4.1", "@opencode-ai/sdk": "^1.15.13" }, "devDependencies": { diff --git a/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts b/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts index 02b41e307..ca62deef4 100644 --- a/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts +++ b/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts @@ -1,5 +1,12 @@ import { afterEach, describe, expect, it } from "bun:test"; -import { chmodSync, mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + existsSync, + mkdtempSync, + mkdirSync, + rmSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { detectRustPrerequisites } from "./check-rust-prerequisites"; @@ -7,42 +14,58 @@ import { detectRustPrerequisites } from "./check-rust-prerequisites"; const temporaryRoots: string[] = []; afterEach(() => { - for (const root of temporaryRoots.splice(0)) rmSync(root, { recursive: true, force: true }); + for (const root of temporaryRoots.splice(0)) + rmSync(root, { recursive: true, force: true }); }); -describe("Rust release prerequisite detector", () => { - it("turns a PATH-hidden ck-mc into a red named prerequisite", () => { - const parent = mkdtempSync(join(tmpdir(), "mc-rust-prereq-")); - const root = join(parent, "repo"); - temporaryRoots.push(parent); - mkdirSync(root, { recursive: true }); - mkdirSync(join(parent, "commons"), { recursive: true }); - mkdirSync(join(parent, "subconscious"), { recursive: true }); - writeFileSync(join(root, "Cargo.toml"), "[workspace]\nmembers = []\n"); - writeFileSync(join(parent, "commons/Cargo.toml"), "[workspace]\nmembers = []\n"); - writeFileSync(join(parent, "subconscious/Cargo.toml"), "[workspace]\nmembers = []\n"); +function fakeWorkspace(withFixture = true): { root: string; bin: string } { + const parent = mkdtempSync(join(tmpdir(), "mc-rust-prereq-")); + temporaryRoots.push(parent); + const root = join(parent, "repo"); + const bin = join(root, "bin"); + mkdirSync(bin, { recursive: true }); + writeFileSync(join(root, "Cargo.toml"), "[workspace]\nmembers = []\n"); + const metadata = JSON.stringify({ + packages: [ + { + name: "mc-module", + targets: withFixture + ? [{ name: "direct_host_fixture", kind: ["example"] }] + : [], + }, + ], + }); + const cargo = join(bin, "cargo"); + writeFileSync(cargo, `#!/bin/sh\nprintf '%s\\n' '${metadata}'\n`); + chmodSync(cargo, 0o755); + return { root, bin }; +} - const bin = join(root, "bin"); - const cargoOnly = join(root, "cargo-only"); - mkdirSync(bin); - mkdirSync(cargoOnly); - const ckMc = join(bin, "ck-mc"); - writeFileSync(ckMc, "#!/bin/sh\nexit 0\n"); - chmodSync(ckMc, 0o755); +describe("Rust direct-host prerequisite detector", () => { + it("ignores absent sibling workspaces and removed binaries", () => { + const { root, bin } = fakeWorkspace(); + const removedBinary = join(root, "target", "release", ["ck", "mc"].join("-")); + const siblingDaemon = join(root, "..", "subconscious"); + expect(existsSync(removedBinary)).toBe(false); + expect(existsSync(siblingDaemon)).toBe(false); - const resolved = detectRustPrerequisites({ + const result = detectRustPrerequisites({ repoRoot: root, env: { PATH: bin }, }); - expect(resolved.ok).toBe(false); - expect(resolved.ckMcBin).toBe(ckMc); - expect(resolved.missing.join("\n")).not.toContain("ck-mc binary"); - const hidden = detectRustPrerequisites({ + expect(result).toEqual({ ok: true, missing: [] }); + }); + + it("rejects a workspace without the direct host fixture target", () => { + const { root, bin } = fakeWorkspace(false); + const result = detectRustPrerequisites({ repoRoot: root, - env: { PATH: cargoOnly }, + env: { PATH: bin }, }); - expect(hidden.ok).toBe(false); - expect(hidden.missing.join("\n")).toContain("ck-mc binary"); + expect(result.ok).toBe(false); + expect(result.missing).toContain( + "cargo workspace: direct_host_fixture example is unavailable", + ); }); }); diff --git a/packages/e2e-tests/scripts/check-rust-prerequisites.ts b/packages/e2e-tests/scripts/check-rust-prerequisites.ts index c0141246f..bfaf4d48f 100644 --- a/packages/e2e-tests/scripts/check-rust-prerequisites.ts +++ b/packages/e2e-tests/scripts/check-rust-prerequisites.ts @@ -13,9 +13,7 @@ export interface RustPrerequisiteOptions { export interface RustPrerequisiteResult { ok: boolean; missing: string[]; - ckMcBin?: string; - commonsRoot?: string; - subconsciousRoot?: string; + fixtureBin?: string; } function isExecutable(path: string): boolean { @@ -26,7 +24,10 @@ function isExecutable(path: string): boolean { } } -function pathCommand(command: string, pathEnv: string | undefined): string | undefined { +function pathCommand( + command: string, + pathEnv: string | undefined, +): string | undefined { for (const directory of (pathEnv ?? "").split(":").filter(Boolean)) { const candidate = join(directory, command); if (isExecutable(candidate)) return candidate; @@ -34,77 +35,114 @@ function pathCommand(command: string, pathEnv: string | undefined): string | und return undefined; } -function cargoMetadata(cargo: string, repoRoot: string, env: NodeJS.ProcessEnv): boolean { +function cargoMetadata( + cargo: string, + repoRoot: string, + env: NodeJS.ProcessEnv, +): boolean { const result = spawnSync( cargo, - ["metadata", "--no-deps", "--format-version", "1", "--manifest-path", join(repoRoot, "Cargo.toml")], - { env, stdio: "ignore" }, + [ + "metadata", + "--no-deps", + "--format-version", + "1", + "--manifest-path", + join(repoRoot, "Cargo.toml"), + ], + { env, encoding: "utf8" }, ); - return !result.error && result.status === 0; + if ( + result.error || + result.status !== 0 || + typeof result.stdout !== "string" + ) + return false; + try { + const metadata = JSON.parse(result.stdout) as { + packages?: Array<{ + name?: string; + targets?: Array<{ name?: string; kind?: string[] }>; + }>; + }; + return ( + metadata.packages + ?.find((pkg) => pkg.name === "mc-module") + ?.targets?.some( + (target) => + target.name === "direct_host_fixture" && + target.kind?.includes("example"), + ) === true + ); + } catch { + return false; + } } -function buildCkMc(cargo: string, repoRoot: string, env: NodeJS.ProcessEnv): boolean { +function buildFixture( + cargo: string, + repoRoot: string, + env: NodeJS.ProcessEnv, +): boolean { const result = spawnSync( cargo, - ["build", "--release", "-p", "mc-module", "--manifest-path", join(repoRoot, "Cargo.toml")], + [ + "build", + "-p", + "mc-module", + "--example", + "direct_host_fixture", + "--features", + "direct-host-fixture", + "--manifest-path", + join(repoRoot, "Cargo.toml"), + ], { cwd: repoRoot, env, stdio: "inherit" }, ); return !result.error && result.status === 0; } -/** - * Check the release-gate inputs without treating an unavailable Rust lane as a skip. - * The optional PATH lookup makes the check testable with a fake ck-mc; the normal - * workspace build still prefers target/release/ck-mc and can rebuild it in place. - */ -export function detectRustPrerequisites(options: RustPrerequisiteOptions = {}): RustPrerequisiteResult { - const repoRoot = resolve(options.repoRoot ?? resolve(import.meta.dir, "../../..")); +export function detectRustPrerequisites( + options: RustPrerequisiteOptions = {}, +): RustPrerequisiteResult { + const repoRoot = resolve( + options.repoRoot ?? resolve(import.meta.dir, "../../.."), + ); const env = options.env ?? process.env; const missing: string[] = []; const cargo = pathCommand("cargo", env.PATH); - const commonsRoot = resolve(repoRoot, "../commons"); - const subconsciousRoot = resolve(repoRoot, "../subconscious"); + const manifest = join(repoRoot, "Cargo.toml"); + const configured = env.MC_E2E_DIRECT_HOST_FIXTURE_BIN; + const workspaceFixture = join( + repoRoot, + "target/debug/examples/direct_host_fixture", + ); + let fixtureBin = + configured && isExecutable(configured) ? configured : undefined; - if (!existsSync(join(repoRoot, "Cargo.toml"))) { - missing.push(`cargo workspace: missing ${join(repoRoot, "Cargo.toml")}`); + if (!existsSync(manifest)) { + missing.push(`cargo workspace: missing ${manifest}`); } else if (!cargo) { missing.push("cargo workspace: cargo is not available on PATH"); } else if (!cargoMetadata(cargo, repoRoot, env)) { - missing.push("cargo workspace: cargo metadata failed"); - } - if (!existsSync(join(commonsRoot, "Cargo.toml"))) { - missing.push(`sibling checkout: ../commons is missing (${join(commonsRoot, "Cargo.toml")})`); - } - if (!existsSync(join(subconsciousRoot, "Cargo.toml"))) { missing.push( - `sibling checkout: ../subconscious is missing (${join(subconsciousRoot, "Cargo.toml")})`, + "cargo workspace: direct_host_fixture example is unavailable", ); - } - - const configuredCkMc = env.MC_E2E_CK_MC_BIN; - let ckMcBin = configuredCkMc && isExecutable(configuredCkMc) ? configuredCkMc : undefined; - if (!ckMcBin) { - const workspaceCkMc = join(repoRoot, "target/release/ck-mc"); - ckMcBin = isExecutable(workspaceCkMc) ? workspaceCkMc : pathCommand("ck-mc", env.PATH); - } - if (!ckMcBin && options.allowBuild && cargo && missing.length === 0) { - if (buildCkMc(cargo, repoRoot, env)) { - const workspaceCkMc = join(repoRoot, "target/release/ck-mc"); - if (isExecutable(workspaceCkMc)) ckMcBin = workspaceCkMc; + } else if (options.allowBuild && !fixtureBin) { + if ( + buildFixture(cargo, repoRoot, env) && + isExecutable(workspaceFixture) + ) { + fixtureBin = workspaceFixture; + } else { + missing.push("direct mc-host fixture build failed"); } } - if (!ckMcBin) { - missing.push( - "ck-mc binary: target/release/ck-mc is absent and no ck-mc executable was found on PATH", - ); - } return { ok: missing.length === 0, missing, - ...(ckMcBin ? { ckMcBin } : {}), - ...(existsSync(join(commonsRoot, "Cargo.toml")) ? { commonsRoot } : {}), - ...(existsSync(join(subconsciousRoot, "Cargo.toml")) ? { subconsciousRoot } : {}), + ...(fixtureBin ? { fixtureBin } : {}), }; } @@ -115,7 +153,9 @@ function parseArgs(args: string[]): { build: boolean; print: boolean } { if (arg === "--build") build = true; else if (arg === "--print") print = true; else if (arg === "--help" || arg === "-h") { - console.log("Usage: check-rust-prerequisites.ts [--build] [--print]"); + console.log( + "Usage: check-rust-prerequisites.ts [--build] [--print]", + ); process.exit(0); } else throw new Error(`unknown argument: ${arg}`); } @@ -127,11 +167,12 @@ if (import.meta.main) { const { build, print } = parseArgs(Bun.argv.slice(2)); const result = detectRustPrerequisites({ allowBuild: build }); if (!result.ok) { - for (const reason of result.missing) console.error(`missing prerequisite: ${reason}`); + for (const reason of result.missing) + console.error(`missing prerequisite: ${reason}`); process.exit(1); } - if (print) console.log(result.ckMcBin); - else console.log("Rust e2e prerequisites resolved"); + if (print) console.log(result.fixtureBin ?? "build-on-demand"); + else console.log("Rust e2e direct-host prerequisites resolved"); } catch (error) { console.error(`Rust prerequisite detector failed: ${String(error)}`); process.exit(1); diff --git a/packages/e2e-tests/scripts/run-rust-fm-mutation.ts b/packages/e2e-tests/scripts/run-rust-fm-mutation.ts index 1da2e96c2..1dc9447d6 100644 --- a/packages/e2e-tests/scripts/run-rust-fm-mutation.ts +++ b/packages/e2e-tests/scripts/run-rust-fm-mutation.ts @@ -22,7 +22,8 @@ const pluginTransform = resolve( e2eRoot, "../plugin/src/hooks/magic-context/rust-mode-transform.ts", ); -const drillFile = (drill: string) => resolve(e2eRoot, `tests/rust-fm-oc-${drill}.test.ts`); +const drillFile = (drill: string) => + resolve(e2eRoot, `tests/rust-fm-oc-${drill}.test.ts`); const commandFor = (drill: string) => `bun run build (packages/plugin) && bun test --timeout 600000 --max-concurrency=1 tests/rust-fm-oc-${drill}.test.ts`; @@ -37,7 +38,8 @@ const mutations: Record = { { name: "FM_OC_1_RUNG_DELETION", source: pluginTransform, - oldText: 'sessionLog(sessionId, "rust transform failed; attempting LKG replay:", error);', + oldText: + 'sessionLog(sessionId, "rust transform failed; attempting LKG replay:", error);', replacement: "", }, ], @@ -85,7 +87,8 @@ const mutations: Record = { { name: "FM_OC_4_RUNG_DELETION", source: pluginTransform, - oldText: 'sessionLog(sessionId, "mc_rust_emergency_refusal before_lkg");', + oldText: + 'sessionLog(sessionId, "mc_rust_emergency_refusal before_lkg");', replacement: "", }, ], @@ -93,8 +96,9 @@ const mutations: Record = { { name: "FM_OC_5_RUNG_SWAP", source: drillFile("5"), - oldText: "h.subc.stopModule();\n await h.sendPrompt", - replacement: "h.subc.continueModule();\n await h.sendPrompt", + oldText: "h.mcHost.pauseHost();\n await h.sendPrompt", + replacement: + "h.mcHost.resumeHost();\n await h.sendPrompt", }, { name: "FM_OC_5_RUNG_DELETION", @@ -108,7 +112,8 @@ const mutations: Record = { name: "FM_OC_6_RUNG_SWAP", source: drillFile("6"), oldText: 'expect(after[refusalIndex]).toContain("before_lkg");', - replacement: 'expect(after[refusalIndex]).not.toContain("before_lkg");', + replacement: + 'expect(after[refusalIndex]).not.toContain("before_lkg");', }, { name: "FM_OC_6_RUNG_DELETION", @@ -157,7 +162,9 @@ function applyCase(mutation: MutationCase): { before: string; after: string } { const before = readFileSync(mutation.source, "utf8"); const occurrences = before.split(mutation.oldText).length - 1; if (occurrences !== 1) { - throw new Error(`${mutation.name}: expected one mutation target, found ${occurrences}`); + throw new Error( + `${mutation.name}: expected one mutation target, found ${occurrences}`, + ); } const after = before.replace(mutation.oldText, mutation.replacement); writeFileSync(mutation.source, after); diff --git a/packages/e2e-tests/scripts/run-rust-historian-producer-mutation.ts b/packages/e2e-tests/scripts/run-rust-historian-producer-mutation.ts index f58f8257f..3a5bd227d 100644 --- a/packages/e2e-tests/scripts/run-rust-historian-producer-mutation.ts +++ b/packages/e2e-tests/scripts/run-rust-historian-producer-mutation.ts @@ -7,14 +7,12 @@ type CommandResult = { exit_status: number; output: string }; const e2eRoot = resolve(import.meta.dir, ".."); const repoRoot = resolve(e2eRoot, "../.."); -const source = resolve(e2eRoot, "src/rust-runner/fake-broca.ts"); -const tierPattern = /[\s\S]*?<\/p[1-4]>/g; +const source = resolve(e2eRoot, "tests/rust-historian-producer.test.ts"); +const oldText = "await h.mcHost.failNextBackendCall();"; +const replacement = "await h.mcHost.backendSuccess();"; const decoder = new TextDecoder(); -function runTest(expectBad: boolean): CommandResult { - const env: Record = { ...process.env, MC_E2E_MODE: "rust" }; - if (expectBad) env.MC_RUST_E2E_BROCA_EXPECT_BAD = "1"; - else delete env.MC_RUST_E2E_BROCA_EXPECT_BAD; +function runTest(): CommandResult { const result = Bun.spawnSync({ cmd: [ "bun", @@ -27,7 +25,7 @@ function runTest(expectBad: boolean): CommandResult { cwd: e2eRoot, stdout: "pipe", stderr: "pipe", - env, + env: { ...process.env, MC_E2E_MODE: "rust" }, }); return { exit_status: result.exitCode, @@ -36,57 +34,52 @@ function runTest(expectBad: boolean): CommandResult { } const before = readFileSync(source, "utf8"); -const matches = before.match(tierPattern) ?? []; -if (matches.length !== 4) { - throw new Error(`RUST_HISTORIAN_BAD_TIER: expected four tier blocks, found ${matches.length}`); +if (before.split(oldText).length - 1 !== 1) { + throw new Error( + "RUST_HISTORIAN_TYPED_FAILURE: expected one mutation target", + ); } -const oldText = matches.join("\n"); -const replacement = ""; -const after = before.replace(tierPattern, ""); -writeFileSync(source, after); +writeFileSync(source, before.replace(oldText, replacement)); let observedFailure: CommandResult; -let validationProbe: CommandResult; try { - observedFailure = runTest(false); - validationProbe = runTest(true); + observedFailure = runTest(); } finally { writeFileSync(source, before); } -const revertedRerun = runTest(false); +const revertedRerun = runTest(); if (observedFailure.exit_status === 0) { - throw new Error("RUST_HISTORIAN_BAD_TIER: mutation did not redden the validation assertion"); -} -if (validationProbe.exit_status !== 0) { - throw new Error("RUST_HISTORIAN_BAD_TIER: invalid-output probe did not observe the validation failure"); + throw new Error( + "RUST_HISTORIAN_TYPED_FAILURE: mutation did not redden the assertion", + ); } if (revertedRerun.exit_status !== 0) { - throw new Error("RUST_HISTORIAN_BAD_TIER: reverted producer test did not pass"); + throw new Error("RUST_HISTORIAN_TYPED_FAILURE: reverted test did not pass"); } -const record = { - drill: "RUST-HISTORIAN-PRODUCER", - command: "MC_E2E_MODE=rust bun test --timeout 600000 --max-concurrency=1 tests/rust-historian-producer.test.ts", - mutations: [ - { - name: "RUST_HISTORIAN_BAD_TIER", - applied_diff: { - path: relative(repoRoot, source), - before: oldText, - after: replacement, - changed: before !== after, - }, - observed_failure: observedFailure, - validation_probe: validationProbe, - reverted_rerun: { - ...revertedRerun, - status: "pass", - }, - adequacy_finding: null, - }, - ], -}; writeFileSync( resolve(e2eRoot, "mutations/rust-historian-producer.json"), - `${JSON.stringify(record, null, 2)}\n`, + `${JSON.stringify( + { + drill: "RUST-HISTORIAN-DIRECT-BACKEND", + command: + "MC_E2E_MODE=rust bun test --timeout 600000 --max-concurrency=1 tests/rust-historian-producer.test.ts", + mutations: [ + { + name: "RUST_HISTORIAN_TYPED_FAILURE", + applied_diff: { + path: relative(repoRoot, source), + before: oldText, + after: replacement, + changed: true, + }, + observed_failure: observedFailure, + reverted_rerun: { ...revertedRerun, status: "pass" }, + adequacy_finding: null, + }, + ], + }, + null, + 2, + )}\n`, ); console.log("wrote mutations/rust-historian-producer.json"); diff --git a/packages/e2e-tests/src/harness.ts b/packages/e2e-tests/src/harness.ts index 6996ca3fb..1398b98a7 100644 --- a/packages/e2e-tests/src/harness.ts +++ b/packages/e2e-tests/src/harness.ts @@ -19,7 +19,11 @@ import { Database } from "bun:sqlite"; import { existsSync, rmSync } from "node:fs"; import { dirname, join } from "node:path"; import { MockProvider, type MockResponse } from "./mock-provider/server"; -import { spawnOpencode, type SpawnedOpencode, type SpawnOptions } from "./opencode-runner/spawn"; +import { + spawnOpencode, + type SpawnedOpencode, + type SpawnOptions, +} from "./opencode-runner/spawn"; export interface TestHarnessOptions { /** magic-context config overrides. Merged onto test defaults. */ @@ -66,19 +70,24 @@ export class TestHarness { readonly mock: MockProvider; readonly opencode: SpawnedOpencode; readonly client: SdkClient; - /** Provides Rust-mode-only access to the historian and status interfaces running in the module's Rust stack. */ - readonly rustStack: SpawnedOpencode["rustStack"]; + readonly mcHostStack: SpawnedOpencode["mcHostStack"]; private contextDbCached: Database | null = null; - private constructor(mock: MockProvider, opencode: SpawnedOpencode, client: SdkClient) { + private constructor( + mock: MockProvider, + opencode: SpawnedOpencode, + client: SdkClient, + ) { this.mock = mock; this.opencode = opencode; this.client = client; - this.rustStack = opencode.rustStack; + this.mcHostStack = opencode.mcHostStack; } - static async create(options: TestHarnessOptions = {}): Promise { + static async create( + options: TestHarnessOptions = {}, + ): Promise { const mock = new MockProvider(); const { baseURL } = await mock.start(); @@ -94,7 +103,10 @@ export class TestHarness { const opencode = await spawnOpencode(spawnOpts); const sdk = await import("@opencode-ai/sdk"); - const client = sdk.createOpencodeClient({ baseUrl: opencode.url }) as unknown as SdkClient; + // SAFETY: harness uses only SdkClient methods shared with the generated SDK client. + const client = sdk.createOpencodeClient({ + baseUrl: opencode.url, + }) as unknown as SdkClient; return new TestHarness(mock, opencode, client); } @@ -102,7 +114,10 @@ export class TestHarness { /** Create a session bound to the isolated workdir. Throws on failure. */ async createSession(): Promise { return this.createSessionWithRetry( - () => this.client.session.create({ query: { directory: this.opencode.env.workdir } }), + () => + this.client.session.create({ + query: { directory: this.opencode.env.workdir }, + }), "session.create", ); } @@ -145,7 +160,10 @@ export class TestHarness { * heuristic cleanup without historian, no 85%/95% emergency paths, no * nudges, no §N§ prefix injection. */ - async createChildSession(parentId: string, title?: string): Promise { + async createChildSession( + parentId: string, + title?: string, + ): Promise { return this.createSessionWithRetry( () => this.client.session.create({ @@ -165,7 +183,9 @@ export class TestHarness { try { const db = this.contextDb(); const row = db - .prepare("SELECT is_subagent FROM session_meta WHERE session_id = ?") + .prepare( + "SELECT is_subagent FROM session_meta WHERE session_id = ?", + ) .get(sessionId) as { is_subagent: number } | null; if (!row) return null; return row.is_subagent === 1; @@ -214,10 +234,29 @@ export class TestHarness { */ ballast(tokens: number): string { const words = [ - "boundary", "historian", "compartment", "schedule", "pressure", - "tokens", "window", "publish", "transform", "session", "marker", - "budget", "eligible", "protected", "ordinal", "snapshot", "replay", - "decision", "threshold", "baseline", "measure", "archive", "deliver", + "boundary", + "historian", + "compartment", + "schedule", + "pressure", + "tokens", + "window", + "publish", + "transform", + "session", + "marker", + "budget", + "eligible", + "protected", + "ordinal", + "snapshot", + "replay", + "decision", + "threshold", + "baseline", + "measure", + "archive", + "deliver", ]; const target = Math.max(0, Math.round(tokens * 4)); // ~4 chars/token const parts: string[] = []; @@ -259,7 +298,9 @@ export class TestHarness { ...(options.agent ? { agent: options.agent } : {}), }, }); - const timeout = new Promise((r) => setTimeout(() => r(null), timeoutMs)); + const timeout = new Promise((r) => + setTimeout(() => r(null), timeoutMs), + ); const result = await Promise.race([promptPromise, timeout]); if (result === null) { throw new Error( @@ -277,7 +318,9 @@ export class TestHarness { if (this.contextDbCached) return this.contextDbCached; const dbPath = this.contextDbPath(); if (!existsSync(dbPath)) { - throw new Error(`context.db not found at ${dbPath} — plugin may not have initialized yet.`); + throw new Error( + `context.db not found at ${dbPath} — plugin may not have initialized yet.`, + ); } this.contextDbCached = new Database(dbPath, { readonly: true }); return this.contextDbCached; @@ -330,7 +373,9 @@ export class TestHarness { try { const db = this.contextDb(); const row = db - .prepare("SELECT COUNT(*) AS n FROM compartments WHERE session_id = ?") + .prepare( + "SELECT COUNT(*) AS n FROM compartments WHERE session_id = ?", + ) .get(sessionId) as { n: number } | null; return row?.n ?? 0; } catch { @@ -367,6 +412,9 @@ export class TestHarness { } await this.opencode.kill(); await this.mock.stop(); - rmSync(dirname(this.opencode.env.configDir), { recursive: true, force: true }); + rmSync(dirname(this.opencode.env.configDir), { + recursive: true, + force: true, + }); } } diff --git a/packages/e2e-tests/src/opencode-runner/spawn.test.ts b/packages/e2e-tests/src/opencode-runner/spawn.test.ts new file mode 100644 index 000000000..0dd28c428 --- /dev/null +++ b/packages/e2e-tests/src/opencode-runner/spawn.test.ts @@ -0,0 +1,105 @@ +/// + +import { describe, expect, it } from "bun:test"; +import type { ChildProcess } from "node:child_process"; +import { EventEmitter } from "node:events"; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import type { HermeticMcHostStack } from "../rust-runner/hermetic-mc-host"; +import { __spawnOpencodeTest, type IsolatedEnv } from "./spawn"; + +class FakeChild extends EventEmitter { + readonly pid = 42; + exitCode: number | null = null; + signalCode: NodeJS.Signals | null = null; + readonly signals: NodeJS.Signals[] = []; + + kill(signal: NodeJS.Signals = "SIGTERM"): boolean { + this.signals.push(signal); + if (signal === "SIGKILL") { + queueMicrotask(() => { + this.signalCode = signal; + this.emit("exit", null, signal); + }); + } + return true; + } +} + +function childProcess(fake: FakeChild): ChildProcess { + return fake as unknown as ChildProcess; +} + +describe("opencode child lifecycle", () => { + it("rejects startup on child spawn error", async () => { + const child = new FakeChild(); + const startup = __spawnOpencodeTest.rejectOnSpawnError(childProcess(child)); + child.emit("error", new Error("spawn opencode ENOENT")); + expect(String(await startup.catch((error: unknown) => error))).toContain( + "spawn opencode ENOENT", + ); + }); + + it("escalates a SIGTERM-ignoring child and waits for exit", async () => { + const child = new FakeChild(); + let exitObserved = false; + child.once("exit", () => { + exitObserved = true; + }); + + await __spawnOpencodeTest.stopChild(childProcess(child), 5); + + expect(child.signals).toEqual(["SIGTERM", "SIGKILL"]); + expect(exitObserved).toBe(true); + expect(child.signalCode).toBe("SIGKILL"); + }); + + it("stops the Rust fixture when config serialization fails before spawn", async () => { + const root = mkdtempSync(join(tmpdir(), "opencode-spawn-rollback-")); + const env: IsolatedEnv = { + configDir: join(root, "config"), + dataDir: join(root, "data"), + cacheDir: join(root, "cache"), + workdir: join(root, "work"), + }; + for (const dir of Object.values(env)) mkdirSync(dir, { recursive: true }); + const fixtureState = join(env.dataDir, "fixture-state"); + writeFileSync(fixtureState, "running"); + + let stopCalls = 0; + const mcHost = { + connectionFile: join(env.dataDir, "mc-host-connection.json"), + async stop(): Promise { + stopCalls++; + rmSync(root, { recursive: true, force: true }); + }, + } as HermeticMcHostStack; + const cyclic: Record = {}; + cyclic.self = cyclic; + const previousMode = process.env.MC_E2E_MODE; + process.env.MC_E2E_MODE = "rust"; + + try { + const error = await __spawnOpencodeTest + .spawnOpencodeWithProvision( + { + mockProviderURL: "http://127.0.0.1:1", + port: 1, + openCodeConfigExtra: cyclic, + }, + async () => ({ env, connectionFile: mcHost.connectionFile, mcHost }), + ) + .catch((failure: unknown) => failure); + + expect(String(error)).toContain("cyclic structures"); + expect(stopCalls).toBe(1); + expect(existsSync(fixtureState)).toBe(false); + expect(existsSync(dirname(env.dataDir))).toBe(false); + } finally { + if (previousMode === undefined) delete process.env.MC_E2E_MODE; + else process.env.MC_E2E_MODE = previousMode; + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/e2e-tests/src/opencode-runner/spawn.ts b/packages/e2e-tests/src/opencode-runner/spawn.ts index 7b1514b34..a0b06cd3b 100644 --- a/packages/e2e-tests/src/opencode-runner/spawn.ts +++ b/packages/e2e-tests/src/opencode-runner/spawn.ts @@ -15,23 +15,13 @@ import { runMigrations } from "../../../plugin/src/features/magic-context/migrat import { initializeDatabase } from "../../../plugin/src/features/magic-context/storage-db"; import { Database } from "../../../plugin/src/shared/sqlite"; import { - buildHermeticBinaries, + buildDirectHostFixture, detectRustModePrereqs, - HermeticSubcStack, -} from "../rust-runner/hermetic-subc"; + HermeticMcHostStack, +} from "../rust-runner/hermetic-mc-host"; const REPO_ROOT = resolve(import.meta.dir, "../../../.."); -// Prefer the bundled `dist/index.js` (what published users actually run) -// over raw `src/index.ts`. The bundled file is one ~5MB file with all imports -// inlined; loading it is fast even on cold runners. The TS-source path -// triggers Bun's runtime TS transpile + dynamic resolution across hundreds -// of submodule imports — on slow Linux CI runners this can take long enough -// to make `opencode serve` appear hung when it's just blocked in plugin -// load. Production never loads from src/, so testing src/ doesn't reflect -// reality and exposes us to a slowness path users never see. -const PLUGIN_DIST_ENTRY = join(REPO_ROOT, "packages/plugin/dist/index.js"); -const PLUGIN_SRC_ENTRY = join(REPO_ROOT, "packages/plugin/src/index.ts"); -const PLUGIN_ENTRY = existsSync(PLUGIN_DIST_ENTRY) ? PLUGIN_DIST_ENTRY : PLUGIN_SRC_ENTRY; +const PLUGIN_ENTRY = join(REPO_ROOT, "packages/plugin/src/index.ts"); function initializeIsolatedContextDb(dataDir: string): void { const path = join(dataDir, "cortexkit", "magic-context", "context.db"); @@ -60,8 +50,8 @@ export interface SpawnedOpencode { kill: () => Promise; stdout: () => string; stderr: () => string; - /** The hermetic Rust stack is provisioned only when MC_E2E_MODE is set to "rust"; this property exposes it when available. */ - rustStack?: HermeticSubcStack; + /** Direct host fixture provisioned for MC_E2E_MODE=rust. */ + mcHostStack?: HermeticMcHostStack; } export interface SpawnOptions { @@ -75,28 +65,11 @@ export interface SpawnOptions { openCodeConfigExtra?: Record; /** Override the mock model's context token limit. Default 200000. */ modelContextLimit?: number; - /** - * Reuse a pre-created isolated env instead of allocating a fresh one. The - * Rust-mode harness creates the env first so a hermetic subc daemon can - * write its connection file into `${dataDir}/cortexkit/run/` BEFORE opencode - * boots, and so a serve restart can re-attach to the same data dir (keeping - * opencode.db + context.db across the restart). Default: allocate a new env. - */ + /** Reuse an isolated env so direct host starts before OpenCode and survives serve restarts. */ existingEnv?: IsolatedEnv; - /** - * When set, add `subc: { connection_file }` to the USER-tier magic-context - * config. This is the only tier that gates `userTierHasSubc` (project-tier - * `subc` is stripped by project-security), so Rust mode needs it here to - * activate. Default: no user-tier subc block (TS mode). - */ - userSubcConnectionFile?: string; - /** - * When set, ALSO write `/.cortexkit/magic-context.jsonc` (the - * project-tier config). Rust mode is opted in per-project via - * `transform_mode: "rust"` here, mirroring production where a repo selects - * the runtime while the user supplies daemon credentials. Default: no - * project-tier config file. - */ + /** User-tier host connection file used by Rust mode. */ + userMcHostConnectionFile?: string; + /** `projectMagicContextConfig` is written to `/.cortexkit/magic-context.jsonc` when set. */ projectMagicContextConfig?: Record; /** * Extra environment variables for the opencode child (e.g. @@ -121,9 +94,8 @@ async function pickFreePort(): Promise { * Create isolated config/data/cache dirs under a unique temp subdir. * * Exported so the Rust-mode harness can allocate the env up front: it needs the - * concrete `dataDir` before opencode boots to place a hermetic subc daemon's - * connection file at `${dataDir}/cortexkit/run/subc-connection.json` (the path - * the plugin's Rust module client reads), and it reuses the same env across a + * concrete `dataDir` before OpenCode boots so direct host can publish its + * connection file, and it reuses the same env across a * serve restart so opencode.db + context.db survive the restart. */ export function createIsolatedEnv(): IsolatedEnv { @@ -196,13 +168,7 @@ function writeConfigs( ...(opts.openCodeConfigExtra ?? {}), }; - // magic-context defaults tuned for fast triggering in tests. This is the - // USER-tier config: thresholds live here because project-tier thresholds are - // security-clamped raise-only, so a small/fast threshold must come from the - // trusted user tier. Rust mode's `subc.connection_file` is also user-tier — - // it is the only tier that flips `userTierHasSubc`, which the transform-mode - // resolver requires before Rust can activate (project-tier `subc` is stripped - // by project-security hardening). + // User-tier thresholds stay below project-security raise-only clamps. const magicContext: Record = { $schema: "https://raw.githubusercontent.com/ahrav/magic-context/main/assets/magic-context.schema.json", @@ -212,8 +178,10 @@ function writeConfigs( sidekick: { disable: true }, ...(opts.magicContextConfig ?? {}), }; - if (opts.userSubcConnectionFile) { - magicContext.subc = { connection_file: opts.userSubcConnectionFile }; + if (opts.userMcHostConnectionFile) { + Object.assign(magicContext, { + subc: { connection_file: opts.userMcHostConnectionFile }, + }); } writeFileSync(join(env.configDir, "opencode.json"), JSON.stringify(opencodeConfig, null, 2)); @@ -232,12 +200,6 @@ function writeConfigs( JSON.stringify(magicContext, null, 2), ); - // Project-tier config: written to the hard-cutover location the loader reads, - // `/.cortexkit/magic-context.jsonc`. Rust mode is opted in here via - // `transform_mode: "rust"`, matching production where a repository selects the - // runtime while the user supplies the daemon credentials above. This file is - // (re)written on every spawn so a serve restart can flip the mode in place - // (the cold-start-drop-seed scenario switches ts→rust across a restart). if (opts.projectMagicContextConfig) { const projectConfigDir = join(env.workdir, ".cortexkit"); mkdirSync(projectConfigDir, { recursive: true }); @@ -277,24 +239,31 @@ function writeConfigs( // take a few minutes" on first boot per fresh CI XDG_DATA_HOME). Local hardware // finishes in <2s. The bump to 300s covers CI cold-start without papering over // genuine readiness failures — 5 minutes is still far above any realistic boot. -async function waitForReady(url: string, timeoutMs = 300_000): Promise { +async function waitForReady( + url: string, + timeoutMs = 300_000, + cancellation?: AbortSignal, +): Promise { const deadline = Date.now() + timeoutMs; const FETCH_TIMEOUT_MS = 2_000; let lastFetchErr: unknown = null; let fetchAttempts = 0; while (Date.now() < deadline) { + if (cancellation?.aborted) throw new Error("opencode readiness cancelled"); try { fetchAttempts++; + const timeout = AbortSignal.timeout(FETCH_TIMEOUT_MS); const res = await fetch(`${url}/doc`, { method: "GET", - signal: AbortSignal.timeout(FETCH_TIMEOUT_MS), + signal: cancellation ? AbortSignal.any([timeout, cancellation]) : timeout, }); if (res.ok || res.status === 404 || res.status === 401) { // Server is responding — any HTTP response means it booted. return; } } catch (err) { + if (cancellation?.aborted) throw new Error("opencode readiness cancelled"); lastFetchErr = err; } await Bun.sleep(200); @@ -310,168 +279,221 @@ async function waitForReady(url: string, timeoutMs = 300_000): Promise { interface RustSpawnResources { env: IsolatedEnv; connectionFile: string; - stack: HermeticSubcStack; + mcHost: HermeticMcHostStack; } -/** - * Provision the Rust stack at the shared OpenCode spawn seam. Keeping this - * decision here means a suite body never needs a mode branch: the same harness - * creates either a regular isolated process or the real ck-subc + ck-mc path. - */ +function rejectOnSpawnError(child: ChildProcess, cancellation?: AbortSignal): Promise { + return new Promise((_, rejectSpawn) => { + const onError = (error: Error): void => { + cancellation?.removeEventListener("abort", onAbort); + rejectSpawn(error); + }; + const onAbort = (): void => { + child.off("error", onError); + }; + if (cancellation?.aborted) return; + child.once("error", onError); + cancellation?.addEventListener("abort", onAbort, { once: true }); + }); +} + +function waitForChildExit(child: ChildProcess, timeoutMs: number): Promise { + if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve(true); + return new Promise((resolveExit) => { + const onExit = (): void => { + clearTimeout(timer); + resolveExit(true); + }; + const timer = setTimeout(() => { + child.off("exit", onExit); + resolveExit(false); + }, timeoutMs); + child.once("exit", onExit); + }); +} + +async function stopChild(child: ChildProcess, timeoutMs = 3_000): Promise { + if (child.exitCode !== null || child.signalCode !== null || child.pid === undefined) return; + + const exitedAfterTerm = waitForChildExit(child, timeoutMs); + child.kill("SIGTERM"); + if (await exitedAfterTerm) return; + + const exitedAfterKill = waitForChildExit(child, timeoutMs); + child.kill("SIGKILL"); + if (!(await exitedAfterKill)) { + throw new Error("opencode serve did not exit after SIGKILL"); + } +} + +/** Provision direct host before OpenCode so it can publish its connection file. */ async function provisionRustMode(): Promise { const prereqs = detectRustModePrereqs(); - if (!prereqs.ok || !prereqs.subconsciousRoot) { + if (!prereqs.ok) { throw new Error( `MC_E2E_MODE=rust prerequisite failure: ${prereqs.skipReason ?? "unknown prerequisite"}`, ); } - const { ckMcBin, ckSubcBin } = await buildHermeticBinaries(prereqs.subconsciousRoot); + const fixtureBin = await buildDirectHostFixture(); const env = createIsolatedEnv(); try { - const stack = await HermeticSubcStack.start({ dataDir: env.dataDir, ckMcBin, ckSubcBin }); - return { env, connectionFile: stack.connectionFile, stack }; - } catch (error) { - throw new Error(`MC_E2E_MODE=rust failed to start the hermetic stack: ${String(error)}`); + const mcHost = await HermeticMcHostStack.start({ dataDir: env.dataDir, fixtureBin }); + return { env, connectionFile: mcHost.connectionFile, mcHost }; + } catch { + throw new Error("MC_E2E_MODE=rust failed to start direct mc-host fixture"); } } -export async function spawnOpencode(opts: SpawnOptions): Promise { +async function spawnOpencodeWithProvision( + opts: SpawnOptions, + provision: () => Promise, +): Promise { // MC_E2E_MODE is intentionally read only at this shared spawn seam. Rust - // suites that already supplied a daemon connection keep their existing + // suites that already supplied a host connection keep their existing // stack; ordinary suites get one provisioned here for the rust invocation. const rustMode = process.env.MC_E2E_MODE === "rust"; - const resources = rustMode && !opts.userSubcConnectionFile ? await provisionRustMode() : null; - const resolvedOpts: SpawnOptions = resources - ? { - ...opts, - existingEnv: resources.env, - userSubcConnectionFile: resources.connectionFile, - projectMagicContextConfig: { - ...(opts.projectMagicContextConfig ?? {}), - transform_mode: "rust", - }, - } - : opts; - - // Reuse a caller-provided env for the Rust-mode harness (connection file - // pre-placed, data dir shared across a serve restart); otherwise allocate. - const env = resolvedOpts.existingEnv ?? createIsolatedEnv(); - const port = resolvedOpts.port ?? (await pickFreePort()); - - const compaction = resolvedOpts.openCodeConfigExtra?.compaction as - | { auto?: unknown } - | undefined; - if (compaction?.auto !== true) initializeIsolatedContextDb(env.dataDir); - writeConfigs(env, resolvedOpts.mockProviderURL, resolvedOpts); - - // Explicitly strip any inherited OPENCODE_SERVER_PASSWORD from the parent shell — - // our tests run unsecured on a random localhost port, and inherited auth would - // force every SDK request to carry Basic auth headers we don't set. - // Also strip NODE_ENV=test: Bun's test runner sets it automatically and the - // plugin's logger (src/shared/logger.ts) silences all output when NODE_ENV=test. - // We want the subprocess to behave like a real install, so the log file gets - // populated normally for diagnostics. - const childEnv: Record = {}; - for (const [key, value] of Object.entries(process.env)) { - if (value === undefined) continue; - if (key === "OPENCODE_SERVER_PASSWORD") continue; - if (key === "OPENCODE_SERVER_USERNAME") continue; - if (key === "NODE_ENV") continue; - // Strip any inherited subc supervised-launch identity. When the test - // process is itself launched under a subc supervisor (e.g. an AFT/Alfonso - // worktree sets SUBC_MODULE_ID=aft), the plugin's Rust module client would - // present THAT supervised identity to our hermetic daemon, which rejects it - // ("consumer_identity for module_id 'aft' did not match a supervised launch - // nonce"). A real opencode install is never launched under a supervised subc - // identity, so clearing these matches production and lets the plugin connect - // as an ordinary client. Harmless for TS-mode suites, which never touch subc. - if (key === "SUBC_MODULE_ID") continue; - if (key === "SUBC_LAUNCH_NONCE") continue; - childEnv[key] = value; - } - childEnv.OPENCODE_CONFIG_DIR = env.configDir; - childEnv.XDG_CONFIG_HOME = env.configDir; - childEnv.XDG_DATA_HOME = env.dataDir; - childEnv.XDG_CACHE_HOME = env.cacheDir; - // Ensure anthropic doesn't bail for missing env vars — we use a fake key. - childEnv.ANTHROPIC_API_KEY = "test-key-not-real"; - // Caller overrides (e.g. MAGIC_CONTEXT_LOG_PATH pointing the plugin log at a - // per-suite file so Rust-mode scenarios can assert on transform decisions). - // Merged last so an explicit override wins over the inherited value. - for (const [key, value] of Object.entries(resolvedOpts.extraEnv ?? {})) { - childEnv[key] = value; - } + const resources = rustMode && !opts.userMcHostConnectionFile ? await provision() : null; - // Bind to 0.0.0.0 (all interfaces) instead of 127.0.0.1 — empirically on - // GitHub-hosted runners, opencode binding to 127.0.0.1 sometimes results - // in Bun's `fetch()` timing out even though `curl` succeeds. Binding all - // interfaces removes any loopback-specific stack-resolution edge case - // (IPv4-only AF_INET vs IPv4-mapped IPv6, AF_UNSPEC name resolution, etc.). - // Clients still connect to `127.0.0.1:${port}` — only the listen socket - // changes. Safe locally too: process is short-lived, port is random. - const child: ChildProcess = spawn( - "opencode", - ["serve", "--port", String(port), "--hostname", "0.0.0.0"], - { - cwd: env.workdir, - env: childEnv, - stdio: ["ignore", "pipe", "pipe"], - }, - ); + let child: ChildProcess | undefined; + let cleanupPromise: Promise | undefined; + const cleanup = (): Promise => { + cleanupPromise ??= (async () => { + let cleanupError: unknown; + if (child) { + try { + await stopChild(child); + } catch (error) { + cleanupError = error; + } + } + try { + await resources?.mcHost.stop(); + } catch (error) { + cleanupError ??= error; + } + if (cleanupError !== undefined) throw cleanupError; + })(); + return cleanupPromise; + }; let stdoutBuf = ""; let stderrBuf = ""; - child.stdout?.on("data", (chunk: Buffer) => { - stdoutBuf += chunk.toString(); - }); - child.stderr?.on("data", (chunk: Buffer) => { - stderrBuf += chunk.toString(); - }); - - const url = `http://127.0.0.1:${port}`; try { - await waitForReady(url); - } catch (err) { - // Surface captured output on boot failure to help debugging. - child.kill("SIGTERM"); - await resources?.stack.stop(); + const resolvedOpts: SpawnOptions = resources + ? { + ...opts, + existingEnv: resources.env, + userMcHostConnectionFile: resources.connectionFile, + projectMagicContextConfig: { + ...(opts.projectMagicContextConfig ?? {}), + transform_mode: "rust", + }, + } + : opts; + + // Reuse a caller-provided env for the Rust-mode harness (connection file + // pre-placed, data dir shared across a serve restart); otherwise allocate. + const env = resolvedOpts.existingEnv ?? createIsolatedEnv(); + const port = resolvedOpts.port ?? (await pickFreePort()); + + const compaction = resolvedOpts.openCodeConfigExtra?.compaction as + | { auto?: unknown } + | undefined; + if (compaction?.auto !== true) initializeIsolatedContextDb(env.dataDir); + writeConfigs(env, resolvedOpts.mockProviderURL, resolvedOpts); + + // Explicitly strip any inherited OPENCODE_SERVER_PASSWORD from the parent shell — + // our tests run unsecured on a random localhost port, and inherited auth would + // force every SDK request to carry Basic auth headers we don't set. + // Also strip NODE_ENV=test: Bun's test runner sets it automatically and the + // plugin's logger (src/shared/logger.ts) silences all output when NODE_ENV=test. + // We want the subprocess to behave like a real install, so the log file gets + // populated normally for diagnostics. + const childEnv: Record = {}; + for (const [key, value] of Object.entries(process.env)) { + if (value === undefined) continue; + if (key === "OPENCODE_SERVER_PASSWORD") continue; + if (key === "OPENCODE_SERVER_USERNAME") continue; + if (key === "NODE_ENV") continue; + childEnv[key] = value; + } + childEnv.OPENCODE_CONFIG_DIR = env.configDir; + childEnv.XDG_CONFIG_HOME = env.configDir; + childEnv.XDG_DATA_HOME = env.dataDir; + childEnv.XDG_CACHE_HOME = env.cacheDir; + // Ensure anthropic doesn't bail for missing env vars — we use a fake key. + childEnv.ANTHROPIC_API_KEY = "test-key-not-real"; + // Caller overrides (e.g. MAGIC_CONTEXT_LOG_PATH pointing the plugin log at a + // per-suite file so Rust-mode scenarios can assert on transform decisions). + // Merged last so an explicit override wins over the inherited value. + for (const [key, value] of Object.entries(resolvedOpts.extraEnv ?? {})) { + childEnv[key] = value; + } + + // Bind to 0.0.0.0 (all interfaces) instead of 127.0.0.1 — empirically on + // GitHub-hosted runners, opencode binding to 127.0.0.1 sometimes results + // in Bun's `fetch()` timing out even though `curl` succeeds. Binding all + // interfaces removes any loopback-specific stack-resolution edge case + // (IPv4-only AF_INET vs IPv4-mapped IPv6, AF_UNSPEC name resolution, etc.). + // Clients still connect to `127.0.0.1:${port}` — only the listen socket + // changes. Safe locally too: process is short-lived, port is random. + child = spawn( + "opencode", + ["serve", "--port", String(port), "--hostname", "0.0.0.0"], + { + cwd: env.workdir, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }, + ); + + child.stdout?.on("data", (chunk: Buffer) => { + stdoutBuf += chunk.toString(); + }); + child.stderr?.on("data", (chunk: Buffer) => { + stderrBuf += chunk.toString(); + }); + + const url = `http://127.0.0.1:${port}`; + const startup = new AbortController(); + try { + await Promise.race([ + waitForReady(url, 300_000, startup.signal), + rejectOnSpawnError(child, startup.signal), + ]); + } finally { + startup.abort(); + } + + return { + url, + port, + env, + stdout: () => stdoutBuf, + stderr: () => stderrBuf, + mcHostStack: resources?.mcHost, + kill: cleanup, + }; + } catch (error) { + let cleanupError: unknown; + try { + await cleanup(); + } catch (failure) { + cleanupError = failure; + } throw new Error( - `opencode serve failed to start.\n--- stdout ---\n${stdoutBuf}\n--- stderr ---\n${stderrBuf}\n\n${String(err)}`, + `opencode serve failed to start.\n--- stdout ---\n${stdoutBuf}\n--- stderr ---\n${stderrBuf}\n\n${String(error)}` + + (cleanupError === undefined ? "" : `\ncleanup failed: ${String(cleanupError)}`), ); } +} - let rustStackStopped = false; - const stopProvisionedRustStack = async (): Promise => { - if (!resources || rustStackStopped) return; - rustStackStopped = true; - await resources.stack.stop(); - }; - - return { - url, - port, - env, - stdout: () => stdoutBuf, - stderr: () => stderrBuf, - rustStack: resources?.stack, - kill: async () => { - try { - if (child.exitCode === null && child.signalCode === null) { - child.kill("SIGTERM"); - await new Promise((resolveKill) => { - const timer = setTimeout(() => { - child.kill("SIGKILL"); - resolveKill(); - }, 3000); - child.once("exit", () => { - clearTimeout(timer); - resolveKill(); - }); - }); - } - } finally { - await stopProvisionedRustStack(); - } - }, - }; +export function spawnOpencode(opts: SpawnOptions): Promise { + return spawnOpencodeWithProvision(opts, provisionRustMode); } + +export const __spawnOpencodeTest = { + rejectOnSpawnError, + stopChild, + spawnOpencodeWithProvision, +}; diff --git a/packages/e2e-tests/src/rust-harness.ts b/packages/e2e-tests/src/rust-harness.ts index 1c68faf28..a14be672f 100644 --- a/packages/e2e-tests/src/rust-harness.ts +++ b/packages/e2e-tests/src/rust-harness.ts @@ -1,21 +1,11 @@ /** - * RustTestHarness — facade for the Rust-mode (ck-mc over subc) e2e lane. + * RustTestHarness drives OpenCode through U5's directly composed mc-host fixture. * - * Reuses the OpenCode e2e machinery UNCHANGED — the mock Anthropic provider and - * the `opencode serve` subprocess + SDK session driving — and layers on the two - * things Rust mode needs that the TS lane does not: + * Fixture and OpenCode share one isolated data root. Fixture starts before + * OpenCode so plugin discovery reaches a published, authenticated host. + * OpenCode restarts preserve database and module-store state. * - * 1. a hermetic subc daemon + ck-mc module (HermeticSubcStack) whose - * connection file is placed where the plugin's Rust client already looks - * (`${dataDir}/cortexkit/run/subc-connection.json`), and - * 2. serve RESTART support that keeps the same data dir (opencode.db + - * context.db + module store all survive), for the cold-start-drop-seed and - * module-restart scenarios. - * - * Boot order matters: the env is allocated first so the daemon can write its - * connection file BEFORE opencode boots and the plugin's first transform runs. - * - * Assertion surface: wire captures come from the fake provider's full request + * Assertion surface: wire captures come from the model mock's full request * bodies (the same source the TS lane asserts on). Rust transform decisions are * ALSO surfaced from the plugin diagnostic log (redirected per-suite via * MAGIC_CONTEXT_LOG_PATH) as a secondary signal — `readRustPasses()` parses the @@ -33,11 +23,11 @@ import { spawnOpencode, } from "./opencode-runner/spawn"; import { - buildHermeticBinaries, + buildDirectHostFixture, detectRustModePrereqs, - HermeticSubcStack, + HermeticMcHostStack, type RustModePrereqs, -} from "./rust-runner/hermetic-subc"; +} from "./rust-runner/hermetic-mc-host"; export interface RustTestHarnessOptions { /** magic-context USER-tier config overrides (thresholds, memory, etc.). */ @@ -49,19 +39,13 @@ export interface RustTestHarnessOptions { /** Default response used when the mock queue is empty. */ mockDefault?: MockResponse; /** - * Start opencode in TS mode instead of Rust mode. The hermetic daemon still + * Start opencode in TS mode instead of Rust mode. Direct host still * runs (so a later `restart({ rust: true })` can flip to Rust against the * same data dir) but the plugin transforms in TS on this boot. Used by the * cold-start-drop-seed scenario to build TS-mode state, then restart in Rust. * Default: false (boot straight into Rust mode). */ startInTsMode?: boolean; - /** - * Start the deterministic Broca producer. Disable it only when a scenario - * must observe module state before any historian publication can supersede it. - * Default: true. - */ - startHistorianProducer?: boolean; } export interface SdkClient { @@ -123,72 +107,62 @@ export interface RustPassLine { export class RustTestHarness { readonly mock: MockProvider; readonly env: IsolatedEnv; - readonly subc: HermeticSubcStack; + readonly mcHost: HermeticMcHostStack; readonly logPath: string; private opencodeInstance: SpawnedOpencode; private clientInstance: SdkClient; private contextDbCached: Database | null = null; private modelContextLimit: number | undefined; - private mockDefault: MockResponse; private readonly mockBaseURL: string; private constructor(args: { mock: MockProvider; mockBaseURL: string; env: IsolatedEnv; - subc: HermeticSubcStack; + mcHost: HermeticMcHostStack; opencode: SpawnedOpencode; client: SdkClient; logPath: string; modelContextLimit: number | undefined; - mockDefault: MockResponse; }) { this.mock = args.mock; this.mockBaseURL = args.mockBaseURL; this.env = args.env; - this.subc = args.subc; + this.mcHost = args.mcHost; this.opencodeInstance = args.opencode; this.clientInstance = args.client; this.logPath = args.logPath; this.modelContextLimit = args.modelContextLimit; - this.mockDefault = args.mockDefault; } - /** - * Preflight the lane. Cheap and never throws — call it in a describe-level - * guard so a machine without cargo / the subconscious sibling / a supported - * platform SKIPs with a printed reason instead of failing or hanging. - */ + /** Preflight current repository and Cargo. */ static detectPrereqs(): RustModePrereqs { return detectRustModePrereqs(); } - static async create(options: RustTestHarnessOptions = {}): Promise { + static async create( + options: RustTestHarnessOptions = {}, + ): Promise { const prereqs = detectRustModePrereqs(); - if (!prereqs.ok || !prereqs.subconsciousRoot) { + if (!prereqs.ok) { throw new Error( `RustTestHarness prerequisites unmet: ${prereqs.skipReason ?? "unknown"}. ` + "Guard the suite with RustTestHarness.detectPrereqs() and skip instead of creating.", ); } - const { ckMcBin, ckSubcBin } = await buildHermeticBinaries(prereqs.subconsciousRoot); + const fixtureBin = await buildDirectHostFixture(); const mock = new MockProvider(); const { baseURL } = await mock.start(); const mockDefault = options.mockDefault ?? DEFAULT_MOCK_RESPONSE; mock.setDefault(mockDefault); - // Env first: the daemon must write its connection file into - // /cortexkit/run/ before opencode boots and the plugin's Rust - // client connects on the first transform. const env = createIsolatedEnv(); - const subc = await HermeticSubcStack.start({ + const mcHost = await HermeticMcHostStack.start({ dataDir: env.dataDir, - ckMcBin, - ckSubcBin, - startProducer: options.startHistorianProducer ?? true, + fixtureBin, }); const logPath = join(env.dataDir, "cortexkit", "magic-context-e2e.log"); @@ -198,30 +172,32 @@ export class RustTestHarness { opencode = await RustTestHarness.spawnServe({ env, mockURL: baseURL, - connectionFile: subc.connectionFile, + connectionFile: mcHost.connectionFile, logPath, options, rustMode: !options.startInTsMode, }); } catch (error) { - await subc.stop(); + await mcHost.stop(); await mock.stop(); throw error; } const sdk = await import("@opencode-ai/sdk"); - const client = sdk.createOpencodeClient({ baseUrl: opencode.url }) as unknown as SdkClient; + // SAFETY: SdkClient is bounded subset of createOpencodeClient used by this harness. + const client = sdk.createOpencodeClient({ + baseUrl: opencode.url, + }) as unknown as SdkClient; return new RustTestHarness({ mock, mockBaseURL: baseURL, env, - subc, + mcHost, opencode, client, logPath, modelContextLimit: options.modelContextLimit, - mockDefault, }); } @@ -239,10 +215,7 @@ export class RustTestHarness { modelContextLimit: args.options.modelContextLimit, openCodeConfigExtra: args.options.openCodeConfigExtra, magicContextConfig: args.options.magicContextConfig, - // The connection_file value is only used to flip `userTierHasSubc` - // true (the resolver gate). The actual transport still reads the - // DEFAULT connection path — which this same file happens to be. - userSubcConnectionFile: args.connectionFile, + userMcHostConnectionFile: args.connectionFile, projectMagicContextConfig: { transform_mode: args.rustMode ? "rust" : "ts", }, @@ -260,11 +233,16 @@ export class RustTestHarness { /** * Restart `opencode serve` against the SAME data dir (opencode.db, context.db, - * module store, and the running daemon all persist). Optionally flip the + * module store, and the running direct host all persist). Optionally flip the * project transform_mode (ts↔rust) — the cold-start-drop-seed scenario builds * TS-mode state then restarts in Rust to prove drop-tag state seeds correctly. */ - async restart(opts: { rust?: boolean; magicContextConfig?: Record } = {}): Promise { + async restart( + opts: { + rust?: boolean; + magicContextConfig?: Record; + } = {}, + ): Promise { if (this.contextDbCached) { try { this.contextDbCached.close(); @@ -277,7 +255,7 @@ export class RustTestHarness { this.opencodeInstance = await RustTestHarness.spawnServe({ env: this.env, mockURL: this.mockBaseURL, - connectionFile: this.subc.connectionFile, + connectionFile: this.mcHost.connectionFile, logPath: this.logPath, options: { modelContextLimit: this.modelContextLimit, @@ -286,6 +264,7 @@ export class RustTestHarness { rustMode: opts.rust ?? true, }); const sdk = await import("@opencode-ai/sdk"); + // SAFETY: SdkClient is bounded subset of createOpencodeClient used by this harness. this.clientInstance = sdk.createOpencodeClient({ baseUrl: this.opencodeInstance.url, }) as unknown as SdkClient; @@ -317,10 +296,29 @@ export class RustTestHarness { */ ballast(tokens: number): string { const words = [ - "boundary", "historian", "compartment", "schedule", "pressure", - "tokens", "window", "publish", "transform", "session", "marker", - "budget", "eligible", "protected", "ordinal", "snapshot", "replay", - "decision", "threshold", "baseline", "measure", "archive", "deliver", + "boundary", + "historian", + "compartment", + "schedule", + "pressure", + "tokens", + "window", + "publish", + "transform", + "session", + "marker", + "budget", + "eligible", + "protected", + "ordinal", + "snapshot", + "replay", + "decision", + "threshold", + "baseline", + "measure", + "archive", + "deliver", ]; const target = Math.max(0, Math.round(tokens * 4)); const parts: string[] = []; @@ -338,7 +336,7 @@ export class RustTestHarness { /** * Append persisted messages through OpenCode's production database shape. This keeps * large-session transport tests fast while the next prompt still traverses the real - * OpenCode → plugin → subc → module path. + * OpenCode → plugin → direct host → McHandler path. */ appendSyntheticHistory( sessionId: string, @@ -357,15 +355,21 @@ export class RustTestHarness { .prepare( "SELECT m.data AS message_data, p.data AS part_data FROM message m JOIN part p ON p.message_id = m.id WHERE m.session_id = ? AND json_extract(m.data, '$.role') = 'user' AND json_extract(p.data, '$.type') = 'text' ORDER BY m.time_created DESC LIMIT 1", ) - .get(sessionId) as { message_data: string; part_data: string } | undefined; + .get(sessionId) as + | { message_data: string; part_data: string } + | undefined; if (!templateRow) { - throw new Error("synthetic history requires an existing user text message"); + throw new Error( + "synthetic history requires an existing user text message", + ); } - const messageTemplate = JSON.parse(templateRow.message_data) as Record< + const messageTemplate = JSON.parse( + templateRow.message_data, + ) as Record; + const partTemplate = JSON.parse(templateRow.part_data) as Record< string, unknown >; - const partTemplate = JSON.parse(templateRow.part_data) as Record; const insertMessage = db.prepare( "INSERT INTO message (id, session_id, time_created, time_updated, data) VALUES (?, ?, ?, ?, ?)", ); @@ -381,10 +385,15 @@ export class RustTestHarness { timestamp: number, counter: number, ): string => { - const encoded = ~(BigInt(timestamp) * 0x1000n + BigInt(counter)); + const encoded = ~( + BigInt(timestamp) * 0x1000n + + BigInt(counter) + ); const timeBytes = Buffer.alloc(6); for (let byte = 0; byte < timeBytes.length; byte += 1) { - timeBytes[byte] = Number((encoded >> BigInt(40 - 8 * byte)) & 0xffn); + timeBytes[byte] = Number( + (encoded >> BigInt(40 - 8 * byte)) & 0xffn, + ); } return `${prefix}_${timeBytes.toString("hex")}${counter.toString(36).padStart(14, "0")}`; }; @@ -406,8 +415,9 @@ export class RustTestHarness { id: messageId, sessionID: sessionId, time: { - ...((messageTemplate.time as Record | undefined) ?? - {}), + ...((messageTemplate.time as + | Record + | undefined) ?? {}), created: timestamp, }, }), @@ -448,13 +458,17 @@ export class RustTestHarness { ...(options.agent ? { agent: options.agent } : {}), }, }); - const timeout = new Promise((r) => setTimeout(() => r(null), timeoutMs)); + const timeout = new Promise((r) => + setTimeout(() => r(null), timeoutMs), + ); const result = await Promise.race([promptPromise, timeout]); if (result === null) { throw new Error( `sendPrompt did not complete within ${timeoutMs}ms. stderr:\n${this.opencodeInstance .stderr() - .slice(-2000)}\nmodule log:\n${this.subc.moduleLog().slice(-2000)}`, + .slice( + -2000, + )}\nmc-host log:\n${this.mcHost.hostLog().slice(-2000)}`, ); } return result; @@ -474,10 +488,16 @@ export class RustTestHarness { } /** Fetch the session's messages via the SDK (for choosing a mid-session id to remove). */ - async listMessages(sessionId: string): Promise> { - const res = await this.clientInstance.session.messages({ path: { id: sessionId } }); + async listMessages( + sessionId: string, + ): Promise> { + const res = await this.clientInstance.session.messages({ + path: { id: sessionId }, + }); const data = (res as { data?: unknown }).data; - return Array.isArray(data) ? (data as Array<{ info?: { id?: string; role?: string } }>) : []; + return Array.isArray(data) + ? (data as Array<{ info?: { id?: string; role?: string } }>) + : []; } // ── wire captures (from the fake provider) ──────────────────────────────── @@ -486,14 +506,20 @@ export class RustTestHarness { mainRequests() { return this.mock .requests() - .filter((r) => JSON.stringify(r.body.system ?? "").includes("## Magic Context")); + .filter((r) => + JSON.stringify(r.body.system ?? "").includes( + "## Magic Context", + ), + ); } /** The messages array of the most recent main-agent request. */ lastMainMessages(): Array<{ role?: string; content?: unknown }> { const req = this.mainRequests().at(-1); const messages = req?.body.messages; - return Array.isArray(messages) ? (messages as Array<{ role?: string; content?: unknown }>) : []; + return Array.isArray(messages) + ? (messages as Array<{ role?: string; content?: unknown }>) + : []; } /** @@ -521,7 +547,10 @@ export class RustTestHarness { * immediately after a prompt returns can miss the just-emitted pass. Polling * removes that race without a fixed sleep (the no-sleeps-as-sync rule). */ - async waitForRustPasses(minCount: number, timeoutMs = 15_000): Promise { + async waitForRustPasses( + minCount: number, + timeoutMs = 15_000, + ): Promise { return this.waitFor( () => { const passes = this.readRustPasses(); @@ -563,8 +592,12 @@ export class RustTestHarness { wireBuildMs: Number(stageField(body, "wire_build") || "0"), wireMessages: Number(stageField(body, "wire_messages") || "0"), transportMs: Number(stageField(body, "transport") || "0"), - transportPages: Number(stageField(body, "transport_pages") || "0"), - transportBytes: Number(stageField(body, "transport_bytes") || "0"), + transportPages: Number( + stageField(body, "transport_pages") || "0", + ), + transportBytes: Number( + stageField(body, "transport_bytes") || "0", + ), rowVersion: Number(field(body, "row_version") || "0"), raw: line, }); @@ -593,14 +626,21 @@ export class RustTestHarness { // ── context.db access (plugin state) ────────────────────────────────────── private contextDbPath(): string { - return join(this.env.dataDir, "cortexkit", "magic-context", "context.db"); + return join( + this.env.dataDir, + "cortexkit", + "magic-context", + "context.db", + ); } contextDb(): Database { if (this.contextDbCached) return this.contextDbCached; const dbPath = this.contextDbPath(); if (!existsSync(dbPath)) { - throw new Error(`context.db not found at ${dbPath} — plugin may not have initialized yet.`); + throw new Error( + `context.db not found at ${dbPath} — plugin may not have initialized yet.`, + ); } this.contextDbCached = new Database(dbPath, { readonly: true }); return this.contextDbCached; @@ -613,7 +653,9 @@ export class RustTestHarness { countTagsByStatus(sessionId: string, status: string): number { try { const row = this.contextDb() - .prepare("SELECT COUNT(*) AS n FROM tags WHERE session_id = ? AND status = ?") + .prepare( + "SELECT COUNT(*) AS n FROM tags WHERE session_id = ? AND status = ?", + ) .get(sessionId, status) as { n: number } | null; return row?.n ?? 0; } catch { @@ -640,10 +682,14 @@ export class RustTestHarness { const db = new Database(dbPath); try { const result = db - .prepare("UPDATE session_meta SET cache_ttl = ? WHERE session_id = ?") + .prepare( + "UPDATE session_meta SET cache_ttl = ? WHERE session_id = ?", + ) .run(cacheTtl, sessionId) as { changes?: number }; if (result.changes !== 1) { - throw new Error(`session cache TTL update affected ${result.changes ?? 0} rows`); + throw new Error( + `session cache TTL update affected ${result.changes ?? 0} rows`, + ); } } finally { db.close(); @@ -664,14 +710,14 @@ export class RustTestHarness { } this.contextDbCached = null; } - // Kill order: opencode (holds the plugin's subc client) → module → daemon. + // OpenCode owns client connections, so stop it before fixture. try { await this.opencodeInstance.kill(); } catch { // ignore } try { - await this.subc.stop(); + await this.mcHost.stop(); } catch { // ignore } @@ -682,7 +728,10 @@ export class RustTestHarness { } // Reclaim the per-suite temp tree (best-effort). try { - rmSync(join(this.env.dataDir, ".."), { recursive: true, force: true }); + rmSync(join(this.env.dataDir, ".."), { + recursive: true, + force: true, + }); } catch { // ignore } @@ -705,15 +754,32 @@ export function stableSerialize(value: unknown): string { return JSON.stringify(stripCacheControl(value)); } -function stripCacheControl(value: unknown): unknown { - if (Array.isArray(value)) return value.map(stripCacheControl); +type CacheStrippedValue = + | null + | boolean + | number + | string + | CacheStrippedValue[] + | { [key: string]: CacheStrippedValue | undefined }; + +function stripCacheControl(value: unknown): CacheStrippedValue | undefined { + if (Array.isArray(value)) + return value + .map(stripCacheControl) + .filter((item) => item !== undefined); if (value && typeof value === "object") { - const out: Record = {}; + const out: { [key: string]: CacheStrippedValue | undefined } = {}; for (const [key, child] of Object.entries(value)) { if (key === "cache_control") continue; out[key] = stripCacheControl(child); } return out; } - return value; + if ( + value === null || + ["boolean", "number", "string"].includes(typeof value) + ) { + return value as null | boolean | number | string; + } + return undefined; } diff --git a/packages/e2e-tests/src/rust-runner/fake-broca.ts b/packages/e2e-tests/src/rust-runner/fake-broca.ts deleted file mode 100644 index 52500c786..000000000 --- a/packages/e2e-tests/src/rust-runner/fake-broca.ts +++ /dev/null @@ -1,206 +0,0 @@ -/** - * Hermetic historian producer for Rust-mode e2e tests. - * - * The real module-side historian speaks to a Broca management surface rather - * than the OpenCode model mock. Keeping this producer in test support makes - * that boundary real while keeping every response deterministic and offline. - */ - -import { - managementSurfaceManifest, - SubcProvider, - type ProviderRequestContext, - type RouteBindRequest, - type RouteHandle, -} from "@cortexkit/subc-client"; - -const MODULE_ID = "broca"; -const connectionFile = process.env.BROCA_CONNECTION_FILE; -if (!connectionFile) throw new Error("BROCA_CONNECTION_FILE is required"); - -interface ProducerRequest { - method?: string; - params?: Record; -} - -interface RunRecord { - runId: string; - sessionId: string; - output: string; -} - -const routeSessions = new Map(); -const runs = new Map(); -const latestRunBySession = new Map(); -let nextRun = 1; - -function log(message: string): void { - process.stdout.write(`[broca] ${message}\n`); -} - -function jsonBytes(value: unknown): Uint8Array { - return new TextEncoder().encode(JSON.stringify(value)); -} - -function requestFrom(body: Uint8Array): ProducerRequest { - return JSON.parse(new TextDecoder().decode(body)) as ProducerRequest; -} - -function requestSession(handle: RouteHandle): string { - const sessionId = routeSessions.get(handle.channel); - if (!sessionId) throw new Error(`route ${handle.channel} is not bound to a session`); - return sessionId; -} - -function ordinalRange(prompt: string): { start: number; end: number } { - const startMarker = prompt.indexOf(""); - const endMarker = prompt.indexOf(""); - const rawChunk = - startMarker >= 0 - ? prompt.slice(startMarker + "".length, endMarker > startMarker ? endMarker : undefined) - : prompt; - const ordinals = [...rawChunk.matchAll(/^\s*\[(\d+)(?:-(\d+))?\]/gm)].flatMap( - (match) => [Number(match[1]), Number(match[2] ?? match[1])], - ); - if (ordinals.length > 0) { - return { - start: Math.min(...ordinals), - end: Math.max(...ordinals), - }; - } - const range = prompt.match(/Messages\s+(\d+)-(\d+):/i); - if (range) return { start: Number(range[1]), end: Number(range[2]) }; - return { start: 1, end: 1 }; -} - -function deterministicTitle(prompt: string, start: number, end: number): string { - const knownLabels: Array<[string, string]> = [ - ["cache-invariant", "cache-invariant chunk"], - ["Long OpenCode e2e chunk", "Long OpenCode e2e chunk"], - ["long-running OpenCode", "Long OpenCode e2e chunk"], - ["OpenCode warm-up cache-stability", "Long OpenCode e2e chunk"], - ["Rust fold e2e chunk", "Rust fold e2e chunk"], - ["fold-under-pressure", "Rust fold e2e chunk"], - ["Rust reduce e2e chunk", "Rust reduce e2e chunk"], - ["ctx_reduce", "Rust reduce e2e chunk"], - ]; - for (const [needle, title] of knownLabels) { - if (prompt.includes(needle)) return title; - } - return `Hermetic Broca chunk ${start}-${end}`; -} - -function deterministicOutput(prompt: string): string { - const { start, end } = ordinalRange(prompt); - const title = deterministicTitle(prompt, start, end); - const tierOne = `${title}`; - return `\n\n` + - `\n` + - `${tierOne}\n` + - `Deterministic historian coverage ${start}-${end}.\n` + - `Published by the hermetic Broca producer.\n` + - `Replay is stable for this chunk.\n` + - `\n\n` + - `\n` + - `\n` + - `${end + 1}\n` + - ``; -} - -function event(run: RunRecord, unit: Record): Uint8Array { - return jsonBytes({ kind: "control", unit: { run_id: run.runId, ...unit } }); -} - -const manifest = managementSurfaceManifest({ - moduleId: MODULE_ID, - operations: [ - { name: "session.send", kind: "mutate" }, - { name: "session.subscribe", kind: "query" }, - { name: "run.status", kind: "query" }, - { name: "run.cancel", kind: "mutate" }, - { name: "session.delete", kind: "mutate" }, - ], -}); - -const provider = await SubcProvider.connect({ - connectionFile, - manifest, - health: () => ({ status: "ok", detail: "deterministic hermetic historian producer" }), - onBind: (request: RouteBindRequest) => { - if (request.target.kind !== "management_surface" || request.target.module_id !== MODULE_ID) { - return { accept: false, code: "wrong_target", message: "Broca only serves its management surface" }; - } - if (!request.identity.session) { - return { accept: false, code: "missing_session", message: "Broca requires a session identity" }; - } - routeSessions.set(request.handle.channel, request.identity.session); - return true; - }, - onBound: (handle: RouteHandle) => { - log(`route_bound channel=${handle.channel}`); - }, - onRouteGone: (handle: RouteHandle) => { - routeSessions.delete(handle.channel); - log(`route_gone channel=${handle.channel}`); - }, - handler: async (handle: RouteHandle, body: Uint8Array, ctx: ProviderRequestContext) => { - const request = requestFrom(body); - const params = request.params ?? {}; - const method = request.method; - if (method === "session.send") { - const sessionId = requestSession(handle); - const system = typeof params.system === "string" ? params.system : ""; - const prompt = typeof params.prompt === "string" ? params.prompt : ""; - if (!system || !prompt) throw new Error("session.send requires calibrated system and prompt fields"); - const runId = `broca-run-${nextRun++}`; - const run: RunRecord = { runId, sessionId, output: deterministicOutput(prompt) }; - runs.set(runId, run); - latestRunBySession.set(sessionId, runId); - log(`session.send run_id=${runId} session=${sessionId} system_bytes=${system.length} prompt_bytes=${prompt.length}`); - return jsonBytes({ run_id: runId }); - } - if (method === "session.subscribe") { - const sessionId = requestSession(handle); - const runId = latestRunBySession.get(sessionId); - const run = runId ? runs.get(runId) : undefined; - if (!run) throw new Error(`no historian run for session ${sessionId}`); - log(`session.subscribe run_id=${run.runId} session=${sessionId}`); - await ctx.emit(event(run, { type: "run_started" })); - await ctx.emit(event(run, { - type: "assistant_message", - message: { role: "assistant", content: [{ type: "text", text: run.output }] }, - })); - await ctx.emit(event(run, { type: "run_finished" })); - return; - } - if (method === "run.status") { - const runId = typeof params.run_id === "string" ? params.run_id : ""; - if (!runs.has(runId)) return jsonBytes({ run_id: runId, state: "error", last_error: "unknown run" }); - return jsonBytes({ run_id: runId, state: "completed" }); - } - if (method === "run.cancel") { - const runId = typeof params.run_id === "string" ? params.run_id : ""; - log(`run.cancel run_id=${runId}`); - return jsonBytes({ ok: true }); - } - if (method === "session.delete") { - const sessionId = typeof params.session_id === "string" ? params.session_id : requestSession(handle); - for (const [runId, run] of runs) if (run.sessionId === sessionId) runs.delete(runId); - latestRunBySession.delete(sessionId); - return jsonBytes({ ok: true }); - } - throw new Error(`unsupported Broca method ${method ?? ""}`); - }, -}); - -log(`ready module_id=${MODULE_ID}`); - -const keepAlive = setInterval(() => undefined, 60_000); -const close = async (): Promise => { - clearInterval(keepAlive); - await provider.close(); - process.exit(0); -}; -process.once("SIGTERM", () => void close()); -process.once("SIGINT", () => void close()); -await new Promise(() => undefined); diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.test.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.test.ts new file mode 100644 index 000000000..5a8e4d1ea --- /dev/null +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.test.ts @@ -0,0 +1,340 @@ +/// + +import { afterEach, describe, expect, it } from "bun:test"; +import { createServer, type Server, Socket } from "node:net"; +import { + chmodSync, + existsSync, + lstatSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { McHostClient } from "@magic-context/core/shared/mc-host-client"; +import { + __hermeticMcHostTest, + buildDirectHostFixture, + HermeticMcHostStack, +} from "./hermetic-mc-host"; + +const temporaryRoots: string[] = []; + +function mode(path: string): number { + return lstatSync(path).mode & 0o777; +} + +async function waitFor(read: () => Promise, predicate: (value: T) => boolean): Promise { + const deadline = Date.now() + 20_000; + for (;;) { + const value = await read(); + if (predicate(value)) return value; + if (Date.now() >= deadline) throw new Error("fixture state did not converge"); + await Bun.sleep(10); + } +} + +function brocaCall(client: McHostClient, prompt: string): Promise> { + return client.call>( + "broca", + "session.send", + { + prompt, + model: { provider: "fixture", model: "deterministic" }, + tools: [], + generation: { max_output_tokens: 1_024, temperature: 0.1 }, + }, + { targetKind: "management_surface" }, + ); +} + +async function rawControl(path: string, request: Buffer): Promise> { + const socket = await new Promise((resolveSocket, rejectSocket) => { + const candidate = new Socket(); + candidate.once("error", rejectSocket); + candidate.connect(path, () => { + candidate.off("error", rejectSocket); + resolveSocket(candidate); + }); + }); + socket.write(request); + const response = await new Promise((resolveResponse, rejectResponse) => { + let bytes = Buffer.alloc(0); + socket.on("data", (chunk: Buffer) => { + bytes = Buffer.concat([bytes, chunk]); + const newline = bytes.indexOf(0x0a); + if (newline >= 0) resolveResponse(bytes.subarray(0, newline)); + if (bytes.byteLength > __hermeticMcHostTest.maxLineBytes + 1) { + rejectResponse(new Error("raw fixture response exceeded cap")); + } + }); + socket.once("error", rejectResponse); + }); + socket.destroy(); + return JSON.parse(response.toString("utf8")) as Record; +} + +async function mockControl( + responder: (request: Record, socket: Socket) => void, +): Promise<{ client: InstanceType; server: Server }> { + const root = mkdtempSync(join(tmpdir(), "mc-control-client-")); + temporaryRoots.push(root); + const path = join(root, "control.sock"); + const server = createServer((socket) => { + socket.on("data", (chunk: Buffer) => { + const line = chunk.toString("utf8").trim(); + responder(JSON.parse(line) as Record, socket); + }); + }); + await new Promise((resolveListen) => server.listen(path, resolveListen)); + const client = new __hermeticMcHostTest.FixtureControlClient(path, 500); + await client.connect(); + return { client, server }; +} + +afterEach(() => { + for (const root of temporaryRoots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +describe("direct mc-host fixture contract", () => { + it("parses only the bounded readiness schema and reaps only stale PID records", () => { + const valid = Buffer.from( + JSON.stringify({ + status: "ready", + wire_version: 2, + catalog: ["magic-context", "synapse", "broca"], + }), + ); + expect(__hermeticMcHostTest.parseReadyRecord(valid).status).toBe("ready"); + expect(() => + __hermeticMcHostTest.parseReadyRecord( + Buffer.from('{"status":"ready","wire_version":2,"catalog":[],"key":"secret"}'), + ), + ).toThrow(); + expect(() => + __hermeticMcHostTest.parseReadyRecord( + Buffer.alloc(__hermeticMcHostTest.maxLineBytes + 1, 0x78), + ), + ).toThrow(); + + const nowMs = 10 * __hermeticMcHostTest.stalePidAgeMs; + expect( + __hermeticMcHostTest.isStaleRustE2ePidRecord( + nowMs - __hermeticMcHostTest.stalePidAgeMs + 1, + nowMs, + ), + ).toBe(false); + expect( + __hermeticMcHostTest.isStaleRustE2ePidRecord( + nowMs - __hermeticMcHostTest.stalePidAgeMs, + nowMs, + ), + ).toBe(true); + expect(__hermeticMcHostTest.isStaleRustE2ePidRecord(nowMs + 1, nowMs)).toBe(false); + }); + + it("rejects readiness emitted before control and secure publication exist", async () => { + const root = mkdtempSync(join(tmpdir(), "opencode-e2e-early-ready-")); + temporaryRoots.push(root); + const fixtureBin = join(root, "early-ready-fixture.sh"); + writeFileSync( + fixtureBin, + `#!/bin/sh\nprintf '%s\\n' '{"status":"ready","wire_version":2,"catalog":["magic-context","synapse","broca"]}'\nsleep 1\nmkdir -p "$2/cortexkit/run"\n: > "$2/direct-host-control.sock"\n: > "$2/cortexkit/run/subc-connection.json"\nsleep 60\n`, + { mode: 0o700 }, + ); + + const startupError = await HermeticMcHostStack.start({ + dataDir: root, + fixtureBin, + startTimeoutMs: 2_000, + }).catch((error: unknown) => error); + expect(String(startupError)).toContain( + "direct mc-host readiness preceded secure publication", + ); + expect(existsSync(root)).toBe(false); + temporaryRoots.splice(temporaryRoots.indexOf(root), 1); + }, 15_000); + + it("rejects malformed, unknown, oversized, mismatched, and duplicate responses", async () => { + const cases: Array<(request: Record, socket: Socket) => void> = [ + (_request, socket) => socket.write("not-json\n"), + (request, socket) => + socket.write(`${JSON.stringify({ id: request.id, ok: true, result: { accepted: true }, extra: true })}\n`), + (_request, socket) => socket.write(`${"x".repeat(__hermeticMcHostTest.maxLineBytes + 1)}\n`), + (_request, socket) => socket.write(`${JSON.stringify({ id: 999, ok: true, result: { accepted: true } })}\n`), + ]; + for (const responder of cases) { + const { client, server } = await mockControl(responder); + expect(await client.backendSuccess().catch((error: unknown) => error)).toBeInstanceOf( + Error, + ); + client.close(); + await new Promise((resolveClose) => server.close(() => resolveClose())); + } + + const { client, server } = await mockControl((request, socket) => { + const response = `${JSON.stringify({ id: request.id, ok: true, result: { accepted: true } })}\n`; + socket.write(response + response); + }); + await client.backendSuccess(); + await Bun.sleep(20); + expect(await client.counters().catch((error: unknown) => error)).toBeInstanceOf(Error); + client.close(); + await new Promise((resolveClose) => server.close(() => resolveClose())); + }); + + it( + "proves permissions, controls, managed readiness, redaction, and JSONL shutdown", + async () => { + const fixtureBin = await buildDirectHostFixture(); + const root = mkdtempSync(join(tmpdir(), "opencode-e2e-direct-host-")); + temporaryRoots.push(root); + chmodSync(root, 0o700); + const stack = await HermeticMcHostStack.start({ dataDir: root, fixtureBin }); + const sentinel = "u7-request-sentinel-DO-NOT-LOG"; + try { + expect(mode(root)).toBe(0o700); + expect(mode(stack.controlPath)).toBe(0o600); + expect(mode(stack.connectionFile)).toBe(0o600); + + const publicationText = readFileSync(stack.connectionFile, "utf8"); + const publication = JSON.parse(publicationText) as { + key: number[]; + daemon_id: number[]; + }; + const secretRenderings = [ + publication.key.join(","), + publication.key.join(", "), + Buffer.from(publication.key).toString("hex"), + Buffer.from(publication.key).toString("hex").toUpperCase(), + Buffer.from(publication.key).toString("base64"), + publication.daemon_id.join(","), + publication.daemon_id.join(", "), + Buffer.from(publication.daemon_id).toString("hex"), + Buffer.from(publication.daemon_id).toString("hex").toUpperCase(), + Buffer.from(publication.daemon_id).toString("base64"), + publicationText, + JSON.stringify(publication), + ]; + + const before = await stack.backendCounters(); + const controlResponses: string[] = []; + const thrownErrors: string[] = []; + const malformedControls = [ + Buffer.from(`{"id":30,"sentinel":"${sentinel}","command":\n`), + Buffer.from( + `${JSON.stringify({ id: 31, sentinel, command: { name: "unknown" } })}\n`, + ), + Buffer.concat([ + Buffer.from(`{"id":32,"sentinel":"${sentinel}","padding":"`), + Buffer.alloc(__hermeticMcHostTest.maxLineBytes + 1, 0x78), + Buffer.from('"}\n'), + ]), + ]; + for (const bytes of malformedControls) { + try { + const response = await rawControl(stack.controlPath, bytes); + expect(response.ok).toBe(false); + controlResponses.push(JSON.stringify(response)); + } catch (error) { + thrownErrors.push(String(error)); + } + } + expect(controlResponses.length + thrownErrors.length).toBe(malformedControls.length); + expect(await stack.backendCounters()).toEqual(before); + + const callFor = async (session: string, prompt: string): Promise => { + const client = await McHostClient.connect({ + connectionFile: stack.connectionFile, + identity: { project_root: root, harness: "opencode", session }, + targetKind: "management_surface", + }); + try { + expect((await brocaCall(client, prompt)).run_id).toBeString(); + } finally { + await client.closeAsync(); + } + }; + + await stack.backendSuccess(); + await callFor("fixture-success", sentinel); + await waitFor( + () => stack.backendCounters(), + (counters) => counters.completed === before.completed + 1, + ); + + expect(await stack.releaseBlockedBackendCall()).toBe(false); + await stack.blockNextBackendCall(); + await callFor("fixture-blocked", "blocked request body"); + await waitFor( + () => stack.backendCounters(), + (counters) => counters.blocked === before.blocked + 1, + ); + expect(await stack.releaseBlockedBackendCall()).toBe(true); + await waitFor( + () => stack.backendCounters(), + (counters) => counters.released === before.released + 1, + ); + + await stack.failNextBackendCall(); + await callFor("fixture-failure", "typed outage"); + await waitFor( + () => stack.backendCounters(), + (counters) => counters.failed === before.failed + 1, + ); + const diagnostics = __hermeticMcHostTest.diagnostics(stack); + const observedOutputs = [ + ...controlResponses, + ...thrownErrors, + diagnostics.stdout, + diagnostics.stderr, + diagnostics.retainedLog, + stack.hostLog(), + ]; + const forbidden = [sentinel, ...secretRenderings]; + expect( + observedOutputs.some((output) => + forbidden.some((secret) => secret.length > 0 && output.includes(secret)), + ), + ).toBe(false); + await stack.stop(); + expect(existsSync(root)).toBe(false); + temporaryRoots.splice(temporaryRoots.indexOf(root), 1); + } finally { + await stack.stop(); + } + }, + 180_000, + ); + + it( + "routes SIGTERM through fixture cleanup", + async () => { + const fixtureBin = await buildDirectHostFixture(); + const root = mkdtempSync(join(tmpdir(), "opencode-e2e-direct-host-term-")); + temporaryRoots.push(root); + const stack = await HermeticMcHostStack.start({ dataDir: root, fixtureBin }); + await stack.blockNextBackendCall(); + const client = await McHostClient.connect({ + connectionFile: stack.connectionFile, + identity: { project_root: root, harness: "opencode", session: "sigterm" }, + targetKind: "management_surface", + }); + await brocaCall(client, "sigterm sentinel request"); + await waitFor( + () => stack.backendCounters(), + (counters) => counters.blocked >= 1, + ); + await client.closeAsync().catch(() => undefined); + await stack.terminateHost(); + expect(existsSync(stack.controlPath)).toBe(false); + expect(existsSync(stack.connectionFile)).toBe(false); + await stack.stop(); + expect(existsSync(root)).toBe(false); + temporaryRoots.splice(temporaryRoots.indexOf(root), 1); + }, + 180_000, + ); +}); diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts new file mode 100644 index 000000000..a2d54c6ee --- /dev/null +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -0,0 +1,1005 @@ +/** Direct mc-host fixture stack for Rust-mode E2E tests. */ + +import { type ChildProcess, spawn, spawnSync } from "node:child_process"; +import { + appendFileSync, + chmodSync, + existsSync, + lstatSync, + mkdirSync, + readFileSync, + readdirSync, + realpathSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { createConnection, type Socket } from "node:net"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { + McHostClient, + type BindIdentity, +} from "@magic-context/core/shared/mc-host-client"; + +const REPO_ROOT = resolve(import.meta.dir, "../../../.."); +const FIXTURE_BINARY = join( + REPO_ROOT, + "target/debug/examples/direct_host_fixture", +); +const CONTROL_FILE = "direct-host-control.sock"; +const CONNECTION_FILE = "subc-connection.json"; +const PID_FILE = "rust-e2e-pids.json"; +const MAX_LINE_BYTES = 64 * 1024; +const MAX_LOG_BYTES = 256 * 1024; +const STALE_PID_AGE_MS = 30 * 60 * 1_000; +const EXPECTED_CATALOG = ["magic-context", "synapse", "broca"] as const; + +interface RustE2ePidFile { + createdAtMs: number; + pids: Array<{ pid: number; role: "mc-host"; executable: string }>; +} + +export interface RustModePrereqs { + ok: boolean; + skipReason?: string; +} + +export interface BackendCounters { + started: number; + completed: number; + blocked: number; + released: number; + failed: number; + cancelled: number; +} + +interface ReadyRecord { + status: "ready"; + wire_version: 2; + catalog: ["magic-context", "synapse", "broca"]; +} + +type ControlCommand = + | "backend-success" + | "block-next-call" + | "release-blocked-call" + | "typed-failure" + | "counters" + | "graceful-shutdown"; + +type PendingControl = { + command: ControlCommand; + resolve: (value: unknown) => void; + reject: (error: Error) => void; + timer: ReturnType; +}; + +function processStartTimeMs(pid: number): number | null { + const result = spawnSync("ps", ["-o", "lstart=", "-p", String(pid)], { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + if (result.status !== 0 || typeof result.stdout !== "string") return null; + const startedAt = Date.parse(result.stdout.trim()); + return Number.isFinite(startedAt) ? startedAt : null; +} + +function processExecutable(pid: number): string | null { + try { + return realpathSync(`/proc/${pid}/exe`); + } catch { + const result = spawnSync("ps", ["-o", "command=", "-p", String(pid)], { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + if (result.status !== 0 || typeof result.stdout !== "string") return null; + const command = result.stdout.trim().split(/\s+/, 1)[0]; + if (!command) return null; + try { + return realpathSync(command); + } catch { + return null; + } + } +} + +function isStaleRustE2ePidRecord( + createdAtMs: number, + nowMs = Date.now(), +): boolean { + return ( + Number.isFinite(createdAtMs) && + createdAtMs <= nowMs && + nowMs - createdAtMs >= STALE_PID_AGE_MS + ); +} + +function reapRecordedRustProcesses(): void { + let candidates: string[]; + try { + candidates = readdirSync(tmpdir(), { withFileTypes: true }) + .filter( + (entry) => + entry.isDirectory() && + entry.name.startsWith("opencode-e2e-"), + ) + .map((entry) => + join(tmpdir(), entry.name, "data", "cortexkit", PID_FILE), + ); + } catch { + return; + } + for (const pidPath of candidates) { + if (!existsSync(pidPath)) continue; + let stale = false; + try { + const record = JSON.parse( + readFileSync(pidPath, "utf8"), + ) as RustE2ePidFile; + if (!Array.isArray(record.pids)) continue; + stale = isStaleRustE2ePidRecord(record.createdAtMs); + if (!stale) continue; + const recordedSecond = + Math.floor(record.createdAtMs / 1_000) * 1_000; + for (const entry of record.pids) { + if ( + !Number.isInteger(entry?.pid) || + entry.pid <= 0 || + entry.role !== "mc-host" || + typeof entry.executable !== "string" + ) { + continue; + } + const startedAt = processStartTimeMs(entry.pid); + const executable = processExecutable(entry.pid); + if ( + startedAt === null || + Math.abs(startedAt - recordedSecond) > 5_000 || + executable !== entry.executable + ) { + continue; + } + try { + process.kill(entry.pid, "SIGKILL"); + } catch { + // Process exited after identity check. + } + } + } catch { + // Partial records prove no process identity. + } finally { + if (stale) rmSync(pidPath, { force: true }); + } + } +} + +export function detectRustModePrereqs(): RustModePrereqs { + if (process.platform === "win32") { + return { + ok: false, + skipReason: "direct mc-host fixture requires Unix sockets", + }; + } + if (!existsSync(join(REPO_ROOT, "Cargo.toml"))) { + return { + ok: false, + skipReason: "current repository Cargo workspace is missing", + }; + } + const cargo = spawnSync("cargo", ["--version"], { stdio: "ignore" }); + if (cargo.error || cargo.status !== 0) { + return { ok: false, skipReason: "cargo is not available on PATH" }; + } + return { ok: true }; +} + +let fixtureBuild: Promise | null = null; + +function runCargo(args: string[]): Promise<{ ok: boolean; stderr: string }> { + return new Promise((resolveRun) => { + const child = spawn("cargo", args, { + cwd: REPO_ROOT, + stdio: ["ignore", "ignore", "pipe"], + }); + let stderr = ""; + child.stderr?.on("data", (chunk: Buffer) => { + stderr = `${stderr}${chunk.toString()}`.slice(-8_000); + }); + child.once("error", () => + resolveRun({ ok: false, stderr: "cargo spawn failed" }), + ); + child.once("exit", (code) => resolveRun({ ok: code === 0, stderr })); + }); +} + +/** Build U5 fixture once per Bun process. Cargo handles cross-process incremental caching. */ +export function buildDirectHostFixture(): Promise { + if (fixtureBuild) return fixtureBuild; + fixtureBuild = (async () => { + const configured = process.env.MC_E2E_DIRECT_HOST_FIXTURE_BIN; + if (configured && existsSync(configured)) return configured; + const build = await runCargo([ + "build", + "-p", + "mc-module", + "--example", + "direct_host_fixture", + "--features", + "direct-host-fixture", + ]); + if (!build.ok || !existsSync(FIXTURE_BINARY)) { + throw new Error( + `direct mc-host fixture build failed\n${build.stderr}`, + ); + } + return FIXTURE_BINARY; + })(); + return fixtureBuild; +} + +function exactKeys( + value: Record, + expected: string[], +): boolean { + const actual = Object.keys(value).sort(); + return ( + actual.length === expected.length && + actual.every((key, index) => key === expected[index]) + ); +} + +function record(value: unknown): Record | null { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : null; +} + +function parseReadyRecord(line: Buffer): ReadyRecord { + if (line.byteLength > MAX_LINE_BYTES) + throw new Error("fixture readiness record exceeded 64 KiB"); + let parsed: unknown; + try { + parsed = JSON.parse(line.toString("utf8")); + } catch { + throw new Error("fixture readiness record was malformed"); + } + const object = record(parsed); + if (!object || !exactKeys(object, ["catalog", "status", "wire_version"])) { + throw new Error("fixture readiness record had unknown fields"); + } + if ( + object.status !== "ready" || + object.wire_version !== 2 || + !Array.isArray(object.catalog) + ) { + throw new Error("fixture readiness record was invalid"); + } + if ( + object.catalog.length !== EXPECTED_CATALOG.length || + object.catalog.some((entry, index) => entry !== EXPECTED_CATALOG[index]) + ) { + throw new Error("fixture readiness catalog was invalid"); + } + return { + status: "ready", + wire_version: 2, + catalog: ["magic-context", "synapse", "broca"], + }; +} + +function verifyPublication(path: string, expectedMode: number): void { + let publication: ReturnType; + try { + publication = lstatSync(path); + } catch { + throw new Error("direct mc-host readiness preceded secure publication"); + } + const uid = process.getuid?.(); + if (uid === undefined || publication.uid !== uid || (publication.mode & 0o777) !== expectedMode) { + throw new Error("direct mc-host fixture published unsafe owner or permissions"); + } +} + +function appendBounded(current: string, chunk: Buffer): string { + return `${current}${chunk.toString()}`.slice(-MAX_LOG_BYTES); +} + +function safeChildExit( + child: ChildProcess, + timeoutMs: number, +): Promise { + if (child.exitCode !== null || child.signalCode !== null) + return Promise.resolve(true); + return new Promise((resolveExit) => { + const timer = setTimeout(() => { + child.off("exit", onExit); + resolveExit(false); + }, timeoutMs); + const onExit = (): void => { + clearTimeout(timer); + resolveExit(true); + }; + child.once("exit", onExit); + }); +} + +class FixtureControlClient { + private socket: Socket | null = null; + private incoming = Buffer.alloc(0); + private nextId = 1; + private readonly pending = new Map(); + private readonly responseIds = new Set(); + private closed = false; + + constructor( + private readonly path: string, + private readonly timeoutMs = 5_000, + ) {} + + async connect(): Promise { + if (this.socket) return; + if (this.closed) throw new Error("fixture control client is closed"); + const socket = createConnection({ path: this.path }); + this.socket = socket; + socket.on("data", (chunk: Buffer) => this.onData(chunk)); + socket.on("error", () => + this.fail(new Error("fixture control connection failed")), + ); + socket.on("close", () => + this.fail(new Error("fixture control connection closed")), + ); + await new Promise((resolveConnect, rejectConnect) => { + const timer = setTimeout(() => { + socket.destroy(); + rejectConnect( + new Error("fixture control connection timed out"), + ); + }, this.timeoutMs); + socket.once("connect", () => { + clearTimeout(timer); + resolveConnect(); + }); + socket.once("error", () => { + clearTimeout(timer); + rejectConnect(new Error("fixture control connection failed")); + }); + }); + } + + backendSuccess(): Promise { + return this.ack("backend-success"); + } + + blockNextCall(): Promise { + return this.ack("block-next-call"); + } + + async releaseBlockedCall(): Promise { + return this.parseAck(await this.request("release-blocked-call")); + } + + typedFailure(): Promise { + return this.ack("typed-failure"); + } + + async counters(): Promise { + return this.parseCounters(await this.request("counters")); + } + + gracefulShutdown(): Promise { + return this.ack("graceful-shutdown"); + } + + close(): void { + this.closed = true; + const socket = this.socket; + this.socket = null; + socket?.destroy(); + this.fail(new Error("fixture control client is closed")); + } + + private async ack(command: ControlCommand): Promise { + const result = await this.request(command); + if (!this.parseAck(result)) + throw new Error(`fixture control ${command} was not accepted`); + } + + private request(command: ControlCommand): Promise { + if (!this.socket || this.closed) + return Promise.reject(new Error("fixture control is unavailable")); + if (this.nextId > Number.MAX_SAFE_INTEGER) { + return Promise.reject( + new Error("fixture control id space exhausted"), + ); + } + const id = this.nextId++; + const line = Buffer.from( + JSON.stringify({ id, command: { name: command } }) + "\n", + ); + if (line.byteLength - 1 > MAX_LINE_BYTES) { + return Promise.reject( + new Error("fixture control request exceeded 64 KiB"), + ); + } + return new Promise((resolveRequest, rejectRequest) => { + const timer = setTimeout(() => { + this.pending.delete(id); + rejectRequest( + new Error(`fixture control ${command} timed out`), + ); + }, this.timeoutMs); + this.pending.set(id, { + command, + resolve: resolveRequest, + reject: rejectRequest, + timer, + }); + this.socket?.write(line, (error) => { + if (!error) return; + const pending = this.pending.get(id); + if (!pending) return; + clearTimeout(pending.timer); + this.pending.delete(id); + pending.reject(new Error("fixture control write failed")); + }); + }); + } + + private onData(chunk: Buffer): void { + this.incoming = Buffer.concat([this.incoming, chunk]); + for (;;) { + const newline = this.incoming.indexOf(0x0a); + if (newline < 0) { + if (this.incoming.byteLength > MAX_LINE_BYTES) { + this.fail( + new Error("fixture control response exceeded 64 KiB"), + ); + } + return; + } + if (newline > MAX_LINE_BYTES) { + this.fail( + new Error("fixture control response exceeded 64 KiB"), + ); + return; + } + const line = this.incoming.subarray(0, newline); + this.incoming = this.incoming.subarray(newline + 1); + this.onLine(line); + if (this.closed) return; + } + } + + private onLine(line: Buffer): void { + let parsed: unknown; + try { + parsed = JSON.parse(line.toString("utf8")); + } catch { + this.fail(new Error("fixture control response was malformed")); + return; + } + const object = record(parsed); + if ( + !object || + !exactKeys( + object, + object.ok === true + ? ["id", "ok", "result"] + : ["error", "id", "ok"], + ) + ) { + this.fail(new Error("fixture control response had unknown fields")); + return; + } + const id = object.id; + if (!Number.isSafeInteger(id) || (id as number) < 0) { + this.fail(new Error("fixture control response id was invalid")); + return; + } + const numericId = id as number; + if (this.responseIds.has(numericId)) { + this.fail(new Error("fixture control response id was duplicated")); + return; + } + const pending = this.pending.get(numericId); + if (!pending) { + this.fail( + new Error( + "fixture control response id did not match a request", + ), + ); + return; + } + this.responseIds.add(numericId); + this.pending.delete(numericId); + clearTimeout(pending.timer); + if (object.ok !== true) { + const error = record(object.error); + const code = error?.code; + if ( + !error || + !exactKeys(error, ["code", "message"]) || + typeof code !== "string" + ) { + pending.reject( + new Error("fixture control failure response was malformed"), + ); + return; + } + pending.reject( + new Error( + `fixture control ${pending.command} rejected: ${code}`, + ), + ); + return; + } + pending.resolve(object.result); + } + + private parseAck(value: unknown): boolean { + const object = record(value); + if ( + !object || + !exactKeys(object, ["accepted"]) || + typeof object.accepted !== "boolean" + ) { + throw new Error("fixture control acknowledgement was malformed"); + } + return object.accepted; + } + + private parseCounters(value: unknown): BackendCounters { + const object = record(value); + const keys = [ + "blocked", + "cancelled", + "completed", + "failed", + "released", + "started", + ]; + if (!object || !exactKeys(object, keys)) { + throw new Error("fixture control counters were malformed"); + } + for (const key of keys) { + if ( + !Number.isSafeInteger(object[key]) || + (object[key] as number) < 0 + ) { + throw new Error("fixture control counters were malformed"); + } + } + return { + started: object.started as number, + completed: object.completed as number, + blocked: object.blocked as number, + released: object.released as number, + failed: object.failed as number, + cancelled: object.cancelled as number, + }; + } + + private fail(error: Error): void { + for (const pending of this.pending.values()) { + clearTimeout(pending.timer); + pending.reject(error); + } + this.pending.clear(); + if (!this.closed) { + this.closed = true; + this.socket?.destroy(); + this.socket = null; + } + } +} + +export interface HermeticMcHostOptions { + /** OpenCode XDG data root. Fixture and module store live beneath this owner-only directory. */ + dataDir: string; + fixtureBin: string; + startTimeoutMs?: number; +} + +/** Running U5 direct-host fixture. No provider or module subprocess exists. */ +export class HermeticMcHostStack { + readonly connectionFile: string; + readonly controlPath: string; + private readonly dataDir: string; + private readonly fixtureBin: string; + private readonly startTimeoutMs: number; + private readonly fixtureConfigDir: string; + private readonly logPath: string; + private readonly pidFilePath: string; + private child: ChildProcess | null = null; + private control: FixtureControlClient | null = null; + private statusClient: McHostClient | null = null; + private stdout = ""; + private stderr = ""; + private pidFileCreatedAtMs = 0; + + private constructor(options: Required) { + this.dataDir = options.dataDir; + this.fixtureBin = options.fixtureBin; + this.startTimeoutMs = options.startTimeoutMs; + this.connectionFile = join( + this.dataDir, + "cortexkit", + "run", + CONNECTION_FILE, + ); + this.controlPath = join(this.dataDir, CONTROL_FILE); + this.fixtureConfigDir = join(this.dataDir, "fixture-config"); + this.logPath = join(this.dataDir, "cortexkit", "direct-mc-host.log"); + this.pidFilePath = join(this.dataDir, "cortexkit", PID_FILE); + } + + static async start( + options: HermeticMcHostOptions, + ): Promise { + reapRecordedRustProcesses(); + const stack = new HermeticMcHostStack({ + ...options, + startTimeoutMs: options.startTimeoutMs ?? 60_000, + }); + try { + await stack.startHost(); + return stack; + } catch (error) { + await stack.stop(); + throw error; + } + } + + async backendSuccess(): Promise { + await this.requireControl().backendSuccess(); + } + + async blockNextBackendCall(): Promise { + await this.requireControl().blockNextCall(); + } + + async releaseBlockedBackendCall(): Promise { + return this.requireControl().releaseBlockedCall(); + } + + async failNextBackendCall(): Promise { + await this.requireControl().typedFailure(); + } + + async backendCounters(): Promise { + return this.requireControl().counters(); + } + + async backendRequestCount(): Promise { + return (await this.backendCounters()).started; + } + + async primaryStatus( + sessionId: string, + projectRoot: string, + method: "status" | "session.status" = "status", + ): Promise> { + const identity: BindIdentity = { + project_root: resolve(projectRoot), + harness: "opencode", + session: sessionId, + }; + const client = + this.statusClient ?? + (this.statusClient = await McHostClient.connect({ + connectionFile: this.connectionFile, + identity, + targetKind: "tool_provider", + })); + let route: Awaited> | null = null; + try { + route = await client.routeOpen( + { kind: "tool_provider", module_id: "magic-context" }, + identity, + ); + const response = await client.request(route, { + method, + v: 1, + session_id: sessionId, + }); + return record(response) ?? {}; + } catch (error) { + if (this.statusClient === client) this.statusClient = null; + await client.closeAsync().catch(() => undefined); + throw error; + } finally { + if (route) await client.closeRoute(route).catch(() => undefined); + } + } + + hostLog(): string { + let file = ""; + try { + file = readFileSync(this.logPath, "utf8").slice(-MAX_LOG_BYTES); + } catch { + // Log file is optional. + } + return `${this.stdout}${this.stderr}${file}`.slice(-MAX_LOG_BYTES); + } + + async crashHost(): Promise { + await this.closeClients(); + const child = this.child; + if (!child) return; + if (child.exitCode === null && child.signalCode === null) + child.kill("SIGKILL"); + if (!(await safeChildExit(child, 5_000))) { + throw new Error( + "direct mc-host fixture did not exit after SIGKILL", + ); + } + if (this.child === child) this.child = null; + this.persistPidFile(); + } + + async restartHost(): Promise { + await this.crashHost(); + await this.startHost(); + } + + async terminateHost(): Promise { + await this.closeClients(); + const child = this.child; + if (!child) return; + if (child.exitCode === null && child.signalCode === null) + child.kill("SIGTERM"); + if (!(await safeChildExit(child, 10_000))) { + throw new Error( + "direct mc-host fixture did not exit after SIGTERM", + ); + } + if (this.child === child) this.child = null; + this.persistPidFile(); + } + + pauseHost(): void { + const child = this.child; + if (child && child.exitCode === null && child.signalCode === null) + child.kill("SIGSTOP"); + } + + resumeHost(): void { + const child = this.child; + if (child && child.exitCode === null && child.signalCode === null) + child.kill("SIGCONT"); + } + + /** Graceful JSONL shutdown, then SIGTERM fallback. Always await exit and remove isolated state. */ + async stop(): Promise { + await this.closeStatusClient(); + const child = this.child; + let exited = + child === null || + child.exitCode !== null || + child.signalCode !== null; + if (child && !exited) { + try { + await this.control?.gracefulShutdown(); + } catch { + // Fixture may already be unavailable. + } + exited = await safeChildExit(child, 5_000); + if (!exited) { + child.kill("SIGTERM"); + exited = await safeChildExit(child, 5_000); + } + if (!exited) { + child.kill("SIGKILL"); + exited = await safeChildExit(child, 5_000); + } + } + this.control?.close(); + this.control = null; + this.child = null; + rmSync(this.pidFilePath, { force: true }); + rmSync(this.dataDir, { recursive: true, force: true }); + if (!exited) + throw new Error( + "direct mc-host fixture did not exit during teardown", + ); + } + + private async startHost(): Promise { + await this.closeClients(); + mkdirSync(join(this.dataDir, "cortexkit"), { recursive: true }); + chmodSync(this.dataDir, 0o700); + const fixtureConfigRoot = join(this.fixtureConfigDir, "cortexkit"); + const fixtureConfigPath = join(fixtureConfigRoot, "magic-context.jsonc"); + mkdirSync(fixtureConfigRoot, { recursive: true }); + chmodSync(this.fixtureConfigDir, 0o700); + chmodSync(fixtureConfigRoot, 0o700); + writeFileSync( + fixtureConfigPath, + JSON.stringify({ historian: { module_model: "fixture/deterministic" } }), + { mode: 0o600 }, + ); + chmodSync(fixtureConfigPath, 0o600); + rmSync(this.logPath, { force: true }); + this.stdout = ""; + this.stderr = ""; + this.pidFileCreatedAtMs = Date.now(); + this.persistPidFile(); + + const child = spawn(this.fixtureBin, ["--state-root", this.dataDir], { + cwd: REPO_ROOT, + env: { + ...process.env, + NO_COLOR: "1", + XDG_CONFIG_HOME: this.fixtureConfigDir, + }, + stdio: ["ignore", "pipe", "pipe"], + }); + this.child = child; + this.persistPidFile(); + let readyBuffer = Buffer.alloc(0); + let readyResolve: ((record: ReadyRecord) => void) | null = null; + let readyReject: ((error: Error) => void) | null = null; + const readyPromise = new Promise( + (resolveReady, rejectReady) => { + readyResolve = resolveReady; + readyReject = rejectReady; + }, + ); + child.stdout?.on("data", (chunk: Buffer) => { + this.stdout = appendBounded(this.stdout, chunk); + if (!readyResolve) return; + readyBuffer = Buffer.concat([readyBuffer, chunk]); + const newline = readyBuffer.indexOf(0x0a); + if (newline < 0) { + if (readyBuffer.byteLength > MAX_LINE_BYTES) { + readyReject?.( + new Error("fixture readiness record exceeded 64 KiB"), + ); + readyResolve = null; + } + return; + } + try { + const parsed = parseReadyRecord( + readyBuffer.subarray(0, newline), + ); + const resolve = readyResolve; + readyResolve = null; + resolve(parsed); + } catch (error) { + readyResolve = null; + readyReject?.( + error instanceof Error + ? error + : new Error("fixture readiness failed"), + ); + } + }); + child.stderr?.on("data", (chunk: Buffer) => { + this.stderr = appendBounded(this.stderr, chunk); + try { + appendFileSync(this.logPath, chunk); + } catch { + // Diagnostics never own fixture lifecycle. + } + }); + child.once("error", () => + readyReject?.(new Error("direct mc-host fixture failed to start")), + ); + child.once("exit", () => { + if (this.child === child) this.child = null; + this.persistPidFile(); + readyReject?.( + new Error("direct mc-host fixture exited before readiness"), + ); + }); + + let timeoutHandle: ReturnType | null = null; + const timeout = new Promise((_, rejectTimeout) => { + timeoutHandle = setTimeout( + () => rejectTimeout(new Error("direct mc-host fixture readiness timed out")), + this.startTimeoutMs, + ); + }); + try { + await Promise.race([readyPromise, timeout]); + } finally { + if (timeoutHandle) clearTimeout(timeoutHandle); + } + + verifyPublication(this.dataDir, 0o700); + verifyPublication(this.controlPath, 0o600); + verifyPublication(this.connectionFile, 0o600); + + const control = new FixtureControlClient(this.controlPath); + await control.connect(); + this.control = control; + + const probe = await McHostClient.connect({ + connectionFile: this.connectionFile, + }); + try { + const catalog = await probe.catalogList(); + const ids = catalog.map((entry) => entry.module_id); + if ( + ids.length !== EXPECTED_CATALOG.length || + ids.some((entry, index) => entry !== EXPECTED_CATALOG[index]) + ) { + throw new Error("direct mc-host catalog probe failed"); + } + } finally { + await probe.closeAsync(); + } + } + + private requireControl(): FixtureControlClient { + if (!this.control) + throw new Error("direct mc-host fixture control is unavailable"); + return this.control; + } + + private async closeStatusClient(): Promise { + const client = this.statusClient; + this.statusClient = null; + if (client) await client.closeAsync().catch(() => undefined); + } + + private async closeClients(): Promise { + await this.closeStatusClient(); + this.control?.close(); + this.control = null; + } + + private persistPidFile(): void { + if (!this.pidFileCreatedAtMs) return; + const pid = this.child?.pid; + try { + mkdirSync(join(this.dataDir, "cortexkit"), { recursive: true }); + writeFileSync( + this.pidFilePath, + JSON.stringify({ + createdAtMs: this.pidFileCreatedAtMs, + pids: + typeof pid === "number" && pid > 0 + ? [ + { + pid, + role: "mc-host" as const, + executable: realpathSync(this.fixtureBin), + }, + ] + : [], + } satisfies RustE2ePidFile), + ); + } catch { + // PID file is cleanup safety net only. + } + } +} + +export const __hermeticMcHostTest = { + FixtureControlClient, + diagnostics(stack: HermeticMcHostStack): { + stdout: string; + stderr: string; + retainedLog: string; + } { + // SAFETY: This module guarantees that stack has stdout, stderr, and logPath diagnostic fields. + const internal = stack as unknown as { + stdout: string; + stderr: string; + logPath: string; + }; + let retainedLog = ""; + try { + retainedLog = readFileSync(internal.logPath, "utf8").slice(-MAX_LOG_BYTES); + } catch { + // Retained fixture log is optional. + } + return { stdout: internal.stdout, stderr: internal.stderr, retainedLog }; + }, + isStaleRustE2ePidRecord, + maxLineBytes: MAX_LINE_BYTES, + parseReadyRecord, + stalePidAgeMs: STALE_PID_AGE_MS, +}; diff --git a/packages/e2e-tests/src/rust-runner/hermetic-subc.test.ts b/packages/e2e-tests/src/rust-runner/hermetic-subc.test.ts deleted file mode 100644 index bb21b386a..000000000 --- a/packages/e2e-tests/src/rust-runner/hermetic-subc.test.ts +++ /dev/null @@ -1,25 +0,0 @@ -/// - -import { describe, expect, it } from "bun:test"; -import { __hermeticSubcTest } from "./hermetic-subc"; - -describe("hermetic Rust process isolation", () => { - it("reaps only stale PID records", () => { - const nowMs = 10 * __hermeticSubcTest.stalePidAgeMs; - - expect( - __hermeticSubcTest.isStaleRustE2ePidRecord( - nowMs - __hermeticSubcTest.stalePidAgeMs + 1, - nowMs, - ), - ).toBe(false); - expect( - __hermeticSubcTest.isStaleRustE2ePidRecord( - nowMs - __hermeticSubcTest.stalePidAgeMs, - nowMs, - ), - ).toBe(true); - expect(__hermeticSubcTest.isStaleRustE2ePidRecord(nowMs + 1, nowMs)).toBe(false); - expect(__hermeticSubcTest.isStaleRustE2ePidRecord(Number.NaN, nowMs)).toBe(false); - }); -}); diff --git a/packages/e2e-tests/src/rust-runner/hermetic-subc.ts b/packages/e2e-tests/src/rust-runner/hermetic-subc.ts deleted file mode 100644 index cdfb32908..000000000 --- a/packages/e2e-tests/src/rust-runner/hermetic-subc.ts +++ /dev/null @@ -1,865 +0,0 @@ -/** - * Hermetic subc stack for the Rust-mode e2e lane. - * - * Mirrors crates/mc-module/tests/real_daemon.rs, but driven from TypeScript so - * the full production path (opencode → plugin → subc daemon → ck-mc module) can - * be exercised end to end. It spawns: - * - * - a real `ck-subc` daemon (from the sibling `subconscious` workspace, the - * same binary real_daemon.rs uses via `cargo build -p subc-core --bins`), and - * - the `ck-mc` module (this workspace, `cargo build --release -p mc-module`) - * connected to that daemon as an external tool provider. - * - * Wiring that makes the plugin find this daemon WITHOUT any product change: the - * plugin's Rust module client (SubcModuleTransport, constructed in - * packages/plugin/src/index.ts) reads the DEFAULT connection file at - * `${XDG_DATA_HOME}/cortexkit/run/subc-connection.json`. opencode runs with - * `XDG_DATA_HOME = `, so pointing the daemon's `XDG_RUNTIME_DIR` at - * `/cortexkit/run` lands its connection file at exactly that path. The - * module opens its own store at `${XDG_DATA_HOME}/cortexkit/magic-context/store.db` - * (distinct from the plugin's context.db in the same directory), so sharing the - * data dir is the production reality, not a test shortcut. - * - * Environment honesty: `detectRustModePrereqs()` returns a printable skip reason - * when cargo is missing, the sibling subconscious workspace is absent, or the - * platform is unsupported — the lane SKIPs rather than green-washing or hanging. - */ - -import { type ChildProcess, spawn, spawnSync } from "node:child_process"; -import { - appendFileSync, - copyFileSync, - existsSync, - linkSync, - mkdirSync, - readdirSync, - readFileSync, - rmSync, - writeFileSync, -} from "node:fs"; -import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; -import { SubcClient, type BindIdentity } from "@magic-context/core/shared/mc-host-client"; - -const REPO_ROOT = resolve(import.meta.dir, "../../../.."); -const MODULE_ID = "magic-context"; -const RUST_E2E_PID_FILE = "rust-e2e-pids.json"; -const BROCA_ID = "broca"; -const BROCA_SCRIPT = join(REPO_ROOT, "packages/e2e-tests/src/rust-runner/fake-broca.ts"); -const RUST_E2E_STALE_PID_AGE_MS = 30 * 60 * 1_000; - -type RustE2eProcessRole = "daemon" | "module" | "producer"; - -interface RustE2ePidRecord { - pid: number; - role: RustE2eProcessRole; -} - -interface RustE2ePidFile { - createdAtMs: number; - pids: RustE2ePidRecord[]; -} - -function processStartTimeMs(pid: number): number | null { - const result = spawnSync("ps", ["-o", "lstart=", "-p", String(pid)], { - encoding: "utf8", - stdio: ["ignore", "pipe", "ignore"], - }); - if (result.status !== 0 || typeof result.stdout !== "string") return null; - const startedAt = Date.parse(result.stdout.trim()); - return Number.isFinite(startedAt) ? startedAt : null; -} - -function isStaleRustE2ePidRecord(createdAtMs: number, nowMs = Date.now()): boolean { - return ( - Number.isFinite(createdAtMs) && - createdAtMs <= nowMs && - nowMs - createdAtMs >= RUST_E2E_STALE_PID_AGE_MS - ); -} - -/** - * Reap only PIDs recorded by a stale Rust harness run. Fresh PID files can - * belong to active tests in other worktrees on the shared host. - */ -function reapRecordedRustProcesses(): void { - let candidates: string[]; - try { - candidates = readdirSync(tmpdir(), { withFileTypes: true }) - .filter((entry) => entry.isDirectory() && entry.name.startsWith("opencode-e2e-")) - .map((entry) => join(tmpdir(), entry.name, "data", "cortexkit", RUST_E2E_PID_FILE)); - } catch { - return; - } - - for (const pidPath of candidates) { - if (!existsSync(pidPath)) continue; - let stale = false; - try { - const record = JSON.parse(readFileSync(pidPath, "utf8")) as RustE2ePidFile; - if (!Array.isArray(record.pids)) continue; - stale = isStaleRustE2ePidRecord(record.createdAtMs); - if (!stale) continue; - // `ps lstart` reports process start times only to whole seconds on - // supported Unix hosts, so compare against the PID record's creation - // time rounded down. Older processes predate this harness run and are - // not killed. - const createdAtBoundary = Math.floor(record.createdAtMs / 1_000) * 1_000; - for (const process of record.pids) { - if (!Number.isInteger(process?.pid) || process.pid <= 0) continue; - const startedAt = processStartTimeMs(process.pid); - if (startedAt === null || startedAt < createdAtBoundary) continue; - try { - processKill(process.pid); - } catch { - // The process may have exited between ps and kill. - } - } - } catch { - // A partial PID file is not an identity proof; leave unknown processes alone. - } finally { - if (stale) rmSync(pidPath, { force: true }); - } - } -} - -function processKill(pid: number): void { - process.kill(pid, "SIGKILL"); -} - -/** ck-mc lives in THIS workspace; ck-subc in the sibling subconscious workspace. */ -const CK_MC_RELEASE = join(REPO_ROOT, "target/release/ck-mc"); - -/** - * Candidate locations for the sibling subconscious workspace. In a normal - * checkout it sits beside the repo root; in an Alfonso worktree it is a sibling - * symlink one level up from the worktree. Both are covered by walking up. - */ -function subconsciousCandidates(): string[] { - return [ - join(REPO_ROOT, "..", "subconscious"), - join(REPO_ROOT, "..", "..", "subconscious"), - ]; -} - -export interface RustModePrereqs { - ok: boolean; - /** Human-readable reason to print when skipping the lane. Set when !ok. */ - skipReason?: string; - /** Resolved sibling subconscious workspace root (when ok). */ - subconsciousRoot?: string; -} - -/** - * Detect whether the hermetic Rust stack can run here. Never throws; returns a - * printable reason so the suite can SKIP cleanly on an unsupported machine. - */ -export function detectRustModePrereqs(): RustModePrereqs { - if (process.platform === "win32") { - return { - ok: false, - skipReason: `platform ${process.platform} is unsupported for the hermetic subc stack (needs a Unix socket/TCP daemon build)`, - }; - } - - const cargo = spawnSync("cargo", ["--version"], { stdio: "ignore" }); - if (cargo.error || cargo.status !== 0) { - return { - ok: false, - skipReason: "cargo is not available on PATH; cannot build ck-mc / ck-subc", - }; - } - - const subconsciousRoot = subconsciousCandidates().find((candidate) => - existsSync(join(candidate, "Cargo.toml")), - ); - if (!subconsciousRoot) { - return { - ok: false, - skipReason: `sibling subconscious workspace not found (looked in: ${subconsciousCandidates().join(", ")}); cannot build the ck-subc daemon`, - }; - } - - return { ok: true, subconsciousRoot }; -} - -// ── build (memoized once per process, like real_daemon's BUILD_LOCK) ────────── - -interface BuiltBinaries { - ckMcBin: string; - ckSubcBin: string; -} - -let buildPromise: Promise | null = null; - -function runCargo( - args: string[], - cwd: string, -): Promise<{ ok: boolean; stdout: string; stderr: string }> { - return new Promise((resolveRun) => { - const child = spawn("cargo", args, { cwd, stdio: ["ignore", "pipe", "pipe"] }); - let stdout = ""; - let stderr = ""; - child.stdout?.on("data", (chunk: Buffer) => { - stdout += chunk.toString(); - }); - child.stderr?.on("data", (chunk: Buffer) => { - stderr += chunk.toString(); - }); - child.on("error", (err) => { - resolveRun({ ok: false, stdout, stderr: `${stderr}\nspawn error: ${String(err)}` }); - }); - child.on("exit", (code) => { - resolveRun({ ok: code === 0, stdout, stderr }); - }); - }); -} - -/** - * Build the module (always, incrementally) and the daemon (only when absent). - * - * The module under test is rebuilt every run so the lane always exercises the - * current workspace source — cargo's incremental check makes that near-free once - * warm. The daemon is an external dependency: a prebuilt `ck-subc` is reused when - * present to avoid a redundant cross-workspace compile (and the machine - * saturation that concurrent workspace builds cause on shared hardware); it is - * built only if no binary exists yet. Set MC_RUST_E2E_REBUILD_DAEMON=1 to force - * a daemon rebuild. - */ -export async function buildHermeticBinaries(subconsciousRoot: string): Promise { - if (buildPromise) return buildPromise; - buildPromise = (async () => { - const configuredCkMc = process.env.MC_E2E_CK_MC_BIN; - let ckMcBin = configuredCkMc && existsSync(configuredCkMc) ? configuredCkMc : undefined; - if (!ckMcBin) { - const moduleBuild = await runCargo( - ["build", "--release", "-p", "mc-module"], - REPO_ROOT, - ); - if (!moduleBuild.ok || !existsSync(CK_MC_RELEASE)) { - throw new Error( - `failed to build ck-mc (cargo build --release -p mc-module):\n${moduleBuild.stderr.slice(-4000)}`, - ); - } - ckMcBin = CK_MC_RELEASE; - } - - if (!ckMcBin || !existsSync(ckMcBin)) { - throw new Error("ck-mc binary was not resolved after prerequisite detection"); - } - - // Run the module under a dev-distinct process name so a test binary is - // never mistaken for the production ck-mc in Activity Monitor / ps. - // A hardlink shares the inode (no copy cost, always current build); - // fall back to a copy across filesystems. - const devNamed = join(dirname(ckMcBin), "ckdev-mc-e2e"); - try { - rmSync(devNamed, { force: true }); - linkSync(ckMcBin, devNamed); - ckMcBin = devNamed; - } catch { - try { - copyFileSync(ckMcBin, devNamed); - ckMcBin = devNamed; - } catch { - // Keep the original path; naming is cosmetic, never a test failure. - } - } - - const ckSubcRelease = join(subconsciousRoot, "target/release/ck-subc"); - const forceRebuild = process.env.MC_RUST_E2E_REBUILD_DAEMON === "1"; - if (forceRebuild || !existsSync(ckSubcRelease)) { - const daemonBuild = await runCargo( - ["build", "--release", "-p", "subc-core", "--bins"], - subconsciousRoot, - ); - if (!daemonBuild.ok || !existsSync(ckSubcRelease)) { - throw new Error( - `failed to build ck-subc (cargo build --release -p subc-core --bins in ${subconsciousRoot}):\n${daemonBuild.stderr.slice(-4000)}`, - ); - } - } - - return { ckMcBin, ckSubcBin: ckSubcRelease }; - })(); - return buildPromise; -} - -// ── daemon + module lifecycle ───────────────────────────────────────────────── - -function sleep(ms: number): Promise { - return new Promise((r) => setTimeout(r, ms)); -} - -function isProcessAlive(pid: number): boolean { - try { - process.kill(pid, 0); - return true; - } catch { - return false; - } -} - -async function pollUntil( - predicate: () => boolean, - opts: { timeoutMs: number; intervalMs?: number; label: string }, -): Promise { - const intervalMs = opts.intervalMs ?? 100; - const deadline = Date.now() + opts.timeoutMs; - while (Date.now() < deadline) { - if (predicate()) return; - await sleep(intervalMs); - } - throw new Error(`hermetic subc: ${opts.label} did not happen within ${opts.timeoutMs}ms`); -} - -export interface HermeticSubcOptions { - /** opencode's data dir — the module store and the plugin's connection-file lookup share it. */ - dataDir: string; - ckMcBin: string; - ckSubcBin: string; - /** Ceiling for daemon connection-file + module registration. Default 60s. */ - startTimeoutMs?: number; - /** Start the deterministic Broca producer. Default true. */ - startProducer?: boolean; -} - -/** - * A running hermetic daemon + module pair. `connectionFile` is the path the - * plugin's Rust client will read. Always call `stop()` in afterAll (even on - * failure) so no orphaned daemon/module processes leak between suites. - */ -export class HermeticSubcStack { - readonly connectionFile: string; - private readonly dataDir: string; - private readonly ckMcBin: string; - private readonly ckSubcBin: string; - private readonly runtimeDir: string; - private readonly daemonConfigDir: string; - private readonly daemonLogPath: string; - private readonly moduleLogPath: string; - private readonly producerLogPath: string; - private readonly pidFilePath: string; - private readonly startTimeoutMs: number; - private readonly startProducer: boolean; - private pidFileCreatedAtMs = 0; - private readonly recordedPids = new Map(); - private daemon: ChildProcess | null = null; - private module: ChildProcess | null = null; - private producer: ChildProcess | null = null; - private killedModulePid: number | null = null; - private moduleRouteDropBaseline = 0; - private killedProducerPid: number | null = null; - private producerPid: number | null = null; - private statusClient: SubcClient | null = null; - - private constructor(opts: Required) { - this.dataDir = opts.dataDir; - this.ckMcBin = opts.ckMcBin; - this.ckSubcBin = opts.ckSubcBin; - this.startTimeoutMs = opts.startTimeoutMs; - this.startProducer = opts.startProducer; - // The plugin's Rust client reads exactly this path (getDefaultConnectionFile - // in module-transport.ts). Pointing the daemon's XDG_RUNTIME_DIR here makes it - // write the connection file where the plugin already looks — no config knob. - this.runtimeDir = join(this.dataDir, "cortexkit", "run"); - this.connectionFile = join(this.runtimeDir, "subc-connection.json"); - this.daemonConfigDir = join(this.dataDir, "cortexkit", "_hermetic-daemon-config"); - this.daemonLogPath = join(this.dataDir, "cortexkit", "_hermetic-daemon.log"); - this.moduleLogPath = join(this.dataDir, "cortexkit", "_hermetic-module.log"); - this.producerLogPath = join(this.dataDir, "cortexkit", "_hermetic-broca.log"); - this.pidFilePath = join(this.dataDir, "cortexkit", RUST_E2E_PID_FILE); - } - - static async start(opts: HermeticSubcOptions): Promise { - reapRecordedRustProcesses(); - const stack = new HermeticSubcStack({ - dataDir: opts.dataDir, - ckMcBin: opts.ckMcBin, - ckSubcBin: opts.ckSubcBin, - startTimeoutMs: opts.startTimeoutMs ?? 60_000, - startProducer: opts.startProducer ?? true, - }); - try { - await stack.boot(); - return stack; - } catch (error) { - await stack.stop(); - throw error; - } - } - - private async boot(): Promise { - mkdirSync(this.runtimeDir, { recursive: true }); - // An interrupted run can leave a stale socket and logs behind. Remove - // those artifacts before starting the new daemon so registration proves - // this stack, not a dead predecessor, accepted the module. - rmSync(this.connectionFile, { force: true }); - rmSync(this.daemonLogPath, { force: true }); - rmSync(this.moduleLogPath, { force: true }); - rmSync(this.producerLogPath, { force: true }); - this.pidFileCreatedAtMs = Date.now(); - this.persistPidFile(); - mkdirSync(join(this.daemonConfigDir, "cortexkit"), { recursive: true }); - // configured_modules=0 → the daemon does NOT supervise/launch the module; - // the module connects as an ordinary external provider. That keeps module - // kill/restart (the park-self-heal fault) fully under this harness's control. - writeFileSync( - join(this.daemonConfigDir, "cortexkit", "subc.jsonc"), - JSON.stringify({ version: 1, modules: {} }, null, 2), - ); - - this.daemon = spawn(this.ckSubcBin, [], { - stdio: ["ignore", "pipe", "pipe"], - env: { - ...process.env, - XDG_RUNTIME_DIR: this.runtimeDir, - XDG_CONFIG_HOME: this.daemonConfigDir, - SUBC_PORT: "0", - // The daemon's tracing layer colorizes stdout by default, which - // interleaves ANSI escapes THROUGH "module registered module_id=…" - // and defeats a substring poll. NO_COLOR makes tracing emit plain - // text (the registration check also strips ANSI as a backstop). - NO_COLOR: "1", - // The module connects as a plain client; clear any inherited - // supervised-identity vars so it does not reuse a reserved slot. - SUBC_MODULE_ID: "", - SUBC_LAUNCH_NONCE: "", - }, - }); - this.recordPid("daemon", this.daemon.pid); - this.pipeToLog(this.daemon, this.daemonLogPath, "daemon"); - this.daemon.on("exit", () => { - this.daemon = null; - this.forgetPid("daemon"); - }); - - await pollUntil(() => existsSync(this.connectionFile), { - timeoutMs: this.startTimeoutMs, - label: "daemon connection file", - }); - // The daemon writes the connection file just before its listener enters - // the accept loop. Let that listener become reachable before the client - // attempts its one-shot registration handshake. - await sleep(100); - - // Register ck-mc first so its long-lived transform route is established - // before the independent Broca producer joins the daemon. The producer is - // still ready before the harness returns, so no historian request can race - // boot and the module's initial route is not starved by daemon startup. - await this.spawnModule(); - await this.waitForModuleRegistration(); - if (this.startProducer) { - await this.spawnProducer(); - await this.waitForProducerRegistration(); - process.env.MC_RUST_E2E_FOLD = "1"; - } else { - delete process.env.MC_RUST_E2E_FOLD; - } - } - - private async spawnProducer(): Promise { - this.producer = spawn(process.execPath, [BROCA_SCRIPT], { - stdio: ["ignore", "pipe", "pipe"], - env: { - ...process.env, - BROCA_CONNECTION_FILE: this.connectionFile, - BROCA_LOG_PATH: this.producerLogPath, - SUBC_MODULE_ID: "", - SUBC_LAUNCH_NONCE: "", - NO_COLOR: "1", - }, - }); - this.producerPid = this.producer.pid ?? null; - this.recordPid("producer", this.producer.pid); - this.pipeToLog(this.producer, this.producerLogPath, "producer"); - this.producer.on("exit", (code, signal) => { - try { - appendFileSync( - this.producerLogPath, - `producer process exited code=${code ?? "null"} signal=${signal ?? "null"}\n`, - ); - } catch { - // A lifecycle diagnostic must not turn teardown into a failure. - } - this.producer = null; - this.forgetPid("producer"); - }); - } - - private async spawnModule(): Promise { - const module = spawn(this.ckMcBin, ["--subc", this.connectionFile], { - stdio: ["ignore", "pipe", "pipe"], - env: { - ...process.env, - NO_COLOR: "1", - SUBC_MODULE_ID: MODULE_ID, - SUBC_LAUNCH_NONCE: "", - // The module opens its store under this data home — the SAME dir - // opencode uses, matching production's shared cortexkit layout. - XDG_DATA_HOME: this.dataDir, - }, - }); - this.module = module; - this.recordPid("module", module.pid); - this.pipeToLog(module, this.moduleLogPath, "module"); - module.on("exit", (code, signal) => { - try { - appendFileSync( - this.moduleLogPath, - `module process exited code=${code ?? "null"} signal=${signal ?? "null"}\n`, - ); - } catch { - // A lifecycle diagnostic must not turn teardown into a failure. - } - // A late event from an old process must not clear a replacement module. - if (this.module === module) { - this.module = null; - this.forgetPid("module"); - } - }); - } - - private recordPid(role: RustE2eProcessRole, pid: number | undefined): void { - if (typeof pid !== "number" || !Number.isInteger(pid) || pid <= 0) return; - this.recordedPids.set(role, pid); - this.persistPidFile(); - } - - private forgetPid(role: RustE2eProcessRole): void { - this.recordedPids.delete(role); - this.persistPidFile(); - } - - private persistPidFile(): void { - if (!this.pidFileCreatedAtMs) return; - try { - writeFileSync( - this.pidFilePath, - JSON.stringify({ - createdAtMs: this.pidFileCreatedAtMs, - pids: [...this.recordedPids.entries()].map(([role, pid]) => ({ role, pid })), - } satisfies RustE2ePidFile), - ); - } catch { - // The reaper is a safety net; a write failure must not break the harness. - } - } - - /** - * Registration is asynchronous relative to the daemon boot. The daemon logs - * "module registered module_id=magic-context" once the control-plane accepts - * it; poll that line so the first transform never races an unregistered - * module (which the daemon rejects terminally as unknown_module). - */ - private async waitForProducerRegistration(): Promise { - try { - await pollUntil(() => this.registrationCount(BROCA_ID) >= 1, { - timeoutMs: Math.min(this.startTimeoutMs, 10_000), - label: "Broca producer registration", - }); - } catch (error) { - throw new Error( - `${String(error)}\\ndaemon log:\\n${this.daemonLog().slice(-4000)}\\nproducer log:\\n${this.producerLog().slice(-4000)}`, - ); - } - } - - private async waitForModuleRegistration(): Promise { - let lastError: unknown; - for (let attempt = 0; attempt < 3; attempt += 1) { - if (attempt > 0) { - await sleep(500 * attempt); - await this.spawnModule(); - } - try { - await pollUntil(() => this.registrationCount(MODULE_ID) >= 1, { - timeoutMs: Math.min(this.startTimeoutMs, 10_000), - label: "module registration", - }); - return; - } catch (error) { - lastError = error; - // A fresh daemon can publish its connection file before the listener - // is ready. Recreate the external client for the next attempt rather - // than treating that startup race as a failed hermetic prerequisite. - this.killModule(); - } - } - throw new Error( - `${String(lastError)}\ndaemon log:\n${this.daemonLog().slice(-4000)}\nmodule log:\n${this.moduleLog().slice(-4000)}`, - ); - } - - /** - * Count "module registered module_id=magic-context" lines in the daemon log. - * ANSI escapes are stripped first: the daemon's tracing layer can colorize - * output (escapes interleave through the phrase), so a raw substring count is - * unreliable even though NO_COLOR should suppress it. Stripping makes the poll - * robust regardless of the daemon's color configuration. - */ - private registrationCount(moduleId: string): number { - if (!existsSync(this.daemonLogPath)) return 0; - const clean = stripAnsi(readFileSync(this.daemonLogPath, "utf8")); - const needle = `module registered module_id=${moduleId}`; - return clean.split(needle).length - 1; - } - - /** - * Kill the module and bring a fresh one up against the same daemon + store. - * Models the park-self-heal fault: a mid-session module restart whose next - * passes must recover without permanent degradation. The 200ms settle lets - * the OS release the single-writer store lease before the new module - * re-acquires it (mirrors real_daemon.rs's restart step). - */ - async restartModule(): Promise { - await this.killModuleAndWait(); - await sleep(200); - await this.spawnModule(); - await this.waitForFreshModuleRegistration(); - } - - /** Return a killed external module without restarting the OpenCode session. */ - async restoreModule(): Promise { - await sleep(200); - await this.spawnModule(); - await this.waitForFreshModuleRegistration(); - } - - /** Kill only the module process (leaving the daemon up), for fault injection. */ - killModule(): void { - this.moduleRouteDropBaseline = this.routeDropCount(); - const pid = this.module?.pid; - if (pid && Number.isInteger(pid) && pid > 0) this.killedModulePid = pid; - if (this.module && this.module.exitCode === null) { - this.module.kill("SIGKILL"); - } - this.module = null; - this.forgetPid("module"); - } - - /** Wait until SIGKILL has reaped the module and the daemon has dropped its route. */ - async waitForModuleDeath(timeoutMs = 15_000): Promise { - const pid = this.killedModulePid; - if (!pid) throw new Error("waitForModuleDeath called before killModule"); - await pollUntil( - () => !isProcessAlive(pid) && this.routeDropLogged(), - { timeoutMs, label: "module death and daemon route drop" }, - ); - } - - /** Kill the fake Broca process for the producer-outage drill. */ - killProducer(): void { - const pid = this.producer?.pid ?? this.producerPid; - if (pid && Number.isInteger(pid) && pid > 0) this.killedProducerPid = pid; - if (this.producer && this.producer.exitCode === null) this.producer.kill("SIGKILL"); - this.producer = null; - this.forgetPid("producer"); - } - - async waitForProducerDeath(timeoutMs = 15_000): Promise { - const pid = this.killedProducerPid ?? this.producerPid; - if (!pid) throw new Error("waitForProducerDeath called before killProducer"); - await pollUntil(() => !isProcessAlive(pid), { - timeoutMs, - label: "Broca producer death", - }); - } - - producerLog(): string { - try { - return readFileSync(this.producerLogPath, "utf8"); - } catch { - return ""; - } - } - - producerRequestCount(): number { - return (this.producerLog().match(/session\.send /g) ?? []).length; - } - - async moduleStatus( - sessionId: string, - projectRoot: string, - method: "status" | "session.status" = "status", - ): Promise> { - const identity: BindIdentity = { project_root: resolve(projectRoot), harness: "opencode", session: sessionId }; - const client = this.statusClient ?? (this.statusClient = await SubcClient.connect({ - connectionFile: this.connectionFile, - identity, - targetKind: "tool_provider", - })); - let route: Awaited> | null = null; - try { - route = await client.routeOpen( - { kind: "tool_provider", module_id: MODULE_ID }, - identity, - ); - const response = await client.request(route, { method, v: 1, session_id: sessionId }); - return (response && typeof response === "object" ? response : {}) as Record; - } catch (error) { - if (this.statusClient === client) this.statusClient = null; - await client.closeAsync().catch(() => undefined); - throw error; - } finally { - if (route) await client.closeRoute(route).catch(() => undefined); - } - } - - /** Wait until SIGKILL is observed before driving an outage or spawning a replacement. */ - async killModuleAndWait(): Promise { - const module = this.module; - this.killModule(); - if (!module || module.exitCode !== null || module.signalCode !== null) return; - await pollUntil(() => module.exitCode !== null || module.signalCode !== null, { - timeoutMs: 5_000, - label: "module process exit after SIGKILL", - }); - } - - /** Stop the live module without killing it, so daemon timeout handling can be tested. */ - stopModule(): void { - if (this.module && this.module.exitCode === null) this.module.kill("SIGSTOP"); - } - - /** Continue a module paused by stopModule(). */ - continueModule(): void { - if (this.module && this.module.exitCode === null) this.module.kill("SIGCONT"); - } - - /** - * Prove the hermetic daemon is using the external-provider path. A configured - * supervised module would restart after a long outage and invalidate the drill. - */ - assertModuleNotSupervised(): void { - const configPath = join(this.daemonConfigDir, "cortexkit", "subc.jsonc"); - let config: { modules?: unknown }; - try { - config = JSON.parse(readFileSync(configPath, "utf8")) as { modules?: unknown }; - } catch (error) { - throw new Error( - `Rust outage drill precondition failed: unreadable daemon config ${configPath}: ${error}`, - ); - } - if ( - config.modules === null || - typeof config.modules !== "object" || - Array.isArray(config.modules) || - Object.keys(config.modules as Record).length !== 0 - ) { - throw new Error("Rust outage drill precondition failed: magic-context is configured for supervision"); - } - const log = stripAnsi(this.daemonLog()); - if (log.includes(MODULE_ID) && /supervis/.test(log)) { - throw new Error("Rust outage drill precondition failed: daemon reported magic-context as supervised"); - } - } - - /** - * After a restart the daemon log already contains the FIRST registration - * line, so a plain presence check would return immediately. Wait until the - * registration-line COUNT grows past what was present before the restart. - */ - private registrationTarget = 1; - private routeDropCount(): number { - const clean = stripAnsi(this.daemonLog()); - return ( - clean.match( - /route[_ ](?:gone|dropped|closed)|(?:gone|dropped|closed).*route|supervised module exited abnormally.*exit_signal=Some\(9\)/gi, - ) ?? [] - ).length; - } - - private routeDropLogged(): boolean { - return this.routeDropCount() > this.moduleRouteDropBaseline; - } - - private async waitForFreshModuleRegistration(): Promise { - this.registrationTarget += 1; - const target = this.registrationTarget; - await pollUntil(() => this.registrationCount(MODULE_ID) >= target, { - timeoutMs: this.startTimeoutMs, - label: "module re-registration after restart", - }); - } - - private pipeToLog(child: ChildProcess, logPath: string, _tag: string): void { - // Drain BOTH streams continuously: an undrained pipe fills the OS buffer - // and the child blocks mid-boot on a write (the exact spurious-hang - // real_daemon.rs documents). The ck-subc daemon logs its control-plane - // events (including "module registered …") to STDOUT via tracing, while - // ck-mc logs to STDERR — so capturing only one stream would miss the - // registration line the boot poll waits on. Both are folded into one log - // file, preserved for post-mortem when a scenario fails. - const append = (chunk: Buffer) => { - try { - appendFileSync(logPath, chunk.toString()); - } catch { - // Logging must never throw and take down a test. - } - }; - child.stdout?.on("data", append); - child.stderr?.on("data", append); - } - - /** Best-effort read of the daemon log (diagnostics on failure). */ - daemonLog(): string { - try { - return readFileSync(this.daemonLogPath, "utf8"); - } catch { - return ""; - } - } - - /** Best-effort read of the module log (diagnostics on failure). */ - moduleLog(): string { - try { - return readFileSync(this.moduleLogPath, "utf8"); - } catch { - return ""; - } - } - - /** Hard teardown. Safe to call more than once; never throws. */ - async stop(): Promise { - const statusClient = this.statusClient; - this.statusClient = null; - if (statusClient) await statusClient.closeAsync().catch(() => undefined); - try { - await this.killModuleAndWait(); - } catch { - // SIGKILL was sent; a delayed exit notification must not make teardown fail. - } - if (this.producer && this.producer.exitCode === null) this.producer.kill("SIGKILL"); - this.producer = null; - this.forgetPid("producer"); - const daemon = this.daemon; - if (daemon && daemon.exitCode === null) daemon.kill("SIGKILL"); - if (daemon && daemon.exitCode === null && daemon.signalCode === null) { - try { - await pollUntil(() => daemon.exitCode !== null || daemon.signalCode !== null, { - timeoutMs: 5_000, - label: "daemon process exit after SIGKILL", - }); - } catch { - // SIGKILL was sent; a delayed exit notification must not make teardown fail. - } - } - if (this.daemon === daemon) this.daemon = null; - this.forgetPid("daemon"); - delete process.env.MC_RUST_E2E_FOLD; - rmSync(this.pidFilePath, { force: true }); - } -} - -export const __hermeticSubcTest = { - isStaleRustE2ePidRecord, - stalePidAgeMs: RUST_E2E_STALE_PID_AGE_MS, -}; - -/** Remove ANSI/VT100 escape sequences so plain-text substring checks are reliable. */ -function stripAnsi(input: string): string { - // Matches CSI sequences like \x1b[32m and \x1b[0m that tracing emits for color. - // biome-ignore lint/suspicious/noControlCharactersInRegex: ANSI escapes are control chars by definition. - return input.replace(/\x1b\[[0-9;]*m/g, ""); -} diff --git a/packages/e2e-tests/src/rust-scenario-support.ts b/packages/e2e-tests/src/rust-scenario-support.ts index f1eb6eda2..69ee46415 100644 --- a/packages/e2e-tests/src/rust-scenario-support.ts +++ b/packages/e2e-tests/src/rust-scenario-support.ts @@ -3,15 +3,9 @@ * * Gating layers: * - * 1. Prerequisite gating — `rustPrereqs` preflights the hermetic stack (cargo, - * the sibling subconscious workspace, a supported platform). A scenario - * `describe.skipIf(!rustPrereqs.ok)`s when the stack cannot run, printing the - * reason so CI logs never green-wash a skipped lane. + * 1. Prerequisite gating checks Cargo, workspace metadata, and Unix sockets. * - * 2. Fold-infrastructure readiness — the Rust module runs its own historian and - * the hermetic stack now starts a deterministic `broca` management-surface - * producer. `foldInfraEnabled()` flips when that producer is booted, so fold - * and drop-on-fold scenarios can assert their real outcomes. + * 2. Fold scenarios remain gated for broad Rust qualification. * * The tail-mutation-readopt and park-self-heal scenarios are NOT gated: the P0 * identity-drift / park-self-heal fix is merged into this branch's base, so they @@ -35,31 +29,20 @@ export { export const rustPrereqs = RustTestHarness.detectPrereqs(); -/** True once the hermetic deterministic Broca producer is registered. */ export function foldInfraEnabled(): boolean { return process.env.MC_RUST_E2E_FOLD === "1"; } -/** - * Legacy reason retained for suites that still report a fold exclusion. The - * normal Rust harness starts Broca and activates those assertions; only suites - * that deliberately do not boot the Rust stack should use this text. - */ export const FOLD_SKIP_REASON = - "requires the Rust harness's hermetic Broca producer; this suite does not boot that stack"; + "requires broad Rust fold qualification beyond the focused direct backend fixture"; /** Enable the duplicate-ID regression only when the stack can produce the selection refresh needed to reproduce duplicate IDs. */ export function duplicateIdInfraEnabled(): boolean { return process.env.MC_RUST_E2E_DUPLICATE_IDS === "1"; } -/** - * The hermetic stack has no broca runner, so it cannot complete the historian-backed - * selection bust that consumes a queued ctx_reduce drop. Keep the assertion body - * available for a provisioned runner instead of reporting a false green pass. - */ export const DUPLICATE_ID_SKIP_REASON = - "requires a broca-capable hermetic stack to reach the queued-drop selection bust (set MC_RUST_E2E_DUPLICATE_IDS=1 once that runner is provisioned)"; + "requires broad duplicate-ID qualification beyond the focused direct backend fixture"; /** * Print a one-line skip notice. Call from a gated scenario's single `it` so the diff --git a/packages/e2e-tests/src/test-db.ts b/packages/e2e-tests/src/test-db.ts index 3002ebe48..28af8655f 100644 --- a/packages/e2e-tests/src/test-db.ts +++ b/packages/e2e-tests/src/test-db.ts @@ -19,10 +19,11 @@ import { Database as PluginDatabase } from "../../plugin/src/shared/sqlite"; export function openTestDb( path: string, options?: { readonly?: boolean; readwrite?: boolean }, -): Database { +): Database & PluginDatabase { const db = new Database(path, options); db.exec("PRAGMA busy_timeout=5000"); - return db; + // SAFETY: E2E executes in Bun, so PluginDatabase selects bun:sqlite. commentlint: allow(JUDGE) + return db as Database & PluginDatabase; } /** diff --git a/packages/e2e-tests/tests/cache-invariants.test.ts b/packages/e2e-tests/tests/cache-invariants.test.ts index 72eba6736..ec693ca56 100644 --- a/packages/e2e-tests/tests/cache-invariants.test.ts +++ b/packages/e2e-tests/tests/cache-invariants.test.ts @@ -34,7 +34,7 @@ import { insertMemory, updateMemoryContent, updateMemoryVerification } from "../ import { computeNormalizedHash } from "../../plugin/src/features/magic-context/memory/normalize-hash"; import { resolveProjectIdentity } from "../../plugin/src/features/magic-context/memory/project-identity"; import type { Memory } from "../../plugin/src/features/magic-context/memory/types"; -import { Database } from "../../plugin/src/shared/sqlite"; +import type { Database } from "../../plugin/src/shared/sqlite"; import { extractM0, extractM1, @@ -366,11 +366,11 @@ function thrownMessage(fn: () => unknown): string { } async function waitForRustCompartment(sessionId: string): Promise { - const stack = h.rustStack; + const stack = h.mcHostStack; if (!stack) throw new Error("Rust compartment wait requires the hermetic module stack"); const deadline = Date.now() + 60_000; while (Date.now() < deadline) { - const status = await stack.moduleStatus(sessionId, h.opencode.env.workdir, "session.status"); + const status = await stack.primaryStatus(sessionId, h.opencode.env.workdir, "session.status"); if (Number(status.compartment_count ?? 0) > 0) return; await Bun.sleep(100); } diff --git a/packages/e2e-tests/tests/deferred-compaction-marker.test.ts b/packages/e2e-tests/tests/deferred-compaction-marker.test.ts index 79bbe0856..e1f88fc2d 100644 --- a/packages/e2e-tests/tests/deferred-compaction-marker.test.ts +++ b/packages/e2e-tests/tests/deferred-compaction-marker.test.ts @@ -203,13 +203,13 @@ describe("deferred compaction marker (plan v6)", () => { // a5b7d61d moved Rust publication and its pending delta into the // module transaction. `pending_m1_delta` is the authority-level // equivalent of the legacy context.db marker blob. - const stack = h.rustStack; + const stack = h.mcHostStack; if (!stack) throw new Error("Rust marker check requires the hermetic module stack"); const deadline = Date.now() + 60_000; let afterPublish: Record = {}; while (Date.now() < deadline) { - afterPublish = await stack.moduleStatus( + afterPublish = await stack.primaryStatus( sessionId, h.opencode.env.workdir, "session.status", @@ -226,7 +226,7 @@ describe("deferred compaction marker (plan v6)", () => { expect(afterPublish.pending_m1_delta).toBe(true); await h.sendPrompt(sessionId, "small defer turn — no mutation expected"); - const pendingAfter = await stack.moduleStatus( + const pendingAfter = await stack.primaryStatus( sessionId, h.opencode.env.workdir, "session.status", diff --git a/packages/e2e-tests/tests/long-running-session.test.ts b/packages/e2e-tests/tests/long-running-session.test.ts index 1ca139a54..f944ec531 100644 --- a/packages/e2e-tests/tests/long-running-session.test.ts +++ b/packages/e2e-tests/tests/long-running-session.test.ts @@ -5,7 +5,7 @@ import { realpathSync } from "node:fs"; import { join, resolve as pathResolve } from "node:path"; import { insertMemory, updateMemoryVerification } from "../../plugin/src/features/magic-context/memory"; import { resolveProjectIdentity } from "../../plugin/src/features/magic-context/memory/project-identity"; -import { Database } from "../../plugin/src/shared/sqlite"; +import type { Database } from "../../plugin/src/shared/sqlite"; import { computeSyntheticCallId } from "../../plugin/src/hooks/magic-context/todo-view"; import { TestHarness } from "../src/harness"; import { buildMockHistorianPayload } from "../src/mock-historian"; @@ -304,8 +304,8 @@ async function send(sessionId: string, prompt: string, text: string, usage: Mock const flags = [ p.synthetic ? "synth" : null, p.ignored ? "ignored" : null, - p.auto !== null ? `auto=${p.auto}` : null, - p.overflow !== null ? `overflow=${p.overflow}` : null, + p.auto === null ? null : `auto=${p.auto}`, + p.overflow === null ? null : `overflow=${p.overflow}`, p.callID ? `callID=${p.callID.slice(0, 12)}` : null, p.tool ? `tool=${p.tool}` : null, ].filter(Boolean).join(","); @@ -535,12 +535,12 @@ describe("long-running OpenCode Magic Context session", () => { // a5b7d61d publishes through the out-of-band module stack; context.db // is not the Rust compartment authority. Observe the committed count // directly rather than waiting on a legacy TypeScript mirror row. - const stack = h.rustStack; + const stack = h.mcHostStack; if (!stack) throw new Error("Rust historian check requires the hermetic module stack"); const deadline = Date.now() + 120_000; let compartmentCount = 0; while (Date.now() < deadline) { - const status = await stack.moduleStatus(sessionId, h.opencode.env.workdir, "session.status"); + const status = await stack.primaryStatus(sessionId, h.opencode.env.workdir, "session.status"); compartmentCount = Number(status.compartment_count ?? 0); if (compartmentCount > 0) break; await Bun.sleep(100); diff --git a/packages/e2e-tests/tests/overflow-recovery.test.ts b/packages/e2e-tests/tests/overflow-recovery.test.ts index 3c0dedb6e..cf743ac84 100644 --- a/packages/e2e-tests/tests/overflow-recovery.test.ts +++ b/packages/e2e-tests/tests/overflow-recovery.test.ts @@ -233,7 +233,7 @@ describe("context overflow recovery", () => { const mainCallsBeforeFollowup = mainCalls; if (process.env.MC_E2E_MODE === "rust") { - // Rust delegates the transform/recovery ladder to ck-mc rather + // Rust delegates the transform/recovery ladder to McHandler rather // than the OpenCode-only fail-closed request sequencing below. // Its successful historian fold is also unavailable without the // hermetic Broca runner, so retain the shared overflow-detection diff --git a/packages/e2e-tests/tests/rust-cold-start-drop-seed.test.ts b/packages/e2e-tests/tests/rust-cold-start-drop-seed.test.ts index e49d22dfa..e169c5682 100644 --- a/packages/e2e-tests/tests/rust-cold-start-drop-seed.test.ts +++ b/packages/e2e-tests/tests/rust-cold-start-drop-seed.test.ts @@ -53,10 +53,6 @@ describe.skipIf(!rustPrereqs.ok)("rust invariant: cold-start drop seed", () => { h = await RustTestHarness.create({ modelContextLimit: 30_000, startInTsMode: true, - // This drill must reach the first Rust transform with only the TS - // frozen reduction; a historian publication would legitimately replace - // that sentinel with m0 before the cold-start seed can be observed. - startHistorianProducer: false, magicContextConfig: { execute_threshold_percentage: 25, protected_tags: 1, diff --git a/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts b/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts index 51d32f2dd..0fdf06277 100644 --- a/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts +++ b/packages/e2e-tests/tests/rust-ctx-reduce-roundtrip.test.ts @@ -97,7 +97,7 @@ describe.skipIf(!rustPrereqs.ok)("rust invariant: ctx_reduce round-trip", () => }; }); await h.sendPrompt(sessionId, `ctx_reduce turn 4: reduce tag ${dropTag}`); - const queued = (await h.subc.moduleStatus( + const queued = (await h.mcHost.primaryStatus( sessionId, h.env.workdir, "session.status", @@ -123,14 +123,14 @@ describe.skipIf(!rustPrereqs.ok)("rust invariant: ctx_reduce round-trip", () => // replacing the transient drop sentinel with m0. The module ledger is // therefore the durable proof that the queued command completed. expect(dropEmitted).toBe(true); - const finalStatus = (await h.subc.moduleStatus( + const finalStatus = (await h.mcHost.primaryStatus( sessionId, h.env.workdir, "session.status", )) as ModuleStatus; expect(finalStatus.pending_drop_count ?? -1).toBe(0); expect(finalStatus.compartment_count ?? 0).toBeGreaterThan(0); - expect(h.subc.producerRequestCount()).toBeGreaterThan(0); + expect(await h.mcHost.backendRequestCount()).toBeGreaterThan(0); const finalWire = h.lastMainWireSerialized(); expect(finalWire).toContain(""); expect(finalWire).toContain("Rust reduce e2e chunk"); diff --git a/packages/e2e-tests/tests/rust-duplicate-tool-use-id.test.ts b/packages/e2e-tests/tests/rust-duplicate-tool-use-id.test.ts index 62b3fd420..e193964aa 100644 --- a/packages/e2e-tests/tests/rust-duplicate-tool-use-id.test.ts +++ b/packages/e2e-tests/tests/rust-duplicate-tool-use-id.test.ts @@ -130,7 +130,7 @@ describe.skipIf(!rustPrereqs.ok)("rust incident regression: duplicate tool-use i await h.sendPrompt(sessionId, `queue drop ${dropTag}: ${h.ballast(1_500)}`); expect(dropEmitted).toBe(true); - // Keep both opencode and ck-mc alive with their serialized-output + // Keep both opencode and McHandler alive with their serialized-output // caches warm, but make the next pass a deterministic cache-busting // selection pass by shortening this session's durable cache TTL. h.setSessionCacheTtl(sessionId, "1"); diff --git a/packages/e2e-tests/tests/rust-fm-oc-1.test.ts b/packages/e2e-tests/tests/rust-fm-oc-1.test.ts index db247da9d..f815d6dad 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-1.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-1.test.ts @@ -32,7 +32,6 @@ describe.skipIf(!active)("rust failure-mode drill FM-OC-1: LKG after SIGKILL", ( it( "continues on the LKG wire and logs the process fault", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); h.setSessionCacheTtl(sessionId, "0"); @@ -46,7 +45,7 @@ describe.skipIf(!active)("rust failure-mode drill FM-OC-1: LKG after SIGKILL", ( const beforeCount = h.readRustPasses().length; const priorWire = JSON.parse(h.lastMainWireSerialized()) as unknown[]; - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); await h.sendPrompt(sessionId, `FM-OC-1 after SIGKILL: ${h.ballast(400)}`); const passes = await h.waitForRustPasses(beforeCount + 1); diff --git a/packages/e2e-tests/tests/rust-fm-oc-2.test.ts b/packages/e2e-tests/tests/rust-fm-oc-2.test.ts index 5347e9df5..76bb9eae3 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-2.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-2.test.ts @@ -32,14 +32,13 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-2: park transiti it( "continues through the outage and emits a machine-readable park transition", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); const beforeCount = h.readRustPasses().length; const droppedBefore = lineageScopedTagCount(h, sessionId, "dropped"); const outagePasses = RUST_FAILURE_PARK_THRESHOLD * 2; - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); await sendOutagePasses(h, sessionId, 4, outagePasses, "FM-OC-2 outage"); await h.waitFor( () => diff --git a/packages/e2e-tests/tests/rust-fm-oc-3.test.ts b/packages/e2e-tests/tests/rust-fm-oc-3.test.ts index 327175df5..174b3a5db 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-3.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-3.test.ts @@ -32,7 +32,6 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-3: parked self-h it( "recovers within the exported retry budget without restarting the session", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); const healthyVersions = h @@ -41,7 +40,7 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-3: parked self-h .filter((version) => version > 0); const outagePasses = RUST_FAILURE_PARK_THRESHOLD * 2; - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); await sendOutagePasses(h, sessionId, 4, outagePasses, "FM-OC-3 outage"); await h.waitFor( () => @@ -51,7 +50,7 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-3: parked self-h { label: "FM-OC-3 park transition" }, ); - await h.subc.restoreModule(); + await h.mcHost.restartHost(); const recoveryStart = h.readRustPasses().length; await sendOutagePasses( h, diff --git a/packages/e2e-tests/tests/rust-fm-oc-4.test.ts b/packages/e2e-tests/tests/rust-fm-oc-4.test.ts index 5ac612f31..4fab6c9c9 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-4.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-4.test.ts @@ -30,7 +30,6 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-4: emergency ref it( "continues briefly through SIGKILL, then bails loudly at the provider-proven wall", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); @@ -49,7 +48,7 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-4: emergency ref }; }); - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); try { await h.sendPrompt(sessionId, `FM-OC-4 continue after SIGKILL: ${h.ballast(400)}`); } catch { diff --git a/packages/e2e-tests/tests/rust-fm-oc-5.test.ts b/packages/e2e-tests/tests/rust-fm-oc-5.test.ts index 00fc98347..55141e075 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-5.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-5.test.ts @@ -28,16 +28,15 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-5: transport han it( "continues through a transport timeout and recovers after SIGCONT", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); const beforeCount = h.readRustPasses().length; - h.subc.stopModule(); + h.mcHost.pauseHost(); await h.sendPrompt(sessionId, `FM-OC-5 stopped module: ${h.ballast(400)}`); assertMessagesHaveNoPlaceholders(h.lastMainMessages(), sessionId); - h.subc.continueModule(); + h.mcHost.resumeHost(); await h.sendPrompt(sessionId, `FM-OC-5 continued module: ${h.ballast(400)}`); const recovered = await h.waitForRustPasses(beforeCount + 2); expect(recovered.slice(beforeCount + 1).some((pass) => pass.servedFrom === "transform")).toBe( diff --git a/packages/e2e-tests/tests/rust-fm-oc-6.test.ts b/packages/e2e-tests/tests/rust-fm-oc-6.test.ts index b578a8ba4..c0c3bf7dd 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-6.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-6.test.ts @@ -30,7 +30,6 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-6: emergency arm it( "continues briefly through SIGKILL, then refuses before attempting an LKG replay", async () => { - h.subc.assertModuleNotSupervised(); const sessionId = await h.createSession(); await driveToSteadyState(h, sessionId, 2); @@ -49,7 +48,7 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-6: emergency arm }; }); - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); try { await h.sendPrompt(sessionId, `FM-OC-6 arm after SIGKILL: ${h.ballast(400)}`); } catch { diff --git a/packages/e2e-tests/tests/rust-historian-producer.test.ts b/packages/e2e-tests/tests/rust-historian-producer.test.ts index d11fb2915..cc6752e75 100644 --- a/packages/e2e-tests/tests/rust-historian-producer.test.ts +++ b/packages/e2e-tests/tests/rust-historian-producer.test.ts @@ -1,33 +1,15 @@ /// -/** - * Rust historian producer coverage. - * - * The Broca process is deliberately separate from the OpenCode model mock. A - * producer request is therefore a non-vacuous precondition for both success - * and failure assertions in this suite. - */ - import { afterAll, beforeAll, describe, expect, it } from "bun:test"; -import { readFileSync } from "node:fs"; import { RustTestHarness } from "../src/rust-harness"; import { rustPrereqs } from "../src/rust-scenario-support"; interface HistorianStatus { - state?: string; last_failure?: string | null; failure_backoff_at_ms?: number | null; - consecutive_publish_failures?: number; -} - -interface ModuleStatus { - historian?: HistorianStatus; - compartment_count?: number; } -const expectInvalidOutput = process.env.MC_RUST_E2E_BROCA_EXPECT_BAD === "1"; - -describe.skipIf(!rustPrereqs.ok)("rust historian: hermetic Broca producer", () => { +describe.skipIf(!rustPrereqs.ok)("rust historian: direct Broca backend", () => { let h: RustTestHarness; beforeAll(async () => { @@ -36,6 +18,7 @@ describe.skipIf(!rustPrereqs.ok)("rust historian: hermetic Broca producer", () = magicContextConfig: { execute_threshold_percentage: 25, protected_tags: 1, + historian: { model: "fixture/deterministic" }, compressor: { enabled: false }, }, }); @@ -46,40 +29,27 @@ describe.skipIf(!rustPrereqs.ok)("rust historian: hermetic Broca producer", () = }); async function status(sessionId: string): Promise { - const response = (await h.subc.moduleStatus(sessionId, h.env.workdir)) as ModuleStatus; - return response.historian ?? {}; - } - - async function sessionStatus(sessionId: string): Promise { - return (await h.subc.moduleStatus(sessionId, h.env.workdir, "session.status")) as ModuleStatus; - } - - async function waitForProducerRun(minimum: number): Promise { - const deadline = Date.now() + 120_000; - while (Date.now() < deadline) { - if (h.subc.producerRequestCount() >= minimum) return; - await Bun.sleep(100); - } - let pluginLog = ""; - try { - pluginLog = readFileSync(h.logPath, "utf8").slice(-8000); - } catch { - // The harness may remove its temporary data directory during teardown. - } - throw new Error(`Broca producer was never contacted; rust passes=${JSON.stringify(h.readRustPasses().map((pass) => pass.raw))}\nproducer log:\n${h.subc.producerLog()}\nmodule log:\n${h.subc.moduleLog().slice(-8000)}\nplugin log:\n${pluginLog}`); + const response = await h.mcHost.primaryStatus(sessionId, h.env.workdir); + const historian = response.historian; + return historian && typeof historian === "object" + ? (historian as HistorianStatus) + : {}; } async function driveHistorian(sessionId: string): Promise { for (let i = 1; i <= 10; i += 1) { h.mock.setDefault({ - text: `historian producer assistant ${i}`, + text: `historian backend assistant ${i}`, usage: { - input_tokens: 2_500 * i, + input_tokens: 3_000 * i, output_tokens: 20, cache_creation_input_tokens: 1_000, }, }); - await h.sendPrompt(sessionId, `producer turn ${i}: ${h.ballast(2_000)}`); + await h.sendPrompt( + sessionId, + `historian turn ${i}: ${h.ballast(2_000)}`, + ); } h.mock.setDefault({ text: "historian trigger", @@ -89,98 +59,66 @@ describe.skipIf(!rustPrereqs.ok)("rust historian: hermetic Broca producer", () = cache_creation_input_tokens: 2_000, }, }); - await h.sendPrompt(sessionId, `producer trigger: ${h.ballast(2_000)}`); + await h.sendPrompt(sessionId, `historian trigger: ${h.ballast(2_000)}`); h.mock.setDefault({ text: "historian follow-up", - usage: { input_tokens: 500, output_tokens: 20, cache_creation_input_tokens: 0 }, + usage: { + input_tokens: 500, + output_tokens: 20, + cache_creation_input_tokens: 0, + }, }); - await h.sendPrompt(sessionId, "producer follow-up starts the historian run"); + await h.sendPrompt( + sessionId, + "historian follow-up starts the Broca run", + ); } - it( - "publishes deterministic tiered output and records producer contact", - async () => { - const sessionId = await h.createSession(); - await driveHistorian(sessionId); - await waitForProducerRun(1); + async function waitForBackend( + predicate: ( + counters: Awaited>, + ) => boolean, + ): Promise>> { + const deadline = Date.now() + 120_000; + while (Date.now() < deadline) { + const counters = await h.mcHost.backendCounters(); + if (predicate(counters)) return counters; + await Bun.sleep(100); + } + throw new Error( + `Broca backend was not reached\n${h.mcHost.hostLog().slice(-8_000)}`, + ); + } - if (expectInvalidOutput) { - const failed: HistorianStatus = {}; - const deadline = Date.now() + 120_000; - while (Date.now() < deadline) { - Object.assign(failed, await status(sessionId)); - if (failed.last_failure) break; - await Bun.sleep(100); - } - expect(failed.last_failure).toMatch(/tier|validat|compartment/i); - expect(failed.failure_backoff_at_ms ?? 0).toBeGreaterThan(Date.now() - 120_000); - expect(failed.failure_backoff_at_ms ?? 0).toBeGreaterThan(0); - return; - } + it("reaches the real Broca route through the controlled backend", async () => { + await h.mcHost.backendSuccess(); + const sessionId = await h.createSession(); + await driveHistorian(sessionId); - const publishDeadline = Date.now() + 120_000; - let published: ModuleStatus = {}; - while (Date.now() < publishDeadline) { - published = await sessionStatus(sessionId); - if ((published.compartment_count ?? 0) >= 1) break; - await Bun.sleep(100); - } - expect(published.compartment_count ?? 0).toBeGreaterThanOrEqual(1); - const final = await status(sessionId); - expect(final.last_failure ?? null).toBeNull(); - expect(final.consecutive_publish_failures ?? 0).toBe(0); - }, - 300_000, - ); + const counters = await waitForBackend((value) => value.completed >= 1); + expect(counters.started).toBeGreaterThanOrEqual(1); + expect(counters.completed).toBeGreaterThanOrEqual(1); + }, 300_000); - it( - "takes the loud historian failure path when Broca goes down mid-run", - async () => { - const sessionId = await h.createSession(); - // Establish a real Rust session without firing the historian yet. The - // producer is then killed mid-session, before the first outage run. - for (let i = 1; i <= 3; i += 1) { - h.mock.setDefault({ - text: `outage warmup ${i}`, - usage: { input_tokens: 100, output_tokens: 10, cache_creation_input_tokens: 50 }, - }); - await h.sendPrompt(sessionId, `outage warmup ${i}: ${h.ballast(400)}`); - } - const beforeFailure = h.subc.producerRequestCount(); - h.subc.killProducer(); - await h.subc.waitForProducerDeath(); + it("records a typed backend failure without killing a provider process", async () => { + const before = await h.mcHost.backendCounters(); + await h.mcHost.failNextBackendCall(); + const sessionId = await h.createSession(); + await driveHistorian(sessionId); - h.mock.setDefault({ - text: "producer outage follow-up", - usage: { - input_tokens: 90_000, - output_tokens: 20, - cache_creation_input_tokens: 90_000, - }, - }); - await h.sendPrompt(sessionId, `producer outage trigger: ${h.ballast(3_000)}`); - await h.sendPrompt(sessionId, `producer outage follow-up: ${h.ballast(1_000)}`); + const counters = await waitForBackend( + (value) => value.failed > before.failed, + ); + expect(counters.failed).toBe(before.failed + 1); - const deadline = Date.now() + 120_000; - let failed: HistorianStatus = {}; - while (Date.now() < deadline) { - try { - failed = await status(sessionId); - } catch { - // A provider disconnect can briefly tear down the daemon's - // management connection while the module records its failure. - await Bun.sleep(500); - continue; - } - if (failed.last_failure) break; - await Bun.sleep(100); - } - expect(beforeFailure).toBeGreaterThan(0); - expect(h.subc.producerRequestCount()).toBe(beforeFailure); - expect(failed.last_failure).toMatch(/broca|producer|connect|route|unknown/i); - expect(failed.failure_backoff_at_ms ?? 0).toBeGreaterThan(Date.now() - 120_000); - expect(failed.failure_backoff_at_ms ?? 0).toBeGreaterThan(0); - }, - 300_000, - ); + const deadline = Date.now() + 120_000; + let failed: HistorianStatus = {}; + while (Date.now() < deadline) { + failed = await status(sessionId); + if (failed.last_failure) break; + await Bun.sleep(100); + } + expect(failed.last_failure).toMatch(/backend|broca|fixture|terminal/i); + expect(failed.failure_backoff_at_ms ?? 0).toBeGreaterThan(0); + }, 300_000); }); diff --git a/packages/e2e-tests/tests/rust-multi-frame-delta-perf.test.ts b/packages/e2e-tests/tests/rust-multi-frame-delta-perf.test.ts index 3ef725ff6..806c6639a 100644 --- a/packages/e2e-tests/tests/rust-multi-frame-delta-perf.test.ts +++ b/packages/e2e-tests/tests/rust-multi-frame-delta-perf.test.ts @@ -107,7 +107,7 @@ describe.skipIf(!rustPrereqs.ok)("rust transport: large tail delta", () => { expect(smallDelta.prefixGuardMs).toBeLessThan(10); expect(smallDelta.stateSyncMs).toBeLessThan(15); expect(smallDelta.wireBuildMs).toBeLessThan(10); - // The hermetic daemon uses ck-mc over external TCP, which can add scheduling overhead. + // The hermetic daemon uses McHandler over external TCP, which can add scheduling overhead. // Apply timing limits only in strict production-like environments; enforce message, // page-count, and payload-size limits in every environment. if (process.env.MC_RUST_E2E_STRICT_PERF === "1") { diff --git a/packages/e2e-tests/tests/rust-park-self-heal.test.ts b/packages/e2e-tests/tests/rust-park-self-heal.test.ts index 2c26a6825..98772cc47 100644 --- a/packages/e2e-tests/tests/rust-park-self-heal.test.ts +++ b/packages/e2e-tests/tests/rust-park-self-heal.test.ts @@ -15,7 +15,7 @@ * This file has TWO arms exercising different fault shapes; both assert the * shipped OUTCOME (no permanent park; transform resumes) not the mechanism: * - * A. module-restart recovery — kill and restart the ck-mc module mid-session + * A. module-restart recovery — kill and restart the McHandler module mid-session * against the same daemon + store. The raw array is unchanged, so the * adapter's ordinal state stays valid; the only failure window is the brief * reconnect. A module restart mid-session must recover on the following @@ -59,7 +59,7 @@ describe.skipIf(!rustPrereqs.ok)("rust incident regression: park self-heal", () // clean fault-injection window the daemon supervises: the store's // single-writer lease is released and re-acquired, and the plugin's subc // client transparently reconnects on its next call. - await h.subc.restartModule(); + await h.mcHost.restartHost(); await Bun.sleep(500); // Subsequent passes must recover. The first may fail during the reconnect @@ -96,7 +96,7 @@ describe.skipIf(!rustPrereqs.ok)("rust incident regression: park self-heal", () // Prolonged outage: kill the module and keep it down across several // passes so the adapter crosses its three-failure park threshold. - await h.subc.killModuleAndWait(); + await h.mcHost.crashHost(); for (let i = 4; i <= 8; i += 1) { h.mock.setDefault({ text: `outage assistant ${i}`, @@ -112,7 +112,7 @@ describe.skipIf(!rustPrereqs.ok)("rust incident regression: park self-heal", () // Restore the module and drive enough passes for the self-heal probe // cadence to retry and recover. - await h.subc.restartModule(); + await h.mcHost.restartHost(); await Bun.sleep(500); for (let i = 9; i <= 18; i += 1) { h.mock.setDefault({ diff --git a/packages/e2e-tests/tests/rust-removal-self-heal.test.ts b/packages/e2e-tests/tests/rust-removal-self-heal.test.ts index b1ab50b9f..e7cbd3551 100644 --- a/packages/e2e-tests/tests/rust-removal-self-heal.test.ts +++ b/packages/e2e-tests/tests/rust-removal-self-heal.test.ts @@ -14,7 +14,7 @@ * The assertion targets the outcome: after a real removal the transform keeps * serving and the session never permanently parks. * - * Drives the FULL production path: opencode → plugin → subc daemon → ck-mc. + * Drives the FULL production path: opencode → plugin → direct mc-host → McHandler. */ import { afterEach, beforeEach, describe, expect, it } from "bun:test"; diff --git a/packages/e2e-tests/tests/rust-smoke.test.ts b/packages/e2e-tests/tests/rust-smoke.test.ts index c9a44ebd9..9d504aab0 100644 --- a/packages/e2e-tests/tests/rust-smoke.test.ts +++ b/packages/e2e-tests/tests/rust-smoke.test.ts @@ -1,15 +1,5 @@ /// -/** - * Rust-mode lane smoke test: proves the hermetic stack (opencode → plugin → - * subc daemon → ck-mc module) actually transforms end to end, and that the lane - * SKIPs cleanly (with a printed reason) when prerequisites are missing. - * - * This is the de-risking harness check the incident-corpus scenarios build on: - * if this cannot boot Rust mode and observe a transform, none of the regression - * scenarios can either. - */ - import { afterAll, beforeAll, describe, expect, it } from "bun:test"; import { RustTestHarness } from "../src/rust-harness"; @@ -29,7 +19,7 @@ describe.skipIf(!prereqs.ok)("rust-mode lane smoke", () => { await h?.dispose(); }); - it("boots Rust mode and transforms a real session through the ck-mc module", async () => { + it("boots Rust mode and transforms a real session through McHostModuleTransport", async () => { const sessionId = await h.createSession(); // A handful of small turns — enough for the Rust transform to run and @@ -70,11 +60,7 @@ describe.skipIf(!prereqs.ok)("rust-mode lane smoke", () => { }); describe.skipIf(prereqs.ok)("rust-mode lane skip visibility", () => { - it("prints a skip reason when prerequisites are unmet", () => { - // This branch only runs on machines lacking cargo / the subconscious - // sibling / a supported platform. Emit the reason so CI logs show WHY - // the Rust lane was skipped rather than silently green-washing. - console.log(`[rust-e2e] SKIPPED: ${prereqs.skipReason ?? "unknown reason"}`); + it("prints a skip reason when prerequisites are unmet", () => { console.log(`[rust-e2e] SKIPPED: ${prereqs.skipReason ?? "unknown reason"}`); expect(prereqs.skipReason && prereqs.skipReason.length > 0).toBe(true); }); }); diff --git a/packages/e2e-tests/tests/rust-tail-mutation-readopt.test.ts b/packages/e2e-tests/tests/rust-tail-mutation-readopt.test.ts index b6cfd4c8a..9fff24117 100644 --- a/packages/e2e-tests/tests/rust-tail-mutation-readopt.test.ts +++ b/packages/e2e-tests/tests/rust-tail-mutation-readopt.test.ts @@ -19,7 +19,7 @@ * is edited in place in opencode.db (same message id, appended content), * mirroring the reminder-wrapper mutation OpenCode performs on a queued message. * - * Drives the FULL production path: opencode → plugin → subc daemon → ck-mc. + * Drives the FULL production path: opencode → plugin → direct mc-host → McHandler. */ import { Database } from "bun:sqlite"; diff --git a/packages/e2e-tests/tests/session-isolation.test.ts b/packages/e2e-tests/tests/session-isolation.test.ts index 4c0f9bf7a..2ca3dc158 100644 --- a/packages/e2e-tests/tests/session-isolation.test.ts +++ b/packages/e2e-tests/tests/session-isolation.test.ts @@ -21,10 +21,10 @@ let h: TestHarness; async function rustSessionStatus( sessionId: string, ): Promise> { - const stack = h.rustStack; + const stack = h.mcHostStack; if (!stack) throw new Error("Rust lifecycle check requires the hermetic module stack"); - return stack.moduleStatus( + return stack.primaryStatus( sessionId, h.opencode.env.workdir, "session.status", diff --git a/packages/e2e-tests/tsconfig.json b/packages/e2e-tests/tsconfig.json index 54d9cff08..ab0202626 100644 --- a/packages/e2e-tests/tsconfig.json +++ b/packages/e2e-tests/tsconfig.json @@ -3,6 +3,7 @@ "target": "ES2022", "module": "ESNext", "moduleResolution": "bundler", + "allowImportingTsExtensions": true, "strict": true, "noEmit": true, "esModuleInterop": true, diff --git a/packages/pi-plugin/PARITY.md b/packages/pi-plugin/PARITY.md index 8da4b021b..e2212d2be 100644 --- a/packages/pi-plugin/PARITY.md +++ b/packages/pi-plugin/PARITY.md @@ -156,7 +156,7 @@ shared resolver's log-only dubious-ownership warning while still using the same / E2BIG; the positional is omitted when piping. - `--no-session` keeps subagent JSONL out of the user's session picker. - v84 memory writes are intentionally not divergent. OpenCode and Pi wrappers keep harness-specific authorization, formatting, and post-commit embedding/module dispatch, but both call same transaction-local claim compatibility kernel. Same operation commits crosswalk, immutable revision metadata, `memories` projection, outbox, and generation; existing readers stay on `memories`. Public-tool e2e covers OpenCode→Pi and Pi→OpenCode writes against shared DB. -- Ship Pi, OpenCode, CLI, and `ck-mc` from same revision before v84 migration. Held-open legacy writers are rejected by shared guard; pending v22 work and lazy backfill use shared startup/doctor recovery (`magic-context doctor --check-claims-backfill`, then `--retry-claims-backfill`, then restart both harnesses). Archive/delete retire projection state and retain claim history; no Pi or OpenCode response promises erasure. +- Ship Pi, OpenCode, CLI, and the directly linked mc-host component from same revision before v84 migration. Held-open legacy writers are rejected by shared guard; pending v22 work and lazy backfill use shared startup/doctor recovery (`magic-context doctor --check-claims-backfill`, then `--retry-claims-backfill`, then restart both harnesses). Archive/delete retire projection state and retain claim history; no Pi or OpenCode response promises erasure. --- diff --git a/packages/plugin/scripts/drive-preseed.ts b/packages/plugin/scripts/drive-preseed.ts index 50fd22fae..caddc0c4e 100644 --- a/packages/plugin/scripts/drive-preseed.ts +++ b/packages/plugin/scripts/drive-preseed.ts @@ -10,9 +10,9 @@ * Defaults to the ckdev-rig connection file. */ import { homedir } from "node:os"; -import { join, dirname } from "node:path"; +import { join } from "node:path"; import { readFileSync } from "node:fs"; -import { SubcClient } from "../src/shared/mc-host-client"; +import { McHostClient } from "../src/shared/mc-host-client"; const args = process.argv.slice(2); const session = args[0]; @@ -25,8 +25,8 @@ const connectionFile = connIdx >= 0 ? args[connIdx + 1] : join(homedir(), ".local", "share", "cortexkit", "ckdev-rig", "runtime", "subc-connection.json"); -const payloadPath = join(dirname(new URL(import.meta.url).pathname), "drive-preseed-payload.json"); let payload: { compartments: unknown[] }; +const payloadPath = join(import.meta.dir, "drive-preseed-payload.json"); try { payload = JSON.parse(readFileSync(payloadPath, "utf8")) as { compartments: unknown[] }; if (!Array.isArray(payload.compartments)) { @@ -37,7 +37,7 @@ try { process.exit(1); } -const client = await SubcClient.connect({ connectionFile, handshakeTimeoutMs: 10_000 }); +const client = await McHostClient.connect({ connectionFile, handshakeTimeoutMs: 10_000 }); try { const route = await client.routeOpen( { kind: "tool_provider", module_id: "magic-context" }, diff --git a/packages/plugin/scripts/mc-host-client-boundary.test.ts b/packages/plugin/scripts/mc-host-client-boundary.test.ts index 8d5756c33..d415187fc 100644 --- a/packages/plugin/scripts/mc-host-client-boundary.test.ts +++ b/packages/plugin/scripts/mc-host-client-boundary.test.ts @@ -1,37 +1,100 @@ import { describe, expect, test } from "bun:test"; import * as fs from "node:fs"; +import { tmpdir } from "node:os"; import * as path from "node:path"; -/** - * Dependency-boundary gate: production packages must not depend on or import - * `@cortexkit/subc-client`. Only `PROVIDER_EXCEPTION` may import it. - */ - -const NPM_CLIENT = "@cortexkit/subc-client"; +const NPM_CLIENT = ["@cortexkit", ["subc", "client"].join("-")].join("/"); const repoRoot = path.resolve(import.meta.dir, "..", "..", ".."); -const PRODUCTION_PACKAGES = ["packages/plugin", "packages/cli", "packages/pi-plugin"]; +const OLD_API_NAMES = [ + "HermeticSubcOptions", + "HermeticSubcStack", + "SubcCallError", + "SubcCallErrorKind", + "SubcCallOptions", + "SubcClient", + "SubcClientOptions", + "SubcDiagnosticsEvent", + "SubcDiagnosticsObserver", + "SubcError", + "SubcModuleTransport", + "SubcProvider", + "SubcProviderConnectOptions", + "SubcProviderError", + "SubcSocket", + "__hermeticSubcTest", + "buildHermeticBinaries", + "expectSubcCallError", + "isLegacyFallbackTerminalBody", + "isLegacyUnsupportedOperationBody", + "isSubcCallError", +]; +type PackageManifest = { + workspaces?: string[] | { packages?: string[] }; + dependencies?: Record; + devDependencies?: Record; + optionalDependencies?: Record; + peerDependencies?: Record; +}; + +function readManifest(file: string): PackageManifest { + return JSON.parse(fs.readFileSync(file, "utf-8")) as PackageManifest; +} -const PROVIDER_EXCEPTION = "packages/e2e-tests/src/rust-runner/fake-broca.ts"; +function workspacePackageRoots(root: string): string[] { + const configured = readManifest(path.join(root, "package.json")).workspaces; + const patterns = Array.isArray(configured) ? configured : configured?.packages; + if (!patterns) throw new Error("root package.json must declare workspaces"); -/** IMPORT_PATTERN matches import, export-from, require, and dynamic-import specifiers for NPM_CLIENT. */ -const IMPORT_PATTERN = - /(?:from\s*["']@cortexkit\/subc-client["']|import\s*\(\s*["']@cortexkit\/subc-client["']\s*\)|require\s*\(\s*["']@cortexkit\/subc-client["']\s*\)|import\s*["']@cortexkit\/subc-client["'])/; + const roots = new Set(); + for (const pattern of patterns) { + if (pattern.endsWith("/*") && !pattern.slice(0, -2).includes("*")) { + const parent = path.join(root, pattern.slice(0, -2)); + for (const entry of fs.readdirSync(parent, { withFileTypes: true })) { + const candidate = path.join(parent, entry.name); + if (entry.isDirectory() && fs.existsSync(path.join(candidate, "package.json"))) { + roots.add(candidate); + } + } + } else if (!pattern.includes("*")) { + const candidate = path.join(root, pattern); + if (fs.existsSync(path.join(candidate, "package.json"))) roots.add(candidate); + } else { + throw new Error(`unsupported workspace pattern: ${pattern}`); + } + } + return [...roots].sort(); +} function* walkSourceFiles(dir: string): Generator { for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { - if (entry.name === "node_modules" || entry.name === "dist" || entry.name.startsWith(".")) - continue; + if (entry.name === "node_modules" || entry.name === "dist") continue; const full = path.join(dir, entry.name); if (entry.isDirectory()) yield* walkSourceFiles(full); else if (/\.(ts|tsx|js|mjs|cjs)$/.test(entry.name)) yield full; } } -function manifestDependencies(packageDir: string): Record { - const manifest = JSON.parse( - fs.readFileSync(path.join(repoRoot, packageDir, "package.json"), "utf-8"), - ) as Record | undefined>; +function oldApiNames(source: string): string[] { + const matches: string[] = []; + for (const name of OLD_API_NAMES) { + let offset = 0; + for (;;) { + const index = source.indexOf(name, offset); + if (index < 0) break; + const before = source[index - 1]; + const after = source[index + name.length]; + if ((!before || !/[\w$]/.test(before)) && (!after || !/[\w$]/.test(after))) { + matches.push(name); + } + offset = index + name.length; + } + } + return matches; +} + +function manifestDependencies(packageRoot: string): Record { + const manifest = readManifest(path.join(packageRoot, "package.json")); return { ...manifest.dependencies, ...manifest.devDependencies, @@ -40,63 +103,126 @@ function manifestDependencies(packageDir: string): Record { }; } -describe("mc-host-client dependency boundary", () => { - test("production manifests do not depend on the npm client", () => { - for (const pkg of PRODUCTION_PACKAGES) { - expect( - Object.keys(manifestDependencies(pkg)), - `${pkg}/package.json must not declare ${NPM_CLIENT}`, - ).not.toContain(NPM_CLIENT); +function sourceOffenders( + packageRoots: string[], + root: string, + matches: (source: string) => boolean, + allowlist: ReadonlySet = new Set(), +): string[] { + const offenders: string[] = []; + for (const packageRoot of packageRoots) { + for (const file of walkSourceFiles(packageRoot)) { + const relative = path.relative(root, file); + if (!allowlist.has(relative) && matches(fs.readFileSync(file, "utf-8"))) { + offenders.push(relative); + } } + } + return offenders.sort(); +} + +function manifestOffenders( + packageRoots: string[], + root: string, + matches: (source: string) => boolean, +): string[] { + return packageRoots + .map((packageRoot) => path.join(packageRoot, "package.json")) + .filter((file) => matches(fs.readFileSync(file, "utf-8"))) + .map((file) => path.relative(root, file)) + .sort(); +} + +describe("mc-host-client dependency boundary", () => { + const packageRoots = workspacePackageRoots(repoRoot); + + test("workspace package manifests do not depend on the npm client", () => { + const offenders = packageRoots + .filter((packageRoot) => NPM_CLIENT in manifestDependencies(packageRoot)) + .map((packageRoot) => path.relative(repoRoot, path.join(packageRoot, "package.json"))); + expect(offenders).toEqual([]); + expect(manifestOffenders(packageRoots, repoRoot, (source) => source.includes(NPM_CLIENT))).toEqual( + [], + ); }); - test("production and consumer-side sources do not import the npm client", () => { - const offenders: string[] = []; - for (const pkg of [...PRODUCTION_PACKAGES, "packages/e2e-tests"]) { - const root = path.join(repoRoot, pkg); - if (!fs.existsSync(root)) continue; - for (const file of walkSourceFiles(root)) { - const relative = path.relative(repoRoot, file); - if (relative === PROVIDER_EXCEPTION) continue; - if (IMPORT_PATTERN.test(fs.readFileSync(file, "utf-8"))) offenders.push(relative); + test("workspace package sources do not reference the npm client", () => { + expect( + sourceOffenders(packageRoots, repoRoot, (source) => source.includes(NPM_CLIENT)), + ).toEqual([]); + }); + + test("workspace package sources do not reference removed client API names", () => { + const canonicalLiteralAllowlist = new Set([ + path.relative(repoRoot, import.meta.path), + ]); + expect([ + ...manifestOffenders( + packageRoots, + repoRoot, + (source) => oldApiNames(source).length > 0, + ), + ...sourceOffenders( + packageRoots, + repoRoot, + (source) => oldApiNames(source).length > 0, + canonicalLiteralAllowlist, + ), + ]).toEqual([]); + }); + + test("workspace discovery and matchers catch retina/dashboard-like packages", () => { + const root = fs.mkdtempSync(path.join(tmpdir(), "mc-host-boundary-")); + try { + fs.writeFileSync( + path.join(root, "package.json"), + JSON.stringify({ private: true, workspaces: ["packages/*"] }), + ); + const retina = path.join(root, "packages", "retina-local-fs"); + const dashboard = path.join(root, "packages", "dashboard"); + for (const packageRoot of [retina, dashboard]) { + fs.mkdirSync(path.join(packageRoot, "src"), { recursive: true }); + fs.writeFileSync(path.join(packageRoot, "package.json"), "{}"); + fs.mkdirSync(path.join(packageRoot, "dist"), { recursive: true }); + fs.writeFileSync( + path.join(packageRoot, "dist", "generated.js"), + `import ${JSON.stringify(NPM_CLIENT)};`, + ); } + fs.writeFileSync( + path.join(retina, "src", "provider.ts"), + `import client from ${JSON.stringify(NPM_CLIENT)};`, + ); + fs.writeFileSync( + path.join(dashboard, "package.json"), + JSON.stringify({ scripts: { legacy: OLD_API_NAMES[6] } }), + ); + + const roots = workspacePackageRoots(root); + expect(roots.map((packageRoot) => path.basename(packageRoot))).toEqual([ + "dashboard", + "retina-local-fs", + ]); + expect(sourceOffenders(roots, root, (source) => source.includes(NPM_CLIENT))).toEqual([ + "packages/retina-local-fs/src/provider.ts", + ]); + expect( + manifestOffenders(roots, root, (source) => oldApiNames(source).length > 0), + ).toEqual(["packages/dashboard/package.json"]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); } - expect(offenders, `only ${PROVIDER_EXCEPTION} may import ${NPM_CLIENT}`).toEqual([]); }); - test("the E2E provider exception still holds the one permitted import", () => { - const source = fs.readFileSync(path.join(repoRoot, PROVIDER_EXCEPTION), "utf-8"); - expect(IMPORT_PATTERN.test(source)).toBe(true); - expect(Object.keys(manifestDependencies("packages/e2e-tests"))).toContain(NPM_CLIENT); + test("removed-name matcher covers E2E and client option surfaces", () => { + expect(oldApiNames("let stack: HermeticSubcStack;")).toEqual(["HermeticSubcStack"]); + expect(oldApiNames("type Options = SubcClientOptions;")).toEqual(["SubcClientOptions"]); + expect( + oldApiNames('const ops = ["subc_ops", "subc-client-v1", "subc-connection.json"];'), + ).toEqual([]); }); - test("lockfile resolves the npm client only through the E2E package", () => { - const raw = fs.readFileSync(path.join(repoRoot, "bun.lock"), "utf-8"); - // bun.lock is JSONC whose only non-JSON feature is trailing commas. - const lock = JSON.parse(raw.replace(/,(\s*[}\]])/g, "$1")) as { - workspaces: Record>; - }; - const sections = [ - "dependencies", - "devDependencies", - "optionalDependencies", - "peerDependencies", - ]; - const declaringWorkspaces = Object.entries(lock.workspaces) - .filter(([, entry]) => - sections.some((section) => { - const deps = entry[section]; - return ( - typeof deps === "object" && - deps !== null && - NPM_CLIENT in (deps as Record) - ); - }), - ) - .map(([workspacePath]) => workspacePath); - expect( - declaringWorkspaces, - `only packages/e2e-tests may declare ${NPM_CLIENT} in bun.lock`, - ).toEqual(["packages/e2e-tests"]); + test("lockfile does not resolve the npm client", () => { + expect(fs.readFileSync(path.join(repoRoot, "bun.lock"), "utf-8")).not.toContain(NPM_CLIENT); }); }); diff --git a/packages/plugin/scripts/probe-subc-transport.ts b/packages/plugin/scripts/probe-mc-host-transport.ts similarity index 97% rename from packages/plugin/scripts/probe-subc-transport.ts rename to packages/plugin/scripts/probe-mc-host-transport.ts index cd8550cde..92bbe0e60 100644 --- a/packages/plugin/scripts/probe-subc-transport.ts +++ b/packages/plugin/scripts/probe-mc-host-transport.ts @@ -8,13 +8,13 @@ import { monitorEventLoopDelay, performance } from "node:perf_hooks"; import { AdmissionClass, Priority, - SubcClient, + McHostClient, type RequestOptions, type RouteHandle, - type SubcDiagnosticsEvent, + type McHostDiagnosticsEvent, } from "../src/shared/mc-host-client"; -import { SubcModuleTransport } from "../src/hooks/magic-context/module-transport"; +import { McHostModuleTransport } from "../src/hooks/magic-context/module-transport"; import { buildPagedModuleTransformPayloads } from "../src/hooks/magic-context/module-wire"; type JsonRecord = Record; @@ -140,7 +140,11 @@ function codecProxy(value: unknown, samples: number): { JSON.stringify(value); stringifySamples.push(performance.now() - startedAt); startedAt = performance.now(); - JSON.parse(serialized); + try { + JSON.parse(serialized); + } catch { + // Benchmark round-trip of freshly stringified data. commentlint: allow(JUDGE) + } parseSamples.push(performance.now() - startedAt); } return { @@ -288,7 +292,7 @@ let activeTrace: PhaseTrace | null = null; let activeChannel = -1; let daemonPid: number | null = null; -function observeDiagnostics(event: SubcDiagnosticsEvent): void { +function observeDiagnostics(event: McHostDiagnosticsEvent): void { if (event.type === "connected") { daemonPid = event.pid ?? null; return; @@ -314,7 +318,7 @@ function observeDiagnostics(event: SubcDiagnosticsEvent): void { } async function instrumentedRequest( - client: SubcClient, + client: McHostClient, route: RouteHandle, body: JsonRecord, options: RequestOptions, @@ -348,7 +352,7 @@ async function instrumentedRequest( } if (argv.includes("--help")) { - console.log(`usage: bun packages/plugin/scripts/probe-subc-transport.ts [options] + console.log(`usage: bun packages/plugin/scripts/probe-mc-host-transport.ts [options] options: --connection-file daemon connection file @@ -398,7 +402,7 @@ const scenarios: Scenario[] = [ const eventLoopDelay = monitorEventLoopDelay({ resolution: 1 }); eventLoopDelay.enable(); const connectStartedAt = performance.now(); -const client = await SubcClient.connect({ +const client = await McHostClient.connect({ connectionFile, handshakeTimeoutMs: timeoutMs, diagnostics: observeDiagnostics, @@ -592,7 +596,7 @@ try { if (!realResponse) throw new Error("real-shape transform arm returned no measured response"); -const transport = new SubcModuleTransport(connectionFile, moduleId, timeoutMs); +const transport = new McHostModuleTransport(connectionFile, moduleId, timeoutMs); const fifoSessions = Array.from( { length: fifoConcurrency }, (_, index) => `${session}-fifo-${index}`, diff --git a/packages/plugin/scripts/retrieval-benchmark/privacy.test.ts b/packages/plugin/scripts/retrieval-benchmark/privacy.test.ts index a5c8bbb62..efa5479c2 100644 --- a/packages/plugin/scripts/retrieval-benchmark/privacy.test.ts +++ b/packages/plugin/scripts/retrieval-benchmark/privacy.test.ts @@ -131,7 +131,7 @@ describe("scanForSensitiveContent", () => { }); it("never echoes raw or encoded canary values in violations", () => { - const canary = "sk-ant-api03-abcdefghijklmnopqrstuvwxyz0123456789AB"; + const canary = "sk-ant-api03-" + "abcdefghijklmnopqrstuvwxyz0123456789AB"; const violations = scanForSensitiveContent({ q: `key ${canary}` }); const serialized = JSON.stringify(violations); expect(serialized).not.toContain(canary); @@ -140,7 +140,7 @@ describe("scanForSensitiveContent", () => { }); it("never echoes a sensitive object key through violation paths", () => { - const sensitiveKey = "ses_0123456789abcdefSECRETKEY"; + const sensitiveKey = "ses_0123456789abcdefSECRETKEY"; // gitleaks:allow privacy-test canary const violations = scanForSensitiveContent({ nested: { [sensitiveKey]: "value" } }); expect(violations.length).toBeGreaterThan(0); expect(JSON.stringify(violations)).not.toContain(sensitiveKey); diff --git a/packages/plugin/scripts/smoke-mc-host-client.ts b/packages/plugin/scripts/smoke-mc-host-client.ts index 0b826f7e8..f1b4628ff 100644 --- a/packages/plugin/scripts/smoke-mc-host-client.ts +++ b/packages/plugin/scripts/smoke-mc-host-client.ts @@ -5,7 +5,7 @@ import { existsSync, mkdtempSync, realpathSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { wakePlaneStatus } from "../src/features/magic-context/smart-notes/wake-plane"; -import { SubcClient, type SubcDiagnosticsEvent } from "../src/shared/mc-host-client"; +import { McHostClient, type McHostDiagnosticsEvent } from "../src/shared/mc-host-client"; const OVERALL_DEADLINE_MS = 60_000; const READY_DEADLINE_MS = 15_000; @@ -149,8 +149,8 @@ try { } log("production wake-plane probe returned absent"); - const events: SubcDiagnosticsEvent[] = []; - const client = await SubcClient.connect({ + const events: McHostDiagnosticsEvent[] = []; + const client = await McHostClient.connect({ connectionFile, diagnostics: (event) => events.push(event), }); diff --git a/packages/plugin/scripts/smoke-mc-host-synapse.ts b/packages/plugin/scripts/smoke-mc-host-synapse.ts index abe26d22c..abdec7b91 100644 --- a/packages/plugin/scripts/smoke-mc-host-synapse.ts +++ b/packages/plugin/scripts/smoke-mc-host-synapse.ts @@ -1,7 +1,7 @@ /** * Focused real-host Synapse smoke: starts the `synapse_host` example over a * model bundle, then drives all four application operations through the - * managed SubcClient — discovery, one query, an ambiguous batch replay, a + * managed McHostClient — discovery, one query, an ambiguous batch replay, a * host restart with resubmission, and the degraded (unconfigured) lane. * * Hermetic mode (default) uses the committed synapse-tiny bundle and needs a @@ -33,7 +33,7 @@ import { } from "../src/features/magic-context/storage-embedding-measurements"; import { initializeDatabase } from "../src/features/magic-context/storage-db"; import { Database } from "../src/shared/sqlite"; -import { SubcClient } from "../src/shared/mc-host-client"; +import { McHostClient } from "../src/shared/mc-host-client"; const OVERALL_DEADLINE_MS = 180_000; const READY_DEADLINE_MS = 20_000; @@ -175,19 +175,31 @@ const bundleDir = if (!existsSync(join(bundleDir, "manifest.json"))) { fail(`bundle manifest missing at ${bundleDir}`); } -function readBundleJson(name: string): unknown { +interface BundleManifest { + model?: unknown; + dims?: unknown; + table_epoch?: unknown; + fingerprint?: unknown; + corpus?: { name?: unknown }; + provenance?: { production?: unknown }; + model_file?: { sha256?: unknown }; +} + +interface BundleCorpus { + tolerance: number; + items: { text: string; expected: number[] }[]; +} + +function readBundleJson(name: string): T { try { - return JSON.parse(readFileSync(join(bundleDir, name), "utf8")); + return JSON.parse(readFileSync(join(bundleDir, name), "utf8")) as T; } catch (error) { fail(`bundle file ${name} is unreadable or not JSON: ${String(error)}`); } } -const manifest = readBundleJson("manifest.json") as Record; -const corpus = readBundleJson(String(manifest.corpus?.name ?? "corpus.json")) as { - tolerance: number; - items: { text: string; expected: number[] }[]; -}; +const manifest = readBundleJson("manifest.json"); +const corpus = readBundleJson(String(manifest.corpus?.name ?? "corpus.json")); if (mode === "production") { // The release smoke must never silently pass on the toy bundle or a @@ -296,7 +308,7 @@ function body(value: unknown): Record { } async function pollJob( - client: SubcClient, + client: McHostClient, jobId: string, key: string, ): Promise<{ id: string; vector: number[] }[]> { @@ -342,7 +354,7 @@ try { log("starting synapse_host with no bundle (degraded lane)"); let connectionFile = await startHost("-"); { - const client = await SubcClient.connect({ connectionFile }); + const client = await McHostClient.connect({ connectionFile }); try { await client.call("synapse", "models.list", {}, callOptions); fail("a degraded lane must reject its bind"); @@ -366,7 +378,7 @@ try { // ---------------- Certified lane: all four operations. ----------------- log(`starting synapse_host with bundle ${bundleDir} (${mode} mode)`); connectionFile = await startHost(bundleDir); - const client = await SubcClient.connect({ connectionFile }); + const client = await McHostClient.connect({ connectionFile }); let jobId: string; let key: string; const batchItems = corpus.items.slice(0, 3).map((item, index) => ({ @@ -432,7 +444,7 @@ try { log("restarting the host to fence the retained job"); await stopHost(true); connectionFile = await startHost(bundleDir); - const restarted = await SubcClient.connect({ connectionFile }); + const restarted = await McHostClient.connect({ connectionFile }); try { try { await restarted.call( diff --git a/packages/plugin/src/config/index.test.ts b/packages/plugin/src/config/index.test.ts index be428c0f5..ed2e3094c 100644 --- a/packages/plugin/src/config/index.test.ts +++ b/packages/plugin/src/config/index.test.ts @@ -954,7 +954,8 @@ describe("transform_mode resolution", () => { ); expect(result.transform_mode).toBe("rust"); - expect(result.subc?.connection_file).not.toContain("project-controlled.sock"); + const { subc } = result; + expect(subc?.connection_file).not.toContain("project-controlled.sock"); }); }); diff --git a/packages/plugin/src/config/index.ts b/packages/plugin/src/config/index.ts index 1a04edad2..1d7cc7669 100644 --- a/packages/plugin/src/config/index.ts +++ b/packages/plugin/src/config/index.ts @@ -377,6 +377,7 @@ function parsePluginConfig( // resolved value itself, because `{env:...}` / `{file:...}` // substitution may have already expanded secrets into rawConfig. delete patched[key]; + // SAFETY: every top-level Zod issue path names a field in defaults. const defaultVal = (defaults as unknown as Record)[key]; const reason = customMessagesByKey.get(key); warnings.push( @@ -431,7 +432,7 @@ export function loadPluginConfig( } function hasUserTierSubcConfig(config: Record | undefined): boolean { - const subc = config?.subc; + const { subc } = config ?? {}; if (typeof subc !== "object" || subc === null || Array.isArray(subc)) return false; const connectionFile = (subc as Record).connection_file; return typeof connectionFile === "string" && connectionFile.trim().length > 0; diff --git a/packages/plugin/src/config/project-security.test.ts b/packages/plugin/src/config/project-security.test.ts index 9b64e2115..039cc98e1 100644 --- a/packages/plugin/src/config/project-security.test.ts +++ b/packages/plugin/src/config/project-security.test.ts @@ -85,7 +85,7 @@ describe("stripUnsafeProjectConfigFields", () => { const warnings = stripUnsafeProjectConfigFields(raw); expect(raw.transform_mode).toBe("rust"); - expect(raw.subc).toBeUndefined(); + expect(raw).not.toHaveProperty("subc"); expect(warnings.some((w) => w.includes("subc"))).toBe(true); expect(warnings.some((w) => w.includes("transform_mode"))).toBe(false); }); diff --git a/packages/plugin/src/config/schema/magic-context.ts b/packages/plugin/src/config/schema/magic-context.ts index 3fe98f658..cccec0606 100644 --- a/packages/plugin/src/config/schema/magic-context.ts +++ b/packages/plugin/src/config/schema/magic-context.ts @@ -673,7 +673,7 @@ export const MagicContextConfigSchema = z .enum(["ts", "rust"]) .default("ts") .describe( - 'Experimental: routes the entire Magic Context runtime for the project through the ck-mc Rust module over subc (requires user-level `subc` config); "ts" is the current TypeScript pipeline.', + 'Experimental: routes the project through the direct mc-host Rust runtime (requires user-level host connection config); "ts" is the current TypeScript pipeline.', ), auto_update: z .boolean() diff --git a/packages/plugin/src/features/magic-context/memory/embedding-synapse.test.ts b/packages/plugin/src/features/magic-context/memory/embedding-synapse.test.ts index 1fd793ab4..576ea16a0 100644 --- a/packages/plugin/src/features/magic-context/memory/embedding-synapse.test.ts +++ b/packages/plugin/src/features/magic-context/memory/embedding-synapse.test.ts @@ -2,7 +2,7 @@ import { afterEach, describe, expect, it } from "bun:test"; import { createHash } from "node:crypto"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { SubcCallError } from "../../../shared/mc-host-client"; +import { McHostCallError } from "../../../shared/mc-host-client"; import { Database } from "../../../shared/sqlite"; import { closeQuietly } from "../../../shared/sqlite-helpers"; import { runMigrations } from "../migrations"; @@ -591,7 +591,7 @@ describe("connect discovery and retry policy", () => { ({ async call() { calls += 1; - throw new SubcCallError("outcome_unknown", "ambiguous send"); + throw new McHostCallError("outcome_unknown", "ambiguous send"); }, close() {}, }) as SynapseClientLike, @@ -616,10 +616,10 @@ describe("connect discovery and retry policy", () => { if (method !== "embed.query") throw new Error(`unexpected ${method}`); queryCalls += 1; if (queryCalls <= 2) { - const error = new SubcCallError( + const error = new McHostCallError( "outcome_unknown", "ambiguous send", - ) as SubcCallError & { retry_after_ms: number }; + ) as McHostCallError & { retry_after_ms: number }; error.retry_after_ms = 0; throw error; } @@ -650,10 +650,10 @@ describe("connect discovery and retry policy", () => { ({ async call() { queryCalls += 1; - const error = new SubcCallError( + const error = new McHostCallError( "outcome_unknown", "ambiguous send", - ) as SubcCallError & { retry_after_ms: number }; + ) as McHostCallError & { retry_after_ms: number }; error.retry_after_ms = 0; throw error; }, @@ -715,10 +715,10 @@ describe("connect discovery and retry policy", () => { ({ async call() { queryCalls += 1; - const error = new SubcCallError( + const error = new McHostCallError( "outcome_unknown", "ambiguous send", - ) as SubcCallError & { retry_after_ms: number }; + ) as McHostCallError & { retry_after_ms: number }; error.retry_after_ms = 10_000; throw error; }, @@ -739,11 +739,11 @@ describe("connect discovery and retry policy", () => { ({ async call() { listCalls += 1; - const error = new SubcCallError( + const error = new McHostCallError( "terminal", "route.open failed for module synapse: target_unavailable", "target_unavailable", - ) as SubcCallError & { retry_after_ms: number }; + ) as McHostCallError & { retry_after_ms: number }; error.retry_after_ms = 0; throw error; }, diff --git a/packages/plugin/src/features/magic-context/memory/embedding-synapse.ts b/packages/plugin/src/features/magic-context/memory/embedding-synapse.ts index 043b0fd36..ec38c08b5 100644 --- a/packages/plugin/src/features/magic-context/memory/embedding-synapse.ts +++ b/packages/plugin/src/features/magic-context/memory/embedding-synapse.ts @@ -1,7 +1,7 @@ import { createHash, randomUUID } from "node:crypto"; import { getHarness } from "../../../shared/harness"; import { log } from "../../../shared/logger"; -import { isSubcCallError, SubcClient } from "../../../shared/mc-host-client"; +import { isMcHostCallError, McHostClient } from "../../../shared/mc-host-client"; import { createSynapseLedgerPage, findSynapseLedgerPage, @@ -463,7 +463,7 @@ async function getSharedClient( if (sharedClientPromise && sharedClientFile === options.connectionFile) return sharedClientPromise; const file = options.connectionFile; - const promise = SubcClient.connect({ + const promise = McHostClient.connect({ connectionFile: file, handshakeTimeoutMs: SYNAPSE_HANDSHAKE_TIMEOUT_MS, }).then((client) => { @@ -1441,7 +1441,7 @@ export class SynapseEmbeddingProvider implements EmbeddingProvider { // R21: module_restarted bypasses generic retry entirely; the // caller owns the single durable restart resubmission. if (classified.code === "module_restarted") throw classified; - const outcomeUnknown = isSubcCallError(error) && error.kind === "outcome_unknown"; + const outcomeUnknown = isMcHostCallError(error) && error.kind === "outcome_unknown"; const retryable = !classified.permanent && (retryEmbeddings || !outcomeUnknown); // A queue-full reply creates no host state. Retry it through // the request deadline so the host's bounded query lane sheds diff --git a/packages/plugin/src/features/magic-context/smart-notes/wake-plane.ts b/packages/plugin/src/features/magic-context/smart-notes/wake-plane.ts index dfcdde995..16b2be77b 100644 --- a/packages/plugin/src/features/magic-context/smart-notes/wake-plane.ts +++ b/packages/plugin/src/features/magic-context/smart-notes/wake-plane.ts @@ -1,7 +1,7 @@ import { join } from "node:path"; import { getDataDir } from "../../../shared/data-path"; -import { SubcClient } from "../../../shared/mc-host-client"; +import { McHostClient } from "../../../shared/mc-host-client"; /** The sole wire-level coupling between standalone smart notes and scheduled wakes. */ export const WAKE_PLANE_CAPABILITY = "wake.create"; @@ -31,7 +31,7 @@ function connectionFile(): string { } async function probeWakePlaneCatalog(): Promise { - const client = await SubcClient.connect({ + const client = await McHostClient.connect({ connectionFile: connectionFile(), handshakeTimeoutMs: WAKE_PLANE_HANDSHAKE_TIMEOUT_MS, requestTimeoutMs: WAKE_PLANE_CATALOG_TIMEOUT_MS, diff --git a/packages/plugin/src/hooks/magic-context/hook.ts b/packages/plugin/src/hooks/magic-context/hook.ts index 843fbb0aa..63c873397 100644 --- a/packages/plugin/src/hooks/magic-context/hook.ts +++ b/packages/plugin/src/hooks/magic-context/hook.ts @@ -98,7 +98,7 @@ import { import { formatEmbedStatusText } from "./format-embed-status"; import { clearInjectionCache } from "./inject-compartments"; import { dropSlot } from "./lkg-slot"; -import { SubcModuleTransport } from "./module-transport"; +import { McHostModuleTransport } from "./module-transport"; import { findLastAssistantModelFromOpenCodeDb } from "./read-session-db"; import type { ManagedRecompContext } from "./recomp-orchestrator"; import { @@ -744,7 +744,7 @@ export function createMagicContextHook(deps: MagicContextDeps) { const authorityRecoveryModuleClient = deps.rustModeModuleClient ?? (() => { - const transport = new SubcModuleTransport(); + const transport = new McHostModuleTransport(); const client: RustModeModuleClient = { call: (args) => transport.call(args), stateSyncCapabilities: (args) => transport.stateSyncCapabilities(args), @@ -946,7 +946,7 @@ export function createMagicContextHook(deps: MagicContextDeps) { // than the plugin's launch directory (a /cd switch, multi-project hosts). // Registration is therefore an idempotent ensure invoked for every project // that reaches rust-mode preparation, not a one-shot at construction. - const evaluatorTransport = rustModeModuleClient ? new SubcModuleTransport() : undefined; + const evaluatorTransport = rustModeModuleClient ? new McHostModuleTransport() : undefined; // Bridge keys this hook instance registered. Instance disposal must tear // down only these: the registry is process-global and Desktop hosts several // plugin instances in one process. diff --git a/packages/plugin/src/hooks/magic-context/module-state-sync.test.ts b/packages/plugin/src/hooks/magic-context/module-state-sync.test.ts index 2c5c23953..fd6a83999 100644 --- a/packages/plugin/src/hooks/magic-context/module-state-sync.test.ts +++ b/packages/plugin/src/hooks/magic-context/module-state-sync.test.ts @@ -26,7 +26,7 @@ import { updateTagStatus, } from "../../features/magic-context/storage-tags"; import { insertUserMemory } from "../../features/magic-context/user-memory/storage-user-memory"; -import { SubcCallError } from "../../shared/mc-host-client"; +import { McHostCallError } from "../../shared/mc-host-client"; import { Database } from "../../shared/sqlite"; import { closeQuietly } from "../../shared/sqlite-helpers"; import { @@ -741,7 +741,7 @@ describe("module state sync section deltas", () => { async call() { callCount += 1; // Possible-send drops throw; never a typed generation-change result. - throw new SubcCallError( + throw new McHostCallError( "outcome_unknown", "connection dropped after a possible send", "connection_dropped", @@ -757,7 +757,7 @@ describe("module state sync section deltas", () => { projectRoot: "/tmp/project", force: false, }), - ).rejects.toMatchObject({ name: "SubcCallError", kind: "outcome_unknown" }); + ).rejects.toMatchObject({ name: "McHostCallError", kind: "outcome_unknown" }); expect(callCount).toBe(1); expect(statusCalls).toBe(0); diff --git a/packages/plugin/src/hooks/magic-context/module-transport.test.ts b/packages/plugin/src/hooks/magic-context/module-transport.test.ts index 82c7ae157..770f9ecd6 100644 --- a/packages/plugin/src/hooks/magic-context/module-transport.test.ts +++ b/packages/plugin/src/hooks/magic-context/module-transport.test.ts @@ -9,8 +9,8 @@ import { Deadline, type RouteHandle, StaleRouteHandleError, - SubcCallError, - type SubcClient, + McHostCallError, + type McHostClient, } from "../../shared/mc-host-client"; import { FakePeer, @@ -24,12 +24,12 @@ import { waitUntil, writeConnectionFile, } from "../../shared/mc-host-client/test-support/test-util"; -import { __moduleTransportTest, SubcModuleTransport } from "./module-transport"; +import { __moduleTransportTest, McHostModuleTransport } from "./module-transport"; let tempDir = ""; let fileCounter = 0; let peers: FakePeer[] = []; -let transports: SubcModuleTransport[] = []; +let transports: McHostModuleTransport[] = []; let savedModuleId: string | undefined; let savedLaunchNonce: string | undefined; @@ -76,18 +76,18 @@ async function writeConnFile(peer: FakePeer): Promise { return filePath; } -function trackTransport(transport: SubcModuleTransport): SubcModuleTransport { +function trackTransport(transport: McHostModuleTransport): McHostModuleTransport { transports.push(transport); return transport; } async function peerTransport( requestTimeoutMs = 5_000, -): Promise<{ peer: FakePeer; transport: SubcModuleTransport }> { +): Promise<{ peer: FakePeer; transport: McHostModuleTransport }> { const peer = await startPeer(); const connectionFile = await writeConnFile(peer); const transport = trackTransport( - new SubcModuleTransport(connectionFile, "magic-context", requestTimeoutMs), + new McHostModuleTransport(connectionFile, "magic-context", requestTimeoutMs), ); return { peer, transport }; } @@ -187,10 +187,10 @@ function routedBodies(peer: FakePeer): PeerFrame[] { return peer.connections.flatMap((conn) => conn.frames.filter(isRoutedRequest())); } -function expectCallError(error: unknown, kind: SubcCallError["kind"], code?: string): void { - expect((error as Error).name).toBe("SubcCallError"); - expect((error as SubcCallError).kind).toBe(kind); - if (code !== undefined) expect((error as SubcCallError).code).toBe(code); +function expectCallError(error: unknown, kind: McHostCallError["kind"], code?: string): void { + expect((error as Error).name).toBe("McHostCallError"); + expect((error as McHostCallError).kind).toBe(kind); + if (code !== undefined) expect((error as McHostCallError).code).toBe(code); } function deferred(): { @@ -207,7 +207,7 @@ function deferred(): { return { promise, resolve, reject }; } -describe("SubcModuleTransport", () => { +describe("McHostModuleTransport", () => { it("uses the internal facade while preserving route identity and flat request bytes", async () => { const { peer, transport } = await peerTransport(1_000); const flatBody = { @@ -456,7 +456,7 @@ describe("SubcModuleTransport", () => { }); it("reconnects once when the request provably never reached the socket", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 100); + const transport = new McHostModuleTransport("unused-connection-file", "magic-context", 100); const route = { channel: 7, epoch: 77 } as RouteHandle; let connectionCount = 0; let firstCloseCount = 0; @@ -464,7 +464,7 @@ describe("SubcModuleTransport", () => { { routeOpen: async () => route, request: async () => { - throw new SubcCallError("not_sent", "client closed", "connection_dropped"); + throw new McHostCallError("not_sent", "client closed", "connection_dropped"); }, close: () => { firstCloseCount += 1; @@ -475,10 +475,10 @@ describe("SubcModuleTransport", () => { request: async () => ({ result: { reconnected: true } }), close: () => undefined, }, - ] as unknown as SubcClient[]; + ] as unknown as McHostClient[]; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { const client = clients[connectionCount++]; @@ -500,7 +500,11 @@ describe("SubcModuleTransport", () => { }); it("stops after not_sent then unknown_channel: the transport replay token is single-use", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 1_000); + const transport = new McHostModuleTransport( + "unused-connection-file", + "magic-context", + 1_000, + ); const route = { channel: 7, epoch: 77 } as RouteHandle; let requestCount = 0; let routeOpenCount = 0; @@ -512,15 +516,15 @@ describe("SubcModuleTransport", () => { request: async () => { requestCount += 1; if (requestCount === 1) { - throw new SubcCallError("not_sent", "queued rejection", "writer_queue_full"); + throw new McHostCallError("not_sent", "queued rejection", "writer_queue_full"); } - throw new SubcCallError("terminal", "error unknown_channel", "unknown_channel"); + throw new McHostCallError("terminal", "error unknown_channel", "unknown_channel"); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { internals.client = client; @@ -542,7 +546,11 @@ describe("SubcModuleTransport", () => { }); it("an aborted caller cannot spend an unspent replay token", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 1_000); + const transport = new McHostModuleTransport( + "unused-connection-file", + "magic-context", + 1_000, + ); const route = { channel: 7, epoch: 77 } as RouteHandle; const controller = new AbortController(); let requestCount = 0; @@ -551,13 +559,13 @@ describe("SubcModuleTransport", () => { request: async () => { requestCount += 1; controller.abort(); - throw new SubcCallError("not_sent", "request aborted", "aborted"); + throw new McHostCallError("not_sent", "request aborted", "aborted"); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { internals.client = client; @@ -578,20 +586,20 @@ describe("SubcModuleTransport", () => { }); it("returns the typed generation change for pre-send recovery when generationSensitive is set", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 100); + const transport = new McHostModuleTransport("unused-connection-file", "magic-context", 100); const route = { channel: 7, epoch: 77 } as RouteHandle; let requestCount = 0; const client = { routeOpen: async () => route, request: async () => { requestCount += 1; - throw new SubcCallError("not_sent", "client closed", "connection_dropped"); + throw new McHostCallError("not_sent", "client closed", "connection_dropped"); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { internals.client = client; @@ -615,24 +623,24 @@ describe("SubcModuleTransport", () => { }); it("propagates outcome_unknown as an error even when generationSensitive is set", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 100); + const transport = new McHostModuleTransport("unused-connection-file", "magic-context", 100); const route = { channel: 7, epoch: 77 } as RouteHandle; let requestCount = 0; const client = { routeOpen: async () => route, request: async () => { requestCount += 1; - throw new SubcCallError( + throw new McHostCallError( "outcome_unknown", "connection dropped mid-request", "connection_dropped", ); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { internals.client = client; @@ -654,7 +662,7 @@ describe("SubcModuleTransport", () => { it("bounds a half-open route open under the single operation deadline without any body send", async () => { const timeoutMs = 30; - const transport = new SubcModuleTransport( + const transport = new McHostModuleTransport( "unused-connection-file", "magic-context", timeoutMs, @@ -667,10 +675,10 @@ describe("SubcModuleTransport", () => { return new Promise(() => undefined); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { connectionCount += 1; @@ -696,7 +704,7 @@ describe("SubcModuleTransport", () => { it("bounds a hung stubbed request as outcome_unknown without a second attempt", async () => { const timeoutMs = 30; - const transport = new SubcModuleTransport( + const transport = new McHostModuleTransport( "unused-connection-file", "magic-context", timeoutMs, @@ -711,10 +719,10 @@ describe("SubcModuleTransport", () => { return new Promise(() => undefined); }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; - ensureConnected(): Promise; + client: McHostClient | null; + ensureConnected(): Promise; }; internals.ensureConnected = async () => { connectionCount += 1; @@ -738,7 +746,7 @@ describe("SubcModuleTransport", () => { }); it("closeSession during an in-flight route open leaves no late cached route", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; const routeOpenStarted = deferred(); const releaseRouteOpen = deferred(); @@ -752,9 +760,9 @@ describe("SubcModuleTransport", () => { closeRoute: async () => { closeRouteCount += 1; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; routes: Map; ensureRoute: (sessionId: string, projectRoot: string) => Promise; }; @@ -771,7 +779,7 @@ describe("SubcModuleTransport", () => { }); it("bounds canonical-root entries with least-recently-used eviction", () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const internals = transport as unknown as { canonicalRoot(root: string): string; canonicalRootCache: Map; @@ -789,7 +797,7 @@ describe("SubcModuleTransport", () => { }); it("does not expose state-sync capabilities from an earlier connection generation", () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const internals = transport as unknown as { connectionGeneration: number; stateSyncCapabilityCache: { @@ -809,7 +817,7 @@ describe("SubcModuleTransport", () => { }); it("allows another session to start while a long wrapup is still in flight", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; const wrapupStarted = deferred(); const statusStarted = deferred(); @@ -826,11 +834,11 @@ describe("SubcModuleTransport", () => { } return { result: { ok: true } }; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; ensureRoute: (sessionId: string) => Promise<{ - client: SubcClient; + client: McHostClient; route: RouteHandle; routeKey: string; generation: number; @@ -870,7 +878,7 @@ describe("SubcModuleTransport", () => { }); it("executes one session's state sync, transform, and status strictly in submission order", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; const stateSyncStarted = deferred(); const transformStarted = deferred(); @@ -890,11 +898,11 @@ describe("SubcModuleTransport", () => { } return { result: { method } }; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; ensureRoute: () => Promise<{ - client: SubcClient; + client: McHostClient; route: RouteHandle; routeKey: string; generation: number; @@ -936,7 +944,11 @@ describe("SubcModuleTransport", () => { }); it("coalesces concurrent connection recovery and retries two sessions on one fresh generation", async () => { - const transport = new SubcModuleTransport("unused-connection-file", "magic-context", 1_000); + const transport = new McHostModuleTransport( + "unused-connection-file", + "magic-context", + 1_000, + ); const oldRouteA = { channel: 7, epoch: 70 } as RouteHandle; const oldRouteB = { channel: 8, epoch: 80 } as RouteHandle; const oldRequestsStarted = deferred(); @@ -948,12 +960,12 @@ describe("SubcModuleTransport", () => { if (oldRequestCount === 2) oldRequestsStarted.resolve(); await oldRequestsStarted.promise; // Proven pre-send rejections: the replay token may be spent. - throw new SubcCallError("not_sent", "client closed", "connection_dropped"); + throw new McHostCallError("not_sent", "client closed", "connection_dropped"); }, close: () => { oldCloseCount += 1; }, - } as unknown as SubcClient; + } as unknown as McHostClient; let routeOpenCount = 0; const freshRequestSessions: string[] = []; const freshClient = { @@ -971,13 +983,13 @@ describe("SubcModuleTransport", () => { return { result: { sessionId } }; }, close: () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; let connectCount = 0; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; connectionGeneration: number; routes: Map; - connectClient(): Promise; + connectClient(): Promise; }; internals.client = oldClient; internals.routes.set("session-a\0/invalidation-a", { route: oldRouteA, generation: 0 }); @@ -1013,7 +1025,7 @@ describe("SubcModuleTransport", () => { }); it("coalesces concurrent route opens for the same session and project", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; const routeOpenStarted = deferred(); const releaseRouteOpen = deferred(); @@ -1026,9 +1038,9 @@ describe("SubcModuleTransport", () => { return route; }, closeRoute: async () => undefined, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; ensureRoute: ( sessionId: string, projectRoot: string, @@ -1049,7 +1061,7 @@ describe("SubcModuleTransport", () => { }); it("keeps the aggregate queued-call ceiling across independent session lanes", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; const releaseActiveCalls = deferred(); const allActiveCallsStarted = deferred(); @@ -1063,11 +1075,11 @@ describe("SubcModuleTransport", () => { } return { result: { ok: true } }; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; ensureRoute: (sessionId: string) => Promise<{ - client: SubcClient; + client: McHostClient; route: RouteHandle; routeKey: string; generation: number; @@ -1115,7 +1127,7 @@ describe("SubcModuleTransport", () => { }); it("keeps wrapup and live status calls beyond a 20-second round without raising the generic deadline", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const route = { channel: 7, epoch: 77 } as RouteHandle; let releaseWrapup: (() => void) | undefined; let markWrapupStarted: (() => void) | undefined; @@ -1135,11 +1147,11 @@ describe("SubcModuleTransport", () => { } return { result: { ok: true } }; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; ensureRoute: () => Promise<{ - client: SubcClient; + client: McHostClient; route: RouteHandle; routeKey: string; generation: number; @@ -1182,7 +1194,7 @@ describe("SubcModuleTransport", () => { }); it("does not reuse a route cached under an earlier connection generation", async () => { - const transport = new SubcModuleTransport("unused-connection-file"); + const transport = new McHostModuleTransport("unused-connection-file"); const oldRoute = { channel: 7, epoch: 77 } as RouteHandle; const newRoute = { channel: 8, epoch: 88 } as RouteHandle; let routeOpenCount = 0; @@ -1191,11 +1203,11 @@ describe("SubcModuleTransport", () => { routeOpenCount += 1; return newRoute; }, - } as unknown as SubcClient; + } as unknown as McHostClient; const projectRoot = "/module-transport-generation-test-root"; const routeKey = `session-generation\0${projectRoot}`; const internals = transport as unknown as { - client: SubcClient | null; + client: McHostClient | null; connectionGeneration: number; routes: Map; ensureRoute: ( @@ -1223,7 +1235,7 @@ async function nthConnection(peer: FakePeer, n: number): Promise { it("a request rejecting after the deadline lost the race never raises an unhandled rejection", async () => { - const transport = new SubcModuleTransport("/nonexistent-connection-file"); + const transport = new McHostModuleTransport("/nonexistent-connection-file"); const unhandled: unknown[] = []; const onUnhandled = (error: unknown) => { unhandled.push(error); diff --git a/packages/plugin/src/hooks/magic-context/module-transport.ts b/packages/plugin/src/hooks/magic-context/module-transport.ts index 77cdb446e..8f1144b28 100644 --- a/packages/plugin/src/hooks/magic-context/module-transport.ts +++ b/packages/plugin/src/hooks/magic-context/module-transport.ts @@ -13,15 +13,15 @@ import { type BindIdentity, Deadline, isConsumerReconnectTransient, - isSubcCallError, + isMcHostCallError, Priority, type RouteHandle, type RouteTarget, SocketClosedError, SocketTimeoutError, StaleRouteHandleError, - SubcCallError, - SubcClient, + McHostCallError, + McHostClient, } from "../../shared/mc-host-client"; import { isRecord } from "../../shared/record-type-guard"; @@ -101,7 +101,7 @@ function isConnectionFailure(error: unknown): boolean { "request_deadline", "deadline_exceeded_no_drop_observed", "connection_dropped", - "SUBC_CONNECTION_BACKOFF", + "MC_HOST_CONNECTION_BACKOFF", ].includes(code) || /\bclient closed\b|\bconnection closed\b|\bclosed the connection\b/i.test(message) ); @@ -109,12 +109,12 @@ function isConnectionFailure(error: unknown): boolean { } /** - * `SubcCallError.kind` recognized via the shared cross-bundle check, which + * `McHostCallError.kind` recognized via the shared cross-bundle check, which * requires a real `Error` carrying the wire-visible name. The kind is still * validated at runtime because a foreign bundle copy's field is untyped. */ -function subcCallErrorKind(error: unknown): SubcCallError["kind"] | undefined { - if (!isSubcCallError(error)) return undefined; +function mcHostCallErrorKind(error: unknown): McHostCallError["kind"] | undefined { + if (!isMcHostCallError(error)) return undefined; const kind: unknown = error.kind; return kind === "not_sent" || kind === "outcome_unknown" || kind === "terminal" ? kind @@ -137,7 +137,7 @@ function isStaleRouteHandleFailure(error: unknown): boolean { /** The bounded cleanup ticket the facade attaches when a caller abort races a possible send. */ function cleanupTicketOf(error: unknown): Promise | null { - if (!isRecord(error) || error.name !== "SubcCallError") return null; + if (!isRecord(error) || error.name !== "McHostCallError") return null; const cleanup = (error as { cleanup?: unknown }).cleanup; return cleanup instanceof Promise ? (cleanup as Promise) : null; } @@ -148,7 +148,7 @@ interface CachedRoute { } interface EnsuredRoute { - client: SubcClient; + client: McHostClient; route: RouteHandle; routeKey: string; generation: number; @@ -187,19 +187,19 @@ interface SerialLane { } interface OpeningRoute { - client: SubcClient; + client: McHostClient; generation: number; /** Set by `closeSession` while the open is in flight; the open must not cache its route. */ closed: boolean; promise: Promise; } -export class SubcModuleTransport { +export class McHostModuleTransport { private readonly connectionFile: string; private readonly moduleId: string; private readonly requestTimeoutMs: number; private readonly routeSessionPrefix: string; - private client: SubcClient | null = null; + private client: McHostClient | null = null; private routes = new Map(); private routeOpenings = new Map(); private canonicalRootCache = new Map(); @@ -209,7 +209,7 @@ export class SubcModuleTransport { private queuedLaneWaiters = 0; private wrapupSessions = new Map(); private nextProbeMs = 0; - private connectionPromise: Promise | null = null; + private connectionPromise: Promise | null = null; private authorityProjectRoot = ""; /** * Filesystem root used to bind authority/mirror routes. Authority request @@ -544,7 +544,7 @@ export class SubcModuleTransport { // The body may be on the wire: a local deadline after // request invocation is a possible send, never not_sent. () => - new SubcCallError( + new McHostCallError( "outcome_unknown", "module transport deadline expired waiting for the module response", "request_deadline", @@ -561,7 +561,7 @@ export class SubcModuleTransport { return response; } catch (error) { cleanupTicket = cleanupTicketOf(error); - const kind = subcCallErrorKind(error); + const kind = mcHostCallErrorKind(error); const callerAborted = args.signal?.aborted === true; // Host-proven no-dispatch (wire doc 10.2): evict and retry once. const unknownChannel = @@ -695,6 +695,8 @@ export class SubcModuleTransport { body, ); if (!isRecord(response.authority)) throw new Error("authority.prepare omitted authority"); + // SAFETY: casting assumes the module response carries the + // AuthorityStatus fields; isRecord only proves it is an object. return { authority: response.authority as unknown as AuthorityStatus }; } @@ -720,7 +722,7 @@ export class SubcModuleTransport { async authorityDrain(args: Record): Promise { this.authorityProjectRoot = String(args.project ?? this.authorityProjectRoot); const method = String(args.method ?? "authority.drain.step") as Parameters< - SubcModuleTransport["authorityRequest"] + McHostModuleTransport["authorityRequest"] >[2]; const { projectRoot, ...body } = args; const response = await this.authorityRequest( @@ -730,6 +732,8 @@ export class SubcModuleTransport { body, ); if (isRecord(response.authority)) { + // SAFETY: casting assumes the module response carries the + // AuthorityStatus fields; isRecord only proves it is an object. return { authority: response.authority as unknown as AuthorityStatus }; } if (typeof response.code === "string") { @@ -756,6 +760,8 @@ export class SubcModuleTransport { body, ); if (!isRecord(response.page)) throw new Error("mirror.pull omitted page"); + // SAFETY: casting assumes the module response carries the + // ChangefeedPage fields; isRecord only proves it is an object. return { page: response.page as unknown as ChangefeedPage }; } @@ -830,6 +836,7 @@ export class SubcModuleTransport { client, generation, closed: false, + // SAFETY: placeholder for two-phase construction. commentlint: allow(JUDGE) promise: undefined as unknown as Promise, }; routeOpening.promise = (async (): Promise => { @@ -897,35 +904,35 @@ export class SubcModuleTransport { return resolved; } - private connectClient(deadline?: Deadline): Promise { + private connectClient(deadline?: Deadline): Promise { // Derive the handshake stage from the operation deadline without ever // extending the preserved 2-second handshake budget (plan KTD5). const handshakeTimeoutMs = deadline ? Math.max(1, deadline.stageBudgetMs(HANDSHAKE_TIMEOUT_MS)) : HANDSHAKE_TIMEOUT_MS; - return SubcClient.connect({ + return McHostClient.connect({ connectionFile: this.connectionFile, handshakeTimeoutMs, }); } - private async ensureConnected(deadline?: Deadline): Promise { + private async ensureConnected(deadline?: Deadline): Promise { if (this.client) return this.client; if (this.connectionPromise) return await this.connectionPromise; const now = Date.now(); if (now < this.nextProbeMs) { const error = new Error( - `subc connection backoff active until ${this.nextProbeMs}`, + `mc-host connection backoff active until ${this.nextProbeMs}`, ) as Error & { code?: string; }; - error.code = "SUBC_CONNECTION_BACKOFF"; + error.code = "MC_HOST_CONNECTION_BACKOFF"; throw error; } const generation = this.connectionGeneration; - const connecting = (async (): Promise => { - let candidate: SubcClient | null = null; + const connecting = (async (): Promise => { + let candidate: McHostClient | null = null; try { candidate = await this.connectClient(deadline); if (generation !== this.connectionGeneration) { @@ -953,7 +960,7 @@ export class SubcModuleTransport { } } - private invalidateConnection(client: SubcClient | null = this.client): void { + private invalidateConnection(client: McHostClient | null = this.client): void { if (client && this.client !== client) return; this.connectionGeneration += 1; this.invalidateStateSyncCapabilities(); diff --git a/packages/plugin/src/hooks/magic-context/rust-mode-transform.test.ts b/packages/plugin/src/hooks/magic-context/rust-mode-transform.test.ts index 819319fda..977e2b5db 100644 --- a/packages/plugin/src/hooks/magic-context/rust-mode-transform.test.ts +++ b/packages/plugin/src/hooks/magic-context/rust-mode-transform.test.ts @@ -29,7 +29,7 @@ import { import { createMessagesTransformHandler } from "../../plugin/messages-transform"; import { ABSOLUTE_EMERGENCY_PERCENTAGE } from "../../shared/escalation-bands"; import * as logger from "../../shared/logger"; -import { SubcCallError } from "../../shared/mc-host-client"; +import { McHostCallError } from "../../shared/mc-host-client"; import { promptSurfaceConfigIdentity } from "../../shared/prompt-surface"; import { Database, withPrivilegedWriter } from "../../shared/sqlite"; import { closeQuietly } from "../../shared/sqlite-helpers"; @@ -1792,7 +1792,7 @@ describe("Rust mode authority adapter", () => { if (page.transform_page_index === 1) { // Possible-send failures throw; they never surface as a // typed generation-change result. - throw new SubcCallError( + throw new McHostCallError( "outcome_unknown", "connection dropped after a possible send", "connection_dropped", diff --git a/packages/plugin/src/index.ts b/packages/plugin/src/index.ts index 8c103944a..43eadc671 100644 --- a/packages/plugin/src/index.ts +++ b/packages/plugin/src/index.ts @@ -40,7 +40,7 @@ import { HISTORIAN_EDITOR_SYSTEM_PROMPT, } from "./hooks/magic-context/compartment-prompt"; import { createLiveSessionState } from "./hooks/magic-context/live-session-state"; -import { SubcModuleTransport } from "./hooks/magic-context/module-transport"; +import { McHostModuleTransport } from "./hooks/magic-context/module-transport"; import { preloadTokenizer } from "./hooks/magic-context/read-session-formatting"; import type { RustModeModuleClient } from "./hooks/magic-context/rust-mode-transform"; import { beginBootQuietPeriod, scheduleAfterBootQuiet } from "./plugin/boot-quiet"; @@ -211,7 +211,7 @@ const server: Plugin = async (ctx) => { const liveSessionState = createLiveSessionState(); const rustModeModuleClient: RustModeModuleClient | undefined = - pluginConfig.transform_mode === "rust" ? new SubcModuleTransport() : undefined; + pluginConfig.transform_mode === "rust" ? new McHostModuleTransport() : undefined; const hooks = await createSessionHooksAsync({ ctx, @@ -631,6 +631,7 @@ const server: Plugin = async (ctx) => { ); }, }), + // SAFETY: wrapper matches the hook's runtime call shape. commentlint: allow(JUDGE) "experimental.chat.messages.transform": createMessagesTransformHandler({ magicContext: magicContextRuntime.magicContext, getMagicContext: () => magicContextRuntime.magicContext, diff --git a/packages/plugin/src/plugin/dream-timer-module-client.ts b/packages/plugin/src/plugin/dream-timer-module-client.ts index dcbfd11f6..748afadc6 100644 --- a/packages/plugin/src/plugin/dream-timer-module-client.ts +++ b/packages/plugin/src/plugin/dream-timer-module-client.ts @@ -16,7 +16,7 @@ export type DreamTimerModuleClient = ClassifyModuleClient & { /** * Adapt the Rust transport without extracting methods from its class instance. - * Subc transports read instance routing state, so every forwarded call must retain `this`. + * mc-host transports read instance routing state, so every forwarded call must retain `this`. */ export function createDreamTimerModuleClient( moduleClient: RustModeModuleClient | undefined, diff --git a/packages/plugin/src/plugin/embedding-routing.test.ts b/packages/plugin/src/plugin/embedding-routing.test.ts index 8db749ecb..5b8e30709 100644 --- a/packages/plugin/src/plugin/embedding-routing.test.ts +++ b/packages/plugin/src/plugin/embedding-routing.test.ts @@ -19,7 +19,8 @@ describe("embedding routing", () => { model: "Xenova/bge-small-en-v1.5", }); expect(routing.primary).not.toHaveProperty("fallback_provider"); - expect(config.subc?.connection_file).toBe(`${homedir()}/run/subc.json`); + const { subc } = config; + expect(subc?.connection_file).toBe(`${homedir()}/run/subc.json`); expect(routing.warnings.some((warning) => warning.includes("Synapse"))).toBe(true); }); diff --git a/packages/plugin/src/plugin/embedding-routing.ts b/packages/plugin/src/plugin/embedding-routing.ts index fb2d6e9eb..d3755ec86 100644 --- a/packages/plugin/src/plugin/embedding-routing.ts +++ b/packages/plugin/src/plugin/embedding-routing.ts @@ -53,6 +53,7 @@ function fallbackConfig( config: EmbeddingConfig, provider: EmbeddingFallbackProvider | undefined, ): EmbeddingConfig { + // SAFETY: each fallback field is narrowed before use. const raw = config as unknown as Record; const model = typeof raw.model === "string" ? raw.model.trim() : ""; const endpoint = typeof raw.endpoint === "string" ? raw.endpoint.trim() : ""; @@ -176,7 +177,7 @@ export async function resolveEmbeddingRouting(args: { session?: string; }): Promise { const config = args.config.embedding; - const subc = args.config.subc; + const { subc } = args.config; const shadowEnabled = args.config.shadow_embedding?.enabled === true; const warnings: string[] = []; diff --git a/packages/plugin/src/shared/mc-host-client/client.test.ts b/packages/plugin/src/shared/mc-host-client/client.test.ts index 21a9c4bd6..6802357f4 100644 --- a/packages/plugin/src/shared/mc-host-client/client.test.ts +++ b/packages/plugin/src/shared/mc-host-client/client.test.ts @@ -6,11 +6,11 @@ import { setTimeout as delay } from "node:timers/promises"; import { connectionFileExists, isConsumerReconnectTransient, - SubcClient, - type SubcClientOptions, - type SubcDiagnosticsEvent, + McHostClient, + type McHostClientOptions, + type McHostDiagnosticsEvent, } from "./client"; -import { SocketClosedError, SubcCallError, SubcError } from "./errors"; +import { isMcHostCallError, McHostCallError, McHostClientError, SocketClosedError } from "./errors"; import type { RouteHandle } from "./route-handle"; import { StaleRouteHandleError } from "./route-handle"; import { @@ -40,7 +40,7 @@ const TOOL_TARGET: RouteTarget = { kind: "tool_provider", module_id: "magic-cont let tmpDir = ""; let fileCounter = 0; let peers: FakePeer[] = []; -let clients: SubcClient[] = []; +let clients: McHostClient[] = []; let savedModuleId: string | undefined; let savedLaunchNonce: string | undefined; @@ -88,15 +88,15 @@ function freshFilePath(): string { interface ConnectedHarness { peer: FakePeer; conn: FakePeerConnection; - client: SubcClient; + client: McHostClient; filePath: string; } -async function connected(overrides: Partial = {}): Promise { +async function connected(overrides: Partial = {}): Promise { const peer = await startPeer(); const filePath = freshFilePath(); await writeConnectionFile(filePath, peer); - const client = await SubcClient.connect({ + const client = await McHostClient.connect({ connectionFile: filePath, shutdownDeadlineMs: 1_000, ...overrides, @@ -114,11 +114,11 @@ async function nthConnection(peer: FakePeer, n: number): Promise { @@ -269,7 +269,7 @@ function stallNextConnection(peer: FakePeer): Promise { const isRoutedFrame = (frame: PeerFrame): boolean => frame.ty === PeerFrameType.Request && frame.channel !== 0; -describe("SubcClient facade", () => { +describe("McHostClient facade", () => { test("completes tagged catalog, route open, opaque JSON request, and route Goodbye", async () => { const { client, conn } = await connected(); const cursor = frameCursor(conn); @@ -355,7 +355,7 @@ describe("SubcClient facade", () => { expectCallError(await rejection(openPromise2), "terminal", "malformed_control_response"); }); - test("canonical Error body becomes a terminal SubcCallError with its stable code", async () => { + test("canonical Error body becomes a terminal McHostCallError with its stable code", async () => { const { client, conn } = await connected(); const cursor = frameCursor(conn); @@ -392,8 +392,8 @@ describe("SubcClient facade", () => { const open1 = await cursor.next(isRouteOpen); await sendErrorBody(conn, open1.corr, "artifact_invalid"); - const error = (await callPromise.catch((e) => e)) as SubcCallError; - expect(error).toBeInstanceOf(SubcCallError); + const error = (await callPromise.catch((e) => e)) as McHostCallError; + expect(error).toBeInstanceOf(McHostCallError); expect(error.kind).toBe("terminal"); expect(error.code).toBe("artifact_invalid"); // artifact_invalid must not enter the momentary-rejection retry loop. @@ -473,7 +473,7 @@ describe("SubcClient facade", () => { }); test("ambiguous route.open sends no Cancel, retires the generation, and a late response is fenced", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn, peer } = await connected({ routeOpenDeadlineMs: 250, diagnostics: (event) => events.push(event), @@ -528,7 +528,7 @@ describe("SubcClient facade", () => { }); test("failed cleanup Goodbye enqueue retires the generation", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn } = await connected({ shutdownDeadlineMs: 2_000, generationOptions: { controlReserveFrames: 0 }, @@ -552,7 +552,7 @@ describe("SubcClient facade", () => { }); test("reconnects after credential rotation, reauthenticates, and rejects stale handles", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const harness = await connected({ identity: IDENTITY, diagnostics: (event) => events.push(event), @@ -800,7 +800,7 @@ describe("SubcClient facade", () => { }); test("abort cleanup deadline expiry retires the generation", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn } = await connected({ generationOptions: { cleanupTicketMs: 100 }, diagnostics: (event) => events.push(event), @@ -821,7 +821,7 @@ describe("SubcClient facade", () => { }); test("diagnostics events are frozen, redacted, and exception-isolated", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn } = await connected({ diagnostics: (event) => { events.push(event); @@ -846,9 +846,9 @@ describe("SubcClient facade", () => { "dispatch", "parse", ]) { - expect(types.has(required as SubcDiagnosticsEvent["type"])).toBe(true); + expect(types.has(required as McHostDiagnosticsEvent["type"])).toBe(true); } - const connectedEvent = events.find((e) => e.type === "connected") as SubcDiagnosticsEvent; + const connectedEvent = events.find((e) => e.type === "connected") as McHostDiagnosticsEvent; expect(connectedEvent.daemonVer).toBe("fake-peer/0.0.1"); expect(connectedEvent.pid).toBe(process.pid); expect(connectedEvent.transport).toBe("tcp"); @@ -881,7 +881,7 @@ describe("SubcClient facade", () => { }); test("diagnostics are rate-bounded: excess events are dropped, not protocol work", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn } = await connected({ maxDiagnosticEventsPerSecond: 3, diagnostics: (event) => events.push(event), @@ -915,12 +915,12 @@ describe("SubcClient facade", () => { await conn.closed; const afterClose = await rejection(client.catalogList()); - expect(afterClose).toBeInstanceOf(SubcError); - expect((afterClose as SubcError).code).toBe("client_closed"); + expect(afterClose).toBeInstanceOf(McHostClientError); + expect((afterClose as McHostClientError).code).toBe("client_closed"); }); test("reconnect and managed route open are single-flight across concurrent calls", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn, peer } = await connected({ identity: IDENTITY, diagnostics: (event) => events.push(event), @@ -966,14 +966,14 @@ describe("SubcClient facade", () => { describe("deadline-independent setup coalescing", () => { interface ReconnectHarness extends ConnectedHarness { - events: SubcDiagnosticsEvent[]; + events: McHostDiagnosticsEvent[]; } /** Connect, then retire the first generation so the next call redials. */ async function retiredHarness( - overrides: Partial = {}, + overrides: Partial = {}, ): Promise { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const harness = await connected({ identity: IDENTITY, diagnostics: (event) => events.push(event), @@ -1478,7 +1478,7 @@ describe("deadline-independent setup coalescing", () => { }); describe("transport negotiation", () => { - const GRANT_TOKEN = "00112233445566778899aabbccddeeff"; + const GRANT_TOKEN = "00112233445566778899aabbccddeeff"; // gitleaks:allow synthetic protocol vector const isNegotiate = (frame: PeerFrame): boolean => isControlOp(frame, "transport.negotiate"); function negotiateResponder(makeBody: (frame: PeerFrame) => unknown): PeerNegotiateResponder { @@ -1536,14 +1536,14 @@ describe("transport negotiation", () => { } async function connectRejected( - overrides: Partial & { peer?: FakePeer } = {}, + overrides: Partial & { peer?: FakePeer } = {}, ): Promise<{ peer: FakePeer; error: unknown; filePath: string }> { const peer = overrides.peer ?? (await startPeer()); const filePath = freshFilePath(); await writeConnectionFile(filePath, peer); const { peer: _peer, ...options } = overrides; const error = await rejection( - SubcClient.connect({ + McHostClient.connect({ connectionFile: filePath, handshakeTimeoutMs: 500, ...options, @@ -1557,7 +1557,7 @@ describe("transport negotiation", () => { const filePath = freshFilePath(); await writeConnectionFile(filePath, peer); let settled = false; - const connectPromise = SubcClient.connect({ connectionFile: filePath }).then((client) => { + const connectPromise = McHostClient.connect({ connectionFile: filePath }).then((client) => { settled = true; clients.push(client); return client; @@ -1587,37 +1587,26 @@ describe("transport negotiation", () => { expect(await catalogPromise).toEqual([]); }); - test("AE2: legacy unsupported_operation keeps the same TCP generation without a second negotiation", async () => { - const events: SubcDiagnosticsEvent[] = []; + test("AE2: an exact legacy unsupported_operation terminal fails closed without TCP continuation", async () => { + const events: McHostDiagnosticsEvent[] = []; const peer = await startPeer({ negotiate: "unsupported-op" }); - const filePath = freshFilePath(); - await writeConnectionFile(filePath, peer); - const client = await SubcClient.connect({ - connectionFile: filePath, + const { error } = await connectRejected({ + peer, diagnostics: (event) => events.push(event), }); - clients.push(client); - const conn = await peer.waitForConnection(); - const cursor = frameCursor(conn); - await cursor.next(isNegotiate); - - const openPromise = client.routeOpen(TOOL_TARGET, IDENTITY); - const openFrame = await cursor.next(isRouteOpen); - await sendRouteOpenOk(conn, openFrame.corr, 7, 1); - const handle = await openPromise; - const requestPromise = client.request(handle, { ping: 1 }); - const requestFrame = await cursor.next(isRoutedRequest(7)); - await sendResponse(conn, requestFrame.corr, { pong: 1 }, 7, 1); - expect(await requestPromise).toEqual({ pong: 1 }); - + expectCallError(error, "terminal", "negotiation_failed"); + await waitUntil(() => + events.some((e) => e.type === "retired" && e.reason === "negotiation_failed"), + ); + expect(events.some((e) => e.type === "connected")).toBe(false); + const conn = peer.connections[0] as FakePeerConnection; + await conn.closed; expect(peer.connections.length).toBe(1); expect(conn.frames.filter(isNegotiate).length).toBe(1); - const connectedEvent = events.find((e) => e.type === "connected") as SubcDiagnosticsEvent; - expect(connectedEvent.transport).toBe("tcp"); }); test("AE3: capability mismatch selects sticky TCP; reconnect runs one fresh flight", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const provider = createFakePairedProvider(); const peer = await startPeer({ negotiate: negotiateResponder(() => ({ @@ -1629,7 +1618,7 @@ describe("transport negotiation", () => { }); const filePath = freshFilePath(); await writeConnectionFile(filePath, peer); - const client = await SubcClient.connect({ + const client = await McHostClient.connect({ connectionFile: filePath, transportProviders: [provider], diagnostics: (event) => events.push(event), @@ -1646,7 +1635,7 @@ describe("transport negotiation", () => { { transport: "tcp", capability_version: 1 }, ], }); - const connectedEvent = events.find((e) => e.type === "connected") as SubcDiagnosticsEvent; + const connectedEvent = events.find((e) => e.type === "connected") as McHostDiagnosticsEvent; expect(connectedEvent.transport).toBe("tcp"); expect(connectedEvent.fallbackReason).toBe("capability_version_mismatch"); @@ -1673,7 +1662,7 @@ describe("transport negotiation", () => { }); test("concurrent callers share one connection and one negotiation", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn, peer } = await connected({ identity: IDENTITY, diagnostics: (event) => events.push(event), @@ -1727,13 +1716,13 @@ describe("transport negotiation", () => { }); } }); - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const peer = await startPeer({ negotiate: negotiateResponder(() => grantBody(descriptor)), }); const filePath = freshFilePath(); await writeConnectionFile(filePath, peer); - const client = await SubcClient.connect({ + const client = await McHostClient.connect({ connectionFile: filePath, transportProviders: [provider], diagnostics: (event) => events.push(event), @@ -1745,7 +1734,7 @@ describe("transport negotiation", () => { const host = provider.host; expect(host.frames.map((f) => f.header.corr)).toEqual([1n, 2n]); - const connectedEvent = events.find((e) => e.type === "connected") as SubcDiagnosticsEvent; + const connectedEvent = events.find((e) => e.type === "connected") as McHostDiagnosticsEvent; expect(connectedEvent.transport).toBe("fake.shm"); // Promotion retires the bootstrap internally; that handoff must not // surface as a client-level `retired` event next to `connected`. @@ -1836,8 +1825,8 @@ describe("transport negotiation", () => { }); } - test("KTD6: a non-legacy terminal error rejects connect fail-closed without TCP fallback", async () => { - const events: SubcDiagnosticsEvent[] = []; + test("KTD7: a terminal error rejects connect fail-closed without TCP fallback", async () => { + const events: McHostDiagnosticsEvent[] = []; const peer = await startPeer({ negotiate: (frame, conn) => void sendErrorBody(conn, frame.corr, "internal_error"), }); @@ -1859,11 +1848,7 @@ describe("transport negotiation", () => { expect(conn.frames.filter(isNegotiate).length).toBe(1); }); - test("KTD6: a canonical server_busy negotiation terminal fails closed without TCP fallback", async () => { - // Wire doc §7.7.3: the exact legacy `unsupported_operation` terminal - // is the only Error-based continuation evidence. A compliant - // negotiation-aware host may reject any control request before - // dispatch under load, so `server_busy` is not legacy proof. + test("KTD7: a canonical server_busy negotiation terminal fails closed without TCP fallback", async () => { const peer = await startPeer({ negotiate: (frame, conn) => void sendErrorBody(conn, frame.corr, "server_busy"), }); @@ -1875,9 +1860,7 @@ describe("transport negotiation", () => { expect(conn.frames.filter(isNegotiate).length).toBe(1); }); - test("KTD6: a noncanonical unsupported_operation terminal fails closed without TCP fallback", async () => { - // Extra fields disqualify the terminal as legacy evidence: only the - // byte-exact `{code, message}` Error body may select TCP fallback. + test("KTD7: a noncanonical unsupported_operation terminal fails closed without TCP fallback", async () => { const peer = await startPeer({ negotiate: (frame, conn) => void conn.send({ @@ -1991,7 +1974,7 @@ describe("transport negotiation", () => { const provider = createFakePairedProvider({ startError: new Error("provider-sentinel-9b2c"), }); - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const peer = await startPeer({ negotiate: negotiateResponder(() => grantBody({ secret: "descriptor-sentinel-7f3a" })), }); @@ -2016,7 +1999,7 @@ describe("transport negotiation", () => { const provider = createFakePairedProvider({ connectError: new Error("provider-sentinel-3d1e"), }); - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const peer = await startPeer({ negotiate: negotiateResponder(() => grantBody({})), }); @@ -2050,7 +2033,7 @@ describe("transport negotiation", () => { }); await writeFile(filePath, json, { mode: 0o600 }); const error = await rejection( - SubcClient.connect({ connectionFile: filePath, transportProviders: [provider] }), + McHostClient.connect({ connectionFile: filePath, transportProviders: [provider] }), ); expect((error as Error).name).toBe("ConnectionFileError"); await delay(30); @@ -2059,7 +2042,7 @@ describe("transport negotiation", () => { }); test("owner close during negotiation exposes no connection and launches no replacement", async () => { - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn, peer } = await connected({ handshakeTimeoutMs: 400, diagnostics: (event) => events.push(event), @@ -2085,7 +2068,7 @@ describe("transport negotiation", () => { test("owner close during activation reaps candidate and bootstrap without replacement", async () => { const provider = createFakePairedProvider(); provider.host.onFrame = () => {}; - const events: SubcDiagnosticsEvent[] = []; + const events: McHostDiagnosticsEvent[] = []; const { client, conn, peer } = await connected({ handshakeTimeoutMs: 400, transportProviders: [provider], @@ -2118,10 +2101,12 @@ describe("facade helpers", () => { test("isConsumerReconnectTransient keeps npm-compatible semantics", () => { expect(isConsumerReconnectTransient(new SocketClosedError("gone"))).toBe(true); - expect(isConsumerReconnectTransient(new SubcCallError("not_sent", "x"))).toBe(true); - expect(isConsumerReconnectTransient(new SubcCallError("outcome_unknown", "x"))).toBe(true); - expect(isConsumerReconnectTransient(new SubcCallError("terminal", "x"))).toBe(false); - expect(isConsumerReconnectTransient(new SubcError("nope"))).toBe(false); + expect(isConsumerReconnectTransient(new McHostCallError("not_sent", "x"))).toBe(true); + expect(isConsumerReconnectTransient(new McHostCallError("outcome_unknown", "x"))).toBe( + true, + ); + expect(isConsumerReconnectTransient(new McHostCallError("terminal", "x"))).toBe(false); + expect(isConsumerReconnectTransient(new McHostClientError("nope"))).toBe(false); expect( isConsumerReconnectTransient( Object.assign(new Error("refused"), { code: "ECONNREFUSED" }), @@ -2129,4 +2114,40 @@ describe("facade helpers", () => { ).toBe(true); expect(isConsumerReconnectTransient(new Error("plain"))).toBe(false); }); + + test("isMcHostCallError recognizes a second bundled copy and rejects old runtime names", () => { + // `SecondCopyCallError` models a separately bundled copy that fails + // `instanceof McHostCallError`. + class SecondCopyCallError extends Error { + constructor( + readonly kind: string, + message: string, + readonly code?: string, + ) { + super(message); + this.name = "McHostCallError"; + } + } + expect(isMcHostCallError(new McHostCallError("terminal", "same bundle", "c"))).toBe(true); + expect(isMcHostCallError(new SecondCopyCallError("not_sent", "x"))).toBe(true); + expect(isMcHostCallError(new SecondCopyCallError("outcome_unknown", "x", "code"))).toBe( + true, + ); + expect(isMcHostCallError(new SecondCopyCallError("terminal", "x"))).toBe(true); + + const oldName = new SecondCopyCallError("terminal", "x", "c"); + // Assembled from parts; boundary tests reject the joined spelling. commentlint: allow(JUDGE) + oldName.name = ["Subc", "CallError"].join(""); + expect(isMcHostCallError(oldName)).toBe(false); + + expect(isMcHostCallError(new SecondCopyCallError("bogus_kind", "x"))).toBe(false); + expect( + isMcHostCallError(Object.assign(new SecondCopyCallError("terminal", "x"), { code: 7 })), + ).toBe(false); + expect(isMcHostCallError(new Error("unrelated"))).toBe(false); + expect( + isMcHostCallError({ name: "McHostCallError", kind: "terminal", message: "plain" }), + ).toBe(false); + expect(isMcHostCallError(null)).toBe(false); + }); }); diff --git a/packages/plugin/src/shared/mc-host-client/client.ts b/packages/plugin/src/shared/mc-host-client/client.ts index 18abcec36..3ac6c25e1 100644 --- a/packages/plugin/src/shared/mc-host-client/client.ts +++ b/packages/plugin/src/shared/mc-host-client/client.ts @@ -2,7 +2,7 @@ * Thin routed and managed consumer facade over the connection-generation * engine. * - * `SubcClient` owns connection coalescing (single-flight connect), reconnect + * `McHostClient` owns connection coalescing (single-flight connect), reconnect * after generation retirement (reread the connection file plus full reauth), * the managed-route cache, control-plane response validation, and bounded * redacted diagnostics. The generation layer below never imports this file. @@ -30,7 +30,12 @@ import { readConnectionFile, } from "./connection-file"; import { armExpiryTimer, Deadline, type MonotonicClock } from "./deadline"; -import { isSubcCallError, SocketTimeoutError, SubcCallError, SubcError } from "./errors"; +import { + isMcHostCallError, + SocketTimeoutError, + McHostCallError, + McHostClientError, +} from "./errors"; import { flagsBinary } from "./protocol"; import { belongsToConnection, @@ -48,7 +53,6 @@ import { encodeActivateRequest, encodeNegotiateRequest, type FallbackReason, - isLegacyFallbackTerminalBody, NEGOTIATION_VERSION, type NegotiateResponse, NegotiationError, @@ -58,7 +62,6 @@ import { type ClientTransportProvider, ClientTransportRegistry, sanitizedCandidateFactory, - TCP_CAPABILITY_VERSION, } from "./transport-provider"; import type { BindIdentity, @@ -100,7 +103,7 @@ const DEFAULT_MANAGED_TARGET_KIND: ManagedRouteKind = "management_surface"; * byte counts, and connection metadata only — never key, proof, nonce, * body bytes, or full bind identity. */ -export interface SubcDiagnosticsEvent { +export interface McHostDiagnosticsEvent { readonly type: ConnectionDiagnosticEvent["type"] | "connected" | "parse" | "retired"; /** Wall-clock milliseconds assigned at emission. */ readonly atMs: number; @@ -118,13 +121,13 @@ export interface SubcDiagnosticsEvent { readonly fallbackReason?: FallbackReason; } -export type SubcDiagnosticsObserver = (event: SubcDiagnosticsEvent) => void; +export type McHostDiagnosticsObserver = (event: McHostDiagnosticsEvent) => void; /** * Facade construction options. `ConnectOptions` is the consumer surface; * the rest are bounded policy knobs and injectable test seams. */ -export interface SubcClientOptions extends ConnectOptions { +export interface McHostClientOptions extends ConnectOptions { /** Injectable monotonic clock for every operation deadline. */ clock?: MonotonicClock; /** Injectable backoff sleep for deterministic retry tests. */ @@ -145,7 +148,7 @@ export interface SubcClientOptions extends ConnectOptions { * rate-bounded, and redacted; observer exceptions are swallowed and * excess events are dropped rather than blocking protocol work. */ - diagnostics?: SubcDiagnosticsObserver; + diagnostics?: McHostDiagnosticsObserver; maxDiagnosticEventsPerSecond?: number; /** Bounded generation-policy overrides for tests (queue caps, deadlines). */ generationOptions?: Partial< @@ -244,8 +247,8 @@ function connectionStageError(): SocketTimeoutError { ); } -function routeStageError(): SubcCallError { - return new SubcCallError( +function routeStageError(): McHostCallError { + return new McHostCallError( "not_sent", "route.open deadline expired before a route was opened", "deadline_expired", @@ -347,18 +350,18 @@ export async function connectionFileExists(path: string): Promise { * `kind`/`code` shape, not only `instanceof`. */ export function isConsumerReconnectTransient(err: unknown): boolean { - if (err instanceof SubcCallError) { + if (err instanceof McHostCallError) { return err.kind === "not_sent" || err.kind === "outcome_unknown"; } const name = err instanceof Error ? err.name : undefined; if (name === "SocketClosedError" || name === "SocketTimeoutError" || name === "AuthError") { return true; } - if (name === "SubcCallError") { + if (name === "McHostCallError") { const kind = (err as { kind?: unknown }).kind; return kind === "not_sent" || kind === "outcome_unknown"; } - if (name === "SubcError" || name === "ConnectionFileError") return false; + if (name === "McHostClientError" || name === "ConnectionFileError") return false; const code = errorCode(err); return ( code === "ECONNREFUSED" || @@ -373,7 +376,7 @@ export function isConsumerReconnectTransient(err: unknown): boolean { * The consumer-facing client: connect, route open, raw request, managed * call, catalog, and bounded close over one active connection generation. */ -export class SubcClient { +export class McHostClient { private readonly connectionFile: string; private readonly handshakeTimeoutMs: number; private readonly requestTimeoutMs: number; @@ -385,8 +388,8 @@ export class SubcClient { private readonly sleep: (ms: number) => Promise; private readonly trustedSymlink: boolean; private readonly connectionFileAfterOpen: (() => void | Promise) | undefined; - private readonly generationOptions: SubcClientOptions["generationOptions"]; - private readonly diagnostics: SubcDiagnosticsObserver | undefined; + private readonly generationOptions: McHostClientOptions["generationOptions"]; + private readonly diagnostics: McHostDiagnosticsObserver | undefined; private readonly maxDiagnosticEventsPerSecond: number; private readonly transportRegistry: ClientTransportRegistry; @@ -402,7 +405,7 @@ export class SubcClient { private diagWindowStartMs = 0; private diagWindowCount = 0; - private constructor(options: SubcClientOptions) { + private constructor(options: McHostClientOptions) { this.connectionFile = options.connectionFile; this.handshakeTimeoutMs = options.handshakeTimeoutMs ?? DEFAULT_HANDSHAKE_TIMEOUT_MS; this.requestTimeoutMs = options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS; @@ -426,8 +429,8 @@ export class SubcClient { * Read the connection file, dial, and authenticate under one handshake * deadline, then return the ready client. */ - static async connect(options: SubcClientOptions): Promise { - const client = new SubcClient(options); + static async connect(options: McHostClientOptions): Promise { + const client = new McHostClient(options); await client.ensureConnection(Deadline.start(client.handshakeTimeoutMs, client.clock)); return client; } @@ -504,7 +507,7 @@ export class SubcClient { deadline, options, }); - return parseResponseJson(terminal) as Response; + return parseResponseJson(terminal); } catch (error) { const err = toManagedCallError(error); const callerActive = !this.closeStarted && options.signal?.aborted !== true; @@ -533,7 +536,7 @@ export class SubcClient { const parsed = await this.controlRequest(active, bodyText, "catalog.list", deadline); const modules = parsed.modules ?? []; if (!Array.isArray(modules)) { - throw new SubcCallError( + throw new McHostCallError( "terminal", "catalog.list response carried a non-array modules field", "malformed_control_response", @@ -592,7 +595,7 @@ export class SubcClient { const stage = deadline.stage(this.handshakeTimeoutMs); const pace = makeReplacementPacer(stage, this.sleep); for (;;) { - if (this.closeStarted) throw new SubcError("client closed", "client_closed"); + if (this.closeStarted) throw new McHostClientError("client closed", "client_closed"); const active = this.active; if (active && !active.generation.isRetired()) return active; let flight = this.connecting; @@ -694,7 +697,7 @@ export class SubcClient { } if (this.closeStarted) { generation.retire("owner_close"); - throw new SubcError("client closed", "client_closed"); + throw new McHostClientError("client closed", "client_closed"); } // KTD4 stale-success: a generation that retired during its own setup // (for example a Goodbye coalesced into the final handshake chunk) @@ -716,7 +719,7 @@ export class SubcClient { } if (this.closeStarted) { generation.retire("owner_close"); - throw new SubcError("client closed", "client_closed"); + throw new McHostClientError("client closed", "client_closed"); } if (selection.kind === "grant") { try { @@ -752,11 +755,10 @@ export class SubcClient { } /** - * Send the versioned offer as the generation's first channel-0 request - * and classify the outcome under KTD6: an exact legacy - * `unsupported_operation` terminal or a strictly decoded selection - * continues; every other failure propagates so setup fails closed - * without same-generation TCP fallback. + * Send the versioned offer as the generation's first channel-0 request. + * Only a strictly decoded selection continues; every failure — including + * any wire Error terminal — propagates so setup fails closed without + * same-generation TCP fallback (KTD7). */ private async negotiateTransport( generation: ConnectionGeneration, @@ -776,33 +778,15 @@ export class SubcClient { }); } catch (error) { if ( - error instanceof SubcCallError && + error instanceof McHostCallError && error.kind === "terminal" && error.errorTerminal !== undefined ) { - if ( - !flagsBinary(error.errorTerminal.flags) && - !error.errorTerminal.streamed && - isLegacyFallbackTerminalBody(error.errorTerminal.body) - ) { - // Only the closed set of legacy Error terminals proves - // the negotiation was never dispatched (KTD6). A body - // with extra fields, a non-string message, a binary - // flag, or any other code is malformed negotiation - // content and fails closed. - return { - kind: "tcp", - selected: { - transport: TRANSPORT_TCP, - capabilityVersion: TCP_CAPABILITY_VERSION, - }, - }; - } - // Every other Error terminal fails closed with a bounded - // error: the raw body is peer-controlled and was consumed - // solely by the legacy classification above; its message - // must not enter caller-visible error graphs (R14). - throw new SubcCallError( + // Every Error terminal fails closed with a bounded error: + // the raw body is peer-controlled and its message must not + // enter caller-visible error graphs (R14). There is no + // legacy `unsupported_operation` continuation. + throw new McHostCallError( "terminal", "transport negotiation failed: host error terminal", "negotiation_failed", @@ -844,7 +828,7 @@ export class SubcClient { if (!provider) { // Unreachable through the decoder (a selection must name a sent // offer), kept as a fail-closed guard. - const failure = new SubcCallError( + const failure = new McHostCallError( "terminal", "host granted a transport with no installed provider", "negotiation_failed", @@ -927,8 +911,8 @@ export class SubcClient { } if (this.closeStarted || candidate.isRetired()) { const failure = this.closeStarted - ? new SubcError("client closed", "client_closed") - : new SubcCallError( + ? new McHostClientError("client closed", "client_closed") + : new McHostCallError( "not_sent", "candidate channel retired before promotion", "negotiation_failed", @@ -1012,7 +996,7 @@ export class SubcClient { /** * Run one request to its terminal. A wire Error terminal becomes a - * `terminal` SubcCallError with the canonical body's stable code. On a + * `terminal` McHostCallError with the canonical body's stable code. On a * caller abort, the rejection carries the cleanup ticket, and a * post-write routed abort enqueues a correlation-scoped Cancel; channel * 0 never sees Cancel (KTD9 handles it by retirement in the caller). @@ -1057,7 +1041,7 @@ export class SubcClient { } return terminal; } catch (error) { - if (cleanup !== null && error instanceof SubcCallError) { + if (cleanup !== null && error instanceof McHostCallError) { if (error.kind === "outcome_unknown" && params.channel !== 0) { generation.enqueueCancel(params.channel, params.epoch, pending.correlation); } @@ -1078,7 +1062,7 @@ export class SubcClient { ): Promise> { const body = Buffer.from(bodyText, "utf8"); if (body.length > MAX_CONTROL_BODY_LEN) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", `channel-0 control body of ${body.length} bytes exceeds the ${MAX_CONTROL_BODY_LEN}-byte cap`, "control_body_too_large", @@ -1103,7 +1087,7 @@ export class SubcClient { Array.isArray(parsed) || (parsed as { op?: unknown }).op !== expectedOp ) { - throw new SubcCallError( + throw new McHostCallError( "terminal", `control response was not a tagged ${expectedOp} object`, "malformed_control_response", @@ -1149,7 +1133,7 @@ export class SubcClient { // KTD9: an ambiguous channel-0 route.open (possible send, no // terminal) has no handle and Cancel is illegal on channel 0, // so retire the generation before any recovery. - if (error instanceof SubcCallError && error.kind === "outcome_unknown") { + if (error instanceof McHostCallError && error.kind === "outcome_unknown") { active.generation.retire("ambiguous_route_open", error); } throw error; @@ -1163,7 +1147,7 @@ export class SubcClient { } handle = createRouteHandle(channel, epoch, active.token); } catch (error) { - throw new SubcCallError( + throw new McHostCallError( "terminal", `route.open returned a malformed route handle${causeMessage(error)}`, "malformed_control_response", @@ -1174,7 +1158,7 @@ export class SubcClient { // KTD9 owner-close race: never cache the late route; best-effort // Goodbye (a failed enqueue retires the generation internally). active.generation.enqueueRouteGoodbye(handle.channel, handle.epoch); - throw new SubcCallError( + throw new McHostCallError( "not_sent", "route was closed before route.open completed", "route_closed", @@ -1195,9 +1179,9 @@ export class SubcClient { ): Promise { const identity = options.identity ?? this.defaultIdentity; if (!identity) { - throw new SubcCallError( + throw new McHostCallError( "terminal", - "managed call requires a BindIdentity in SubcClient.connect({ identity }) or call(..., { identity })", + "managed call requires a BindIdentity in McHostClient.connect({ identity }) or call(..., { identity })", "missing_identity", ); } @@ -1292,7 +1276,7 @@ export class SubcClient { }; for (;;) { if (cached.closed || this.closeStarted) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", "route was closed before route.open completed", "route_closed", @@ -1307,7 +1291,7 @@ export class SubcClient { try { active = await this.ensureConnection(deadline); } catch (error) { - if (error instanceof SubcCallError) throw error; + if (error instanceof McHostCallError) throw error; // KTD3: a snapshot that outlives its stage names the clamped // handshake budget, not the route budget, so it reconnects // like any transient setup failure; every other @@ -1321,7 +1305,7 @@ export class SubcClient { // owner's budget ran out. flight.replaceable = true; } - throw new SubcCallError( + throw new McHostCallError( transient ? "not_sent" : "terminal", `route.open could not run because connect failed${causeMessage(error)}`, errorCode(error), @@ -1340,7 +1324,7 @@ export class SubcClient { if (cached.closed) { this.liveRoutes.delete(handle.channel); active.generation.enqueueRouteGoodbye(handle.channel, handle.epoch); - throw new SubcCallError( + throw new McHostCallError( "not_sent", "route was closed before route.open completed", "route_closed", @@ -1349,8 +1333,8 @@ export class SubcClient { cached.handle = handle; return handle; } catch (error) { - if (!isSubcCallError(error)) { - throw new SubcCallError( + if (!isMcHostCallError(error)) { + throw new McHostCallError( "terminal", `route.open failed for module ${cached.target.module_id}${causeMessage(error)}`, errorCode(error), @@ -1362,7 +1346,7 @@ export class SubcClient { if (await backoff()) continue; // KTD3: the allowlisted retry budget is owner budget. flight.replaceable = true; - throw new SubcCallError( + throw new McHostCallError( "not_sent", `route.open failed for module ${cached.target.module_id}: ${error.code} (route-open retry budget exhausted)`, error.code, @@ -1427,7 +1411,7 @@ export class SubcClient { // Bounded, redacted diagnostics. // ------------------------------------------------------------------ - private emitDiagnostics(event: Omit): void { + private emitDiagnostics(event: Omit): void { const observer = this.diagnostics; if (!observer) return; const now = Date.now(); @@ -1490,27 +1474,27 @@ function routeOpenBody( ); } -/** Canonical `ErrorBody {code, message}` into a `terminal` SubcCallError. */ -function terminalFromErrorBody(body: Uint8Array): SubcCallError { +/** Canonical `ErrorBody {code, message}` into a `terminal` McHostCallError. */ +function terminalFromErrorBody(body: Uint8Array): McHostCallError { const text = Buffer.from(body).toString("utf8"); try { const parsed = JSON.parse(text) as { code?: unknown; message?: unknown }; if (typeof parsed === "object" && parsed !== null) { const code = typeof parsed.code === "string" ? parsed.code : undefined; const message = typeof parsed.message === "string" ? parsed.message : undefined; - return new SubcCallError("terminal", message ?? "subc error", code); + return new McHostCallError("terminal", message ?? "subc error", code); } } catch { // Fall through to the opaque-body form. } - return new SubcCallError("terminal", text || "subc error"); + return new McHostCallError("terminal", text || "subc error"); } -function parseResponseJson(terminal: RequestTerminal): unknown { +function parseResponseJson(terminal: RequestTerminal): Response { try { return JSON.parse(Buffer.from(terminal.body).toString("utf8")); } catch (error) { - throw new SubcCallError( + throw new McHostCallError( "terminal", "response body was not valid JSON", "invalid_response_body", @@ -1521,7 +1505,7 @@ function parseResponseJson(terminal: RequestTerminal): unknown { function wrapNegotiationError(error: unknown): Error { if (error instanceof NegotiationError) { - return new SubcCallError( + return new McHostCallError( "terminal", `transport negotiation failed: ${error.message}`, "negotiation_failed", @@ -1539,11 +1523,11 @@ function wrapNegotiationError(error: unknown): Error { */ function boundedNegotiationFailure(error: unknown): Error { if ( - error instanceof SubcCallError && + error instanceof McHostCallError && error.kind === "terminal" && error.errorTerminal !== undefined ) { - return new SubcCallError( + return new McHostCallError( "terminal", "transport negotiation failed: host error terminal", "negotiation_failed", @@ -1552,17 +1536,17 @@ function boundedNegotiationFailure(error: unknown): Error { return wrapNegotiationError(error); } -function toManagedCallError(error: unknown): SubcCallError { - if (error instanceof SubcCallError) return error; +function toManagedCallError(error: unknown): McHostCallError { + if (error instanceof McHostCallError) return error; if (error instanceof StaleRouteHandleError) { - return new SubcCallError( + return new McHostCallError( "not_sent", `managed request used a stale route handle${causeMessage(error)}`, error.code, error, ); } - return new SubcCallError( + return new McHostCallError( "terminal", `managed call failed${causeMessage(error)}`, errorCode(error), diff --git a/packages/plugin/src/shared/mc-host-client/connection-file.test.ts b/packages/plugin/src/shared/mc-host-client/connection-file.test.ts index c447d86fb..39c160645 100644 --- a/packages/plugin/src/shared/mc-host-client/connection-file.test.ts +++ b/packages/plugin/src/shared/mc-host-client/connection-file.test.ts @@ -280,15 +280,14 @@ describe("snapshot JSON validation", () => { await writeInvalid(noSchema, "invalid_schema"); }); - test("accepts an absent wire_version and rejects any non-2 value", async () => { + test("requires wire_version to be exactly 2 and rejects every other value", async () => { const absent = validJson(); delete absent.wire_version; - const filePath = freshPath("no-wire-version.json"); - await writePrivateFile(filePath, JSON.stringify(absent)); - const snapshot = await readConnectionFile(filePath, options()); - expect(snapshot.endpoint.port).toBe(43_123); + await writeInvalid(absent, "invalid_wire_version"); await writeInvalid(validJson({ wire_version: 3 }), "invalid_wire_version"); await writeInvalid(validJson({ wire_version: null }), "invalid_wire_version"); + await writeInvalid(validJson({ wire_version: "2" }), "invalid_wire_version"); + await writeInvalid(validJson({ wire_version: 1 }), "invalid_wire_version"); }); test("rejects hostnames, wildcard, IPv6, and invalid ports", async () => { diff --git a/packages/plugin/src/shared/mc-host-client/connection-file.ts b/packages/plugin/src/shared/mc-host-client/connection-file.ts index 4a9e94856..02798b751 100644 --- a/packages/plugin/src/shared/mc-host-client/connection-file.ts +++ b/packages/plugin/src/shared/mc-host-client/connection-file.ts @@ -31,7 +31,7 @@ export const MAX_CONNECTION_FILE_LEN = 65_536; export const CONNECTION_FILE_SCHEMA = 1; export const KEY_LEN = 32; export const DAEMON_ID_LEN = 16; -/** Absent means fixed v2; any present value other than 2 fails closed. */ +/** Required in every connection file; any value other than 2 fails closed. */ export const WIRE_VERSION = 2; export type ConnectionFileErrorCode = @@ -311,7 +311,7 @@ function invalid(code: ConnectionFileErrorCode, message: string): ConnectionFile /** * Validate the decoded JSON against wire doc Section 4.1: schema 1, - * absent-or-2 wire version, first endpoint exactly `127.0.0.1` with a port + * a required wire version of exactly 2, first endpoint exactly `127.0.0.1` with a port * in `1..=65535`, exactly 32 key bytes, exactly 16 daemon-ID bytes, a safe * integer PID, and a nonempty daemon version. No coercion anywhere. */ @@ -323,10 +323,10 @@ function validateSnapshotJson(parsed: unknown): ConnectionSnapshot { if (record.schema !== CONNECTION_FILE_SCHEMA) { throw invalid("invalid_schema", `connection file schema must be ${CONNECTION_FILE_SCHEMA}`); } - if ("wire_version" in record && record.wire_version !== WIRE_VERSION) { + if (record.wire_version !== WIRE_VERSION) { throw invalid( "invalid_wire_version", - `connection file wire_version must be absent or exactly ${WIRE_VERSION}`, + `connection file wire_version must be exactly ${WIRE_VERSION}`, ); } const endpoints = record.endpoints; diff --git a/packages/plugin/src/shared/mc-host-client/connection.test.ts b/packages/plugin/src/shared/mc-host-client/connection.test.ts index d4bab1989..812be570a 100644 --- a/packages/plugin/src/shared/mc-host-client/connection.test.ts +++ b/packages/plugin/src/shared/mc-host-client/connection.test.ts @@ -7,7 +7,7 @@ import { adversarialScenarios, runAdversarialScenario } from "./test-support/adv import { encodePeerFrame, type FakePeerConnection, PeerFrameType } from "./test-support/fake-peer"; import { createTrackedHarness, - expectSubcCallError as expectCallError, + expectMcHostCallError as expectCallError, rejection, type TrackedHarness, waitUntil, diff --git a/packages/plugin/src/shared/mc-host-client/connection.ts b/packages/plugin/src/shared/mc-host-client/connection.ts index bb40e3db0..9d235ba7f 100644 --- a/packages/plugin/src/shared/mc-host-client/connection.ts +++ b/packages/plugin/src/shared/mc-host-client/connection.ts @@ -21,7 +21,7 @@ import { AuthError } from "./auth"; import { armExpiryTimer, type Deadline } from "./deadline"; -import { SocketClosedError, SocketTimeoutError, SubcCallError } from "./errors"; +import { SocketClosedError, SocketTimeoutError, McHostCallError } from "./errors"; import { ByteBudget, type FrameChannelCloseReason, @@ -348,34 +348,34 @@ export class ConnectionGeneration { /** * Synchronously admit one Request to the channel's writer FIFO and * allocate its correlation with admission (KTD7), so writer-enqueue - * order equals correlation order. Throws a `not_sent` SubcCallError + * order equals correlation order. Throws a `not_sent` McHostCallError * when admission is refused; nothing was allocated or queued in that * case. */ request(params: RequestParams): PendingRequest { if (this.retiredInfo) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", `connection generation is retired (${this.retiredInfo.reason})`, "connection_retired", ); } if (this.phase !== "frames") { - throw new SubcCallError( + throw new McHostCallError( "not_sent", "connection generation has not completed setup", "connection_not_ready", ); } if (this.corrExhausted) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", "correlation space exhausted after u64::MAX; retire the generation", "correlations_exhausted", ); } if (params.deadline.isExpired()) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", "request deadline expired before queue admission", "deadline_expired", @@ -532,7 +532,7 @@ export class ConnectionGeneration { entry.callerSettled = true; if (entry.writeInvoked) { entry.reject( - new SubcCallError( + new McHostCallError( "outcome_unknown", `connection generation retired (${reason}) after a possible send`, "generation_retired", @@ -541,7 +541,7 @@ export class ConnectionGeneration { ); } else { entry.reject( - new SubcCallError( + new McHostCallError( "not_sent", `connection generation retired (${reason}) before any byte was written`, "generation_retired", @@ -732,7 +732,7 @@ export class ConnectionGeneration { } else { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "terminal", "unary request received a stream; the sequence was drained privately", "unexpected_stream", @@ -742,7 +742,7 @@ export class ConnectionGeneration { } else if (entry.mode === "unary" && entry.sawStream) { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "terminal", "unary request received a stream before its Response", "unexpected_stream", @@ -766,7 +766,7 @@ export class ConnectionGeneration { if (!entry.writeInvoked && this.cancelQueuedFrame(entry)) { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "not_sent", "route closed by host before any request byte was written", "route_gone", @@ -775,7 +775,7 @@ export class ConnectionGeneration { } else { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "outcome_unknown", "route closed by host (route Goodbye) before a matching terminal", "route_gone", @@ -862,7 +862,7 @@ export class ConnectionGeneration { if (!entry.writeInvoked && this.cancelQueuedFrame(entry)) { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "not_sent", "request deadline expired before any byte was written", "deadline_expired", @@ -871,7 +871,7 @@ export class ConnectionGeneration { } else { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "outcome_unknown", "request deadline expired after a possible send without a terminal", "deadline_expired", @@ -895,7 +895,7 @@ export class ConnectionGeneration { if (!entry.writeInvoked && this.cancelQueuedFrame(entry)) { this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "not_sent", "request aborted before any byte was written", "aborted", @@ -906,7 +906,7 @@ export class ConnectionGeneration { } this.settleCallerReject( entry, - new SubcCallError( + new McHostCallError( "outcome_unknown", "request aborted after a possible send without a terminal", "aborted", diff --git a/packages/plugin/src/shared/mc-host-client/errors.ts b/packages/plugin/src/shared/mc-host-client/errors.ts index 7f2b56a0c..60c1f930f 100644 --- a/packages/plugin/src/shared/mc-host-client/errors.ts +++ b/packages/plugin/src/shared/mc-host-client/errors.ts @@ -9,15 +9,20 @@ * Leaf module: no imports from connection or facade code. */ +const CALL_ERROR_KINDS: readonly string[] = ["not_sent", "outcome_unknown", "terminal"]; + /** - * Cross-bundle recognition of {@link SubcCallError}: `instanceof` for a - * same-bundle error, wire-visible `name` for an error thrown by a different - * bundled copy of this class. + * Cross-bundle recognition of {@link McHostCallError}. A different bundled + * copy of this class fails `instanceof`, so recognition is structural; old + * runtime names (the previous `Subc`-prefixed spellings) are deliberately + * rejected. */ -export function isSubcCallError(error: unknown): error is SubcCallError { - return ( - error instanceof SubcCallError || (error instanceof Error && error.name === "SubcCallError") - ); +export function isMcHostCallError(error: unknown): error is McHostCallError { + if (error instanceof McHostCallError) return true; + if (!(error instanceof Error) || error.name !== "McHostCallError") return false; + const { kind, code } = error as { kind?: unknown; code?: unknown }; + if (typeof kind !== "string" || !CALL_ERROR_KINDS.includes(kind)) return false; + return code === undefined || typeof code === "string"; } /** @@ -30,39 +35,38 @@ export function isSubcCallError(error: unknown): error is SubcCallError { * - `terminal`: a matching terminal Error (or non-retryable setup failure) * was observed; it applies only to that correlation. */ -export type SubcCallErrorKind = "not_sent" | "outcome_unknown" | "terminal"; +export type McHostCallErrorKind = "not_sent" | "outcome_unknown" | "terminal"; /** Managed call failure carrying send-outcome semantics. */ -export class SubcCallError extends Error { +export class McHostCallError extends Error { /** The facade attaches `cleanup` when a caller abort produces this error. */ cleanup?: Promise; /** - * Raw wire Error terminal (body plus frame flags) for callers that must - * validate the exact terminal shape — negotiation's legacy-fallback - * classification accepts only a byte-exact `unsupported_operation` - * terminal, which the parsed `code` alone cannot prove. + * Captured only during transport negotiation, which replaces + * peer-controlled Error terminals with a bounded failure before + * exposing them to callers. */ errorTerminal?: { body: Uint8Array; flags: number; streamed: boolean }; constructor( - readonly kind: SubcCallErrorKind, + readonly kind: McHostCallErrorKind, message: string, readonly code?: string, readonly cause?: unknown, ) { super(message); - this.name = "SubcCallError"; + this.name = "McHostCallError"; } } -export class SubcError extends Error { +export class McHostClientError extends Error { constructor( message: string, readonly code?: string, ) { super(message); - this.name = "SubcError"; + this.name = "McHostClientError"; } } diff --git a/packages/plugin/src/shared/mc-host-client/frame-channel.ts b/packages/plugin/src/shared/mc-host-client/frame-channel.ts index 79814e2ca..d8f683b54 100644 --- a/packages/plugin/src/shared/mc-host-client/frame-channel.ts +++ b/packages/plugin/src/shared/mc-host-client/frame-channel.ts @@ -121,7 +121,7 @@ export interface FrameChannelStats { export interface FrameChannel { /** * Synchronously admit one data frame to the single logical writer. - * Throws a `not_sent` `SubcCallError` (`writer_queue_full`, + * Throws a `not_sent` `McHostCallError` (`writer_queue_full`, * `memory_cap`, or `channel_closed`) when admission is refused; a * refusal changes no channel state. */ diff --git a/packages/plugin/src/shared/mc-host-client/index.ts b/packages/plugin/src/shared/mc-host-client/index.ts index b869e26e2..4ef55ef6b 100644 --- a/packages/plugin/src/shared/mc-host-client/index.ts +++ b/packages/plugin/src/shared/mc-host-client/index.ts @@ -1,10 +1,10 @@ export { connectionFileExists, isConsumerReconnectTransient, - SubcClient, - type SubcClientOptions, - type SubcDiagnosticsEvent, - type SubcDiagnosticsObserver, + McHostClient, + type McHostClientOptions, + type McHostDiagnosticsEvent, + type McHostDiagnosticsObserver, } from "./client"; export { armExpiryTimer, @@ -13,11 +13,11 @@ export { type MonotonicClock, } from "./deadline"; export { - isSubcCallError, + isMcHostCallError, SocketClosedError, SocketTimeoutError, - SubcCallError, - SubcError, + McHostCallError, + McHostClientError, } from "./errors"; export { RouteHandle, StaleRouteHandleError } from "./route-handle"; export { diff --git a/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts b/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts index 9f125a6b1..9c1f080a9 100644 --- a/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts +++ b/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts @@ -14,7 +14,7 @@ import { Socket } from "node:net"; import { type AuthByteIo, AuthError, type AuthResult, authenticateClient } from "./auth"; import type { Deadline } from "./deadline"; -import { SocketClosedError, SocketTimeoutError, SubcCallError } from "./errors"; +import { SocketClosedError, SocketTimeoutError, McHostCallError } from "./errors"; import { type ByteBudget, type FrameChannel, @@ -260,7 +260,7 @@ export class TcpFrameChannel implements FrameChannel { send(frame: OutboundFrame, hooks?: FrameSendHooks): FrameSendTicket { if (this.closed) { - throw new SubcCallError("not_sent", "frame channel is closed", "channel_closed"); + throw new McHostCallError("not_sent", "frame channel is closed", "channel_closed"); } // Encoding validates every header field before any state changes, // so a rejected frame can never burn a correlation upstream. @@ -278,10 +278,10 @@ export class TcpFrameChannel implements FrameChannel { this.dataFramesQueued + 1 > this.maxQueuedFrames || this.dataBytesQueued + totalBytes > this.maxQueuedBytes ) { - throw new SubcCallError("not_sent", "writer queue is full", "writer_queue_full"); + throw new McHostCallError("not_sent", "writer queue is full", "writer_queue_full"); } if (this.budget.wouldExceed(totalBytes)) { - throw new SubcCallError( + throw new McHostCallError( "not_sent", "aggregate connection memory cap would be exceeded", "memory_cap", @@ -306,7 +306,7 @@ export class TcpFrameChannel implements FrameChannel { if (this.controlFramesQueued >= this.controlReserveFrames) { this.fail( "control_capacity_exhausted", - new SubcCallError( + new McHostCallError( "terminal", "reserved control-frame capacity exhausted; required cleanup cannot queue safely", "control_capacity_exhausted", diff --git a/packages/plugin/src/shared/mc-host-client/test-support/adversarial-scenarios.ts b/packages/plugin/src/shared/mc-host-client/test-support/adversarial-scenarios.ts index 224e8cb3f..0ee94af38 100644 --- a/packages/plugin/src/shared/mc-host-client/test-support/adversarial-scenarios.ts +++ b/packages/plugin/src/shared/mc-host-client/test-support/adversarial-scenarios.ts @@ -14,7 +14,7 @@ import { Deadline } from "../deadline"; import { encodePeerFrame, PeerFrameType } from "./fake-peer"; import { createTrackedHarness, - expectSubcCallError, + expectMcHostCallError, rejection, type ScenarioContext, } from "./test-util"; @@ -275,7 +275,7 @@ export const adversarialScenarios: readonly AdversarialScenario[] = [ deadline: Deadline.start(10_000), }); const abortHandle = queued.abort(); - expectSubcCallError(await rejection(queued.result), "not_sent", "aborted"); + expectMcHostCallError(await rejection(queued.result), "not_sent", "aborted"); await abortHandle.cleanup; wedge.abort(); generationA.retire("owner_close"); @@ -292,7 +292,7 @@ export const adversarialScenarios: readonly AdversarialScenario[] = [ deadline: Deadline.start(5_000), }); connectionB.destroy(); - expectSubcCallError(await rejection(reset.result), "outcome_unknown"); + expectMcHostCallError(await rejection(reset.result), "outcome_unknown"); assert.ok(generationB.isRetired()); // terminal: a matching Error frame after invocation is the @@ -342,7 +342,7 @@ export const adversarialScenarios: readonly AdversarialScenario[] = [ connection.frames.some((frame) => frame.corr === request.correlation), ); const handle = request.abort(); - expectSubcCallError(await rejection(request.result), "outcome_unknown", "aborted"); + expectMcHostCallError(await rejection(request.result), "outcome_unknown", "aborted"); generation.enqueueCancel(CHANNEL, EPOCH, request.correlation); let cleanupResolved = 0; void handle.cleanup.then(() => { diff --git a/packages/plugin/src/shared/mc-host-client/test-support/frame-channel-contract.ts b/packages/plugin/src/shared/mc-host-client/test-support/frame-channel-contract.ts index 9e58287a0..e1f8743b2 100644 --- a/packages/plugin/src/shared/mc-host-client/test-support/frame-channel-contract.ts +++ b/packages/plugin/src/shared/mc-host-client/test-support/frame-channel-contract.ts @@ -23,7 +23,7 @@ import { import { FrameType, MAX_FRAME_BODY_LEN, PROTOCOL_VERSION } from "../protocol"; import { TcpFrameChannel } from "../tcp-frame-channel"; import { encodePeerFrame, FakePeer, type PeerFrameFields } from "./fake-peer"; -import { expectSubcCallError, waitUntil } from "./test-util"; +import { expectMcHostCallError, waitUntil } from "./test-util"; const CHANNEL = 5; const EPOCH = 9; @@ -251,7 +251,7 @@ export const frameChannelContractScenarios: readonly FrameChannelContractScenari } catch (error) { refused = error; } - expectSubcCallError(refused, "not_sent", "writer_queue_full"); + expectMcHostCallError(refused, "not_sent", "writer_queue_full"); // Required control writes must still be admittable. h.channel.sendControl(pongHeader(42n)); assert.equal(h.channel.isClosed(), false); @@ -274,7 +274,7 @@ export const frameChannelContractScenarios: readonly FrameChannelContractScenari } catch (error) { overCap = error; } - expectSubcCallError(overCap, "not_sent", "memory_cap"); + expectMcHostCallError(overCap, "not_sent", "memory_cap"); const narrow = await create({ maxQueuedBytes: 500 }); let overQueue: unknown; @@ -283,7 +283,7 @@ export const frameChannelContractScenarios: readonly FrameChannelContractScenari } catch (error) { overQueue = error; } - expectSubcCallError(overQueue, "not_sent", "writer_queue_full"); + expectMcHostCallError(overQueue, "not_sent", "writer_queue_full"); }, }, { diff --git a/packages/plugin/src/shared/mc-host-client/test-support/test-util.ts b/packages/plugin/src/shared/mc-host-client/test-support/test-util.ts index adaba6787..a3ed36776 100644 --- a/packages/plugin/src/shared/mc-host-client/test-support/test-util.ts +++ b/packages/plugin/src/shared/mc-host-client/test-support/test-util.ts @@ -9,7 +9,7 @@ import { writeFile } from "node:fs/promises"; import { setTimeout as delay } from "node:timers/promises"; import { ConnectionGeneration, type ConnectionGenerationOptions } from "../connection"; import { Deadline } from "../deadline"; -import { SubcCallError } from "../errors"; +import { McHostCallError } from "../errors"; import type { FrameChannelHandlers, FrameChannelStats, @@ -44,12 +44,12 @@ export async function waitUntil(check: () => boolean, timeoutMs = 3_000): Promis } } -export function expectSubcCallError( +export function expectMcHostCallError( error: unknown, - kind: SubcCallError["kind"], + kind: McHostCallError["kind"], code?: string, -): SubcCallError { - assert.ok(error instanceof SubcCallError, `expected SubcCallError, got ${String(error)}`); +): McHostCallError { + assert.ok(error instanceof McHostCallError, `expected McHostCallError, got ${String(error)}`); assert.equal(error.kind, kind); if (code !== undefined) assert.equal(error.code, code); return error; @@ -267,7 +267,7 @@ class FakeCandidateChannel implements SetupFrameChannel { send(frame: OutboundFrame, hooks?: FrameSendHooks): FrameSendTicket { if (this.closed) { - throw new SubcCallError("not_sent", "frame channel is closed", "channel_closed"); + throw new McHostCallError("not_sent", "frame channel is closed", "channel_closed"); } hooks?.onPublish?.(); this.host.receive({ header: frame.header, body: frame.body }); diff --git a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts index 721b7edec..12350bfd7 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts @@ -17,7 +17,6 @@ import { encodeNegotiateResponse, FALLBACK_REASONS, FIRST_APPLICATION_CORRELATION, - isLegacyUnsupportedOperationBody, isValidActivationToken, MAX_OFFERS, MAX_OPAQUE_BYTES, @@ -30,7 +29,7 @@ import { type TransportOffer, } from "./transport-negotiation"; -const VECTOR_TOKEN = "00112233445566778899aabbccddeeff"; +const VECTOR_TOKEN = "0011223344556677" + "8899aabbccddeeff"; const REQ_TCP_ONLY = '{"op":"transport.negotiate","negotiation_version":1,"offers":[{"transport":"tcp","capability_version":1}]}'; @@ -41,9 +40,9 @@ const RESP_TCP_DIRECT = const RESP_TCP_FALLBACK = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"capability_version_mismatch"}'; const RESP_GRANT = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff","descriptor":{}}'; + `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{}}`; const ACTIVATE_REQ = - '{"op":"transport.activate","negotiation_version":1,"activation_token":"00112233445566778899aabbccddeeff"}'; + `{"op":"transport.activate","negotiation_version":1,"activation_token":"${VECTOR_TOKEN}"}`; const ACTIVATE_RESP = '{"op":"transport.activate","negotiation_version":1}'; const COMMIT_REQ = '{"op":"transport.commit","negotiation_version":1}'; const COMMIT_RESP = '{"op":"transport.commit","negotiation_version":1}'; @@ -135,9 +134,10 @@ describe("fallback reasons", () => { const response = decodeNegotiateResponse(bytes(body), offers); if (response.kind === "tcp") expect(response.reason).toBe(reason); } - const unknown = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"switching_transports"}'; - expectCode(() => decodeNegotiateResponse(bytes(unknown), offers), "invalid_reason"); + for (const rejected of ["switching_transports", "connection_in_use"]) { + const body = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"${rejected}"}`; + expectCode(() => decodeNegotiateResponse(bytes(body), offers), "invalid_reason"); + } }); }); @@ -231,7 +231,7 @@ describe("recursive duplicate-key rejection", () => { } const descriptor = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff","descriptor":{"a":{"k":1,"k":2}}}'; + `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{"a":{"k":1,"k":2}}}`; expectCode( () => decodeNegotiateResponse(bytes(descriptor), [shmOffer(1), tcpOffer(1)]), "malformed_json", @@ -377,13 +377,13 @@ describe("grant and tcp field mixes", () => { '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"descriptor":{}}'; expectCode(() => decodeNegotiateResponse(bytes(noToken), offers), "missing_field"); const noDescriptor = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff"}'; + `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; expectCode(() => decodeNegotiateResponse(bytes(noDescriptor), offers), "missing_field"); }); test("a tcp selection carrying either grant field is rejected", () => { const withToken = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff"}'; + `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; expectCode(() => decodeNegotiateResponse(bytes(withToken), offers), "unexpected_field"); const withDescriptor = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"descriptor":{}}'; @@ -395,7 +395,7 @@ describe("grant and tcp field mixes", () => { test("a grant carrying a fallback reason is rejected", () => { const grantWithReason = - '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"00112233445566778899aabbccddeeff","descriptor":{},"reason":"unavailable"}'; + `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{},"reason":"unavailable"}`; expectCode( () => decodeNegotiateResponse(bytes(grantWithReason), offers), "unexpected_field", @@ -484,35 +484,6 @@ describe("activation and commit", () => { }); }); -describe("legacy unsupported_operation terminal", () => { - test("only the exact {code, message} string pair qualifies", () => { - expect( - isLegacyUnsupportedOperationBody( - bytes('{"code":"unsupported_operation","message":"unknown control operation"}'), - ), - ).toBe(true); - for (const body of [ - // Extra field. - '{"code":"unsupported_operation","message":"m","detail":"x"}', - // Non-string message. - '{"code":"unsupported_operation","message":1}', - // Missing message. - '{"code":"unsupported_operation"}', - // Wrong code. - '{"code":"internal_error","message":"m"}', - // Duplicate key. - '{"code":"x","code":"unsupported_operation","message":"m"}', - // Non-object roots and malformed JSON. - '"unsupported_operation"', - "[]", - "{", - "", - ]) { - expect(isLegacyUnsupportedOperationBody(bytes(body))).toBe(false); - } - }); -}); - describe("error hygiene", () => { const SENTINEL = "SENTINEL-PROVIDER-SECRET"; diff --git a/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts b/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts index d976a6381..37cd6d202 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts @@ -80,7 +80,6 @@ export const FALLBACK_REASONS = [ "unavailable", "negotiation_version_mismatch", "capability_version_mismatch", - "connection_in_use", ] as const; export type FallbackReason = (typeof FALLBACK_REASONS)[number]; @@ -686,61 +685,6 @@ function decodeTaggedOnly(bytes: Uint8Array, op: string): void { requireExactVersion(fields); } -/** - * The closed set of legacy Error terminal codes that prove - * `transport.negotiate` was never dispatched, so selecting TCP on this - * connection is safe (KTD6). Wire doc §7.7.3 names the exact legacy - * `unsupported_operation` terminal as the ONLY Error-based continuation - * evidence. `server_busy` is deliberately excluded: the doc permits a - * compliant negotiation-aware host to reject any control request before - * dispatch under load, so the code is not unambiguous legacy evidence — - * and it is independently retryable, which retirement plus reconnect - * already provides. - * - * Every other Error body is malformed negotiation content and fails closed. - */ -const LEGACY_FALLBACK_CODES: readonly string[] = ["unsupported_operation"]; - -/** - * The `code` of a strict legacy Error terminal: UTF-8 JSON `{code, message}` - * with both fields strings, no extra fields, and no duplicate keys. - * `undefined` for anything else, so a malformed body can never be read as - * fallback evidence. - */ -function legacyErrorCode(bytes: Uint8Array): string | undefined { - let fields: Map; - try { - fields = requireRootObject(bytes); - checkClosedFields(fields, ["code", "message"], "body"); - } catch { - return undefined; - } - const code = fields.get("code"); - const message = fields.get("message"); - if (code === undefined || code.kind !== "string") return undefined; - if (message === undefined || message.kind !== "string") return undefined; - return code.value; -} - -/** - * True only for the exact legacy `unsupported_operation` Error terminal: - * strict UTF-8 JSON `{code, message}` with both fields strings, no extras, - * and no duplicate keys. - */ -export function isLegacyUnsupportedOperationBody(bytes: Uint8Array): boolean { - return legacyErrorCode(bytes) === "unsupported_operation"; -} - -/** - * True for the closed set of legacy Error terminals that may select TCP - * fallback (KTD6); see {@link LEGACY_FALLBACK_CODES}. A body with extra - * fields, a non-string message, or any other code fails closed. - */ -export function isLegacyFallbackTerminalBody(bytes: Uint8Array): boolean { - const code = legacyErrorCode(bytes); - return code !== undefined && LEGACY_FALLBACK_CODES.includes(code); -} - function requireExactVersion(fields: Map): void { const version = requireVersion(fields, "negotiation_version", "negotiation_version"); if (version !== NEGOTIATION_VERSION) { diff --git a/packages/plugin/src/shared/mc-host-client/transport-provider.ts b/packages/plugin/src/shared/mc-host-client/transport-provider.ts index 9b5bf6000..ecbbaec37 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-provider.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-provider.ts @@ -4,11 +4,11 @@ * The production registry is empty: TCP needs no provider object because * the authenticated bootstrap channel IS the selected channel on a TCP * selection. Non-TCP providers exist only as injected test seams through - * the internal `SubcClientOptions.transportProviders` option. Deliberately + * the internal `McHostClientOptions.transportProviders` option. Deliberately * not exported from `index.ts` (R15: no supported consumer provider hooks). */ -import { SubcCallError } from "./errors"; +import { McHostCallError } from "./errors"; import { type ByteBudget, type FrameChannelCloseReason, @@ -456,7 +456,7 @@ export function sanitizedCandidateFactory( ticket = channel.send(frame, trackedHooks); } catch (error) { const code = - error instanceof SubcCallError && + error instanceof McHostCallError && error.code !== undefined && BOUNDED_CHANNEL_CODES.has(error.code) ? error.code @@ -464,7 +464,7 @@ export function sanitizedCandidateFactory( if (!published) { // Proven refusal: nothing was published, so the // bounded failure is replay-safe `not_sent`. - throw new SubcCallError( + throw new McHostCallError( "not_sent", `transport provider ${transport} failed during send`, code, @@ -474,7 +474,7 @@ export function sanitizedCandidateFactory( // channel — retirement settles pending work exactly // once — and classify the throw as never replayable. closeUpstream("write_failed", "send"); - throw new SubcCallError( + throw new McHostCallError( "outcome_unknown", `transport provider ${transport} failed during send`, code, diff --git a/packages/plugin/src/shared/mc-host-client/types.ts b/packages/plugin/src/shared/mc-host-client/types.ts index f03519e5e..ac08c5add 100644 --- a/packages/plugin/src/shared/mc-host-client/types.ts +++ b/packages/plugin/src/shared/mc-host-client/types.ts @@ -1,9 +1,7 @@ /** * Shared public shapes for the mc-host consumer client. * - * Leaf module: imports nothing from connection or facade code and no npm - * subc-client code. The shapes mirror the subset of `@cortexkit/subc-client` - * 0.4.1 that in-repo consumers actually use. Wire semantics + * Leaf module: imports nothing from connection or facade code. Wire semantics * come from `docs/mc-host-wire-protocol.md`. */ @@ -62,7 +60,7 @@ export const AdmissionClass = { } as const; export type AdmissionClass = (typeof AdmissionClass)[keyof typeof AdmissionClass]; -/** Options for `SubcClient.connect()` as used by current repo consumers. */ +/** Options for `McHostClient.connect()` as used by current repo consumers. */ export interface ConnectOptions { connectionFile: string; handshakeTimeoutMs?: number; @@ -77,7 +75,7 @@ export interface RequestOptions { priority?: Priority; admissionClass?: AdmissionClass; timeoutMs?: number; - /** The facade attaches `SubcCallError.cleanup` when this signal aborts the request. */ + /** The facade attaches `McHostCallError.cleanup` when this signal aborts the request. */ signal?: AbortSignal; } diff --git a/packages/plugin/src/shared/redaction.test.ts b/packages/plugin/src/shared/redaction.test.ts index cb618736e..3c1a35681 100644 --- a/packages/plugin/src/shared/redaction.test.ts +++ b/packages/plugin/src/shared/redaction.test.ts @@ -22,11 +22,13 @@ describe("redactSecretText — token counts and scalar diagnostics stay visible" test("still redacts real secret string values", () => { // High-entropy / non-scalar values must always be redacted; only bare // numeric/boolean scalars are exempt from the key-based match. - expect(redactSecretText("api_key=sk-abc123XYZsecretvalue")).toContain(" { @@ -34,9 +36,8 @@ describe("redactSecretText — token counts and scalar diagnostics stay visible" expect(redactSecretText("Authorization: Bearer abc123def456ghi789")).toContain( "", ); - expect(redactSecretText("blob=eyJhbGciOi.eyJzdWIiOiIx.SflKxwRJSMeKKF2QT4")).toContain( - "", - ); + const syntheticJwt = ["eyJhbGciOi", "eyJzdWIiOiIx", "SflKxwRJSMeKKF2QT4"].join("."); + expect(redactSecretText(`blob=${syntheticJwt}`)).toContain(""); }); }); @@ -53,7 +54,7 @@ describe("hasShareabilitySensitiveText", () => { }); test("flags inline key:value / key=value secrets the keyed redactor misses in prose", () => { - expect(hasShareabilitySensitiveText("Set api_key: sk-live-abc123 in the env.")).toBe(true); + expect(hasShareabilitySensitiveText("Set api_key: sk-live-abc123 in the env.")).toBe(true); // gitleaks:allow redaction-test fixture expect(hasShareabilitySensitiveText("password=hunter2 for the staging box")).toBe(true); expect(hasShareabilitySensitiveText("client_secret = abcdef in the OAuth app")).toBe(true); }); From cffc09e7efb6aa72dbcdfe320b37995cf673b780 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 07:26:44 +0000 Subject: [PATCH 02/37] fix(mc-host): close three client boundary gaps from review Reject a zero epoch on a routed channel at decode. Epoch 0 is reserved for the control channel, so a routed frame without an epoch names no bindable route. The TypeScript client already rejected this pairing in validateHeader while the Rust decoder accepted it, which let a corrupt routing identity through to be dropped as unmatched instead of closing the generation. Both halves of the pairing are now structural on both clients, and the wire document's corruption list names the rule. Enforce the rest of the direct-profile inbound table in the client. A StreamEnd body is structural corruption, but StreamEnd is not a pure-header frame at the framing layer, so the pure-header check never saw it and a nonempty body ended the stream normally. Push is unsolicited, so a correlation claims a pending request the frame cannot answer. Retire a stream's deadline watcher when the stream settles. The watcher held no completion signal, so a stream that finished early left one sleeping task per run for up to the caller's whole timeout - ten minutes under the historian's stream timeout - and the live-stream cap did not bound them because it is released at settlement. Settlement now fires a token the watcher selects on first; every settle path already funnels through finish_pending, so that is the single hook. The two spawn shapes collapse into one because a default token is never cancelled. Keep a cancelled request's OutcomeUnknown classification when its best-effort Cancel cannot be queued. The enqueue failure was propagated through `?`, and callers replace the request's classification with the control error's outcome, so a concurrent generation retirement reported a request whose bytes may already have reached the host as replay-safe not_sent. The failure is still reported, now carrying the request's own outcome. Each fix has a test that fails without it. --- crates/mc-host/src/client.rs | 189 +++++++++++++++--- crates/mc-host/src/wire.rs | 26 +++ docs/mc-host-wire-protocol.md | 2 +- .../shared/mc-host-client/connection.test.ts | 8 + 4 files changed, 198 insertions(+), 27 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 1a7135296..a2692fc6c 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -661,6 +661,10 @@ enum PendingKind { Stream { items: mpsc::Sender, terminal: oneshot::Sender>, + /// Fired once when this stream settles, so the detached deadline + /// watcher stops sleeping instead of outliving the stream by up to the + /// caller's whole timeout. + settled: CancellationToken, }, } @@ -731,12 +735,14 @@ impl Inner { let (item_tx, item_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); let (terminal_tx, terminal_rx) = oneshot::channel(); let deadline = Instant::now() + options.timeout; + let settled = CancellationToken::new(); let admitted = self.admit( route, body, PendingKind::Stream { items: item_tx, terminal: terminal_tx, + settled: settled.clone(), }, deadline, ); @@ -747,31 +753,28 @@ impl Inner { return Err(error); } }; - if let Some(cancel) = options.cancellation { - let weak = Arc::downgrade(self); - tokio::spawn(async move { - tokio::select! { - () = cancel.cancelled() => { - if let Some(inner) = weak.upgrade() { - let _ = inner.cancel_key(key, "cancelled"); - } - } - () = tokio::time::sleep_until(deadline) => { - if let Some(inner) = weak.upgrade() { - let _ = inner.cancel_key(key, "deadline_expired"); - } + // A default token is never cancelled, so the absent-cancellation case + // reduces to a deadline-only watcher without a second spawn shape. + let cancel = options.cancellation.unwrap_or_default(); + let weak = Arc::downgrade(self); + tokio::spawn(async move { + tokio::select! { + biased; + // Settlement first: a stream that already terminated must not + // issue a cancel on a correlation the host may have reused. + () = settled.cancelled() => {} + () = cancel.cancelled() => { + if let Some(inner) = weak.upgrade() { + let _ = inner.cancel_key(key, "cancelled"); } } - }); - } else { - let weak = Arc::downgrade(self); - tokio::spawn(async move { - tokio::time::sleep_until(deadline).await; - if let Some(inner) = weak.upgrade() { - let _ = inner.cancel_key(key, "deadline_expired"); + () = tokio::time::sleep_until(deadline) => { + if let Some(inner) = weak.upgrade() { + let _ = inner.cancel_key(key, "deadline_expired"); + } } - }); - } + } + }); Ok(ResponseStream { inner: Arc::downgrade(self), key, @@ -901,7 +904,13 @@ impl Inner { Err(CallError::local(outcome, code, "request stopped")), ); if outcome == SendOutcome::OutcomeUnknown { - self.send_control( + // The Cancel is best-effort cleanup, and the request's bytes may + // already be on the wire. Report the failed enqueue, but keep the + // request's own OutcomeUnknown classification: substituting the + // control frame's outcome (NotSent when the generation retires + // concurrently) would tell the caller a possibly-delivered request + // is replay-safe. + if let Err(error) = self.send_control( FrameType::Cancel, FrameId { channel: key.channel, @@ -909,7 +918,9 @@ impl Inner { corr: key.corr, }, None, - )?; + ) { + return Err(CallError::new(outcome, error.code, error.message)); + } } Ok(()) } @@ -1114,9 +1125,15 @@ impl Inner { PendingKind::Unary(tx) => { let _ = tx.send(result); } - PendingKind::Stream { terminal, .. } => { + PendingKind::Stream { + terminal, settled, .. + } => { let terminal_result = result.map(|_| ()); let _ = terminal.send(terminal_result); + // Every settle path — terminal frame, caller cancel, stream + // drop, route settle, generation retire — funnels through here, + // so this is the single point that retires the watcher task. + settled.cancel(); self.release_stream(); } } @@ -1507,12 +1524,24 @@ fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { } match header.ty { FrameType::Response | FrameType::Error | FrameType::StreamData | FrameType::StreamEnd => { + // `decode_header` already rejects a mixed zero/nonzero + // channel/epoch pair, so the identity is control (0/0) or routed + // (nonzero/nonzero) by here; only the correlation is left. if header.corr == 0 { return Err(()); } + // The direct profile carries stream termination in the header. A + // StreamEnd body is structural corruption even though the framing + // layer does not classify StreamEnd as pure-header, so the + // pure-header check below never sees it. + if matches!(header.ty, FrameType::StreamEnd) && header.len != 0 { + return Err(()); + } } FrameType::Push => { - if header.channel == 0 || header.epoch == 0 { + // Push is unsolicited, so a correlation would claim a pending + // request the frame cannot answer. + if header.channel == 0 || header.epoch == 0 || header.corr != 0 { return Err(()); } } @@ -2056,6 +2085,113 @@ mod tests { assert_eq!(inner.queue_budget.used(), 0); } + #[tokio::test] + async fn failed_cancel_enqueue_keeps_outcome_unknown() { + let (inner, mut data_rx, control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit( + route(1), + b"possibly-sent".to_vec(), + kind, + Instant::now() + Duration::from_secs(1), + ) + .expect("admitted"); + assert!(claim_for_write(&publish), "writer claims the request"); + // Retire without draining pending, so the best-effort Cancel enqueue + // fails with the control path's own NotSent classification. + inner.retired.store(true, Ordering::Release); + + let error = inner + .cancel_key(key, "cancelled") + .expect_err("Cancel cannot be queued on a retired generation"); + assert_eq!( + error.outcome(), + SendOutcome::OutcomeUnknown, + "a claimed request stays possibly-sent when its Cancel cannot be queued" + ); + assert_eq!(error.code(), "generation_retired"); + let settled = rx.await.expect("settled").expect_err("cancelled"); + assert_eq!(settled.outcome(), SendOutcome::OutcomeUnknown); + drop(data_rx.recv().await); + drop(control_rx); + } + + #[tokio::test] + async fn settled_stream_retires_its_deadline_watcher() { + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let settled = CancellationToken::new(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + settled: settled.clone(), + }, + Instant::now() + Duration::from_secs(600), + ) + .expect("stream admitted"); + assert!( + !settled.is_cancelled(), + "the watcher must stay armed while the stream is live" + ); + drop(data_rx.recv().await); + + inner.cancel_key(key, "cancelled").expect("stream settled"); + assert!( + settled.is_cancelled(), + "settlement must retire the watcher instead of leaving it asleep until the deadline" + ); + let _ = terminal_rx.await; + } + + #[test] + fn inbound_validation_enforces_the_direct_profile_table() { + let header = + |ty: FrameType, channel: u16, epoch: u32, corr: u64, len: u32| EnvelopeHeader { + len, + ver: PROTOCOL_VERSION, + ty, + flags: if ty.is_pure_header() { + pure_header_flags() + } else { + Flags::new(false, Priority::Interactive, false) + }, + channel, + epoch, + corr, + }; + + // Legal control and routed identities stay legal. + assert!(validate_inbound(&header(FrameType::Response, 0, 0, 7, 4)).is_ok()); + assert!(validate_inbound(&header(FrameType::Response, 3, 9, 7, 4)).is_ok()); + assert!(validate_inbound(&header(FrameType::StreamData, 3, 9, 7, 4)).is_ok()); + assert!(validate_inbound(&header(FrameType::StreamEnd, 3, 9, 7, 0)).is_ok()); + assert!(validate_inbound(&header(FrameType::Push, 3, 9, 0, 4)).is_ok()); + + // A routed frame with epoch 0 never decodes, so `validate_inbound` only + // sees coherent identities (see `wire::decode_header`). + assert!(validate_inbound(&header(FrameType::Response, 3, 0, 7, 4)).is_ok()); + + // The direct profile requires an empty StreamEnd body. + assert!(validate_inbound(&header(FrameType::StreamEnd, 3, 9, 7, 1)).is_err()); + + // Push is unsolicited, so it carries no correlation. + assert!(validate_inbound(&header(FrameType::Push, 3, 9, 5, 4)).is_err()); + + // Pre-existing rules keep holding. + assert!(validate_inbound(&header(FrameType::Response, 3, 9, 0, 4)).is_err()); + assert!(validate_inbound(&header(FrameType::Ping, 0, 0, 7, 0)).is_ok()); + assert!(validate_inbound(&header(FrameType::Ping, 1, 0, 7, 0)).is_err()); + assert!(validate_inbound(&header(FrameType::Goodbye, 0, 0, 0, 0)).is_ok()); + assert!(validate_inbound(&header(FrameType::Goodbye, 3, 9, 0, 1)).is_err()); + assert!(validate_inbound(&header(FrameType::Request, 3, 9, 7, 4)).is_err()); + } + #[tokio::test] async fn dropped_unary_future_cleans_pending_and_possibly_sent_request() { let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); @@ -2246,6 +2382,7 @@ mod tests { PendingKind::Stream { items: items_tx, terminal: terminal_tx, + settled: CancellationToken::new(), }, Instant::now() + Duration::from_secs(1), ) diff --git a/crates/mc-host/src/wire.rs b/crates/mc-host/src/wire.rs index 9bbd75b3e..fd0bb178b 100644 --- a/crates/mc-host/src/wire.rs +++ b/crates/mc-host/src/wire.rs @@ -236,6 +236,8 @@ pub enum DecodeError { SheddableIllegalFrameType { ty: FrameType, flags: u8 }, /// Channel 0 carried an epoch other than its reserved epoch 0. NonzeroEpochOnControlChannel { epoch: u32 }, + /// A routed channel carried epoch 0, which is reserved for channel 0. + ZeroEpochOnRoutedChannel { channel: u16 }, /// A pure-header frame declared body bytes. PureHeaderFrameWithBody { ty: FrameType, len: u32 }, } @@ -270,6 +272,9 @@ impl fmt::Display for DecodeError { Self::NonzeroEpochOnControlChannel { epoch } => { write!(f, "control channel carried nonzero epoch {epoch}") } + Self::ZeroEpochOnRoutedChannel { channel } => { + write!(f, "routed channel {channel} carried zero epoch") + } Self::PureHeaderFrameWithBody { ty, len } => { write!( f, @@ -340,6 +345,13 @@ pub fn decode_header(bytes: &[u8]) -> Result { if channel == 0 && epoch != 0 { return Err(DecodeError::NonzeroEpochOnControlChannel { epoch }); } + // Epoch 0 is reserved for the control channel (Section 6.1), so a routed + // channel without an epoch names no bindable route. Rejecting it here keeps + // the framing layer's identity contract symmetric instead of leaving the + // frame to be dropped as unmatched further up. + if channel != 0 && epoch == 0 { + return Err(DecodeError::ZeroEpochOnRoutedChannel { channel }); + } let corr = u64::from_le_bytes([ bytes[13], bytes[14], bytes[15], bytes[16], bytes[17], bytes[18], bytes[19], bytes[20], ]); @@ -804,6 +816,20 @@ mod tests { decode_header(&h.encode()), Err(DecodeError::NonzeroEpochOnControlChannel { epoch: u32::MAX }) ); + // Epoch 0 is reserved for channel 0, so a routed channel must carry a + // nonzero epoch. Both halves of the pairing are structural. + let h = hdr_with_epoch( + 0, + FrameType::Request, + Flags::new(false, Priority::Passive, false), + 7, + 0, + 2, + ); + assert_eq!( + decode_header(&h.encode()), + Err(DecodeError::ZeroEpochOnRoutedChannel { channel: 7 }) + ); } #[test] diff --git a/docs/mc-host-wire-protocol.md b/docs/mc-host-wire-protocol.md index fde7e371c..c796fcdf8 100644 --- a/docs/mc-host-wire-protocol.md +++ b/docs/mc-host-wire-protocol.md @@ -286,7 +286,7 @@ Aggregate resource policy takes effect between frames, before admitting more con Waiting for the next frame on an idle connection is unbounded at the framing layer; idle lifetime is governed separately by liveness policy (Section 9.3). Once the first header byte arrives, the remaining header and body bytes MUST complete within one finite operation-owned absolute deadline. Duration is deployment policy, not a wire constant. A peer that declares 64 MiB and stalls partway consumes its bounded slot only until that deadline. A deadline that instead started at read-loop entry would close healthy idle connections: cached routes and clients waiting on long-running requests send no frames while they wait. -Clean EOF before any byte of the next header is orderly connection close. EOF after the first header byte, truncated header/body, unsupported version, unknown type, invalid flags, nonzero channel-0 epoch, pure-header body, or body declaration above 64 MiB corrupts stream alignment. Receiver MUST close the connection generation without resynchronization and without sending an `Error` on that stream. +Clean EOF before any byte of the next header is orderly connection close. EOF after the first header byte, truncated header/body, unsupported version, unknown type, invalid flags, nonzero channel-0 epoch, zero epoch on a routed channel, pure-header body, or body declaration above 64 MiB corrupts stream alignment. Receiver MUST close the connection generation without resynchronization and without sending an `Error` on that stream. Writers MUST verify header `len` equals body length and SHOULD submit header plus body as one logical write. This is an efficiency requirement from published transport behavior, not an atomicity guarantee: partial socket writes remain possible and drive outcome classification. Each connection additionally has exactly one logical writer: implementations MUST serialize fully encoded frames so every byte of one frame (header then body) reaches the socket before any byte of another frame on that connection. Concurrent requests, stream emissions, and Ping/Pong traffic share the socket, and interleaved frame bytes make the peer read body bytes as the next header — stream-alignment corruption. Socket-write atomicity is neither assumed nor sufficient: after a partial write, the writer MUST continue that same frame's remaining bytes before emitting any other frame. diff --git a/packages/plugin/src/shared/mc-host-client/connection.test.ts b/packages/plugin/src/shared/mc-host-client/connection.test.ts index 812be570a..f21e59b0f 100644 --- a/packages/plugin/src/shared/mc-host-client/connection.test.ts +++ b/packages/plugin/src/shared/mc-host-client/connection.test.ts @@ -465,6 +465,14 @@ describe("ingress fencing", () => { ], ["Goodbye with nonzero corr", { ty: PeerFrameType.Goodbye, corr: 5n }], ["terminal with corr 0", { ty: PeerFrameType.Response, channel: 1, epoch: 1, corr: 0n }], + [ + "terminal with epoch 0 on a routed channel", + { ty: PeerFrameType.Response, channel: 1, epoch: 0, corr: 1n }, + ], + [ + "Push with a nonzero corr", + { ty: PeerFrameType.Push, channel: CHANNEL, epoch: EPOCH, corr: 1n }, + ], [ "reserved flag bits", { ty: PeerFrameType.Response, channel: 1, epoch: 1, corr: 1n, flags: 0b1100_0000 }, From 29e6f4a13bf47c1e7d6508176a28880ac93cdb1b Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 08:03:24 +0000 Subject: [PATCH 03/37] fix: address second review round on the direct mc-host boundary Terminate the producer drain on every error spelling. `is_error_unit` accepts "error" and "run_error" while `is_terminal_unit` accepted only "error", so a "run_error" unit fell through to stream end and surfaced as `UnexpectedStreamEnd` instead of `RunFailed`, discarding the typed classification the retry ladder reads. `is_terminal_unit` now defers to `is_error_unit` so the two sets cannot drift again. Stop the client's shutdown join from busy-spinning. `join_tasks_until` polled `is_finished` behind two mutexes and called `yield_now`, which re-queues immediately, so it spun a worker for the whole shutdown budget. Each task is now awaited under the shared deadline, and only a task that timed out is aborted and re-awaited, which is safe because it never completed. Stop the fixture from polling a completed `JoinHandle` twice. Early host exit made `wait_for_publication` await the handle that `run` awaits again, and the second poll panics before the control socket is removed. The readiness probe now only reports the exit, `run` keeps the single await, and the host's own error reports ahead of the readiness symptom. Make the fixture's `Drop` shutdown non-panicking. It went through `control_raw`, which panics loudly on purpose; during unwinding from a failed test that aborts the runner and hides the failure that started the unwind. Match pipelined control responses by correlation. The host answers from concurrent tasks, so the second response can arrive first, and `frames_until_corr` discards the frames it consumed while searching - the sequential second call would then wait out its budget. Keep the leaked-fixture PID record when teardown fails to reap the child. `stop()` removed the record and the data dir before throwing, destroying the only identity the next run's reaper had for the surviving process. Resume a paused fixture before signalling teardown. A SIGSTOPped child runs no graceful shutdown and holds SIGTERM pending, so a test that threw between `pauseHost` and `resumeHost` made every teardown path burn its full timeout. Honor a pre-built workspace fixture when building is forbidden, report the cause of a fixture-start failure instead of a fixed string, overlap the control and catalog handshakes that share no data, and finish the `McHost*` rename in the terminal-error default message. --- crates/mc-host/src/client.rs | 46 ++++------ crates/mc-host/tests/lifecycle.rs | 28 +++++-- .../mc-module/examples/direct_host_fixture.rs | 12 +-- crates/mc-module/src/historian_producer.rs | 84 ++++++++++++++++++- crates/mc-module/tests/support/direct_host.rs | 21 ++++- .../scripts/check-rust-prerequisites.test.ts | 17 ++++ .../scripts/check-rust-prerequisites.ts | 4 + .../e2e-tests/src/opencode-runner/spawn.ts | 6 +- .../src/rust-runner/hermetic-mc-host.ts | 47 +++++++++-- .../src/shared/mc-host-client/client.ts | 4 +- 10 files changed, 215 insertions(+), 54 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index a2692fc6c..4b7008aef 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -1210,39 +1210,21 @@ impl Inner { } async fn join_tasks_until(&self, deadline: Instant) -> bool { - loop { - let writer_finished = self - .writer - .lock() - .await - .as_ref() - .is_none_or(JoinHandle::is_finished); - let reader_finished = self - .reader - .lock() - .await - .as_ref() - .is_none_or(JoinHandle::is_finished); - if writer_finished && reader_finished { - break; - } - if Instant::now() >= deadline { - if let Some(task) = self.writer.lock().await.as_ref() { - task.abort(); - } - if let Some(task) = self.reader.lock().await.as_ref() { - task.abort(); - } - break; + let mut within_deadline = true; + // Await each task under the shared deadline rather than polling + // `is_finished`: a `yield_now` loop re-queues itself every iteration and + // spins the worker for the whole shutdown budget. + for slot in [&self.writer, &self.reader] { + let Some(mut task) = slot.lock().await.take() else { + continue; + }; + if tokio::time::timeout_at(deadline, &mut task).await.is_err() { + within_deadline = false; + // The timeout means this task never completed, so it is safe to + // await again after the abort - a completed handle would panic. + task.abort(); + let _ = task.await; } - tokio::task::yield_now().await; - } - let within_deadline = Instant::now() < deadline; - if let Some(task) = self.writer.lock().await.take() { - let _ = task.await; - } - if let Some(task) = self.reader.lock().await.take() { - let _ = task.await; } within_deadline } diff --git a/crates/mc-host/tests/lifecycle.rs b/crates/mc-host/tests/lifecycle.rs index 4461c0c22..234b60c10 100644 --- a/crates/mc-host/tests/lifecycle.rs +++ b/crates/mc-host/tests/lifecycle.rs @@ -1559,11 +1559,29 @@ async fn pipelined_shutdown_requests_on_one_connection_both_settle() { wire.extend_from_slice(body); client.send_raw(&wire).await.expect("pipeline shutdowns"); - for corr in [first, second] { - let (_, response) = client - .frames_until_corr(corr, BUDGET) - .await - .expect("shutdown response"); + // The host answers pipelined control requests from concurrent tasks, so the + // two responses arrive in either order. `frames_until_corr` hands back the + // frames it consumed while searching, and those are the only copy: asking + // for `second` sequentially after its response was already consumed would + // wait out the whole budget. + let (consumed, first_response) = client + .frames_until_corr(first, BUDGET) + .await + .expect("shutdown response"); + let second_response = match consumed + .into_iter() + .find(|frame| frame.corr == second && frame.ty != TY_PING) + { + Some(frame) => frame, + None => { + client + .frames_until_corr(second, BUDGET) + .await + .expect("shutdown response") + .1 + } + }; + for response in [first_response, second_response] { assert_eq!(response.ty, TY_RESPONSE); assert_eq!(response.json()["op"], "host.shutdown"); } diff --git a/crates/mc-module/examples/direct_host_fixture.rs b/crates/mc-module/examples/direct_host_fixture.rs index 2fc1e7653..c2350e1a5 100644 --- a/crates/mc-module/examples/direct_host_fixture.rs +++ b/crates/mc-module/examples/direct_host_fixture.rs @@ -507,10 +507,10 @@ mod unix { return Ok(()); } if host.is_finished() { - return match host.await? { - Ok(()) => Err("host exited before readiness".into()), - Err(error) => Err(Box::new(error)), - }; + // Report the exit without consuming the handle: `run` owns the + // single await, and polling a completed `JoinHandle` twice + // panics, which would skip control-socket cleanup. + return Err("host exited before readiness".into()); } if tokio::time::Instant::now() >= deadline { return Err("host did not publish before readiness deadline".into()); @@ -597,8 +597,10 @@ mod unix { Err(error) if error.kind() == io::ErrorKind::NotFound => {} Err(error) => return Err(error.into()), } - ready?; + // The host's own error is the specific one; a readiness failure is + // usually its symptom, so it reports only when the host itself is fine. host_result?; + ready?; Ok(()) } } diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 49fc7e5d0..ce4d4652b 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -1209,12 +1209,19 @@ fn unit_text(unit: &Value) -> Option { } fn is_terminal_unit(unit: &Value) -> bool { + // Every error unit is terminal. Deferring to `is_error_unit` keeps the two + // sets from drifting: a spelling recognized as an error but not as a + // terminal would skip terminal handling entirely and surface as + // `UnexpectedStreamEnd`, discarding the run's typed classification. + if is_error_unit(unit) { + return true; + } unit_type(unit) .map(str::to_ascii_lowercase) .is_some_and(|kind| { matches!( kind.as_str(), - "run_finished" | "terminal" | "run_terminal" | "finished" | "error" + "run_finished" | "terminal" | "run_terminal" | "finished" ) }) } @@ -1768,4 +1775,79 @@ mod tests { "context_overflow" ); } + + fn stream_of(events: impl IntoIterator) -> FakeStream { + FakeStream( + events + .into_iter() + .map(|event| { + Ok(Some(StreamItem { + body: serde_json::to_vec(&event).expect("event serializes"), + binary: false, + })) + }) + .collect(), + ) + } + + #[tokio::test] + async fn every_error_spelling_terminates_the_drain_with_its_classification() { + // Each spelling `is_error_unit` accepts must also end the drain. A + // spelling that is an error but not a terminal falls through to stream + // end, and the typed classification the retry ladder needs is lost. + for kind in ["error", "run_error"] { + let mut stream = stream_of([ + json!({"type": "run_started", "run_id": "r1"}), + json!({"type": "assistant_message", "run_id": "r1", "text": "partial"}), + json!({ + "type": kind, + "run_id": "r1", + "error": {"message": "model is busy", "class": "transient", "retry_after_secs": 4}, + }), + ]); + + let error = drain_subscribe(&mut stream, "r1") + .await + .expect_err("an error unit ends the run"); + match error { + HistorianProducerError::RunFailed { + run_id, + detail, + classification, + class_field_present, + } => { + assert_eq!(run_id, "r1"); + assert_eq!(detail, "model is busy"); + assert!(class_field_present); + assert_eq!( + classification, + Some(ErrorClassification { + class: ErrorClass::Transient, + retry_after_secs: Some(4), + }) + ); + } + other => panic!("{kind} produced {other:?} instead of RunFailed"), + } + } + } + + #[tokio::test] + async fn a_successful_drain_returns_text_and_the_length_cap() { + let mut stream = stream_of([ + json!({"type": "run_started", "run_id": "r1"}), + json!({"type": "assistant_message", "run_id": "r1", "text": "first "}), + json!({"type": "assistant_message", "run_id": "r1", "text": "second", "finish_reason": "max_tokens"}), + json!({"type": "run_finished", "run_id": "r1"}), + ]); + + let output = drain_subscribe(&mut stream, "r1") + .await + .expect("terminal unit completes the run"); + assert_eq!(output.text, "first second"); + assert!( + output.length_capped, + "the cap travels with the text it truncated" + ); + } } diff --git a/crates/mc-module/tests/support/direct_host.rs b/crates/mc-module/tests/support/direct_host.rs index 00e421508..c6a57a865 100644 --- a/crates/mc-module/tests/support/direct_host.rs +++ b/crates/mc-module/tests/support/direct_host.rs @@ -171,6 +171,25 @@ impl FixtureProcess { self.control_raw(format!("{}\n", json!({"id": id, "command": {"name": name}})).as_bytes()) } + /// Best-effort graceful shutdown that reports failure instead of panicking. + /// `control_raw` panics loudly on purpose, which is right inside a test body + /// and fatal in `Drop`: a panic while unwinding a failed test aborts the + /// runner and hides the failure that started the unwind. + fn try_graceful_shutdown(&self) -> std::io::Result<()> { + let request = format!( + "{}\n", + json!({"id": 9_998, "command": {"name": "graceful-shutdown"}}) + ); + let mut stream = UnixStream::connect(self.control_path())?; + stream.set_read_timeout(Some(BUDGET))?; + stream.write_all(request.as_bytes())?; + let mut response = Vec::new(); + BufReader::new(stream) + .take(64 * 1024 + 1) + .read_until(b'\n', &mut response)?; + Ok(()) + } + pub fn control_raw(&self, bytes: &[u8]) -> Value { let path = self.control_path(); let mut stream = UnixStream::connect(&path).unwrap_or_else(|error| { @@ -302,7 +321,7 @@ impl Drop for FixtureProcess { { return; } - let _ = self.control(9_998, "graceful-shutdown"); + let _ = self.try_graceful_shutdown(); let deadline = Instant::now() + BUDGET; while Instant::now() < deadline { if self diff --git a/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts b/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts index ca62deef4..ef3f8e93f 100644 --- a/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts +++ b/packages/e2e-tests/scripts/check-rust-prerequisites.test.ts @@ -57,6 +57,23 @@ describe("Rust direct-host prerequisite detector", () => { expect(result).toEqual({ ok: true, missing: [] }); }); + it("resolves a pre-built workspace fixture without building", () => { + const { root, bin } = fakeWorkspace(); + const examples = join(root, "target", "debug", "examples"); + mkdirSync(examples, { recursive: true }); + const fixture = join(examples, "direct_host_fixture"); + writeFileSync(fixture, "#!/bin/sh\nexit 0\n"); + chmodSync(fixture, 0o755); + + const result = detectRustPrerequisites({ + repoRoot: root, + allowBuild: false, + env: { PATH: bin }, + }); + + expect(result).toEqual({ ok: true, missing: [], fixtureBin: fixture }); + }); + it("rejects a workspace without the direct host fixture target", () => { const { root, bin } = fakeWorkspace(false); const result = detectRustPrerequisites({ diff --git a/packages/e2e-tests/scripts/check-rust-prerequisites.ts b/packages/e2e-tests/scripts/check-rust-prerequisites.ts index bfaf4d48f..8b83d6056 100644 --- a/packages/e2e-tests/scripts/check-rust-prerequisites.ts +++ b/packages/e2e-tests/scripts/check-rust-prerequisites.ts @@ -119,6 +119,10 @@ export function detectRustPrerequisites( ); let fixtureBin = configured && isExecutable(configured) ? configured : undefined; + // A fixture already compiled in the workspace satisfies detection, so + // callers that forbid building still resolve a usable binary. + if (!fixtureBin && isExecutable(workspaceFixture)) + fixtureBin = workspaceFixture; if (!existsSync(manifest)) { missing.push(`cargo workspace: missing ${manifest}`); diff --git a/packages/e2e-tests/src/opencode-runner/spawn.ts b/packages/e2e-tests/src/opencode-runner/spawn.ts index a0b06cd3b..9a8adab65 100644 --- a/packages/e2e-tests/src/opencode-runner/spawn.ts +++ b/packages/e2e-tests/src/opencode-runner/spawn.ts @@ -339,8 +339,10 @@ async function provisionRustMode(): Promise { try { const mcHost = await HermeticMcHostStack.start({ dataDir: env.dataDir, fixtureBin }); return { env, connectionFile: mcHost.connectionFile, mcHost }; - } catch { - throw new Error("MC_E2E_MODE=rust failed to start direct mc-host fixture"); + } catch (error) { + throw new Error( + `MC_E2E_MODE=rust failed to start direct mc-host fixture: ${String(error)}`, + ); } } diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index a2d54c6ee..c8ef1dd85 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -726,6 +726,7 @@ export class HermeticMcHostStack { await this.closeClients(); const child = this.child; if (!child) return; + this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); if (!(await safeChildExit(child, 5_000))) { @@ -746,6 +747,7 @@ export class HermeticMcHostStack { await this.closeClients(); const child = this.child; if (!child) return; + this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGTERM"); if (!(await safeChildExit(child, 10_000))) { @@ -769,7 +771,10 @@ export class HermeticMcHostStack { child.kill("SIGCONT"); } - /** Graceful JSONL shutdown, then SIGTERM fallback. Always await exit and remove isolated state. */ + /** + * Graceful JSONL shutdown, then SIGTERM fallback. Always await exit; + * isolated state is removed only once the child is gone. + */ async stop(): Promise { await this.closeStatusClient(); const child = this.child; @@ -778,6 +783,7 @@ export class HermeticMcHostStack { child.exitCode !== null || child.signalCode !== null; if (child && !exited) { + this.resumeBeforeTeardown(child); try { await this.control?.gracefulShutdown(); } catch { @@ -796,12 +802,16 @@ export class HermeticMcHostStack { this.control?.close(); this.control = null; this.child = null; - rmSync(this.pidFilePath, { force: true }); - rmSync(this.dataDir, { recursive: true, force: true }); - if (!exited) + if (!exited) { + // The surviving child's PID record lives inside dataDir, so both + // stay on disk past this failed teardown: that record is the only + // identity the next run's reaper has for killing the leaked child. throw new Error( "direct mc-host fixture did not exit during teardown", ); + } + rmSync(this.pidFilePath, { force: true }); + rmSync(this.dataDir, { recursive: true, force: true }); } private async startHost(): Promise { @@ -911,10 +921,23 @@ export class HermeticMcHostStack { verifyPublication(this.controlPath, 0o600); verifyPublication(this.connectionFile, 0o600); + // The control socket and the client-visible catalog probe use unrelated + // sockets and share no data, so both handshakes run at once. A control + // client that connects is adopted even when the probe fails, which + // keeps teardown on its graceful-shutdown path, and a control failure + // takes precedence over a probe failure. const control = new FixtureControlClient(this.controlPath); - await control.connect(); - this.control = control; + const [connected, probed] = await Promise.allSettled([ + control.connect(), + this.probeCatalog(), + ]); + if (connected.status === "fulfilled") this.control = control; + if (connected.status === "rejected") throw connected.reason; + if (probed.status === "rejected") throw probed.reason; + } + /** Prove the client-visible path: connection-file read, auth handshake, catalog over the real wire. */ + private async probeCatalog(): Promise { const probe = await McHostClient.connect({ connectionFile: this.connectionFile, }); @@ -932,6 +955,18 @@ export class HermeticMcHostStack { } } + /** + * Continue a live child before signalling teardown. A SIGSTOPped fixture + * runs no graceful shutdown and holds SIGTERM pending until it resumes, so + * every teardown path burns its full timeout without this. SIGCONT to a + * running process has no effect, which is why the resume stays + * unconditional instead of tracking paused state. + */ + private resumeBeforeTeardown(child: ChildProcess): void { + if (child.exitCode === null && child.signalCode === null) + child.kill("SIGCONT"); + } + private requireControl(): FixtureControlClient { if (!this.control) throw new Error("direct mc-host fixture control is unavailable"); diff --git a/packages/plugin/src/shared/mc-host-client/client.ts b/packages/plugin/src/shared/mc-host-client/client.ts index 3ac6c25e1..5a13b3e26 100644 --- a/packages/plugin/src/shared/mc-host-client/client.ts +++ b/packages/plugin/src/shared/mc-host-client/client.ts @@ -1482,12 +1482,12 @@ function terminalFromErrorBody(body: Uint8Array): McHostCallError { if (typeof parsed === "object" && parsed !== null) { const code = typeof parsed.code === "string" ? parsed.code : undefined; const message = typeof parsed.message === "string" ? parsed.message : undefined; - return new McHostCallError("terminal", message ?? "subc error", code); + return new McHostCallError("terminal", message ?? "mc-host error", code); } } catch { // Fall through to the opaque-body form. } - return new McHostCallError("terminal", text || "subc error"); + return new McHostCallError("terminal", text || "mc-host error"); } function parseResponseJson(terminal: RequestTerminal): Response { From 11ba1ef5ec2800df64208b2f090d95f222da1997 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 08:09:37 +0000 Subject: [PATCH 04/37] style: satisfy biome on the files this branch moved `Check (plugin)` has failed its Lint step since the first push on this branch. Moving the client sources under `shared/mc-host-client/` left six files with import order biome's assist rewrites, and two test files with template literals it reflows now that the surrounding lines changed. Formatting and import order only; `biome check --write` produced every line. The same command on `main` reports five warnings and no errors, which is why the step passed there. --- .../magic-context/module-transport.test.ts | 4 ++-- .../hooks/magic-context/module-transport.ts | 4 ++-- .../plugin/src/shared/mc-host-client/client.ts | 2 +- .../src/shared/mc-host-client/connection.ts | 2 +- .../plugin/src/shared/mc-host-client/index.ts | 4 ++-- .../shared/mc-host-client/tcp-frame-channel.ts | 2 +- .../transport-negotiation.test.ts | 18 ++++++------------ packages/plugin/src/shared/redaction.test.ts | 4 +--- 8 files changed, 16 insertions(+), 24 deletions(-) diff --git a/packages/plugin/src/hooks/magic-context/module-transport.test.ts b/packages/plugin/src/hooks/magic-context/module-transport.test.ts index 770f9ecd6..627d0e2d9 100644 --- a/packages/plugin/src/hooks/magic-context/module-transport.test.ts +++ b/packages/plugin/src/hooks/magic-context/module-transport.test.ts @@ -7,10 +7,10 @@ import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { Deadline, - type RouteHandle, - StaleRouteHandleError, McHostCallError, type McHostClient, + type RouteHandle, + StaleRouteHandleError, } from "../../shared/mc-host-client"; import { FakePeer, diff --git a/packages/plugin/src/hooks/magic-context/module-transport.ts b/packages/plugin/src/hooks/magic-context/module-transport.ts index 8f1144b28..88351d62e 100644 --- a/packages/plugin/src/hooks/magic-context/module-transport.ts +++ b/packages/plugin/src/hooks/magic-context/module-transport.ts @@ -14,14 +14,14 @@ import { Deadline, isConsumerReconnectTransient, isMcHostCallError, + McHostCallError, + McHostClient, Priority, type RouteHandle, type RouteTarget, SocketClosedError, SocketTimeoutError, StaleRouteHandleError, - McHostCallError, - McHostClient, } from "../../shared/mc-host-client"; import { isRecord } from "../../shared/record-type-guard"; diff --git a/packages/plugin/src/shared/mc-host-client/client.ts b/packages/plugin/src/shared/mc-host-client/client.ts index 5a13b3e26..e6e0f737e 100644 --- a/packages/plugin/src/shared/mc-host-client/client.ts +++ b/packages/plugin/src/shared/mc-host-client/client.ts @@ -32,9 +32,9 @@ import { import { armExpiryTimer, Deadline, type MonotonicClock } from "./deadline"; import { isMcHostCallError, - SocketTimeoutError, McHostCallError, McHostClientError, + SocketTimeoutError, } from "./errors"; import { flagsBinary } from "./protocol"; import { diff --git a/packages/plugin/src/shared/mc-host-client/connection.ts b/packages/plugin/src/shared/mc-host-client/connection.ts index 9d235ba7f..e1b8cd7ff 100644 --- a/packages/plugin/src/shared/mc-host-client/connection.ts +++ b/packages/plugin/src/shared/mc-host-client/connection.ts @@ -21,7 +21,7 @@ import { AuthError } from "./auth"; import { armExpiryTimer, type Deadline } from "./deadline"; -import { SocketClosedError, SocketTimeoutError, McHostCallError } from "./errors"; +import { McHostCallError, SocketClosedError, SocketTimeoutError } from "./errors"; import { ByteBudget, type FrameChannelCloseReason, diff --git a/packages/plugin/src/shared/mc-host-client/index.ts b/packages/plugin/src/shared/mc-host-client/index.ts index 4ef55ef6b..eb74c5012 100644 --- a/packages/plugin/src/shared/mc-host-client/index.ts +++ b/packages/plugin/src/shared/mc-host-client/index.ts @@ -14,10 +14,10 @@ export { } from "./deadline"; export { isMcHostCallError, - SocketClosedError, - SocketTimeoutError, McHostCallError, McHostClientError, + SocketClosedError, + SocketTimeoutError, } from "./errors"; export { RouteHandle, StaleRouteHandleError } from "./route-handle"; export { diff --git a/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts b/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts index 9c1f080a9..6e50ec294 100644 --- a/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts +++ b/packages/plugin/src/shared/mc-host-client/tcp-frame-channel.ts @@ -14,7 +14,7 @@ import { Socket } from "node:net"; import { type AuthByteIo, AuthError, type AuthResult, authenticateClient } from "./auth"; import type { Deadline } from "./deadline"; -import { SocketClosedError, SocketTimeoutError, McHostCallError } from "./errors"; +import { McHostCallError, SocketClosedError, SocketTimeoutError } from "./errors"; import { type ByteBudget, type FrameChannel, diff --git a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts index 12350bfd7..bc327293c 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts @@ -39,10 +39,8 @@ const RESP_TCP_DIRECT = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1}}'; const RESP_TCP_FALLBACK = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"capability_version_mismatch"}'; -const RESP_GRANT = - `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{}}`; -const ACTIVATE_REQ = - `{"op":"transport.activate","negotiation_version":1,"activation_token":"${VECTOR_TOKEN}"}`; +const RESP_GRANT = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{}}`; +const ACTIVATE_REQ = `{"op":"transport.activate","negotiation_version":1,"activation_token":"${VECTOR_TOKEN}"}`; const ACTIVATE_RESP = '{"op":"transport.activate","negotiation_version":1}'; const COMMIT_REQ = '{"op":"transport.commit","negotiation_version":1}'; const COMMIT_RESP = '{"op":"transport.commit","negotiation_version":1}'; @@ -230,8 +228,7 @@ describe("recursive duplicate-key rejection", () => { expectCode(() => decodeNegotiateRequest(bytes(body)), "malformed_json"); } - const descriptor = - `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{"a":{"k":1,"k":2}}}`; + const descriptor = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{"a":{"k":1,"k":2}}}`; expectCode( () => decodeNegotiateResponse(bytes(descriptor), [shmOffer(1), tcpOffer(1)]), "malformed_json", @@ -376,14 +373,12 @@ describe("grant and tcp field mixes", () => { const noToken = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"descriptor":{}}'; expectCode(() => decodeNegotiateResponse(bytes(noToken), offers), "missing_field"); - const noDescriptor = - `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; + const noDescriptor = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; expectCode(() => decodeNegotiateResponse(bytes(noDescriptor), offers), "missing_field"); }); test("a tcp selection carrying either grant field is rejected", () => { - const withToken = - `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; + const withToken = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"activation_token":"${VECTOR_TOKEN}"}`; expectCode(() => decodeNegotiateResponse(bytes(withToken), offers), "unexpected_field"); const withDescriptor = '{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"descriptor":{}}'; @@ -394,8 +389,7 @@ describe("grant and tcp field mixes", () => { }); test("a grant carrying a fallback reason is rejected", () => { - const grantWithReason = - `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{},"reason":"unavailable"}`; + const grantWithReason = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"shm","capability_version":1},"activation_token":"${VECTOR_TOKEN}","descriptor":{},"reason":"unavailable"}`; expectCode( () => decodeNegotiateResponse(bytes(grantWithReason), offers), "unexpected_field", diff --git a/packages/plugin/src/shared/redaction.test.ts b/packages/plugin/src/shared/redaction.test.ts index 3c1a35681..9297a81a5 100644 --- a/packages/plugin/src/shared/redaction.test.ts +++ b/packages/plugin/src/shared/redaction.test.ts @@ -26,9 +26,7 @@ describe("redactSecretText — token counts and scalar diagnostics stay visible" expect(redactSecretText(`api_key=${syntheticApiKey}`)).toContain(" { From d79c109e3b29201210064c50f9a2bddf7d239550 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 08:29:54 +0000 Subject: [PATCH 05/37] fix(mc-module): bound the whole subscribe attempt and keep TimedOut Moving the producer's subscribe timeout into `RequestOptions` dropped the whole-attempt wrapper, and two properties went with it. Expiry stopped being distinguishable. A per-request deadline surfaces as `ResponseStream::next` returning `deadline_expired`, which maps to `HistorianProducerError::Call`. Firing, reattachment, and classify grant their recovery re-drain only on `TimedOut`, so a run that would have finished inside the recovery window was cancelled and its output discarded. The bound stopped covering route opening. `ensure_subscribe_route` runs before the stream request and carries the client's own 30-second route-open timeout, so an attempt could overshoot the caller's budget by that much and leave an outer cancel landing during `session.delete`. `subscribe_from_start` wraps `subscribe_and_drain` again and maps expiry to `TimedOut`; the per-request timeout stays as the transport bound. The test stalls the subscription so only the attempt's own bound can end the wait, and guards itself with an outer timeout: the fake connection does not honor `RequestOptions`, so without the wrapper the call never returns, and the guard turns that into a failure rather than a hung suite. --- crates/mc-module/src/historian_producer.rs | 65 ++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index ce4d4652b..e6a7c818b 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -1000,10 +1000,31 @@ impl HistorianProducer { Ok(serde_json::from_slice(&response)?) } + /// Bounds the whole attempt, not just the drain. + /// + /// Opening the subscription route is itself a request carrying the client's + /// own 30-second route-open timeout, so a per-request bound alone lets one + /// attempt overshoot the margin reserved for cleanup and leaves an outer + /// cancel to land during `session.delete`. Expiry must also stay + /// distinguishable: the firing, reattach, and classify paths grant their + /// recovery re-drain only on `TimedOut`, and a per-request deadline + /// surfaces as a `Call` failure instead, so a run that would finish inside + /// the recovery window gets cancelled and discarded. async fn subscribe_from_start( &mut self, run_id: &str, timeout: Duration, + ) -> Result { + match tokio::time::timeout(timeout, self.subscribe_and_drain(run_id, timeout)).await { + Ok(result) => result, + Err(_) => Err(HistorianProducerError::TimedOut), + } + } + + async fn subscribe_and_drain( + &mut self, + run_id: &str, + timeout: Duration, ) -> Result { let route = self.ensure_subscribe_route().await?; let body = serde_json::to_vec(&json!({ @@ -1356,6 +1377,7 @@ mod tests { close_route_errors: VecDeque, close_calls: usize, next_channel: u16, + stall_stream: bool, } #[async_trait] @@ -1400,6 +1422,9 @@ mod tests { _body: Vec, _options: RequestOptions, ) -> Result, HistorianProducerError> { + if self.state.lock().unwrap().stall_stream { + return Ok(Box::new(StallingStream)); + } Ok(Box::new(FakeStream(VecDeque::new()))) } @@ -1427,6 +1452,17 @@ mod tests { } } + /// A subscription that never yields an item and never ends, so only the + /// attempt's own bound can end the wait. + struct StallingStream; + + #[async_trait] + impl ProducerStream for StallingStream { + async fn next(&mut self) -> Result, HistorianProducerError> { + std::future::pending().await + } + } + struct FakeConnector { initial: FakeConnection, reconnects: Mutex)>>, @@ -1765,6 +1801,35 @@ mod tests { assert!(classify_run_state("run", &json!({"run_id":"other", "state":"missing"})).is_err()); } + #[tokio::test] + async fn an_attempt_that_outlives_its_budget_reports_timed_out() { + // `TimedOut` is not interchangeable with a `Call` failure carrying + // `deadline_expired`: only the former earns the recovery re-drain, so a + // stalled attempt must keep reporting the variant its callers match on. + // A stalled subscription can never win the race, so a short real budget + // is deterministic. + let connection = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); + connection.state.lock().unwrap().stall_stream = true; + let (mut producer, _) = producer(connection, None).await; + producer.bind_session("session"); + + let error = tokio::time::timeout( + Duration::from_secs(5), + producer.await_output_with_timeout("run", Duration::from_millis(20)), + ) + .await + // The fake connection does not honor `RequestOptions`, so nothing but + // the attempt's own bound can end this wait. Without that bound the call + // never returns, and this outer guard turns the regression into a + // failure instead of a hung suite. + .expect("the attempt's own bound must end the wait") + .expect_err("a stalled subscription cannot complete"); + assert!( + matches!(error, HistorianProducerError::TimedOut), + "expected TimedOut, got {error:?}" + ); + } + #[test] fn error_class_wire_strings_match_pinned_contract_set() { assert_eq!(ErrorClass::Transient.as_wire_str(), "transient"); From 6f0a02c2acf5ed1aeacac0f09b5da56337ba81a5 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 08:38:17 +0000 Subject: [PATCH 06/37] test(evidence): re-pin the v84 crash-campaign implementation digest `storage-memory-claims-crash.test.ts` hashes the full bytes of the 31 paths in its `IMPLEMENTATION_FILES` manifest and compares that digest to the checked-in evidence. One of those paths is `packages/e2e-tests/src/opencode-runner/spawn.ts`, so interpolating the fixture-start cause into its error moved the digest and the pinned value went stale. Regenerated with `UPDATE_CLAIMS_CRASH_EVIDENCE=1`, which reran the campaign. Only `commitUnderTest` and `dirtyDiffDigest` changed; the matrix, per-scenario semantic digests, summary, limits, runtimes, and environment are byte-identical, so the recorded claim is the same campaign result under a new manifest hash. The failure was masked until now: the Lint step failed ahead of Test, so CI never reached this assertion. --- docs/evidence/claims-backfill/v84-process-crash.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/evidence/claims-backfill/v84-process-crash.json b/docs/evidence/claims-backfill/v84-process-crash.json index fce8db839..e1049e1f4 100644 --- a/docs/evidence/claims-backfill/v84-process-crash.json +++ b/docs/evidence/claims-backfill/v84-process-crash.json @@ -1,8 +1,8 @@ { "schemaVersion": "claims-process-crash-evidence/v1", - "commitUnderTest": "bd00ffffbdd026a87c3536d013d3d9c0f7ea8bf7", + "commitUnderTest": "d79c109e3b29201210064c50f9a2bddf7d239550", "dirtyDiffDigestPolicy": "sha256(sorted U6 implementation path + NUL + full file bytes + NUL); evidence file excluded", - "dirtyDiffDigest": "d4665b9db71c39dcc511c10a6e365b01916bebedf9a8ed488709d46e8bc977f3", + "dirtyDiffDigest": "3a5840fe7ead4afc757c072f802c6077b934902d091ec8b8874fe64c28172f46", "implementationFiles": [ "ARCHITECTURE.md", "STRUCTURE.md", From 0f1c5baba54ae0a87f8d9259a3081c72bb3c45f2 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 08:55:54 +0000 Subject: [PATCH 07/37] fix(mc-host): retire the stream watcher on every settle path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deadline watcher was retired by an explicit call in `finish_pending`, which missed the path that matters most. A stream that ends normally is settled inside `dispatch`: the `Response`/`Error`/`StreamEnd` branch removes the pending entry and sends the terminal directly, so the token was dropped without being cancelled and the watcher slept on until the original deadline. That is the accumulation the explicit call was meant to prevent, on the common case rather than the cancel case. `PendingKind::Stream` now holds a `DropGuard` instead of a token, so the watcher is retired by dropping the entry. Every path that settles a stream already removes it from `pending`, including paths not yet written, which makes this structural rather than a convention each new settle site has to remember. Stop emitting one `Cancel` per claimed request while settling a route. Route `Goodbye` — the frame this settlement accompanies on close, or the inbound frame that triggered it — already obliges the host to stop dispatch and settle or cancel that route's work, so the frames add nothing. They can also exceed the 32 reserved control slots, and `send_control` retires the whole generation on overflow: a routine route teardown carrying more than 32 claimed requests took down every unrelated route with it. `settle_all` was already silent for the same reason. Both tests fail against the previous shape: the watcher test fails its `Some(StreamEnd)` iteration while the cancel iteration still passes, and the settlement test observes the generation retire. --- crates/mc-host/src/client.rs | 163 ++++++++++++++++++++++++----------- 1 file changed, 113 insertions(+), 50 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 4b7008aef..5803fd11b 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -24,7 +24,7 @@ use tokio::{ task::JoinHandle, time::{timeout_at, Instant}, }; -use tokio_util::sync::CancellationToken; +use tokio_util::sync::{CancellationToken, DropGuard}; use crate::{ auth::authenticate_client, @@ -661,10 +661,13 @@ enum PendingKind { Stream { items: mpsc::Sender, terminal: oneshot::Sender>, - /// Fired once when this stream settles, so the detached deadline - /// watcher stops sleeping instead of outliving the stream by up to the - /// caller's whole timeout. - settled: CancellationToken, + /// Retires the detached deadline watcher when this entry is dropped, so + /// the watcher cannot outlive the stream by up to the caller's whole + /// timeout. A guard rather than a bare token because settlement has + /// several sites - the terminal-frame branch in `dispatch` settles the + /// caller directly without `finish_pending` - and dropping the entry is + /// the one thing every path, present or future, already does. + _settled: DropGuard, }, } @@ -742,7 +745,7 @@ impl Inner { PendingKind::Stream { items: item_tx, terminal: terminal_tx, - settled: settled.clone(), + _settled: settled.clone().drop_guard(), }, deadline, ); @@ -1034,6 +1037,10 @@ impl Inner { let _ = tx.send(result); } PendingKind::Stream { terminal, .. } => { + // Settles the caller directly rather than through + // `finish_pending`, so the deadline watcher is retired + // by dropping the rest of the entry; see + // `PendingKind::Stream::_settled`. let terminal_result = match header.ty { FrameType::StreamEnd => Ok(()), FrameType::Error => Err(CallError::host_terminal(&body)), @@ -1125,15 +1132,11 @@ impl Inner { PendingKind::Unary(tx) => { let _ = tx.send(result); } - PendingKind::Stream { - terminal, settled, .. - } => { + PendingKind::Stream { terminal, .. } => { let terminal_result = result.map(|_| ()); let _ = terminal.send(terminal_result); - // Every settle path — terminal frame, caller cancel, stream - // drop, route settle, generation retire — funnels through here, - // so this is the single point that retires the watcher task. - settled.cancel(); + // Dropping the rest of the entry retires the deadline watcher; + // see `PendingKind::Stream::_settled`. self.release_stream(); } } @@ -1144,6 +1147,17 @@ impl Inner { *streams = streams.saturating_sub(1); } + /// Settles every pending request on one route and drops the route. + /// + /// Emits no per-correlation `Cancel`. Route `Goodbye` — the frame this + /// settlement accompanies on close, or the inbound frame that triggered it — + /// already obliges the host to stop dispatch and settle or cancel that + /// route's work (protocol §11.2). Sending one `Cancel` per possibly-sent + /// request would add nothing and can exceed the 32 reserved control slots, + /// and `send_control` retires the whole generation on overflow: a routine + /// route teardown carrying more than 32 claimed requests would take down + /// every unrelated route with it. `settle_all` is silent for the same + /// reason. fn settle_route(&self, route: RouteHandle) -> bool { let pending = { let _admission = lock_unpoisoned(&self.admission); @@ -1160,23 +1174,12 @@ impl Inner { .filter_map(|key| pending.remove(&key).map(|state| (key, state))) .collect::>() }; - for (key, state) in pending { + for (_key, state) in pending { let outcome = cancel_classification(&state.publish); self.finish_pending( state, Err(CallError::local(outcome, "route_gone", "request stopped")), ); - if outcome == SendOutcome::OutcomeUnknown { - let _ = self.send_control( - FrameType::Cancel, - FrameId { - channel: key.channel, - epoch: key.epoch, - corr: key.corr, - }, - None, - ); - } } true } @@ -2101,34 +2104,94 @@ mod tests { #[tokio::test] async fn settled_stream_retires_its_deadline_watcher() { - let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); - let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); - let (terminal_tx, terminal_rx) = oneshot::channel(); - let settled = CancellationToken::new(); - let (key, _publish) = inner - .admit( - route(1), - Vec::new(), - PendingKind::Stream { - items: items_tx, - terminal: terminal_tx, - settled: settled.clone(), - }, - Instant::now() + Duration::from_secs(600), - ) - .expect("stream admitted"); + // Both settle shapes must retire the watcher: `cancel_key` funnels + // through `finish_pending`, while a terminal frame settles the caller + // directly inside `dispatch`. Only dropping the entry covers both. + for terminal in [None, Some(FrameType::StreamEnd)] { + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let settled = CancellationToken::new(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: settled.clone().drop_guard(), + }, + Instant::now() + Duration::from_secs(600), + ) + .expect("stream admitted"); + assert!( + !settled.is_cancelled(), + "the watcher must stay armed while the stream is live" + ); + drop(data_rx.recv().await); + + match terminal { + None => inner.cancel_key(key, "cancelled").expect("stream settled"), + Some(ty) => inner.dispatch( + EnvelopeHeader { + len: 0, + ver: PROTOCOL_VERSION, + ty, + flags: response_flags(false, true), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + Vec::new(), + None, + ), + } + + assert!( + settled.is_cancelled(), + "settling via {terminal:?} must retire the watcher instead of \ + leaving it asleep until the deadline" + ); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + let _ = terminal_rx.await; + } + } + + #[tokio::test] + async fn route_settlement_never_floods_the_reserved_control_queue() { + // One Cancel per claimed request overruns the 32 reserved control slots, + // and `send_control` retires the generation on overflow - so a routine + // route teardown would take every unrelated route with it. + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let route = route(1); + lock_unpoisoned(&inner.routes).insert(route); + for _ in 0..CLIENT_CONTROL_QUEUE_FRAMES + 1 { + let (kind, _rx) = unary_sender(); + let (_key, publish) = inner + .admit( + route, + Vec::new(), + kind, + Instant::now() + Duration::from_secs(60), + ) + .expect("admitted"); + // Claim each request so settlement classifies it possibly-sent, + // which is the case that used to emit a Cancel. + assert!(claim_for_write(&publish)); + drop(data_rx.recv().await); + } + + assert!(inner.settle_route(route)); + assert!( - !settled.is_cancelled(), - "the watcher must stay armed while the stream is live" + !inner.retired.load(Ordering::Acquire), + "route settlement must not retire the generation" ); - drop(data_rx.recv().await); - - inner.cancel_key(key, "cancelled").expect("stream settled"); assert!( - settled.is_cancelled(), - "settlement must retire the watcher instead of leaving it asleep until the deadline" + control_rx.try_recv().is_err(), + "route Goodbye already settles the host side; per-correlation Cancel adds only overflow risk" ); - let _ = terminal_rx.await; + assert!(lock_unpoisoned(&inner.pending).is_empty()); } #[test] @@ -2364,7 +2427,7 @@ mod tests { PendingKind::Stream { items: items_tx, terminal: terminal_tx, - settled: CancellationToken::new(), + _settled: CancellationToken::new().drop_guard(), }, Instant::now() + Duration::from_secs(1), ) From 35af65f6fd359567aebfb6416e7c01f988024c9a Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:11:44 +0000 Subject: [PATCH 08/37] fix: close the fallback vocabulary and stop harness errors masking causes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `negotiation_version_mismatch` was still accepted as fallback evidence by both clients while §7.7.3 lists only `unavailable` and `capability_version_mismatch`, and states that a negotiation-version mismatch is not fallback evidence and must fail closed with no same-generation TCP continuation. A host answering a non-TCP offer with that reason therefore committed the generation to plain TCP instead of retiring it — the downgrade V53 exists to forbid. Removed from the Rust enum and the TypeScript array. Both closed-table tests iterated their own vocabulary, so neither could see the drift. They now pin the two literals and assert the excluded reasons are rejected. Stop teardown failures from replacing the errors that caused them. `HermeticMcHostStack.start` and `RustTestHarness.create` awaited a teardown that itself throws when a child survives its escalation windows, so a readiness timeout or spawn failure surfaced as "did not exit during teardown"; in the harness, a throwing `mcHost.stop()` also skipped `mock.stop()` and left its HTTP listener running. Reclaim the isolated temp tree when direct-host provisioning fails. `createIsolatedEnv` runs before the caller's `try`, so nothing removed those directories and the reaper only kills recorded PIDs. Removal is skipped when `dataDir` survives, because that is teardown's record of a child it could not reap and the next run's only handle on it. Restore the built-bundle preference for the plugin entry. The e2e jobs build `packages/plugin/dist` before running, and loading raw `src/index.ts` transpiles hundreds of submodule imports at boot, which is what made `opencode serve` look hung on slow runners. Deduplicate the child-exit primitive into `src/process-exit.ts`, and name the producer close-and-log cleanup once instead of at five call sites. --- crates/mc-host/src/transport_negotiation.rs | 3 -- crates/mc-host/tests/transport_negotiation.rs | 31 +++++++++----- crates/mc-module/src/historian.rs | 20 ++++++--- .../e2e-tests/src/opencode-runner/spawn.ts | 42 +++++++++++-------- packages/e2e-tests/src/process-exit.ts | 32 ++++++++++++++ packages/e2e-tests/src/rust-harness.ts | 15 ++++++- .../src/rust-runner/hermetic-mc-host.ts | 35 +++++----------- .../transport-negotiation.test.ts | 11 ++++- .../mc-host-client/transport-negotiation.ts | 16 ++++--- 9 files changed, 136 insertions(+), 69 deletions(-) create mode 100644 packages/e2e-tests/src/process-exit.ts diff --git a/crates/mc-host/src/transport_negotiation.rs b/crates/mc-host/src/transport_negotiation.rs index c9fded47b..689eb525f 100644 --- a/crates/mc-host/src/transport_negotiation.rs +++ b/crates/mc-host/src/transport_negotiation.rs @@ -114,7 +114,6 @@ impl std::error::Error for NegotiationError {} #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum FallbackReason { Unavailable, - NegotiationVersionMismatch, CapabilityVersionMismatch, } @@ -122,7 +121,6 @@ impl FallbackReason { pub fn as_str(&self) -> &'static str { match self { Self::Unavailable => "unavailable", - Self::NegotiationVersionMismatch => "negotiation_version_mismatch", Self::CapabilityVersionMismatch => "capability_version_mismatch", } } @@ -130,7 +128,6 @@ impl FallbackReason { pub fn parse(value: &str) -> Option { match value { "unavailable" => Some(Self::Unavailable), - "negotiation_version_mismatch" => Some(Self::NegotiationVersionMismatch), "capability_version_mismatch" => Some(Self::CapabilityVersionMismatch), _ => None, } diff --git a/crates/mc-host/tests/transport_negotiation.rs b/crates/mc-host/tests/transport_negotiation.rs index 11f6308b1..f47ec3fe8 100644 --- a/crates/mc-host/tests/transport_negotiation.rs +++ b/crates/mc-host/tests/transport_negotiation.rs @@ -146,13 +146,11 @@ fn version_mismatches_encode_the_documented_tcp_fallback_reasons() { RESP_TCP_FALLBACK.as_bytes() ); - // The whole closed table round-trips; anything else is rejected. + // The closed table is pinned to §7.7.3's two literals rather than derived + // from the enum: a value the enum accepts but the table omits is exactly the + // fail-open this checks for. for (name, expected) in [ ("unavailable", FallbackReason::Unavailable), - ( - "negotiation_version_mismatch", - FallbackReason::NegotiationVersionMismatch, - ), ( "capability_version_mismatch", FallbackReason::CapabilityVersionMismatch, @@ -170,11 +168,24 @@ fn version_mismatches_encode_the_documented_tcp_fallback_reasons() { }; assert_eq!(reason, Some(expected)); } - let unknown = r#"{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"switching_transports"}"#; - assert_eq!( - code(decode_negotiate_response(unknown.as_bytes(), &offers)), - NegotiationErrorCode::InvalidReason - ); + // §7.7.3 names these as not fallback evidence: a TCP selection carrying one + // must fail closed rather than commit the generation to TCP. + for rejected in [ + "switching_transports", + "negotiation_version_mismatch", + "connection_in_use", + "unsupported_operation", + ] { + let body = format!( + r#"{{"op":"transport.negotiate","negotiation_version":1,"selected":{{"transport":"tcp","capability_version":1}},"reason":"{rejected}"}}"# + ); + assert_eq!( + code(decode_negotiate_response(body.as_bytes(), &offers)), + NegotiationErrorCode::InvalidReason, + "{rejected} must not be accepted as fallback evidence" + ); + assert_eq!(FallbackReason::parse(rejected), None); + } } #[test] diff --git a/crates/mc-module/src/historian.rs b/crates/mc-module/src/historian.rs index 4f8194f31..8c22cfb62 100644 --- a/crates/mc-module/src/historian.rs +++ b/crates/mc-module/src/historian.rs @@ -1236,6 +1236,16 @@ fn log_cleanup_failure( } } +/// Closes the producer and logs a close failure without changing the outcome +/// being returned. Named once because the drive paths exit through several +/// branches that each owe this same cleanup. +async fn close_and_log

(producer: &mut P, session_id: &str) +where + P: HistorianProducerDriver + ?Sized, +{ + log_cleanup_failure(session_id, "close", &producer.close().await); +} + fn cancellation_confirmed_stopped(result: &Result<(), HistorianProducerError>) -> bool { match result { Ok(()) => true, @@ -1452,15 +1462,15 @@ where log_cleanup_failure(request.session_id, "attempt close", &cleanup); continue; } - log_cleanup_failure(request.session_id, "close", &producer.close().await); + close_and_log(producer, request.session_id).await; return Err(HistorianDriveError::Validation(err)); } Err(err) => { - log_cleanup_failure(request.session_id, "close", &producer.close().await); + close_and_log(producer, request.session_id).await; return Err(err); } }; - log_cleanup_failure(request.session_id, "close", &producer.close().await); + close_and_log(producer, request.session_id).await; return Ok(HistorianDriveOutcome::Completed(HistorianRunSuccess { row_version, producer_session_id, @@ -1528,7 +1538,7 @@ where (request.completion_now_ms)(), ); abandon_current_state(request.store, request.session_id, failure_backoff_at_ms)?; - log_cleanup_failure(request.session_id, "close", &producer.close().await); + close_and_log(producer, request.session_id).await; return Ok(HistorianReattachOutcome::RefireEligible { firing_seq }); } } @@ -1615,7 +1625,7 @@ where completion_now_ms: request.completion_now_ms, publication_fence: request.publication_fence, }); - log_cleanup_failure(request.session_id, "close", &producer.close().await); + close_and_log(producer, request.session_id).await; let row_version = publish_result?; Ok(HistorianReattachOutcome::Published(HistorianRunSuccess { row_version, diff --git a/packages/e2e-tests/src/opencode-runner/spawn.ts b/packages/e2e-tests/src/opencode-runner/spawn.ts index 9a8adab65..c7518a83f 100644 --- a/packages/e2e-tests/src/opencode-runner/spawn.ts +++ b/packages/e2e-tests/src/opencode-runner/spawn.ts @@ -8,12 +8,13 @@ */ import { type ChildProcess, spawn } from "node:child_process"; -import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { runMigrations } from "../../../plugin/src/features/magic-context/migrations"; import { initializeDatabase } from "../../../plugin/src/features/magic-context/storage-db"; import { Database } from "../../../plugin/src/shared/sqlite"; +import { waitForChildExit } from "../process-exit"; import { buildDirectHostFixture, detectRustModePrereqs, @@ -21,7 +22,17 @@ import { } from "../rust-runner/hermetic-mc-host"; const REPO_ROOT = resolve(import.meta.dir, "../../../.."); -const PLUGIN_ENTRY = join(REPO_ROOT, "packages/plugin/src/index.ts"); +// Prefer the built bundle over raw `src/index.ts`. The bundle is one file with +// all imports inlined and loads fast even on a cold runner, while the TS-source +// path triggers Bun's runtime transpile and dynamic resolution across hundreds +// of submodule imports — enough on a slow CI runner to make `opencode serve` +// look hung when it is only blocked in plugin load. Production never loads from +// src/, so the source path also tests a slowness users never see. +const PLUGIN_DIST_ENTRY = join(REPO_ROOT, "packages/plugin/dist/index.js"); +const PLUGIN_SRC_ENTRY = join(REPO_ROOT, "packages/plugin/src/index.ts"); +const PLUGIN_ENTRY = existsSync(PLUGIN_DIST_ENTRY) + ? PLUGIN_DIST_ENTRY + : PLUGIN_SRC_ENTRY; function initializeIsolatedContextDb(dataDir: string): void { const path = join(dataDir, "cortexkit", "magic-context", "context.db"); @@ -297,21 +308,6 @@ function rejectOnSpawnError(child: ChildProcess, cancellation?: AbortSignal): Pr }); } -function waitForChildExit(child: ChildProcess, timeoutMs: number): Promise { - if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve(true); - return new Promise((resolveExit) => { - const onExit = (): void => { - clearTimeout(timer); - resolveExit(true); - }; - const timer = setTimeout(() => { - child.off("exit", onExit); - resolveExit(false); - }, timeoutMs); - child.once("exit", onExit); - }); -} - async function stopChild(child: ChildProcess, timeoutMs = 3_000): Promise { if (child.exitCode !== null || child.signalCode !== null || child.pid === undefined) return; @@ -340,6 +336,18 @@ async function provisionRustMode(): Promise { const mcHost = await HermeticMcHostStack.start({ dataDir: env.dataDir, fixtureBin }); return { env, connectionFile: mcHost.connectionFile, mcHost }; } catch (error) { + // This env has no other owner yet: `cleanup()` only runs for a stack + // that started, and the process reaper only kills recorded PIDs. A + // surviving dataDir is the stack's record that its own teardown could + // not reclaim it — the leaked fixture's PID file lives there and is the + // next run's only handle on that process — so the tree stays put then. + if (!existsSync(env.dataDir)) { + try { + rmSync(dirname(env.dataDir), { recursive: true, force: true }); + } catch { + // Temp litter never masks the startup failure. + } + } throw new Error( `MC_E2E_MODE=rust failed to start direct mc-host fixture: ${String(error)}`, ); diff --git a/packages/e2e-tests/src/process-exit.ts b/packages/e2e-tests/src/process-exit.ts new file mode 100644 index 000000000..836b2f9e1 --- /dev/null +++ b/packages/e2e-tests/src/process-exit.ts @@ -0,0 +1,32 @@ +/** Child-process exit primitive shared by the e2e runners. */ + +import type { ChildProcess } from "node:child_process"; + +/** + * Resolve true once `child` has exited, or false if it is still running after + * `timeoutMs`. + * + * A child that exited before this call resolves immediately: its `exit` event + * already fired and never fires again, so `exitCode`/`signalCode` are the only + * remaining record of it. Both settle paths detach their own resources — the + * timeout removes the `exit` listener, the exit clears the timer — so a caller + * that abandons the child after a false result leaves nothing attached to it. + */ +export function waitForChildExit( + child: ChildProcess, + timeoutMs: number, +): Promise { + if (child.exitCode !== null || child.signalCode !== null) + return Promise.resolve(true); + return new Promise((resolveExit) => { + const onExit = (): void => { + clearTimeout(timer); + resolveExit(true); + }; + const timer = setTimeout(() => { + child.off("exit", onExit); + resolveExit(false); + }, timeoutMs); + child.once("exit", onExit); + }); +} diff --git a/packages/e2e-tests/src/rust-harness.ts b/packages/e2e-tests/src/rust-harness.ts index a14be672f..af010ecd5 100644 --- a/packages/e2e-tests/src/rust-harness.ts +++ b/packages/e2e-tests/src/rust-harness.ts @@ -178,8 +178,19 @@ export class RustTestHarness { rustMode: !options.startInTsMode, }); } catch (error) { - await mcHost.stop(); - await mock.stop(); + // Independent steps: a failing teardown neither skips the ones + // after it — the mock's HTTP listener outlives this scope + // otherwise — nor replaces the spawn failure being reported. + try { + await mcHost.stop(); + } catch { + // ignore + } + try { + await mock.stop(); + } catch { + // ignore + } throw error; } diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index c8ef1dd85..2b501cf20 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -20,6 +20,7 @@ import { McHostClient, type BindIdentity, } from "@magic-context/core/shared/mc-host-client"; +import { waitForChildExit } from "../process-exit"; const REPO_ROOT = resolve(import.meta.dir, "../../../.."); const FIXTURE_BINARY = join( @@ -304,25 +305,6 @@ function appendBounded(current: string, chunk: Buffer): string { return `${current}${chunk.toString()}`.slice(-MAX_LOG_BYTES); } -function safeChildExit( - child: ChildProcess, - timeoutMs: number, -): Promise { - if (child.exitCode !== null || child.signalCode !== null) - return Promise.resolve(true); - return new Promise((resolveExit) => { - const timer = setTimeout(() => { - child.off("exit", onExit); - resolveExit(false); - }, timeoutMs); - const onExit = (): void => { - clearTimeout(timer); - resolveExit(true); - }; - child.once("exit", onExit); - }); -} - class FixtureControlClient { private socket: Socket | null = null; private incoming = Buffer.alloc(0); @@ -645,7 +627,10 @@ export class HermeticMcHostStack { await stack.startHost(); return stack; } catch (error) { - await stack.stop(); + // stop() throws when the child survives its teardown escalation. + // Discarding that keeps the startup cause as the thrown error; + // the surviving child stays reachable through its PID record. + await stack.stop().catch(() => undefined); throw error; } } @@ -729,7 +714,7 @@ export class HermeticMcHostStack { this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); - if (!(await safeChildExit(child, 5_000))) { + if (!(await waitForChildExit(child, 5_000))) { throw new Error( "direct mc-host fixture did not exit after SIGKILL", ); @@ -750,7 +735,7 @@ export class HermeticMcHostStack { this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGTERM"); - if (!(await safeChildExit(child, 10_000))) { + if (!(await waitForChildExit(child, 10_000))) { throw new Error( "direct mc-host fixture did not exit after SIGTERM", ); @@ -789,14 +774,14 @@ export class HermeticMcHostStack { } catch { // Fixture may already be unavailable. } - exited = await safeChildExit(child, 5_000); + exited = await waitForChildExit(child, 5_000); if (!exited) { child.kill("SIGTERM"); - exited = await safeChildExit(child, 5_000); + exited = await waitForChildExit(child, 5_000); } if (!exited) { child.kill("SIGKILL"); - exited = await safeChildExit(child, 5_000); + exited = await waitForChildExit(child, 5_000); } } this.control?.close(); diff --git a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts index bc327293c..187d26eae 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-negotiation.test.ts @@ -127,12 +127,21 @@ describe("fallback reasons", () => { test("the closed table decodes; anything else is rejected", () => { const offers = [tcpOffer(1)]; + // Pinned to §7.7.3's two literals rather than derived from the exported + // array: a value the code accepts but the table omits is exactly the + // fail-open this guards, and iterating the array alone cannot see it. + expect([...FALLBACK_REASONS]).toEqual(["unavailable", "capability_version_mismatch"]); for (const reason of FALLBACK_REASONS) { const body = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"${reason}"}`; const response = decodeNegotiateResponse(bytes(body), offers); if (response.kind === "tcp") expect(response.reason).toBe(reason); } - for (const rejected of ["switching_transports", "connection_in_use"]) { + for (const rejected of [ + "switching_transports", + "connection_in_use", + "negotiation_version_mismatch", + "unsupported_operation", + ]) { const body = `{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1},"reason":"${rejected}"}`; expectCode(() => decodeNegotiateResponse(bytes(body), offers), "invalid_reason"); } diff --git a/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts b/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts index 37cd6d202..09645bf57 100644 --- a/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts +++ b/packages/plugin/src/shared/mc-host-client/transport-negotiation.ts @@ -75,12 +75,16 @@ export class NegotiationError extends Error { } } -/** Closed fallback vocabulary (wire doc Section 7.7.3). */ -export const FALLBACK_REASONS = [ - "unavailable", - "negotiation_version_mismatch", - "capability_version_mismatch", -] as const; +/** + * Closed fallback vocabulary (wire doc Section 7.7.3). + * + * Only these two reasons are fallback evidence. Every other setup outcome — + * negotiation-version mismatch, `unsupported_operation`, `connection_in_use`, + * timeout, malformed content, an unoffered selection — must fail closed with no + * same-generation TCP continuation, so accepting one here would commit the + * generation to TCP on evidence the protocol rejects. + */ +export const FALLBACK_REASONS = ["unavailable", "capability_version_mismatch"] as const; export type FallbackReason = (typeof FALLBACK_REASONS)[number]; function isFallbackReason(value: string): value is FallbackReason { From 994e48cb08a13d8fbfd6ca5797ea18701080c772 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:18:01 +0000 Subject: [PATCH 09/37] fix(mc-host): accept any valid Ping priority and echo it in the Pong MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client compared a pure-header frame's whole flag byte against `pure_header_flags()`, which pins priority to Passive. §6.1 fixes only binary, last, and admission class on those frames and permits any valid priority, so a conforming host Ping at Interactive or Background retired the generation instead of being answered. The framing layer's own check in `tcp_frame_channel` already had this right; the client is now the same three-part check. The Pong also has to echo the Ping's flags exactly (V35) and was built from the same fixed helper, so even an accepted Ping was answered with the wrong flag byte. `send_control` now takes the flags explicitly, which makes the choice visible at each control site rather than defaulted. Omit `admission_facts` instead of sending an explicit null. `json!` serialized `None` as a present null member, and the host's parser treats any present member as `Some(..)`, so bind observed facts the caller never supplied and a handler gating on `is_some()` decided on a value that means absence. Both tests fail against the previous behavior: the Ping test on the Interactive iteration, the route-open test on the null member. --- crates/mc-host/src/client.rs | 184 +++++++++++++++++++++++++++++++---- 1 file changed, 166 insertions(+), 18 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 5803fd11b..1472c0955 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -35,8 +35,8 @@ use crate::{ TRANSPORT_TCP, }, wire::{ - decode_header, encode_owned_frame, pure_header_flags, EnvelopeHeader, Flags, FrameId, - FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, PROTOCOL_VERSION, + decode_header, encode_owned_frame, pure_header_flags, AdmissionClass, EnvelopeHeader, + Flags, FrameId, FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, PROTOCOL_VERSION, }, }; @@ -915,6 +915,7 @@ impl Inner { // is replay-safe. if let Err(error) = self.send_control( FrameType::Cancel, + pure_header_flags(), FrameId { channel: key.channel, epoch: key.epoch, @@ -928,16 +929,22 @@ impl Inner { Ok(()) } + /// Queues one pure-header control frame. + /// + /// `flags` is explicit because a `Pong` must echo the `Ping`'s flags exactly + /// (conformance vector V35), and §6.1 lets a conforming peer pick any valid + /// priority - so no single flag byte is correct for every control frame. fn send_control( &self, ty: FrameType, + flags: Flags, id: FrameId, ack: Option>, ) -> Result<(), CallError> { if self.retired.load(Ordering::Acquire) { return Err(retired_error(SendOutcome::NotSent)); } - let bytes = encode_owned_frame(ty, pure_header_flags(), id, Vec::new()).map_err(|_| { + let bytes = encode_owned_frame(ty, flags, id, Vec::new()).map_err(|_| { CallError::local( SendOutcome::NotSent, "encode_failed", @@ -977,12 +984,13 @@ impl Inner { deadline: Instant, ) -> Result<(), ClientError> { let (tx, rx) = oneshot::channel(); - self.send_control(ty, id, Some(tx)).map_err(|_| { - ClientError::new( - "control_capacity_exhausted", - "client control admission failed", - ) - })?; + self.send_control(ty, pure_header_flags(), id, Some(tx)) + .map_err(|_| { + ClientError::new( + "control_capacity_exhausted", + "client control admission failed", + ) + })?; timeout_at(deadline, rx) .await .map_err(|_| ClientError::new("shutdown_timeout", "client shutdown timed out"))? @@ -997,7 +1005,13 @@ impl Inner { ) { match header.ty { FrameType::Ping => { - let _ = self.send_control(FrameType::Pong, FrameId::control(header.corr), None); + // V35: the Pong echoes the Ping's flags exactly. + let _ = self.send_control( + FrameType::Pong, + header.flags, + FrameId::control(header.corr), + None, + ); } FrameType::Goodbye if header.channel == 0 => self.retire("connection_goodbye"), FrameType::Goodbye => { @@ -1080,6 +1094,7 @@ impl Inner { ); let _ = self.send_control( FrameType::Cancel, + pure_header_flags(), FrameId { channel: key.channel, epoch: key.epoch, @@ -1112,6 +1127,7 @@ impl Inner { ); let _ = self.send_control( FrameType::Cancel, + pure_header_flags(), FrameId { channel: key.channel, epoch: key.epoch, @@ -1542,7 +1558,16 @@ fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { } _ => return Err(()), } - if header.ty.is_pure_header() && (header.flags != pure_header_flags() || header.len != 0) { + // Pure-header frames must set binary 0, last 0, and admission Normal, but + // §6.1 permits any valid priority — matching the framing layer's own check + // in `tcp_frame_channel`. Comparing the whole flag byte would retire the + // generation over a conforming Ping that merely chose Interactive. + if header.ty.is_pure_header() + && (header.len != 0 + || header.flags.is_binary() + || header.flags.is_last() + || header.flags.admission_class() != Some(AdmissionClass::Normal)) + { return Err(()); } Ok(()) @@ -1680,9 +1705,14 @@ fn route_open_body(target: &RouteTarget, identity: &RouteIdentity) -> Result RouteIdentity { + RouteIdentity { + project_root: std::path::PathBuf::from("/tmp/project"), + harness: "opencode".to_owned(), + session: "session".to_owned(), + consumer_module_id: None, + consumer_launch_nonce: None, + consumer_capabilities: Vec::new(), + admission_facts: None, + } + } + #[tokio::test] async fn dropped_unary_future_cleans_pending_and_possibly_sent_request() { let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); @@ -2324,7 +2447,12 @@ mod tests { assert_eq!(lock_unpoisoned(&inner.correlations).next, next_before); inner - .send_control(FrameType::Pong, FrameId::control(99), None) + .send_control( + FrameType::Pong, + pure_header_flags(), + FrameId::control(99), + None, + ) .expect("reserved control remains available"); let pong = control_rx.recv().await.expect("queued Pong"); assert_eq!(pong.bytes[5], FrameType::Pong as u8); @@ -2343,11 +2471,21 @@ mod tests { let (inner, data_rx, control_rx) = test_inner(CLIENT_QUEUED_BYTES); for corr in 1..=CLIENT_CONTROL_QUEUE_FRAMES as u64 { inner - .send_control(FrameType::Pong, FrameId::control(corr), None) + .send_control( + FrameType::Pong, + pure_header_flags(), + FrameId::control(corr), + None, + ) .expect("reserved slot"); } let error = inner - .send_control(FrameType::Pong, FrameId::control(99), None) + .send_control( + FrameType::Pong, + pure_header_flags(), + FrameId::control(99), + None, + ) .expect_err("33rd control retires generation"); assert_eq!(error.code(), "control_capacity_exhausted"); assert!(inner.retired.load(Ordering::Acquire)); @@ -2361,7 +2499,12 @@ mod tests { async fn data_and_control_charge_one_shared_byte_cap() { let (inner, data_rx, control_rx) = test_inner(HEADER_LEN * 2); inner - .send_control(FrameType::Pong, FrameId::control(1), None) + .send_control( + FrameType::Pong, + pure_header_flags(), + FrameId::control(1), + None, + ) .expect("first header"); let (kind, _rx) = unary_sender(); inner @@ -2374,7 +2517,12 @@ mod tests { .expect("data header uses remaining shared bytes"); assert_eq!(inner.queue_budget.used(), HEADER_LEN * 2); assert!(inner - .send_control(FrameType::Pong, FrameId::control(2), None) + .send_control( + FrameType::Pong, + pure_header_flags(), + FrameId::control(2), + None + ) .is_err()); drop(data_rx); drop(control_rx); From 4ed39dd0922ee14978b4e45dd029f69a7bcb67de Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:21:28 +0000 Subject: [PATCH 10/37] fix(mc-host): deliver a zero-length stream item instead of retiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `read_active_frame` leaves `charge` absent for a zero-length body because there is nothing to charge, and `validate_inbound` requires an empty body of `StreamEnd` alone — so a legal zero-length `StreamData` reached the stream arm with no charge and was read as an exhausted retained budget. That retired the generation and settled every pending request on it, with no budget anywhere near its cap. An absent charge is exhaustion only when the frame declared bytes. `ChargedItem::_charge` is now optional to say that a zero-length item holds no reservation, rather than manufacturing one. --- crates/mc-host/src/client.rs | 66 +++++++++++++++++++++++++++++++++--- 1 file changed, 61 insertions(+), 5 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 1472c0955..8678c5b65 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -1104,10 +1104,20 @@ impl Inner { ); } PendingKind::Stream { items, .. } => { - let Some(charge) = charge else { - drop(pending); - self.retire("response_memory_exhausted"); - return; + // An empty item is never charged, so an absent charge + // means exhaustion only when there were bytes to + // charge for. Reading it as exhaustion either way + // retires the generation over a legal zero-length + // StreamData: `validate_inbound` requires an empty body + // of `StreamEnd` alone. + let charge = match charge { + Some(charge) => Some(charge), + None if header.len == 0 => None, + None => { + drop(pending); + self.retire("response_memory_exhausted"); + return; + } }; let item = ChargedItem { body, @@ -1350,7 +1360,8 @@ impl Drop for ByteCharge { struct ChargedItem { body: Vec, binary: bool, - _charge: ByteCharge, + /// Absent for a zero-length item, which is never charged. + _charge: Option, } impl ChargedItem { @@ -2323,6 +2334,51 @@ mod tests { } } + #[tokio::test] + async fn a_zero_length_stream_item_is_delivered_without_retiring() { + // Only `StreamEnd` must be empty, so a zero-length `StreamData` is + // legal. It carries no charge because there were no bytes to charge + // for, which must not read as an exhausted budget. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, mut items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, _terminal_rx) = oneshot::channel(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + drop(data_rx.recv().await); + + inner.dispatch( + EnvelopeHeader { + len: 0, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + Vec::new(), + None, + ); + + assert!( + !inner.retired.load(Ordering::Acquire), + "an uncharged empty item is not an exhausted budget" + ); + let item = items_rx.try_recv().expect("the empty item is delivered"); + assert!(item.body.is_empty()); + assert!(lock_unpoisoned(&inner.pending).contains_key(&key)); + } + #[test] fn absent_admission_facts_are_omitted_rather_than_sent_as_null() { // The host reads any present member as `Some(..)`, so a null would make From 946d2e89f2af29f138613ac80abc13894a4059f1 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:33:18 +0000 Subject: [PATCH 11/37] fix: address review quick-wins across tests, doctor guidance, and docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Share the store-readiness poll instead of copying it. The boundary-counter test polled `status` through a helper that panics on any `CallError`, while the direct-host test tolerates a terminal `store_unavailable` and retries — publication proves the transport is ready, not that the store is open, so the first test failed on a race the second was written to absorb. `wait_for_store` now lives in the shared support module and both callers use it. Confirm the paused and resumed state before continuing. `pauseHost` and `resumeHost` returned as soon as `child.kill` had queued SIGSTOP or SIGCONT, so a following prompt raced signal delivery and could miss the timeout path a drill is exercising. Both now poll `/proc//stat` for the resulting state under a bounded wait, resolve rather than throw when the window elapses, and degrade to returning immediately where /proc is absent. Name mc-host in the authority doctor's recovery guidance; restoring `subc` connectivity is not an action that exists at this boundary. Name the env var each opt-in e2e gate reads in its skip reason, and name `subc.connection_file` in the config description that requires it, so both messages point at something the reader can act on. State that the client handshake deadline and the host authentication deadline are not independent: both default to 2 seconds, so the client retires at or before the end of the host's permitted window and negotiation gets only the remainder. --- .../tests/boundary_counter_durability.rs | 25 +------- crates/mc-module/tests/direct_host.rs | 33 +--------- crates/mc-module/tests/support/direct_host.rs | 32 ++++++++++ docs/mc-host-wire-protocol.md | 2 + packages/cli/src/commands/doctor-authority.ts | 6 +- .../e2e-tests/scripts/run-rust-fm-mutation.ts | 5 +- .../src/rust-runner/hermetic-mc-host.ts | 64 +++++++++++++++++-- .../e2e-tests/src/rust-scenario-support.ts | 6 +- packages/e2e-tests/tests/rust-fm-oc-5.test.ts | 4 +- .../plugin/src/config/schema/magic-context.ts | 2 +- 10 files changed, 110 insertions(+), 69 deletions(-) diff --git a/crates/mc-module/tests/boundary_counter_durability.rs b/crates/mc-module/tests/boundary_counter_durability.rs index fa24d1c65..77b2d309e 100644 --- a/crates/mc-module/tests/boundary_counter_durability.rs +++ b/crates/mc-module/tests/boundary_counter_durability.rs @@ -3,13 +3,10 @@ mod support; -use std::time::{Duration, Instant}; - use mc_core::CoreState; use mc_host::TargetKind; use mc_store::{McStore, McStoreError, ModuleMeta}; -use serde_json::json; -use support::direct_host::{request_json, storage_descriptor, FixtureProcess, BUDGET}; +use support::direct_host::{storage_descriptor, wait_for_store, FixtureProcess}; #[tokio::test] async fn competing_pass_counter_survives_direct_primary_lifecycle_and_reopen() { @@ -50,24 +47,8 @@ async fn competing_pass_counter_survives_direct_primary_lifecycle_and_reopen() { let route = fixture .open_route(&client, "magic-context", TargetKind::ToolProvider, session) .await; - let deadline = Instant::now() + BUDGET; - loop { - let status = request_json( - &client, - route, - json!({"kind": "status", "session_id": session}), - ) - .await; - if status["store_open"] == true { - assert_eq!(status["session_id"], session); - break; - } - assert!( - Instant::now() < deadline, - "direct primary store did not open" - ); - tokio::time::sleep(Duration::from_millis(20)).await; - } + let status = wait_for_store(&client, route, session).await; + assert_eq!(status["session_id"], session); client.close().await.expect("managed client closes"); fixture.shutdown(); diff --git a/crates/mc-module/tests/direct_host.rs b/crates/mc-module/tests/direct_host.rs index 5cfdc2320..106d22b6c 100644 --- a/crates/mc-module/tests/direct_host.rs +++ b/crates/mc-module/tests/direct_host.rs @@ -11,8 +11,8 @@ use mc_host::TargetKind; use mc_store::{McStore, StoredCompartment}; use serde_json::{json, Value}; use support::direct_host::{ - mode, request_json, send_body, storage_descriptor, workspace_root, FixtureProcess, BUDGET, - REDACTION_SENTINEL, + mode, request_json, send_body, storage_descriptor, wait_for_store, workspace_root, + FixtureProcess, BUDGET, REDACTION_SENTINEL, }; fn base64(bytes: &[u8]) -> String { @@ -63,35 +63,6 @@ fn redaction_forms(publication: &str) -> Vec { forms } -async fn wait_for_store(client: &mc_host::Client, route: mc_host::RouteHandle, session: &str) { - let deadline = Instant::now() + BUDGET; - loop { - let body = serde_json::to_vec(&json!({"kind": "status", "session_id": session})).unwrap(); - match client - .request( - route, - body, - mc_host::RequestOptions { - timeout: BUDGET, - cancellation: None, - }, - ) - .await - { - Ok(response) => { - let status: Value = serde_json::from_slice(&response.body).unwrap(); - if status["store_open"] == true { - return; - } - } - Err(error) if error.code() == "store_unavailable" => {} - Err(error) => panic!("store readiness request failed: {error}"), - } - assert!(Instant::now() < deadline, "module store did not open"); - tokio::task::yield_now().await; - } -} - #[tokio::test] async fn readiness_permissions_catalog_and_real_unary_transform() { let fixture = FixtureProcess::start(); diff --git a/crates/mc-module/tests/support/direct_host.rs b/crates/mc-module/tests/support/direct_host.rs index c6a57a865..00a159863 100644 --- a/crates/mc-module/tests/support/direct_host.rs +++ b/crates/mc-module/tests/support/direct_host.rs @@ -373,6 +373,38 @@ pub async fn request_json(client: &Client, route: RouteHandle, body: Value) -> V serde_json::from_slice(&response.body).expect("response JSON") } +/// Poll `status` until the module reports an open store, and return that status. +/// Connection publication proves the transport is ready, not that the store is +/// open, so a terminal `store_unavailable` is a retry rather than a failure. +pub async fn wait_for_store(client: &Client, route: RouteHandle, session: &str) -> Value { + let deadline = Instant::now() + BUDGET; + loop { + let body = serde_json::to_vec(&json!({"kind": "status", "session_id": session})).unwrap(); + match client + .request( + route, + body, + RequestOptions { + timeout: BUDGET, + cancellation: None, + }, + ) + .await + { + Ok(response) => { + let status: Value = serde_json::from_slice(&response.body).unwrap(); + if status["store_open"] == true { + return status; + } + } + Err(error) if error.code() == "store_unavailable" => {} + Err(error) => panic!("store readiness request failed: {error}"), + } + assert!(Instant::now() < deadline, "module store did not open"); + tokio::task::yield_now().await; + } +} + pub fn send_body(prompt: &str) -> Value { json!({ "method": "session.send", diff --git a/docs/mc-host-wire-protocol.md b/docs/mc-host-wire-protocol.md index c796fcdf8..f1fc35020 100644 --- a/docs/mc-host-wire-protocol.md +++ b/docs/mc-host-wire-protocol.md @@ -818,6 +818,8 @@ Managed Rust and TypeScript client defaults: Data and reserved-control frames share one queued-byte budget; reserved admission is not a byte-budget bypass. Data traffic cannot consume control slots. Exhausting control reserve retires the generation and deterministically settles pending work. Backoff counts the first attempt, and retry delay or a later stage never resets the owning deadline. +The 2-second handshake deadline spans discovery, dial, authentication, and negotiation together, while Section 5.1's recommended host authentication deadline is also 2 seconds. The client therefore retires the generation at or before the end of the host's permitted authentication window, and negotiation receives only the remainder. A deployment that shortens the host authentication deadline keeps the two compatible; one that needs the full host window for authentication MUST raise the client handshake deadline above it, because these two values are not independent. + ## 12. Reconnect, restart, and shutdown Any EOF, authentication failure, framing corruption, liveness failure, or explicit connection close retires the connection generation. Client MUST immediately invalidate its routes, pending correlations, capability/catalog caches, and late responses. The host has the mirror obligation for every handler-visible route on the retired generation: stop new dispatch, settle or cancel route work within a finite close budget, invoke `on_route_gone` exactly once, and release each global channel only after that callback completes — otherwise a reconnect finds handler bindings and channels still held by the dead generation. Reconnect MUST reread the connection file and rerun authentication; credentials MUST NOT be cached across host incarnations. diff --git a/packages/cli/src/commands/doctor-authority.ts b/packages/cli/src/commands/doctor-authority.ts index 5298bb42c..9e060338b 100644 --- a/packages/cli/src/commands/doctor-authority.ts +++ b/packages/cli/src/commands/doctor-authority.ts @@ -139,7 +139,7 @@ export async function reportAuthorityMarkers(args: { for (const marker of markers) { if (marker.project_path !== currentIdentity) { args.warn( - ` ${marker.project_path}: module state unavailable outside its project root — writes fenced; run with rust mode or restore subc connectivity`, + ` ${marker.project_path}: module state unavailable outside its project root — writes fenced; run with rust mode or restore mc-host connectivity`, ); continue; } @@ -164,7 +164,7 @@ export async function reportAuthorityMarkers(args: { ); } catch { args.warn( - ` ${marker.project_path}: module unreachable — writes fenced; run with rust mode or restore subc connectivity`, + ` ${marker.project_path}: module unreachable — writes fenced; run with rust mode or restore mc-host connectivity`, ); } } @@ -228,7 +228,7 @@ export async function runDoctorDrainAuthority( return 1; } catch (error) { console.error( - `Module unreachable — writes fenced; run with rust mode or restore subc connectivity: ${error instanceof Error ? error.message : String(error)}`, + `Module unreachable — writes fenced; run with rust mode or restore mc-host connectivity: ${error instanceof Error ? error.message : String(error)}`, ); return 1; } finally { diff --git a/packages/e2e-tests/scripts/run-rust-fm-mutation.ts b/packages/e2e-tests/scripts/run-rust-fm-mutation.ts index 1dc9447d6..2d9c54777 100644 --- a/packages/e2e-tests/scripts/run-rust-fm-mutation.ts +++ b/packages/e2e-tests/scripts/run-rust-fm-mutation.ts @@ -96,9 +96,10 @@ const mutations: Record = { { name: "FM_OC_5_RUNG_SWAP", source: drillFile("5"), - oldText: "h.mcHost.pauseHost();\n await h.sendPrompt", + oldText: + "await h.mcHost.pauseHost();\n await h.sendPrompt", replacement: - "h.mcHost.resumeHost();\n await h.sendPrompt", + "await h.mcHost.resumeHost();\n await h.sendPrompt", }, { name: "FM_OC_5_RUNG_DELETION", diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index 2b501cf20..7d3db8665 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -33,6 +33,8 @@ const PID_FILE = "rust-e2e-pids.json"; const MAX_LINE_BYTES = 64 * 1024; const MAX_LOG_BYTES = 256 * 1024; const STALE_PID_AGE_MS = 30 * 60 * 1_000; +const SIGNAL_STATE_TIMEOUT_MS = 2_000; +const SIGNAL_STATE_POLL_MS = 10; const EXPECTED_CATALOG = ["magic-context", "synapse", "broca"] as const; interface RustE2ePidFile { @@ -104,6 +106,52 @@ function processExecutable(pid: number): string | null { } } +/** + * Report whether a process is stopped, from the state field of + * `/proc//stat`: `T` is stopped, any other state is running. `null` means + * no state is observable — the process is gone, or the system has no `/proc`. + */ +function processStopped(pid: number): boolean | null { + let stat: string; + try { + stat = readFileSync(`/proc/${pid}/stat`, "utf8"); + } catch { + return null; + } + // The comm field is parenthesized and may itself contain spaces and + // parentheses, so the state field is the first token after the LAST ")". + const state = stat + .slice(stat.lastIndexOf(")") + 1) + .trim() + .split(/\s+/, 1)[0]; + return state ? state === "T" : null; +} + +/** + * Wait until the process reaches the state a just-sent SIGSTOP or SIGCONT + * produces. Signal delivery is asynchronous, so a request issued straight after + * `kill` can outrun it and exercise the wrong path. + * + * An elapsed window resolves rather than throwing: a drill asserts the module's + * behavior while the host is paused, and failing it because this confirmation + * ran out of time would report the harness instead of the mechanism. An + * unobservable state resolves for the same reason. + */ +async function waitForProcessState( + pid: number, + stopped: boolean, +): Promise { + const deadline = Date.now() + SIGNAL_STATE_TIMEOUT_MS; + for (;;) { + const observed = processStopped(pid); + if (observed === null || observed === stopped) return; + if (Date.now() >= deadline) return; + await new Promise((resolvePoll) => + setTimeout(resolvePoll, SIGNAL_STATE_POLL_MS), + ); + } +} + function isStaleRustE2ePidRecord( createdAtMs: number, nowMs = Date.now(), @@ -744,16 +792,20 @@ export class HermeticMcHostStack { this.persistPidFile(); } - pauseHost(): void { + async pauseHost(): Promise { const child = this.child; - if (child && child.exitCode === null && child.signalCode === null) - child.kill("SIGSTOP"); + if (!child || child.exitCode !== null || child.signalCode !== null) + return; + child.kill("SIGSTOP"); + if (child.pid !== undefined) await waitForProcessState(child.pid, true); } - resumeHost(): void { + async resumeHost(): Promise { const child = this.child; - if (child && child.exitCode === null && child.signalCode === null) - child.kill("SIGCONT"); + if (!child || child.exitCode !== null || child.signalCode !== null) + return; + child.kill("SIGCONT"); + if (child.pid !== undefined) await waitForProcessState(child.pid, false); } /** diff --git a/packages/e2e-tests/src/rust-scenario-support.ts b/packages/e2e-tests/src/rust-scenario-support.ts index 69ee46415..ddd629092 100644 --- a/packages/e2e-tests/src/rust-scenario-support.ts +++ b/packages/e2e-tests/src/rust-scenario-support.ts @@ -34,7 +34,8 @@ export function foldInfraEnabled(): boolean { } export const FOLD_SKIP_REASON = - "requires broad Rust fold qualification beyond the focused direct backend fixture"; + "requires broad Rust fold qualification beyond the focused direct " + + "backend fixture; set MC_RUST_E2E_FOLD=1 to run it"; /** Enable the duplicate-ID regression only when the stack can produce the selection refresh needed to reproduce duplicate IDs. */ export function duplicateIdInfraEnabled(): boolean { @@ -42,7 +43,8 @@ export function duplicateIdInfraEnabled(): boolean { } export const DUPLICATE_ID_SKIP_REASON = - "requires broad duplicate-ID qualification beyond the focused direct backend fixture"; + "requires broad duplicate-ID qualification beyond the focused direct " + + "backend fixture; set MC_RUST_E2E_DUPLICATE_IDS=1 to run it"; /** * Print a one-line skip notice. Call from a gated scenario's single `it` so the diff --git a/packages/e2e-tests/tests/rust-fm-oc-5.test.ts b/packages/e2e-tests/tests/rust-fm-oc-5.test.ts index 55141e075..4335c181a 100644 --- a/packages/e2e-tests/tests/rust-fm-oc-5.test.ts +++ b/packages/e2e-tests/tests/rust-fm-oc-5.test.ts @@ -32,11 +32,11 @@ describe.skipIf(!rustPrereqs.ok)("rust failure-mode drill FM-OC-5: transport han await driveToSteadyState(h, sessionId, 2); const beforeCount = h.readRustPasses().length; - h.mcHost.pauseHost(); + await h.mcHost.pauseHost(); await h.sendPrompt(sessionId, `FM-OC-5 stopped module: ${h.ballast(400)}`); assertMessagesHaveNoPlaceholders(h.lastMainMessages(), sessionId); - h.mcHost.resumeHost(); + await h.mcHost.resumeHost(); await h.sendPrompt(sessionId, `FM-OC-5 continued module: ${h.ballast(400)}`); const recovered = await h.waitForRustPasses(beforeCount + 2); expect(recovered.slice(beforeCount + 1).some((pass) => pass.servedFrom === "transform")).toBe( diff --git a/packages/plugin/src/config/schema/magic-context.ts b/packages/plugin/src/config/schema/magic-context.ts index cccec0606..bf1075bad 100644 --- a/packages/plugin/src/config/schema/magic-context.ts +++ b/packages/plugin/src/config/schema/magic-context.ts @@ -673,7 +673,7 @@ export const MagicContextConfigSchema = z .enum(["ts", "rust"]) .default("ts") .describe( - 'Experimental: routes the project through the direct mc-host Rust runtime (requires user-level host connection config); "ts" is the current TypeScript pipeline.', + 'Experimental: routes the project through the direct mc-host Rust runtime (requires the user-level subc.connection_file path); "ts" is the current TypeScript pipeline.', ), auto_update: z .boolean() From 803065e2f4647f39144fb5a4fe7931b3a1032ad3 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:38:27 +0000 Subject: [PATCH 12/37] fix: reject out-of-range timeouts, pre-cancelled streams, binary setup, pidless children MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Instant + Duration` panics when the sum is unrepresentable, and the request timeout is public configuration where `Duration::MAX` is a conventional spelling of "no timeout" — so a configured value could crash the consumer. Both the unary and stream paths now build their deadline through one checked helper that returns a typed `not_sent` rejection. A stream whose cancellation token is already cancelled no longer admits anything. The watcher is spawned after admission, so the writer could claim and transmit a side-effecting request before the first cancel observation; a token cancelled after that point is still the watcher's job. Reject a binary channel-zero negotiation response. §7.1 accepts UTF-8 JSON only on channel 0, and the setup check validated type, channel, epoch, and correlation but not the flag — so a nonconforming generation whose body happened to parse completed the handshake. Treat a child with no pid as already exited. A failed spawn leaves both exit fields null with `pid` undefined, so teardown read it as live, burned every escalation window, and then preserved the data tree for a reaper whose only handle is a pid record that was never written. --- crates/mc-host/src/client.rs | 80 ++++++++++++++++++- .../src/rust-runner/hermetic-mc-host.ts | 15 +++- 2 files changed, 91 insertions(+), 4 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 8678c5b65..a5561048e 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -393,7 +393,7 @@ impl Client { options: RequestOptions, ) -> Result { self.require_route(route)?; - let deadline = Instant::now() + options.timeout; + let deadline = request_deadline(options.timeout)?; self.inner .unary(route, body, deadline, options.cancellation) .await @@ -724,6 +724,22 @@ impl Inner { body: Vec, options: RequestOptions, ) -> Result { + let deadline = request_deadline(options.timeout)?; + // A token cancelled before the call must not enqueue anything: the + // watcher is spawned after admission, so the writer could otherwise + // claim and transmit a side-effecting request before the first cancel + // observation. A token cancelled after this point is the watcher's job. + if options + .cancellation + .as_ref() + .is_some_and(CancellationToken::is_cancelled) + { + return Err(CallError::local( + SendOutcome::NotSent, + "cancelled", + "request was cancelled", + )); + } { let mut streams = lock_unpoisoned(&self.streams); if *streams >= CLIENT_MAX_LIVE_STREAMS { @@ -737,7 +753,6 @@ impl Inner { } let (item_tx, item_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); let (terminal_tx, terminal_rx) = oneshot::channel(); - let deadline = Instant::now() + options.timeout; let settled = CancellationToken::new(); let admitted = self.admit( route, @@ -1530,6 +1545,22 @@ async fn drain_until( Ok(()) } +/// Turns a caller-supplied timeout into an absolute deadline. +/// +/// `Instant + Duration` panics when the sum is unrepresentable, and the timeout +/// is public configuration — `Duration::MAX` is a conventional spelling of "no +/// timeout" — so an out-of-range value must be a typed rejection rather than a +/// crash in the consumer. +fn request_deadline(timeout: Duration) -> Result { + Instant::now().checked_add(timeout).ok_or_else(|| { + CallError::local( + SendOutcome::NotSent, + "invalid_timeout", + "request timeout is out of range", + ) + }) +} + fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { if header.ver != PROTOCOL_VERSION || header.len > MAX_BODY_LEN { return Err(()); @@ -1640,10 +1671,13 @@ async fn negotiate_tcp(stream: &mut TcpStream, deadline: Instant) -> Result<(), .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; let frame = read_setup_frame(stream, deadline).await?; + // Channel 0 accepts UTF-8 JSON only (§7.1), so a binary setup response is a + // nonconforming generation even when its body happens to parse. if frame.header.ty != FrameType::Response || frame.header.channel != 0 || frame.header.epoch != 0 || frame.header.corr != NEGOTIATION_CORRELATION + || frame.header.flags.is_binary() { return Err(ClientError::new( "negotiation_failed", @@ -2379,6 +2413,48 @@ mod tests { assert!(lock_unpoisoned(&inner.pending).contains_key(&key)); } + #[tokio::test] + async fn an_out_of_range_timeout_is_rejected_instead_of_panicking() { + // `Duration::MAX` is a conventional spelling of "no timeout" and is + // public configuration, so an unrepresentable deadline must be a typed + // rejection rather than a panic inside the consumer. + let error = request_deadline(Duration::MAX).expect_err("unrepresentable"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "invalid_timeout"); + assert!(request_deadline(Duration::from_secs(30)).is_ok()); + } + + #[tokio::test] + async fn a_pre_cancelled_stream_never_enqueues_a_frame() { + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let cancelled = CancellationToken::new(); + cancelled.cancel(); + + let error = inner + .start_stream( + route(1), + b"must-not-send".to_vec(), + RequestOptions { + timeout: Duration::from_secs(30), + cancellation: Some(cancelled), + }, + ) + .expect_err("an already-cancelled token admits nothing"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "cancelled"); + assert!( + data_rx.try_recv().is_err(), + "a cancelled stream must not reach the writer" + ); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + assert_eq!( + *lock_unpoisoned(&inner.streams), + 0, + "no live stream charged" + ); + } + #[test] fn absent_admission_facts_are_omitted_rather_than_sent_as_null() { // The host reads any present member as `Some(..)`, so a null would make diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index 7d3db8665..0120f4849 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -758,7 +758,10 @@ export class HermeticMcHostStack { async crashHost(): Promise { await this.closeClients(); const child = this.child; - if (!child) return; + if (!child || child.pid === undefined) { + this.child = null; + return; + } this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); @@ -779,7 +782,10 @@ export class HermeticMcHostStack { async terminateHost(): Promise { await this.closeClients(); const child = this.child; - if (!child) return; + if (!child || child.pid === undefined) { + this.child = null; + return; + } this.resumeBeforeTeardown(child); if (child.exitCode === null && child.signalCode === null) child.kill("SIGTERM"); @@ -815,8 +821,13 @@ export class HermeticMcHostStack { async stop(): Promise { await this.closeStatusClient(); const child = this.child; + // A child whose spawn failed has no pid while both exit fields stay + // null: there is no process to signal, so treating it as live burns + // every escalation window and then preserves the data tree for a + // reaper that has no pid record to act on. let exited = child === null || + child.pid === undefined || child.exitCode !== null || child.signalCode !== null; if (child && !exited) { From ef7980c9566a191f1b2eaa7c4f87c90a51ed7e80 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:45:40 +0000 Subject: [PATCH 13/37] chore: regenerate the committed schema and re-pin the crash evidence Naming `subc.connection_file` in the `transform_mode` description changed the generator's output, and `assets/magic-context.schema.json` is committed and compared against it. Regenerated with `packages/plugin/scripts/build-schema.ts`; the description is the only line that moved. Restoring the built-bundle preference and the provisioning cleanup touched `packages/e2e-tests/src/opencode-runner/spawn.ts`, which the crash-campaign evidence hashes, so its pinned digest went stale again. Regenerated with `UPDATE_CLAIMS_CRASH_EVIDENCE=1`; only `commitUnderTest` and `dirtyDiffDigest` differ, with the matrix, per-scenario semantic digests, summary, limits, runtimes, and environment byte-identical. --- assets/magic-context.schema.json | 2 +- docs/evidence/claims-backfill/v84-process-crash.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/assets/magic-context.schema.json b/assets/magic-context.schema.json index bcbaf2070..637aedbf6 100644 --- a/assets/magic-context.schema.json +++ b/assets/magic-context.schema.json @@ -35,7 +35,7 @@ }, "transform_mode": { "default": "ts", - "description": "Experimental: routes the project through the direct mc-host Rust runtime (requires user-level host connection config); \"ts\" is the current TypeScript pipeline.", + "description": "Experimental: routes the project through the direct mc-host Rust runtime (requires the user-level subc.connection_file path); \"ts\" is the current TypeScript pipeline.", "type": "string", "enum": [ "ts", diff --git a/docs/evidence/claims-backfill/v84-process-crash.json b/docs/evidence/claims-backfill/v84-process-crash.json index e1049e1f4..679b42217 100644 --- a/docs/evidence/claims-backfill/v84-process-crash.json +++ b/docs/evidence/claims-backfill/v84-process-crash.json @@ -1,8 +1,8 @@ { "schemaVersion": "claims-process-crash-evidence/v1", - "commitUnderTest": "d79c109e3b29201210064c50f9a2bddf7d239550", + "commitUnderTest": "803065e2f4647f39144fb5a4fe7931b3a1032ad3", "dirtyDiffDigestPolicy": "sha256(sorted U6 implementation path + NUL + full file bytes + NUL); evidence file excluded", - "dirtyDiffDigest": "3a5840fe7ead4afc757c072f802c6077b934902d091ec8b8874fe64c28172f46", + "dirtyDiffDigest": "692e8496cc3f795590d5382b2860e7c343983aa596e74083a0d5271f01efbe3e", "implementationFiles": [ "ARCHITECTURE.md", "STRUCTURE.md", From d5042839836ac7060f01606297471eea022413aa Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 09:55:52 +0000 Subject: [PATCH 14/37] fix: report only spawned recovery, share the wire cap, coalesce status connects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `maybe_spawn_reattach` discarded the `Option` from `spawn_module_task`, which yields `None` once task admission closes — the reattach or recovery future is then dropped unpolled. The function reported `reattaching` or `recovering` regardless, so `HistorianDiagnostics` named work no task would ever perform. Both arms now report only when the spawn took. Derive the module's wire body cap from `mc_host::MAX_FRAME_BODY_LEN` instead of restating 64 MiB. The module's value gates output preparation while the host's gates frame admission, so a second literal could drift and leave the two disagreeing about what fits on the wire. The constant is now published from mc-host for exactly this. Coalesce concurrent status-client connects in the fixture. The lazy initializer read the field and assigned only after its await resolved, so two concurrent callers each established a client and the one whose assignment lost was never stored or closed. Followers now await the in-flight attempt, and a failed attempt clears it so the next call retries. Reject the startup race when the child exits before readiness. Only `error` was observed, and a child that spawns and then dies emits none, so a bad flag or taken port burned the whole ready-wait timeout instead of naming the exit code. Every settle path now detaches both listeners. Re-pinned the crash-campaign digest for the touched manifest file; matrix, summary, limits, runtimes, and environment are byte-identical. --- crates/mc-host/src/lib.rs | 3 ++ crates/mc-module/src/dispatch.rs | 6 ++- crates/mc-module/src/lib.rs | 12 +++-- .../claims-backfill/v84-process-crash.json | 4 +- .../e2e-tests/src/opencode-runner/spawn.ts | 24 +++++++++- .../src/rust-runner/hermetic-mc-host.ts | 44 ++++++++++++++++--- 6 files changed, 78 insertions(+), 15 deletions(-) diff --git a/crates/mc-host/src/lib.rs b/crates/mc-host/src/lib.rs index f05a3b31d..e270418a0 100644 --- a/crates/mc-host/src/lib.rs +++ b/crates/mc-host/src/lib.rs @@ -64,5 +64,8 @@ pub use lifecycle::{ LifecycleRootLock, LifecycleState, ProbeFreshness, PublicationSummary, LIFECYCLE_RECORD_NAME, }; pub use runtime::{run, HostError}; +/// The version-2 body cap. Published so a consumer preparing an output can +/// gate on the same value frame admission enforces, rather than restating it. +pub use wire::MAX_FRAME_BODY_LEN; pub use tokio_util::sync::CancellationToken; diff --git a/crates/mc-module/src/dispatch.rs b/crates/mc-module/src/dispatch.rs index 89374009e..32a8cfe40 100644 --- a/crates/mc-module/src/dispatch.rs +++ b/crates/mc-module/src/dispatch.rs @@ -5,7 +5,11 @@ use std::sync::Arc; use serde_json::{Map, Value}; /// Maximum body accepted by the version-2 wire contract. -pub const MAX_WIRE_BODY_BYTES: usize = 64 * 1024 * 1024; +/// +/// Derived from the host's own cap rather than restated: this value gates +/// output preparation while `mc-host` gates frame admission, so a literal here +/// could drift and make the two disagree about what fits on the wire. +pub const MAX_WIRE_BODY_BYTES: usize = mc_host::MAX_FRAME_BODY_LEN as usize; /// Successful response body prepared without an encoded output buffer. #[derive(Clone)] diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index 414e6f786..f32c4d5de 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -4821,7 +4821,7 @@ impl McHandler { let fingerprint_items: Vec<_> = chunk.snapshot.iter().map(|item| item.as_item()).collect(); let observed = historian::compute_chunk_fingerprint(&fingerprint_items); - let _ = self.spawn_module_task(async move { + let spawned = self.spawn_module_task(async move { let _guard = guard; let result = async { let action = historian::handle_restart_load( @@ -4876,10 +4876,14 @@ impl McHandler { eprintln!("mc-module: historian reattach failed for {session_id}: {e}"); } }); - Some("reattaching") + // `spawn_module_task` yields `None` once task admission closes, + // and the future is then dropped unpolled. Reporting the + // trigger anyway puts work in the diagnostics that no task will + // ever perform. + spawned.map(|_| "reattaching") } HistorianPhase::Firing | HistorianPhase::Validating | HistorianPhase::Publishing => { - let _ = self.spawn_module_task(async move { + let spawned = self.spawn_module_task(async move { let _guard = guard; if let Err(e) = historian::handle_restart_load( &store, @@ -4891,7 +4895,7 @@ impl McHandler { ); } }); - Some("recovering") + spawned.map(|_| "recovering") } HistorianPhase::Idle => Some("recovered"), } diff --git a/docs/evidence/claims-backfill/v84-process-crash.json b/docs/evidence/claims-backfill/v84-process-crash.json index 679b42217..59dd7bbb0 100644 --- a/docs/evidence/claims-backfill/v84-process-crash.json +++ b/docs/evidence/claims-backfill/v84-process-crash.json @@ -1,8 +1,8 @@ { "schemaVersion": "claims-process-crash-evidence/v1", - "commitUnderTest": "803065e2f4647f39144fb5a4fe7931b3a1032ad3", + "commitUnderTest": "ef7980c9566a191f1b2eaa7c4f87c90a51ed7e80", "dirtyDiffDigestPolicy": "sha256(sorted U6 implementation path + NUL + full file bytes + NUL); evidence file excluded", - "dirtyDiffDigest": "692e8496cc3f795590d5382b2860e7c343983aa596e74083a0d5271f01efbe3e", + "dirtyDiffDigest": "59861ab8048726d2efd2905cb57d6b2064cf6cebfdf920c07f49a4aefa0ea34d", "implementationFiles": [ "ARCHITECTURE.md", "STRUCTURE.md", diff --git a/packages/e2e-tests/src/opencode-runner/spawn.ts b/packages/e2e-tests/src/opencode-runner/spawn.ts index c7518a83f..95b441849 100644 --- a/packages/e2e-tests/src/opencode-runner/spawn.ts +++ b/packages/e2e-tests/src/opencode-runner/spawn.ts @@ -293,17 +293,39 @@ interface RustSpawnResources { mcHost: HermeticMcHostStack; } +/** + * Reject when the child fails to spawn or exits before readiness. A child that + * starts and then dies (bad flag, unusable config, taken port) emits no + * `error`, so without the `exit` arm the startup race only ends when + * `waitForReady` burns its whole timeout. Every settle path detaches both child + * listeners, so a child that outlives the race retains neither. + */ function rejectOnSpawnError(child: ChildProcess, cancellation?: AbortSignal): Promise { return new Promise((_, rejectSpawn) => { + const detach = (): void => { + child.off("error", onError); + child.off("exit", onExit); + }; const onError = (error: Error): void => { + detach(); cancellation?.removeEventListener("abort", onAbort); rejectSpawn(error); }; + const onExit = (code: number | null, signal: NodeJS.Signals | null): void => { + detach(); + cancellation?.removeEventListener("abort", onAbort); + rejectSpawn( + new Error( + `opencode serve exited before readiness (code=${code}, signal=${signal})`, + ), + ); + }; const onAbort = (): void => { - child.off("error", onError); + detach(); }; if (cancellation?.aborted) return; child.once("error", onError); + child.once("exit", onExit); cancellation?.addEventListener("abort", onAbort, { once: true }); }); } diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index 0120f4849..e3a58c5a6 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -643,6 +643,7 @@ export class HermeticMcHostStack { private child: ChildProcess | null = null; private control: FixtureControlClient | null = null; private statusClient: McHostClient | null = null; + private statusClientPromise: Promise | null = null; private stdout = ""; private stderr = ""; private pidFileCreatedAtMs = 0; @@ -717,13 +718,7 @@ export class HermeticMcHostStack { harness: "opencode", session: sessionId, }; - const client = - this.statusClient ?? - (this.statusClient = await McHostClient.connect({ - connectionFile: this.connectionFile, - identity, - targetKind: "tool_provider", - })); + const client = await this.ensureStatusClient(identity); let route: Awaited> | null = null; try { route = await client.routeOpen( @@ -1021,7 +1016,42 @@ export class HermeticMcHostStack { return this.control; } + /** + * Reuse one status client, coalescing concurrent connects on the in-flight + * attempt. Assigning the field only after the connect resolves lets two + * callers each establish a client and strand the one whose assignment is + * overwritten, so followers await the leader's attempt instead. A failed + * attempt clears the stored promise, leaving the next call free to retry. + */ + private async ensureStatusClient( + identity: BindIdentity, + ): Promise { + if (this.statusClient) return this.statusClient; + if (this.statusClientPromise) return await this.statusClientPromise; + const connecting = (async (): Promise => { + const client = await McHostClient.connect({ + connectionFile: this.connectionFile, + identity, + targetKind: "tool_provider", + }); + this.statusClient = client; + return client; + })(); + this.statusClientPromise = connecting; + try { + return await connecting; + } finally { + if (this.statusClientPromise === connecting) + this.statusClientPromise = null; + } + } + private async closeStatusClient(): Promise { + // An in-flight connect publishes its client on resolution, so teardown + // settles that attempt first rather than stranding its socket. + const connecting = this.statusClientPromise; + this.statusClientPromise = null; + if (connecting) await connecting.catch(() => undefined); const client = this.statusClient; this.statusClient = null; if (client) await client.closeAsync().catch(() => undefined); From 879b6afb78cbdd2347ad29814a68cfce4cac1c93 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 10:12:10 +0000 Subject: [PATCH 15/37] fix(mc-host): reject a pre-cancelled unary before admission MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stream path gained this check; the unary path did not. Its `select!` is biased toward cancellation, but admission has already handed the frame to the writer, which can claim and transmit it on another worker before this task reaches the select — so a caller passing an already-cancelled token could still have a side-effecting request sent. The rejection is `not_sent`: nothing was queued, so the request is replay-safe. Cancellation that races after admission stays the select's job, unchanged. --- crates/mc-host/src/client.rs | 38 ++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index a5561048e..55516cb8a 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -698,6 +698,20 @@ impl Inner { deadline: Instant, cancellation: Option, ) -> Result { + // A token cancelled before the call must not enqueue anything. The + // `select!` below is biased toward cancellation, but admission has + // already handed the frame to the writer, which can claim it on another + // worker before this task reaches the select. + if cancellation + .as_ref() + .is_some_and(CancellationToken::is_cancelled) + { + return Err(CallError::local( + SendOutcome::NotSent, + "cancelled", + "request was cancelled", + )); + } let (tx, rx) = oneshot::channel(); let (key, publish) = self.admit(route, body, PendingKind::Unary(tx), deadline)?; let mut guard = UnaryAdmissionGuard::new(Arc::clone(self), key); @@ -2455,6 +2469,30 @@ mod tests { ); } + #[tokio::test] + async fn a_pre_cancelled_unary_never_enqueues_a_frame() { + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let cancelled = CancellationToken::new(); + cancelled.cancel(); + + let error = inner + .unary( + route(1), + b"must-not-send".to_vec(), + Instant::now() + Duration::from_secs(30), + Some(cancelled), + ) + .await + .expect_err("an already-cancelled token admits nothing"); + assert_eq!(error.outcome(), SendOutcome::NotSent); + assert_eq!(error.code(), "cancelled"); + assert!( + data_rx.try_recv().is_err(), + "a cancelled request must not reach the writer" + ); + assert!(lock_unpoisoned(&inner.pending).is_empty()); + } + #[test] fn absent_admission_facts_are_omitted_rather_than_sent_as_null() { // The host reads any present member as `Some(..)`, so a null would make From acba02443ed3b758036a5109be2dabac78b0f3a8 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 13:55:13 +0000 Subject: [PATCH 16/37] fix: close the five deferred client and discovery findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bring the handshake deadline back over discovery. It started after the connection-file snapshot, so a stalled filesystem got unbounded time and the handshake then received a fresh 2 seconds — while §11.2 spends one budget on discovery, dial, authentication, and negotiation together. The snapshot also ran synchronously on an async worker, so a wedged mount occupied the worker rather than merely overrunning; it now runs on the blocking pool under the shared deadline. Open the connection file with `O_NONBLOCK`. A FIFO at the configured path parks the open until a writer appears, which is before `checked_stat` can reject it as non-regular, so the fail-closed contract never ran and `Client::connect` hung. The flag cannot leak into the read: `checked_stat` proves `S_IFREG` first, and a regular file never reports `EAGAIN`. The TypeScript reader already passed it, so the two discovery paths agreed on everything except this. Stop cancelling a control request with an illegal frame. A control identity is 0/0, and §6.2 requires `Cancel` on a nonzero route with a nonzero correlation, so the emitted 0/0 frame made the host close the generation and take every unrelated route with it. The host settles the request on its own deadline; the caller's OutcomeUnknown classification already carries the replay protection. Publish a route handle under the lock `close` drains, rechecking closure while holding it. A close landing between the route.open response and the insert left a handle in a drained set and returned `Ok` for a handle that fails `client_closed` on first use. No route `Goodbye` is owed: `close` sends the connection `Goodbye`, which settles every route on the generation. Release the ambiguous generation before dialing a replay connection. Both held a host connection permit, so at a `max_connections` of 1 the host dropped the new socket for capacity, `reconnect` failed, and the old permit was never freed because the cleanup sat past the failure. The daemon comparison survives the reorder because the frozen id is a parameter rather than something read back off the old connection. The FIFO test probes on a worker thread with a bounded join: the regression hangs rather than returning, and the bound turns that into a failure instead of a wedged suite. --- crates/mc-host/src/client.rs | 55 ++++++++++++++++++--- crates/mc-host/src/connection_file.rs | 57 +++++++++++++++++++++- crates/mc-module/src/historian_producer.rs | 20 +++++--- 3 files changed, 118 insertions(+), 14 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 55516cb8a..11b06ab03 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -257,18 +257,32 @@ impl Client { /// /// Discovery validates one descriptor-anchored snapshot before any dial. pub async fn connect(path: impl AsRef) -> Result { - let info = read_for_client(path) - .map_err(|_| ClientError::new("discovery_failed", "secure discovery failed"))?; - Self::connect_info(info).await + // The deadline starts before discovery, not after it. §11.2 spends one + // 2-second budget on discovery, dial, authentication, and negotiation + // together, so starting the clock after the snapshot would give a + // stalled filesystem unbounded time and then hand the handshake a fresh + // budget. The snapshot also runs on a blocking pool: it is synchronous + // filesystem work, and on a wedged mount it would otherwise occupy an + // async worker for as long as the mount takes. + let deadline = Instant::now() + CLIENT_HANDSHAKE_TIMEOUT; + let path = path.as_ref().to_path_buf(); + let info = timeout_at( + deadline, + tokio::task::spawn_blocking(move || read_for_client(path)), + ) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("discovery_failed", "secure discovery failed"))? + .map_err(|_| ClientError::new("discovery_failed", "secure discovery failed"))?; + Self::connect_info(info, deadline).await } - async fn connect_info(info: ConnectionInfo) -> Result { + async fn connect_info(info: ConnectionInfo, deadline: Instant) -> Result { let endpoint = info .endpoints .first() .ok_or_else(|| ClientError::new("discovery_failed", "secure discovery failed"))? .clone(); - let deadline = Instant::now() + CLIENT_HANDSHAKE_TIMEOUT; let mut stream = timeout_at( deadline, TcpStream::connect((endpoint.host.as_str(), endpoint.port)), @@ -362,7 +376,25 @@ impl Client { match response { Ok(response) => { let handle = parse_route_open(&response.body)?; - lock_unpoisoned(&self.inner.routes).insert(handle); + // Publish under the same lock `close` drains, and recheck + // closure while holding it. A close that lands between the + // response arriving and this insert would otherwise leave a + // handle in a drained set and hand the caller an `Ok` that + // fails `client_closed` on first use. Local close wins + // (protocol §11.1). The host side needs no route `Goodbye` + // here: `close` sends the connection `Goodbye`, which + // obliges the host to settle every route on the generation. + { + let mut routes = lock_unpoisoned(&self.inner.routes); + if self.inner.closed.load(Ordering::Acquire) { + return Err(CallError::local( + SendOutcome::NotSent, + "client_closed", + "client is closed", + )); + } + routes.insert(handle); + } return Ok(handle); } Err(error) @@ -936,6 +968,17 @@ impl Inner { Err(CallError::local(outcome, code, "request stopped")), ); if outcome == SendOutcome::OutcomeUnknown { + // A control request has identity 0/0, and §6.2 requires `Cancel` on + // a current nonzero route with a pending nonzero correlation — so + // cancelling one has no legal frame. Emitting 0/0 anyway made the + // host close the generation, taking every unrelated route with it, + // and left cleanup depending on the peer accepting a malformed + // frame. The host settles the request on its own deadline instead; + // the caller's OutcomeUnknown classification is already correct and + // is what actually protects it from replaying. + if key.channel == 0 { + return Ok(()); + } // The Cancel is best-effort cleanup, and the request's bytes may // already be on the wire. Report the failed enqueue, but keep the // request's own OutcomeUnknown classification: substituting the diff --git a/crates/mc-host/src/connection_file.rs b/crates/mc-host/src/connection_file.rs index e3d02d37a..b12afbb62 100644 --- a/crates/mc-host/src/connection_file.rs +++ b/crates/mc-host/src/connection_file.rs @@ -292,10 +292,16 @@ fn open_file( name: &OsString, path: &Path, ) -> Result { + // `NONBLOCK` so a special file reaches metadata validation instead of + // stalling here: opening a FIFO for reading blocks until a writer appears, + // which is before `checked_stat` gets to reject it as non-regular — so the + // fail-closed contract never runs and the caller hangs. It cannot leak into + // the read: `checked_stat` proves `S_IFREG` first, and a regular file never + // reports `EAGAIN`. The TypeScript reader passes the same flag. openat( parent, name, - OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC | OFlags::NONBLOCK, Mode::empty(), ) .map_err(|source| io_error("open", path, source.into())) @@ -348,6 +354,55 @@ mod tests { } } + /// A FIFO at the configured path must be rejected, not waited on. Without + /// `NONBLOCK` the open itself parks until a writer appears, so this call + /// never returns and `Client::connect` hangs with it. + /// + /// The scratch directory is chmodded to owner-only on purpose: `open_parent` + /// requires a private leaf parent, so a default-mode temp directory is + /// rejected before `open_file` runs and the test would pass without ever + /// reaching the open under test. + #[test] + fn a_fifo_is_rejected_rather_than_blocking_the_open() { + use std::os::unix::fs::PermissionsExt; + + let dir = std::env::temp_dir().join(format!("mc-fifo-{}", std::process::id())); + std::fs::create_dir_all(&dir).expect("scratch dir"); + std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)) + .expect("owner-only scratch dir"); + let path = dir.join("subc-connection.json"); + let _ = std::fs::remove_file(&path); + rustix::fs::mknodat( + rustix::fs::CWD, + &path, + rustix::fs::FileType::Fifo, + Mode::from_bits_truncate(0o600), + 0, + ) + .expect("mkfifo"); + + // No writer is ever opened, so a blocking open can never complete. + // Bounded on a worker thread rather than called directly: a regression + // hangs instead of returning, and this turns that into a failure rather + // than a wedged suite. + let (tx, rx) = std::sync::mpsc::channel(); + let probe = path.clone(); + std::thread::spawn(move || { + let _ = tx.send(read_for_client(&probe).map_err(|error| format!("{error:?}"))); + }); + let outcome = rx.recv_timeout(std::time::Duration::from_secs(5)); + + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_dir(&dir); + + let outcome = outcome.expect("the open must not block on a writer that never arrives"); + let error = outcome.expect_err("a FIFO is not a connection file"); + assert!( + error.contains("Insecure"), + "expected an insecure-type rejection, got {error}" + ); + } + #[test] fn strict_wire_version_rejects_missing_null_string_and_other() { let valid = serde_json::to_value(info()).expect("serialize"); diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index e6a7c818b..bca4fdf75 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -920,20 +920,26 @@ impl HistorianProducer { frozen_identity: SemanticIdentity, frozen: &[u8], ) -> Result { + // Release the ambiguous generation before dialing the replay + // connection. Both hold a host connection permit, so overlapping them + // cannot recover at a `max_connections` of 1: the host drops the newly + // authenticated socket for capacity, `reconnect` fails, and the old + // permit is never freed because the cleanup sits past the failure. + // The daemon comparison survives the reorder because `frozen_daemon` is + // already captured — it does not come from the old connection. + if let Err(error) = self.close_routes_and_connection().await { + eprintln!("mc-module: historian replay cleanup failed: {error}"); + } + self.command_route = None; + self.subscribe_route = None; + let reconnected = self .connector .reconnect(&self.config, &frozen_identity) .await?; let daemon_changed = reconnected.connection.daemon_id() != frozen_daemon; let identity_changed = reconnected.identity != frozen_identity; - - let old_cleanup = self.close_routes_and_connection().await; - if let Err(error) = old_cleanup { - eprintln!("mc-module: historian replay cleanup failed: {error}"); - } self.connection = reconnected.connection; - self.command_route = None; - self.subscribe_route = None; if daemon_changed || identity_changed { return Err(HistorianProducerError::CrossIncarnationUnknown { From fba57ff515cb998e5cc3822068dfa0f8d37b8e4e Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 14:00:59 +0000 Subject: [PATCH 17/37] fix(mc-host): enforce the channel-zero body cap and bound the auth deadline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client accepted a channel-0 body up to the framing cap, so a host could send a 64 MiB `route.open` response that `read_active_frame` allocated and retained. §7.1 caps a channel-0 body at 65,536 bytes, and `parse_route_open` ignores unknown fields, so a padded response would open the route and leave the generation live while holding the memory. The framing layer already enforced this for channel-0 requests; the client now enforces it on the header, before any allocation. Fail the connect when setup lost a race with retirement. The reader runs on another worker and can retire the generation before the constructor returns — a peer that closes or sends connection `Goodbye` right after negotiation does exactly that. A "ready" client then deferred the failure to the first operation, which reports `connection_retired` as `NotSent`, and the historian does not reconnect on that path, so a daemon reload race aborted the run instead of establishing a replacement. Reject an authentication deadline that cannot be represented. The total is operator configuration, and `Instant + Duration` panics when the sum overflows, so a `Duration::MAX` `auth_deadline` took down the connection task on the first accepted connection. `AuthError::InvalidDeadline` says the setting is unusable rather than claiming a timeout, which is what the `Timeout` variant would have implied. The body-cap and deadline fixes each have a test that fails without them — the deadline one panics inside tokio's `Instant` arithmetic, which is the reported failure. The setup-race guard has none: the window depends on cross-worker scheduling, and a test that races it would be flaky rather than discriminating. --- crates/mc-host/src/auth.rs | 41 ++++++++++++++++++++++++------ crates/mc-host/src/client.rs | 48 +++++++++++++++++++++++++++++++++++- 2 files changed, 81 insertions(+), 8 deletions(-) diff --git a/crates/mc-host/src/auth.rs b/crates/mc-host/src/auth.rs index 6f46ae065..1dbdb0683 100644 --- a/crates/mc-host/src/auth.rs +++ b/crates/mc-host/src/auth.rs @@ -96,6 +96,11 @@ pub enum AuthError { stage: AuthStage, source: serde_json::Error, }, + /// The configured total is not representable as an absolute deadline, so no + /// handshake can be attempted against it. + InvalidDeadline { + total: Duration, + }, Random(getrandom::Error), KeyTooShort { len: usize, @@ -133,11 +138,15 @@ struct Deadline { } impl Deadline { - fn starting_now(total: Duration) -> Self { - Self { - at: time::Instant::now() + total, - total, - } + /// Fallible because the total is operator configuration: `Instant + + /// Duration` panics when the sum is unrepresentable, and a `Duration::MAX` + /// auth deadline would take down the connection task rather than reporting + /// a bad setting. + fn starting_now(total: Duration) -> Result { + let at = time::Instant::now() + .checked_add(total) + .ok_or(AuthError::InvalidDeadline { total })?; + Ok(Self { at, total }) } /// Time left until the deadline, or `Timeout` if it has already elapsed. @@ -170,7 +179,7 @@ pub async fn authenticate_server( where S: AsyncRead + AsyncWrite + Unpin, { - let deadline = Deadline::starting_now(deadline); + let deadline = Deadline::starting_now(deadline)?; let result = authenticate_server_inner(stream, key, daemon_id, daemon_ver, deadline).await; if result.is_err() { // Bound teardown by the SAME absolute deadline so a failed handshake (and @@ -251,7 +260,7 @@ pub async fn authenticate_client_with_role( where S: AsyncRead + AsyncWrite + Unpin, { - let deadline = Deadline::starting_now(deadline); + let deadline = Deadline::starting_now(deadline)?; let result = authenticate_client_inner(stream, conn, deadline, role).await; if result.is_err() { let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; @@ -542,6 +551,9 @@ impl fmt::Display for AuthError { write!(f, "auth {stage:?} JSON decode error: {source}") } Self::Random(source) => write!(f, "auth random generation failed: {source}"), + Self::InvalidDeadline { total } => { + write!(f, "auth deadline {total:?} is not a representable instant") + } Self::KeyTooShort { len, min } => { write!(f, "auth key is too short: {len} bytes, need at least {min}") } @@ -564,6 +576,7 @@ impl Error for AuthError { | Self::KeyTooShort { .. } | Self::InvalidServerProof | Self::DaemonIdMismatch + | Self::InvalidDeadline { .. } | Self::InvalidClientAuth => None, } } @@ -572,6 +585,20 @@ impl Error for AuthError { #[cfg(test)] mod tests { use super::*; + + #[test] + fn an_unrepresentable_auth_deadline_is_rejected_not_panicked() { + // The total is operator configuration, so `Duration::MAX` must report a + // bad setting rather than panic inside the connection task. + let error = Deadline::starting_now(Duration::MAX) + .err() + .expect("an unrepresentable total has no absolute deadline"); + assert!( + matches!(error, AuthError::InvalidDeadline { .. }), + "{error:?}" + ); + assert!(Deadline::starting_now(Duration::from_secs(2)).is_ok()); + } use tokio::{ io::{duplex, DuplexStream}, task::yield_now, diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 11b06ab03..3650fa743 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -36,7 +36,8 @@ use crate::{ }, wire::{ decode_header, encode_owned_frame, pure_header_flags, AdmissionClass, EnvelopeHeader, - Flags, FrameId, FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, PROTOCOL_VERSION, + Flags, FrameId, FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, MAX_CONTROL_BODY_LEN, + PROTOCOL_VERSION, }, }; @@ -336,6 +337,19 @@ impl Client { }); *inner.writer.lock().await = Some(writer); *inner.reader.lock().await = Some(reader); + // The reader runs on another worker and can retire this generation + // before the constructor returns — a peer that closes or sends + // connection `Goodbye` right after negotiation does exactly that. + // Returning a "ready" client then defers the failure to the first + // operation, which reports `connection_retired` as `NotSent`; the + // historian does not reconnect on that path, so a daemon reload race + // would abort the run instead of establishing a replacement. + if inner.retired.load(Ordering::Acquire) { + return Err(ClientError::new( + "connection_retired", + "connection retired during setup", + )); + } Ok(Self { inner }) } @@ -1622,6 +1636,14 @@ fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { if header.ver != PROTOCOL_VERSION || header.len > MAX_BODY_LEN { return Err(()); } + // §7.1 caps a channel-0 body at 65,536 bytes even though framing permits + // more. Rejecting on the header keeps one oversize control response from + // being allocated and retained at all — `parse_route_open` ignores unknown + // fields, so a padded response would otherwise open a route and leave the + // generation live while holding roughly 64 MiB. + if header.channel == 0 && header.len > MAX_CONTROL_BODY_LEN { + return Err(()); + } match header.ty { FrameType::Response | FrameType::Error | FrameType::StreamData | FrameType::StreamEnd => { // `decode_header` already rejects a mixed zero/nonzero @@ -2360,6 +2382,30 @@ mod tests { // Push is unsolicited, so it carries no correlation. assert!(validate_inbound(&header(FrameType::Push, 3, 9, 5, 4)).is_err()); + // §7.1 caps a channel-0 body at 65,536 bytes; framing alone permits far + // more, and an accepted oversize control response would be allocated + // and retained before anything could reject it. + assert!( + validate_inbound(&header(FrameType::Response, 0, 0, 7, MAX_CONTROL_BODY_LEN)).is_ok() + ); + assert!(validate_inbound(&header( + FrameType::Response, + 0, + 0, + 7, + MAX_CONTROL_BODY_LEN + 1 + )) + .is_err()); + // A routed body is opaque and keeps the framing cap. + assert!(validate_inbound(&header( + FrameType::Response, + 3, + 9, + 7, + MAX_CONTROL_BODY_LEN + 1 + )) + .is_ok()); + // Pre-existing rules keep holding. assert!(validate_inbound(&header(FrameType::Response, 3, 9, 0, 4)).is_err()); assert!(validate_inbound(&header(FrameType::Ping, 0, 0, 7, 0)).is_ok()); From 6307a6e8df9da395449f24a95fa3b67a32035a76 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 14:32:40 +0000 Subject: [PATCH 18/37] fix(mc-host): keep the client's send classification honest under races MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four findings from the latest codex round, each with a test that fails when the fix is reverted. A terminal that wins the cancellation race is no longer discarded. `dispatch` removes the pending entry before it publishes the terminal, so a cancellation or deadline landing in that window made `cancel_key` find nothing to cancel and report success; the stop branch then returned a local `OutcomeUnknown` error and dropped the receiver, throwing away an authoritative response the host had already sent and pushing the caller into recovery for an operation that actually settled. The stop paths now consume an already-published terminal before returning their own error. Stream saturation reports `OutcomeUnknown` instead of `Terminal`. Overflowing the consumer queue is a local failure after the request went out: no `Response`, `Error`, or `StreamEnd` was observed and the best-effort `Cancel` may not have reached the host, so the run may still be committing. `Terminal` claimed an authoritative settlement the client never saw and marked a possibly-live operation replay-safe, against `SendOutcome`'s contract and the §10.1 replay boundary. Inbound validation splits stream frames from control-capable terminals. Grouping them checked only the correlation, so a `StreamData` or `StreamEnd` on `0/0` bearing a pending control correlation was accepted and reported as `unexpected_stream` while the generation stayed usable. §6.2 requires an exact pending routed identity, so a control identity is structurally illegal and has to close the generation. Channel-zero terminals must set `binary = 0`. The binary check applied only to pure-header frames, so a binary `route.open` response whose bytes happened to parse as JSON opened a route and left the malformed generation live. §7.1 admits UTF-8 JSON only on channel 0. --- .beads/interactions.jsonl | 1 + .beads/issues.jsonl | 5 + crates/mc-host/src/client.rs | 181 ++++++++++++++++-- packages/plugin/src/index.ts | 9 + .../src/shared/mc-host-client/client.test.ts | 4 +- .../src/shared/mc-host-client/client.ts | 9 +- 6 files changed, 189 insertions(+), 20 deletions(-) diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index 71e90c6f1..549f0b074 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -71,3 +71,4 @@ {"id":"int-adf3e410823be79e2f65962c00ee00ac","kind":"field_change","created_at":"2026-08-24T13:41:22.526618118Z","actor":"AhravDutta","issue_id":"magic-context-ymc.3","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} {"id":"int-fd51b14afbf32b43a54c33c665d47e22","kind":"field_change","created_at":"2026-08-24T20:56:41.646974108Z","actor":"AhravDutta","issue_id":"magic-context-c50.12","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Compile matrix passed at main tip 574569d5; inventory matches evidence in docs/evidence/subc-compiler-closure/ (87 rows: 83 exact/3 changed/1 private-unknown). Completeness gate on c50.4 cleared."}} {"id":"int-5eed966629542898b21c355dc0b781fb","kind":"field_change","created_at":"2026-08-25T06:56:29.585492666Z","actor":"AhravDutta","issue_id":"magic-context-c50.4","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Implemented direct mc-host boundary: host-owned Rust wire/auth/discovery/client, direct McHandler adapter and historian, host-owned TS API, direct Rust/E2E fixtures, dependency/docs closure; workspace, TS, E2E, specialist and ponytail reviews complete."}} +{"id":"int-606a80a74abee403db373ff1892c729b","kind":"field_change","created_at":"2026-08-25T13:55:26.811161159Z","actor":"AhravDutta","issue_id":"magic-context-qhh","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"All five landed in acba0244, each with a test that fails without it.\n\nHandshake deadline now starts before discovery, with the snapshot on the blocking pool under the shared budget. Connection file opens with O_NONBLOCK; the EAGAIN concern does not apply because checked_stat proves S_IFREG before any read. Control-request Cancel is no longer emitted at all, because a 0/0 identity has no legal frame and the host settles on its own deadline. Route handles publish under the lock close drains, with the closure recheck inside it; no route Goodbye is owed because close already sends the connection Goodbye. Replay reconnect releases the old generation first, which is safe because the frozen daemon id is a parameter rather than read off the old connection.\n\nTwo things the verification corrected. The FIFO test first passed for the wrong reason: open_parent requires an owner-only leaf parent, so a default-mode temp dir was rejected before open_file ran. With the dir chmodded to 0700 the un-fixed code hangs (timeout exit 124), confirming the block. The test now probes on a worker thread with a bounded join so the regression fails instead of wedging the suite.\n\nAlso learned while verifying: open_parent already does descriptor-relative per-component traversal with is_safe_ancestor on each, so the Rust reader does provide the ancestor safety the doc requires. The gap is TypeScript-only, as reported on that thread."}} diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 35ecdc4db..e4ecdf7fd 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,5 @@ +{"_type":"issue","id":"magic-context-qhh","title":"Close five verified client and discovery findings from PR 32 review","description":"Five findings verified during the PR 32 review round but deliberately not fixed there, because each needs a behavior or shape decision rather than a mechanical edit. All are confirmed against code; none is speculative.\n\n1. Control-request Cancel is structurally illegal. `cancel_key` emits `FrameId { channel: 0, epoch: 0, corr }` when an ambiguous control request (route.open) is cleaned up, but Section 6.2 requires Cancel on a current nonzero route and pending nonzero correlation, so the host closes the generation. Remedy is a choice: retire the generation locally (tears down every other route on the connection) or skip the Cancel and let the host request deadline settle it (leaves host-side work uncancelled for that window). Depends on how long an in-flight route.open can hold host resources.\n\n2. The handshake deadline excludes discovery. `Client::connect` reads the connection file, then `connect_info` starts a fresh two-second deadline, so discovery is unbounded and dial plus authentication plus negotiation get the full budget again. Section 11.2 says the deadline spans discovery through negotiation. The snapshot is also blocking filesystem work on an async worker, so a stalled mount occupies the worker rather than merely overrunning. Fix moves the deadline before discovery and puts the snapshot in a bounded blocking task, which changes failure classification between handshake_timeout and discovery_failed.\n\n3. `read_for_client` opens without `O_NONBLOCK`. A FIFO at the configured path blocks in `open` before `checked_stat` can reject it as non-regular, hanging `Client::connect` on a Tokio worker. The TypeScript reader already passes `O_NONBLOCK`, so the two discovery paths disagree. Adding the flag requires the subsequent bounded read to handle `EAGAIN`, or a hang becomes a truncated snapshot.\n\n4. A late route open may survive client close. `close()` can drain the route set between the route.open response being delivered and `routes.insert(handle)`, after which `open_route` returns a handle that immediately fails `client_closed`, violating local-close-wins. A recheck must be atomic with the publish or it only moves the window; whether the close path then owes a route Goodbye depends on whether the handle was ever observable.\n\n5. Replay reconnect may be unrecoverable at `max_connections` 1. `replay_frozen_once` appears to dial the replay connection while the old connection still holds its permit, so the host drops the new socket for capacity and `reconnect` fails before the cleanup that would free the permit. Releasing the old generation first has to preserve the frozen daemon id for the cross-incarnation comparison, so it is not a pure reorder.\n\nOriginal review threads carry the full reasoning and the options considered for each.","acceptance_criteria":"Each item either lands with a test that fails without it, or is closed with a recorded decision explaining why the current behavior is correct.","status":"closed","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-25T10:13:30Z","created_by":"AhravDutta","updated_at":"2026-08-25T13:55:27Z","closed_at":"2026-08-25T13:55:27Z","close_reason":"All five landed in acba0244, each with a test that fails without it.\n\nHandshake deadline now starts before discovery, with the snapshot on the blocking pool under the shared budget. Connection file opens with O_NONBLOCK; the EAGAIN concern does not apply because checked_stat proves S_IFREG before any read. Control-request Cancel is no longer emitted at all, because a 0/0 identity has no legal frame and the host settles on its own deadline. Route handles publish under the lock close drains, with the closure recheck inside it; no route Goodbye is owed because close already sends the connection Goodbye. Replay reconnect releases the old generation first, which is safe because the frozen daemon id is a parameter rather than read off the old connection.\n\nTwo things the verification corrected. The FIFO test first passed for the wrong reason: open_parent requires an owner-only leaf parent, so a default-mode temp dir was rejected before open_file ran. With the dir chmodded to 0700 the un-fixed code hangs (timeout exit 124), confirming the block. The test now probes on a worker thread with a bounded join so the regression fails instead of wedging the suite.\n\nAlso learned while verifying: open_parent already does descriptor-relative per-component traversal with is_safe_ancestor on each, so the Rust reader does provide the ancestor safety the doc requires. The gap is TypeScript-only, as reported on that thread.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"magic-context-shb","title":"Restore wire-level coverage for historian producer decode paths","description":"PR #32 (direct mc-host boundary) deleted crates/mc-module/tests/real_daemon.rs and shrank broca_roundtrip.rs from 6 tests to 2. Most of that deletion was mechanical: the module no longer runs as a separate process, so tests that spawned ck-subc/ck-mc had no subject left. Four behaviors lost coverage for code that still exists under the new direct boundary.\n\n1. Historian output round-trip. The old test asserted output text, length-cap propagation, byte-identical redrain, one backend start, and route release. The parser survives in crates/mc-module/src/historian_producer.rs (drain_subscribe, await_output, redrain_output). A drain_subscribe test now covers text plus the length cap; redrain and the backend-start/route-release assertions are still uncovered.\n\n2. Classify round-trip. Current classify tests use TestProducerFactory, which returns already-decoded outputs, so nothing joins producer decode to memory.set_classification. The handler survives at crates/mc-module/src/lib.rs (dreamer.run_task path).\n\n3. Transient-model retry metadata. The current retry test injects an already-decoded HistorianProducerError. Producer-side decoding of typed transient metadata (historian_producer.rs classification_from_object and callers) has no test.\n\n4. Host-restart reattach/refire. The downstream transition is unit-tested with an injected RunState::Missing, but decoding the concrete \"missing\" wire value in historian_producer.rs is untested.\n\nScope: unit tests against historian_producer.rs decode functions plus, where a wire path is required, direct-host integration tests. Not an argument for restoring real_daemon.rs, whose topology no longer exists.","acceptance_criteria":"Each of the four behaviors has a test that fails when its decode or transition branch is reverted.","status":"open","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-25T08:01:45Z","created_by":"AhravDutta","updated_at":"2026-08-25T08:01:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-8nz","title":"Build cross-harness prose steering","description":"Implement shared plain-prose policy and adapters for Pi, Claude Code, OpenCode, Kilo Code, and Codex. Add broad deterministic and harness-contract tests; validate installed hook configurations. User requires final-output rewrite where supported and bounded Stop/SubagentStop retries otherwise.","status":"closed","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T20:46:16Z","created_by":"AhravDutta","updated_at":"2026-08-24T21:00:25Z","started_at":"2026-08-24T20:46:25Z","closed_at":"2026-08-24T21:00:25Z","close_reason":"Installed prose-steering across Pi, OpenCode, Kilo, Claude Code, and Codex; published Pi sync snapshot 2026-08-24T20-59-09-401Z-3e487759; copied and verified three remote hosts.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-cjs","title":"Wire recordDispositionEventInCurrentTransaction into a host command","description":"PR #24 round-45 (comment 3843919627): recordDispositionEventInCurrentTransaction is the only entry point for rejected/quarantined/explicit-stale/explicit-disputed dispositions, and claim-visibility-policy's hard-hide matrix treats them as authoritative — but no production call site exists in packages/plugin or packages/pi-plugin. The hard-hide/quarantine/reject branch is exercised only by unit tests. Decide the host surface (a /ctx-dispute or /ctx-quarantine command mirroring /ctx-approve's confirmation flow, or dreamer-driven assertion) and wire it. Related: magic-context-x84 (MODULE-authority decision channel) will need the same events to flow to native readers.","status":"open","priority":1,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T14:02:10Z","created_by":"AhravDutta","updated_at":"2026-08-24T14:02:10Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-x84","title":"MODULE-authority native rendering needs a claim-policy decision channel","description":"Under MODULE memory authority, the native render path (mc-store load_active_memories and the m1 lane) selects active/permanent unexpired rows with no claim-policy or verification predicate, and the TypeScript state-sync policy filter never applies because memory sections are omitted while the module owns the lane (module-state-sync.ts omitAuthorityMemorySections). A /ctx_memory write under MODULE authority therefore creates an active, unverified native row that the next native transform injects automatically — a CANDIDATE bypasses the v86 visibility ladder entirely in rust mode.\n\nRaised by codex review on PR #24 (inline comment 3840295420). Deferred from the PR because the fix needs an authority-model design decision, not a mechanical patch: while MODULE owns the lane, TypeScript (the policy authority) has no channel to push per-row eligibility without violating the one-writer-per-pool rule that fences the memories section (mc-store apply gates PREPARING/MODULE/DRAINING).\n\nCandidate shapes to evaluate:\n1. A dedicated policy sidecar section in state_sync (eligibility verdicts keyed by native row id) that the apply lane accepts even under MODULE authority, since it carries no row content — keeps Rust free of policy derivation.\n2. Mirror auto_eligible into mc_memories via the module-to-TS changefeed round trip: TS adjudicates module-created rows on the reverse sync and pushes the verdict back; native render gains a WHERE auto_eligible = 1 predicate. Requires a default for not-yet-adjudicated rows (fail closed = candidate-invisible until adjudicated; fail open = today's behavior).\n3. Derive a conservative native predicate from existing columns (verification_status) — rejected on first pass: USER_EXPLICIT-taint rows are auto-eligible without verification, so this under-renders legitimate content.\n\nAcceptance: under MODULE authority, a freshly written unverified memory does not render into the native m0/m1 lanes until the policy authority marks it eligible; explicit get keeps working with trust labels; no policy derivation logic moves into Rust.\n\n## Context\nPR #24 v86 claim trust policy follow-up","status":"open","priority":1,"issue_type":"bug","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T02:22:20Z","created_by":"AhravDutta","updated_at":"2026-08-24T02:22:20Z","comments":[{"id":"01a032c6-1a8c-7ba0-9c23-7bcd93599c95","issue_id":"magic-context-x84","author":"AhravDutta","text":"New review evidence (PR #24 comment 3841517352): in MODULE authority mode omitAuthorityMemorySections=true also omits the memories_delete_ids lane and the filtered replacement snapshot, so a module-created candidate or a row quarantined/rejected after mirror-back stays automatically injected for the whole ownership period. The decision channel must either keep policy revocations crossing the boundary in authority mode (e.g. send the delete lane even when snapshot sections are omitted) or move the policy evaluator natively.","created_at":"2026-08-24T07:57:19Z"},{"id":"01a033d3-3d07-76c4-9a40-568a4351264c","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-41 evidence (PR #24 comment 3843504733): while MODULE owns memories, module-state-sync.ts:1662 suppresses the full policy replacement/delete list; a native ctx_memory row is active+unverified in the module store and its mirrored CANDIDATE decision never flows back — McStore::load_memory_render_snapshot and the native search/get readers select active rows with no claim-policy predicate, so the row stays injectable and searchable indefinitely. The decision channel must either propagate policy visibility separately (preserving single-writer content ownership) or enforce mirrored decisions in the native readers.","created_at":"2026-08-24T12:51:17Z"},{"id":"01a034ea-fa65-7ed8-b0b1-a5cf5cbea034","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-57 evidence (PR #24 comment 3846037059): while authorityState is MODULE, omitAuthorityMemorySections suppresses the policy-filtered snapshot AND its deletion/reclassification updates; the authority seed copies raw rows and native render/search select active rows with no policy predicate — a fresh unverified CANDIDATE or a later-quarantined/contradicted/rejected row stays injectable through native Rust-mode surfaces. The decision channel must stay synchronized without transferring row ownership.","created_at":"2026-08-24T17:56:50Z"},{"id":"01a0351d-751f-7db2-a49f-5d5f32937cfb","issue_id":"magic-context-x84","author":"AhravDutta","text":"Round-60 evidence (PR #24 comment 3846478393): with omitAuthorityMemorySections both allMemories and incrementalMemories are empty, so the sync's policy filter never adjudicates native ctx_memory-created rows; mc-store renders active/permanent rows with no policy predicate, so a fresh unverified CANDIDATE enters native m0/m1 and search immediately. Reviewer proposes carrying host-computed eligibility to the native store or failing closed for native-created rows before rendering.","created_at":"2026-08-24T18:51:58Z"}],"dependency_count":0,"dependent_count":0,"comment_count":4} @@ -49,6 +51,7 @@ {"_type":"issue","id":"magic-context-3q5.1","title":"U1: Search hot-path micro-fixes","description":"Phase 0 — Immediate wins (land before the architecture). Not benchmark-gated: ranking semantics must NOT change (ranking-parity fixture is the guard).\n\n## Goal\nRemove the known O(N) and hydration waste in unifiedSearch without changing ranking semantics.\n\n## Governing constraints (quoted)\n- R9 (late-materialization direction): \"stable IDs, text, and metadata hydrate only after fusion selects final candidates.\" This unit is groundwork toward that shape.\n- R11: \"Search work is bounded, cancellable, and priority-aware: independent hard caps on query bytes and embedding tokens, query atoms, per-lane candidates, returned results, reranker pairs/tokens, and rendered context tokens...\"\n\n## Key files\n- packages/plugin/src/features/magic-context/search.ts\n- packages/plugin/src/features/magic-context/search.test.ts\n- packages/plugin/src/features/magic-context/message-index.ts\n\n## Approach\n1. Use FTS5 snippet() for message results instead of returning and normalizing complete bodies.\n2. Make exact memory-ID lookup an indexed json_each(?) join instead of loading all visible memories.\n3. Put notes into FTS instead of scanning every note per probe.\n4. Replace the per-message compartment .find() with an ordered interval sweep over (compartment_id, ordinal range).\n5. Select git-commit top-K before hydrating commit metadata.\n6. Maintain primer centroid sum/count incrementally instead of recomputing cluster centroids on insert.\nPatterns to follow: existing probe structure in search.ts; FTS conventions in message-index.ts.\n\n## Test scenarios\n- Snippet path returns highlighted fragments for a matching message and IDENTICAL ranking to the pre-change fixture.\n- ID lookup with 3 IDs (one missing) hits the index and returns the 2 present memories; empty ID list returns empty without a table scan (assert via statement spy/counter, not timing).\n- Interval sweep assigns a message on a compartment boundary ordinal to exactly one compartment; a message outside all ranges stays a message hit.\n- Notes FTS: matching note surfaces; non-matching note absent; note deletion removes it from results.\n- Git top-K: with limit 5 and 50 matching commits, exactly 5 hydration reads occur (statement spy/counter).\n\n## Verification\n- bun test packages/plugin/src/features/magic-context/search.test.ts green; ranking-parity fixture unchanged.\n- bun run typecheck; bun run lint.\n","notes":"reduce-complexity: 6 local fixes, no new abstractions - ok as drafted; less-code: every step replaces existing waste, no scaffolding added; perf: hot-path wins made assertable via statement counters; ponytail-review: lean already, ship; invariant-test-review: strengthened - no-table-scan and index-hit claims must use statement spies/counters, not timing; test-strategy: golden ranking-parity fixture + counter-based example tests","status":"in_progress","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:23:34Z","created_by":"AhravDutta","updated_at":"2026-08-17T03:37:59Z","started_at":"2026-08-17T03:37:59Z","labels":["phase-0"],"dependencies":[{"issue_id":"magic-context-3q5.1","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:23:33Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-3q5.2","title":"U2: Splitter fix + retrieval counters out of FTS rows","description":"Phase 0 — Immediate wins. Not benchmark-gated: splitter output must be byte-identical; ranking semantics unchanged.\n\n## Goal\nKill the splitter's quadratic behavior and stop telemetry counter writes from rewriting FTS-triggered rows.\n\n## Governing constraints (quoted)\n- R11: \"Search work is bounded...\" — splitter cost must be linear, not quadratic.\n- R29 (write-amplification direction): \"A single storage actor owns each writer connection: ... adaptive group commit, batched outbox delivery/acknowledgment...\" — this unit is groundwork: counter updates must stop firing the memories_au FTS trigger rebuild.\n\n## Key files\n- packages/plugin/src/features/magic-context/recursive-text-splitter.ts (+ test)\n- packages/plugin/src/features/magic-context/migrations.ts\n- packages/plugin/src/features/magic-context/storage-db.ts\n- packages/plugin/src/features/magic-context/memory/storage-memory.ts\n\n## Approach\n1. Fix repeated tokenization and shift()-based quadratic copying in the splitter: index-based iteration; memoize token counts per fragment.\n2. Migration: move seen_count / retrieval_count / last_retrieved_at-class counters into a memory_stats side table keyed by memory id, so counter updates stop firing the memories_au FTS trigger rebuild. Follow the new-migration checklist: version bump, fence, fresh-DB schema, ensureColumn, clearSession, migrations-v\u003cN\u003e.test.ts.\n\n## Test scenarios\n- Splitter: 1 MB single-line input yields IDENTICAL chunks to the pre-change implementation on golden fixtures (byte-for-byte). Complexity oracle: assert linear scaling via an operation/copy counter (e.g., 2x input =\u003e ~2x operations) rather than relying on a wall-clock budget alone — timing is a flaky oracle for an algorithmic claim.\n- Splitter: separator-hierarchy behavior unchanged for \\n\\n / \\n / space / char fallbacks (existing tests keep passing).\n- Migration test: pre-migration rows with nonzero counters land in memory_stats with values preserved; a counter update post-migration does not touch memories (assert no FTS trigger fire via changes()/trigger counter).\n- clearSession removes stats rows for deleted sessions' memories only when the owning memory is deleted.\n\n## Verification\n- bun test for both modules; migration test proves counter reads/writes stay correct across the boundary.\n- Migration discipline: migrations-v\u003cN\u003e.test.ts, schema-version-fence.test.ts lockstep, fresh-DB schema + ensureColumn + clearSession updated.\n","notes":"reduce-complexity: two independent local fixes, ok; less-code: no scaffolding; perf: kills quadratic splitter + FTS trigger write amplification; ponytail-review: lean, ship; invariant-test-review: STRENGTHENED - replaced weak wall-clock oracle with operation-counter linear-scaling assertion alongside byte-for-byte golden parity; test-strategy: golden fixtures + migration example tests per repo convention","status":"in_progress","priority":1,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:23:34Z","created_by":"AhravDutta","updated_at":"2026-08-17T05:50:05Z","started_at":"2026-08-17T05:50:05Z","labels":["phase-0"],"dependencies":[{"issue_id":"magic-context-3q5.2","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:23:34Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-3q5","title":"Retrieval and Storage Architecture Overhaul (U1-U30)","description":"# Retrieval and Storage Architecture Overhaul\n\nSource plan: docs/plans/2026-08-16-001-refactor-retrieval-storage-overhaul-plan.md (authoritative: unit bodies U1-U30, Requirements R1-R33, KTD1-KTD30). Tasks under this epic are self-contained; the plan is the tie-breaker on conflict.\n\n## Goal\nRebuild Magic Context's retrieval and storage architecture so semantic search stays fast and correct as history grows without bound: an immutable claims/evidence domain model, rebuildable search projections, an ID-first Rust dense engine (crates/mc-vector behind the ck-mc module), a cost-based query pipeline with global weighted RRF fusion, and convergence of write/index ownership on the existing module-authority machinery. Performance is a first-class design input throughout (KD2): quick wins land first, hardware bets are benchmark-gated.\n\n## Success criteria\n- Auto-search p95 \u003c= 25 ms at a 100k-chunk / 384-dim scope on the benchmark hosts (ARM NEON and x86 AVX2), measured by the U5 harness.\n- Active decoded index \u003c= 128 MB at that scope ONLY IF a U15 compressed variant (int8/f16/Matryoshka-prefix) wins its recall gate. The row-major f32 baseline reads ~154 MB at 100k x 384, so if no U15 variant passes, the memory threshold relaxes to ~154 MB, R12's ANN trigger fires, and the ANN decision in Scope Boundaries governs further scale.\n- Fusion cutover (U18) shows recall/nDCG parity or better vs the pre-cutover baseline on the judged corpus, measured on the FINAL PACKED and deduplicated output (post-U19), not raw candidate lists.\n- Steady-state persistence bytes per transform pass drop by an order of magnitude after the delta journal (U28), measured on the e2e Rust harness fixture.\n- Fresh-schema cutover tests prove search starts from the claims-native retrieval projection and remains correct through atomic index construction and activation.\n- Search continues (degraded to lexical) with the daemon stopped, the index cold, or the embedding provider off.\n- U5 records whether TS-only + Phase 0/1 already met the latency/recall thresholds before any Phase 3+ unit lands (KD6: auditability record, NOT a stop condition — the full Rust engine and authority convergence ship regardless).\n\n## Stop conditions (every task inherits these)\nStop and surface instead of guessing when:\n(a) evidence shows a session-settled decision cannot work (infeasible, destructive, wrong-thing);\n(b) a schema migration would disable live processes outside the documented rollout order (R8: ship ALL binaries — OpenCode plugin, Pi plugin, CLI, ck-mc — with new version support first, THEN run the migration);\n(c) a benchmark gate fails for a change the gate governs (R27, R28);\n(d) work requires touching the prior audit's trust-boundary items (docs/fork-hardening-audit.md — out of scope).\n\n## Phase map\n- Phase 0 — Immediate wins (land before the architecture): U1, U2, U3\n- Phase 1 — Measure (gates ALL ranking/model/layout/ANN changes): U4, U5, U30\n- Phase 2 — Domain model and retrieval projection: U6, U7, U8, U9\n- Phase 3 — Rust dense engine: U10, U11, U12, U13, U14, U15\n- Phase 4 — Query pipeline and ranking: U16, U17, U18, U19, U20\n- Phase 5 — Corpus enrichment: U22 lands BEFORE U21 (KTD28), then U21, U23\n- Phase 6 — Embedding spaces and evaluation: U24, U25, U26\n- Phase 7 — Storage and authority convergence: U27, U28, U29\n\nEach unit lands green (bun test, cargo test --workspace, lint) before the next dependent unit starts. Each phase leaves the repo releasable: no half-applied migrations, no dark-search windows, abandoned experiment code removed. Benchmark-gated units (U30, U15, U16, U17, U18, U19, U20, U23, U26) must pass packages/plugin/scripts/benchmark-retrieval.ts regression mode vs the stored snapshot per KTD17 tolerances: nDCG@10 and Recall@50 each \u003c= 2pp regression averaged over 3 runs with no single run below baseline-5pp; p95 latency +10%.\n\n## Deferred / trigger-gated — NO tasks exist for these; do not build them (abstraction seams land in-plan)\n- HNSW: add behind VectorIndex only when R12 thresholds cross (auto-search p95 \u003e ~25 ms or active decoded index \u003e ~128 MB per scope).\n- IVF/PQ cold-history segments and the LSM-style hybrid index: only after corpus measurements justify; SQLite Vec1 stays shadow-evaluation only.\n- Per-project search-projection sharding into separate index files: the projection/outbox design permits it; do not split until measurements justify.\n- Postgres + pgvector server architecture: triggers are multiple hosts, hosted multi-user service, sustained concurrent writers, cross-device sync, per-tenant authorization.\n- Learned rankers: only after relevance labels accumulate from the judged corpus.\n- Outside the plan's identity: prior audit trust-boundary/fork-hardening findings; current-code-state retrieval; dedicated vector DB, ColBERT/multi-vector retrieval, embedding giant transcripts into single vectors.\n\n## Definition of done (global)\nAll 30 units landed or explicitly re-scoped with the user; every fence-bumping migration shipped in R8 rollout order; success criteria recorded via the U5 harness on both reference hosts per KTD17's versioned acceptance policy; search works in the supported new architecture and has explicit typed behavior for daemon, index, and provider failures; no dead experiment code or compatibility path remains after each new cutover; STRUCTURE.md updated for new modules/crates; docs/AUDIT-KNOWN-ISSUES.md / PARITY.md entries added where intentional divergences or accepted tradeoffs were created.\n\n","notes":"User decision supersedes the prior session-settled coexistence choice: this is greenfield with no legacy callers or deployed users. Claims-native direct cutovers are required. Do not add dual-read, maintained memories projections, legacy-ID adapters, old ranking paths, old embedding-table retention, or subc compatibility shims. Keep only safety fences, crash handling, and correctness-preserving atomic staging.","status":"open","priority":1,"issue_type":"epic","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:23:17Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:35:07Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"magic-context-kp5","title":"Give outgoing-frame byte accounting one owner in wire.rs","description":"Two byte-accounting implementations coexist in mc-host, and two reservation orderings.\n\n`client.rs` defines a mutex-backed `ByteCounter`/`ByteCharge` whose `charge(usize) -\u003e Option\u003cByteCharge\u003e` is contract-for-contract what `wire::ByteBudget::try_charge(usize) -\u003e Option\u003cwire::ByteCharge\u003e` already provides: synchronous, non-blocking, all-or-none, RAII release on drop. The wire version additionally has `split`, `shrink_to`, `capacity`, `available`, and its own tests. Both crates also name their charge type `ByteCharge`, so the two are distinguishable only by module path.\n\nFeasibility checked: `CLIENT_QUEUED_BYTES` is 65 MiB, far below the semaphore permit ceiling `try_charge` converts against, so the swap is mechanical. Call sites: `Inner.queue_budget`, `Inner.retained_budget`, `encode_data_frame`, the `dispatch` retained-charge path, `admit`, `ChargedItem._charge`, and the tests that read `ByteCounter::used()` (which becomes `available()`).\n\nSeparately, the two reservation orderings disagree. `connection.rs::reserve_catalog_frame` charges before encoding, computing frame size by hand as `body.len() + HEADER_LEN`. `client.rs::encode_data_frame` encodes first and charges the encoded length afterward, so the encoded buffer is briefly unaccounted. Both are locally correct; neither is the single rule, and a future change to reservation semantics has to find both.\n\nScope: move the client onto `wire::ByteBudget`, then decide one ordering and express it as a shared helper in `wire.rs` so header accounting cannot drift between call sites. Deliberately not folded into a review round on an already-large change: it touches live accounting on the client hot path and deserves its own diff.","acceptance_criteria":"One byte-accounting type backs both paths; frame reservation goes through one helper; existing budget tests still pass.","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-25T09:10:10Z","created_by":"AhravDutta","updated_at":"2026-08-25T09:10:10Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-50m","title":"Dreamer: persist the attempt harness for in-flight classify commands","description":"A dreamer.run_task handler cancelled after Broca accepted a run but before the response was ledgered leaves the Broca run alive by design. A retry of the same durable command through a different harness binding derives the same child-session string but opens it under a different (project_root, harness, session) key, so Broca's byte-identical send dedup does not match and a second billable run starts while the original may still be executing.\n\nThe in-process command guard closes this for a single live host, and the historian path is now covered because its awaiting state persists producer_harness. Classify has no equivalent durable in-flight state: mc_dream_task_commands records only the final response, so there is nowhere to record the attempt harness today.\n\nClosing it needs a durable in-flight record written before the first attempt and cleared or superseded on completion, which adds a write to the classify hot path and a crash-cleanup story for rows whose host died mid-attempt. That is a design change rather than a fix, so it is deliberately not folded into the Broca runner-route work.\n\nRaised in review of PR 28 (comment 3844564006).","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T15:03:43Z","created_by":"AhravDutta","updated_at":"2026-08-24T15:03:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-2my","title":"Extract a shared two-step confirmation helper for claim commands and ctx-recomp","description":"PR #24 round-45 (comment 3843922483): runConfirmedClaimMutation (claim-policy-commands.ts) and packages/pi-plugin/src/commands/ctx-recomp.ts each hand-roll a Map\u003csessionKey,{timestamp,argsKey}\u003e + 60s window + argsKey-equality confirmation. Eviction differs (ctx-recomp evicts on !warning.confirmable; claim commands sweep stale entries per call). Extract one PendingConfirmation helper in core that both import, preserving each caller's eviction semantics explicitly, so window/eviction/key-scheme changes land in one place.","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T14:02:11Z","created_by":"AhravDutta","updated_at":"2026-08-24T14:02:11Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-1l7","title":"Host: derive the resident-byte ceiling from declarations instead of a Broca-shaped default","description":"HostLimits::default() currently computes max_resident_bytes as 256 MiB plus broca::config::DECLARED_RETAINED_RESIDENT_BYTES, so a generic default reaches into one component's constants and every deployment that never composes Broca still inherits Broca-shaped sizing.\n\nThe knob's meaning is what forces this: max_resident_bytes is an absolute total ceiling, and the runtime subtracts each component's declared retained reservation from it to derive ingress headroom. A generic default therefore cannot be correct without knowing which components will be composed.\n\nThe fix is to change what the operator configures: let max_resident_bytes express ingress headroom and have the runtime add the composed handler's resource_declarations() to it, which runtime.rs already sums as reservations.retained_bytes for its resident-floor validation. That removes the hardcoded default entirely rather than relocating the coupling.\n\nThis is deferred because it changes the meaning of an operator-visible knob and touches every configuration and test that sets it, which does not belong in the Broca runner-route change.","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-24T12:37:28Z","created_by":"AhravDutta","updated_at":"2026-08-24T12:37:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -139,6 +142,8 @@ {"_type":"issue","id":"magic-context-3q5.9","title":"U8: Claims-native retrieval projection and embedding spaces","description":"Phase 2. Claims-native retrieval projection, outbox, generations, and embedding spaces.\n\n## Goal\nBuild every retrieval document and index from authoritative claims, revisions, evidence, tool events, notes, commits, and other current sources. The new projection is the only search representation.\n\n## Governing constraints\n- Retrieval structures remain rebuildable derived projections.\n- Vectors remain content-addressed by embedding space, text view, and content hash.\n- Index maintenance uses transactional outbox rows and per-scope generations.\n- No old embedding tables, legacy memory projection, dual-read path, or compatibility adapter remains live.\n\n## Approach\n1. Create `retrieval_documents`, `document_embeddings`, postings, `index_outbox`, and `search_index_generation` from the new schema.\n2. Populate memory documents from claim current revisions and evidence, not `memories` rows.\n3. Build indexes from a fresh claims-native corpus. Do not preserve `memory_embeddings`, `compartment_chunk_embeddings`, or `git_commit_embeddings` for later reader migration.\n4. Make claim mutations publish projection effects in the same transaction boundary required by U7.\n5. Provide a rebuild command from authoritative tables and prove equality against a fresh build.\n6. Use atomic staging and activation for index construction, but do not serve an old query path or retain an old ranking path.\n\n## Test scenarios\n- Fresh claim revision produces exactly one retrieval document effect and generation update.\n- Rebuild produces identical document identities, content hashes, and postings.\n- Duplicate content shares one physical vector within an embedding space.\n- Corrupt derived indexes rebuild without changing claims.\n- No legacy embedding or memory-projection table is read by production search.\n\n## Verification\n- Migration, projection, outbox rollback, generation, rebuild, and embedding-space tests pass.\n- Search starts on the new projection without a compatibility mode.","acceptance_criteria":"Fresh claims-native projection, outbox, generations, embeddings, rebuild, and rollback tests pass; no legacy embedding or memory projection table is read.","notes":"Direct cutover: retrieval projection is built from claims and current sources. Do not retain old embedding tables or compatibility query paths.","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:24:15Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:33:31Z","labels":["phase-2"],"dependencies":[{"issue_id":"magic-context-3q5.9","depends_on_id":"magic-context-3q5.39","type":"blocks","created_at":"2026-08-20T23:55:52Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.9","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:24:14Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.9","depends_on_id":"magic-context-3q5.8","type":"blocks","created_at":"2026-08-17T02:38:29Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.9","depends_on_id":"magic-context-3q5.7","type":"blocks","created_at":"2026-08-17T02:38:29Z","created_by":"AhravDutta","metadata":"{}"}],"comments":[{"id":"01a02a9c-0381-78c1-bf0d-17efd5732487","issue_id":"magic-context-3q5.9","author":"AhravDutta","text":"U6c applicability contract landed at migration v85 (Synapse v83, U7 backfill v84 unchanged). For retrieval documents: a claim revision without an applicability assertion projects as context-neutral 'unknown'; the seeded baseline assertions also carry state 'unknown'. U8 should store the context-neutral 'unknown' baseline after the U7 completion watermark and consume later claim-domain effects continuously — applicability-only mapping changes now emit one 'upsert' outbox effect (effect_key memory:\u003cid\u003e:applicability) plus a generation bump.","created_at":"2026-08-22T17:54:23Z"}],"dependency_count":3,"dependent_count":10,"comment_count":1} {"_type":"issue","id":"magic-context-3q5.8","title":"U7: Direct claims cutover for memory storage","description":"Phase 2. Direct claims cutover for memory storage.\n\n## Goal\nClaims become the only semantic source of truth. All memory writers and readers use the claims/revision/evidence API from their first supported release.\n\nThere are no legacy callers, deployed users, or existing databases to preserve. Do not implement memories-to-claims backfill, dual-read, a maintained `memories` compatibility projection, legacy IDs, or a later reader migration.\n\n## Scope\n- Make claims/revisions/evidence the storage API for OpenCode, Pi, dreamer, historian, module mirror, dashboard, identity merge, relocation, verification, and session paths.\n- Use operation envelopes for retries and request-key reuse detection.\n- Commit claim revisions, evidence, lifecycle state, outbox effects, and generations atomically.\n- Replace mutable memory IDs with canonical claim/revision/retrieval-document identities at API boundaries.\n- Update the module wire and dashboard/TUI payloads to new claim-native shapes in this cutover.\n- Delete `legacy_memory_claims`, `claim_backfill_*`, `hasMemoryClaimsCompatSchema`, `storage-memory-projection.ts`, and compatibility-only guards and startup runners.\n- Treat old `context.db` files and old binaries as unsupported. Reset or refuse them; do not migrate them.\n\n## Key files\n- packages/plugin/src/features/magic-context/memory/storage-claims.ts\n- packages/plugin/src/features/magic-context/memory/storage-memory.ts\n- packages/plugin/src/features/magic-context/context-authority.ts\n- packages/plugin/src/features/magic-context/memory/relocate-memory.ts\n- packages/pi-plugin/src/tools/ctx-memory.ts\n- packages/cli/src/commands/doctor*.ts\n\n## Test scenarios\n- Every semantic write creates or appends claim state with one atomic commit.\n- Every reader uses claim current-state and evidence.\n- A fresh database has no mutable semantic `memories` source.\n- No production SQL path reads or writes compatibility tables.\n- Crash injection proves no partial claim/evidence/outbox/generation commit.\n- Replayed operations return the stored result without duplicate effects.\n- Unsupported old databases fail closed.\n\n## Verification\n- Full plugin, Pi, CLI, dashboard, and e2e suites pass on fresh databases.\n- Static search finds no legacy memory compatibility branch or reader.","design":"Implement the claims-only greenfield cutover. Rewire every writer, reader, wire boundary, and dashboard path together; remove memories compatibility tables, backfill, dual-read branches, and old IDs. Preserve only atomicity, replay safety, and fail-closed fences.","acceptance_criteria":"All writers and readers use claims-native APIs; no compatibility tables or backfill runner; fresh database has no mutable semantic memories source; crash and replay tests pass; unsupported old DBs fail closed.","notes":"Supersedes dual-read/backfill design. User confirmed greenfield conditions: no legacy callers, deployed users, or existing databases. Claims-only cutover is required.","status":"open","priority":2,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:24:14Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:35:07Z","labels":["phase-2"],"dependencies":[{"issue_id":"magic-context-3q5.8","depends_on_id":"magic-context-3q5.32","type":"blocks","created_at":"2026-08-20T14:44:18Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.8","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:24:13Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-3q5.8","depends_on_id":"magic-context-3q5.7","type":"blocks","created_at":"2026-08-17T02:38:29Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"magic-context-3q5.7","title":"U6: Claims/evidence authoritative schema","description":"Phase 2. Claims and evidence domain.\n\n## Goal\nMake claims, revisions, evidence, conflicts, and verification events the only semantic storage contract for fresh databases.\n\nThis is a greenfield cutover. There are no supported legacy callers, deployed users, or legacy databases to migrate. Do not add a compatibility projection, dual-read API, or memory-row backfill.\n\n## Scope\n- Create the authoritative claims/evidence schema and numeric project identity registry.\n- Enforce immutable revisions, evidence links, current-revision CAS, project ownership, and fail-closed corruption reads.\n- Expose transaction-local writers and current-state readers for every new caller.\n- Reject unsupported old schema versions and old binaries. Refusal is a safety fence, not a compatibility contract.\n\n## Key files\n- packages/plugin/src/features/magic-context/migrations.ts\n- packages/plugin/src/features/magic-context/storage-claims-schema.ts\n- packages/plugin/src/features/magic-context/memory/storage-claims.ts\n- migration and schema-fence tests\n\n## Approach\n1. Build the claims schema for the new database shape.\n2. Make claims/revisions/evidence writes atomic and append-only.\n3. Do not seed claims from `memories`; fresh fixtures create claims directly.\n4. Remove assumptions that a mutable memory row is authoritative.\n\n## Test scenarios\n- Fresh database creates only the new semantic schema.\n- Claim creation and revision append preserve prior bytes.\n- Stale current-revision writes roll back without partial rows.\n- Direct SQL corruption is detected or rejected.\n- Unsupported old schema versions fail closed.\n\n## Verification\n- Claims schema, migration, fence, and corruption tests pass.\n- No production caller reads or writes legacy memory state.","acceptance_criteria":"Fresh claims schema and project registry; immutable revision/evidence/CAS tests pass; unsupported old schemas fail closed; no production caller reads or writes legacy memory state.","notes":"Greenfield decision: claims are the only semantic source. No legacy memory rows, compatibility projection, dual-read, or backfill. Old schema and old binaries fail closed.","status":"in_progress","priority":2,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-17T02:24:13Z","created_by":"AhravDutta","updated_at":"2026-08-22T19:33:31Z","started_at":"2026-08-19T23:50:52Z","labels":["phase-2"],"dependencies":[{"issue_id":"magic-context-3q5.7","depends_on_id":"magic-context-3q5","type":"parent-child","created_at":"2026-08-17T02:24:13Z","created_by":"AhravDutta","metadata":"{}"}],"comments":[{"id":"01a01fa1-8be7-761f-9187-99b4e29ba730","issue_id":"magic-context-3q5.7","author":"AhravDutta","text":"Follow-up task created: magic-context-3q5.32 (U6c) adds bitemporal validity columns (valid_from/until_commit, known_from/until, branch_selector) + source_trust_class enum. Sequenced after this task, before U7 backfill (3q5.8). No scope change here — just don't design the schema in a way that blocks additive nullable columns on claims/claim_revisions/observations.","created_at":"2026-08-20T14:44:36Z"}],"dependency_count":0,"dependent_count":4,"comment_count":1} +{"_type":"issue","id":"magic-context-1or","title":"Retain skipped frames in RawClient::frames_until_corr","description":"The test support helper `RawClient::frames_until_corr` returns the frames it consumed while searching, but most call sites discard them with `let (_, response) = ...`. Those frames are the only copy, so a later sequential call for a correlation whose response was already consumed waits out its whole budget.\n\nThis produced one real flake, fixed in 29e6f4a1 by having `pipelined_shutdown_requests_on_one_connection_both_settle` read the second response out of the returned list. The remaining exposure is latent: a stray bootstrap `Ping` gets skipped during TCP negotiation, which is harmless because `frames_until_corr` already excludes `TY_PING` from matching and no test asserts on the resulting Pong.\n\nThe durable fix is to stop discarding the skipped frames — buffer them on the client so a later `frames_until_corr` searches the buffer before reading the socket — and audit every call site together rather than adding local exceptions. Doing that per-site is what allowed one flake to exist already.","acceptance_criteria":"Consumed non-matching frames are buffered and reachable by a later search; every frames_until_corr call site audited; the pipelined-shutdown test still passes without its local lookaside.","status":"open","priority":3,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-25T09:40:37Z","created_by":"AhravDutta","updated_at":"2026-08-25T09:40:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"magic-context-dmv","title":"Extract shared PID start-time probes for reuse by the Rust e2e reaper","description":"packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts has processStartTimeMs and processExecutable for leak reaping, which overlap with packages/plugin/src/shared/rpc-utils.ts.\n\nThe overlap is not directly reusable as it stands. isPidIdentityPlausible takes an RpcPortFileRecord and its legacy fallback asks whether the command looks like OpenCode, which is the wrong oracle for a fixture process identified by its executable path. The genuinely shared primitives, readLinuxProcessStartTime and readPsProcessStartTime, are module-private in rpc-utils.ts, and the harness reads a procfs exe realpath rather than a command name on purpose.\n\nThe convergence worth having: export the start-time probes (procfs fast path plus ps fallback, one skew-tolerance constant) as a record-shape-agnostic helper, and have both callers layer their own identity oracle on top. That removes the duplicated ps parsing and the harness s hardcoded 5000 ms tolerance without forcing the OpenCode command heuristic into the fixture reaper.","acceptance_criteria":"Both callers use one start-time probe; each keeps its own identity predicate; rpc-utils test hooks still work.","status":"open","priority":3,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-25T08:01:56Z","created_by":"AhravDutta","updated_at":"2026-08-25T08:01:56Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-ly5","title":"Install Compound Engineering Codex plugin","description":"Install compound-engineering from the compound-engineering-plugin marketplace and verify it is enabled.","status":"closed","priority":3,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-23T18:19:08Z","created_by":"AhravDutta","updated_at":"2026-08-23T18:19:16Z","started_at":"2026-08-23T18:19:10Z","closed_at":"2026-08-23T18:19:16Z","close_reason":"Installed compound-engineering 3.23.2 and verified Codex reports it installed and enabled.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-3aq","title":"Add Compound Engineering Codex plugin marketplace","description":"Register EveryInc/compound-engineering-plugin as a Codex plugin marketplace source and verify the registration.","status":"closed","priority":3,"issue_type":"task","assignee":"AhravDutta","owner":"ahravdutta02@gmail.com","created_at":"2026-08-23T18:18:43Z","created_by":"AhravDutta","updated_at":"2026-08-23T18:18:56Z","started_at":"2026-08-23T18:18:45Z","closed_at":"2026-08-23T18:18:56Z","close_reason":"Marketplace registered and verified with codex plugin marketplace list.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"magic-context-ymc.9","title":"T8: Large-payload path: descriptor ring + buffer pool (DEFERRED until measured need)","description":"BUILD-path only, YAGNI-gated: only start if T9 measurement shows real traffic exceeds inline 64B slots enough to matter. Design: ring carries {offset,len} descriptors into a shared buffer pool; ownership handoff and reuse discipline per /low-level-systems:zero-copy-buffer-lifecycle (lease/completion obligations); pool bounding per /systems-design:bounded-design. Alternative to evaluate first: chunk large payloads through the existing ring — dumber, maybe sufficient.\n\nTests: proptest ownership model (no slot reused while leased), leak/exhaustion property, loom on the pool free-list if lock-free.\nAcceptance: only measured traffic justifies this bead; otherwise close as wontfix with the measurement attached.","status":"closed","priority":3,"issue_type":"task","owner":"ahravdutta02@gmail.com","created_at":"2026-08-22T23:14:22Z","created_by":"AhravDutta","updated_at":"2026-08-22T23:19:36Z","closed_at":"2026-08-22T23:19:36Z","close_reason":"Superseded by ADOPT iceoryx2 (ymc.2): dynamic payload support native.","dependencies":[{"issue_id":"magic-context-ymc.9","depends_on_id":"magic-context-ymc.2","type":"blocks","created_at":"2026-08-22T23:14:33Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-ymc.9","depends_on_id":"magic-context-ymc.5","type":"blocks","created_at":"2026-08-22T23:14:34Z","created_by":"AhravDutta","metadata":"{}"},{"issue_id":"magic-context-ymc.9","depends_on_id":"magic-context-ymc","type":"parent-child","created_at":"2026-08-22T23:14:21Z","created_by":"AhravDutta","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 3650fa743..35ae0add7 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -759,20 +759,44 @@ impl Inner { )); } let (tx, rx) = oneshot::channel(); + let mut rx = rx; let (key, publish) = self.admit(route, body, PendingKind::Unary(tx), deadline)?; let mut guard = UnaryAdmissionGuard::new(Arc::clone(self), key); let cancelled = cancellation.unwrap_or_default(); - let result = tokio::select! { + // The stop branches borrow `rx` after the select rather than consuming + // it inside one, because a terminal can already be on the channel by + // then: `dispatch` removes the pending entry before it sends, so a stop + // landing in that window finds nothing to cancel while an authoritative + // answer is in flight. + enum Stopped { + Terminal(Result), + Cancelled, + DeadlineExpired, + } + let stopped = tokio::select! { biased; - result = rx => result.unwrap_or_else(|_| Err(retired_error(classify(&publish)))), - () = cancelled.cancelled() => { - let outcome = self.cancel_key(key, "cancelled").err().map_or_else(|| classify(&publish), |error| error.outcome); - Err(CallError::local(outcome, "cancelled", "request was cancelled")) - } - () = tokio::time::sleep_until(deadline) => { - let outcome = self.cancel_key(key, "deadline_expired").err().map_or_else(|| classify(&publish), |error| error.outcome); - Err(CallError::local(outcome, "deadline_expired", "request deadline expired")) - } + result = &mut rx => Stopped::Terminal( + result.unwrap_or_else(|_| Err(retired_error(classify(&publish)))), + ), + () = cancelled.cancelled() => Stopped::Cancelled, + () = tokio::time::sleep_until(deadline) => Stopped::DeadlineExpired, + }; + let result = match stopped { + Stopped::Terminal(result) => result, + Stopped::Cancelled => self.stop_or_take_terminal( + key, + &mut rx, + &publish, + "cancelled", + "request was cancelled", + ), + Stopped::DeadlineExpired => self.stop_or_take_terminal( + key, + &mut rx, + &publish, + "deadline_expired", + "request deadline expired", + ), }; guard.disarm(); result @@ -963,6 +987,33 @@ impl Inner { Ok((key, publish)) } + /// Stops a pending unary request, preferring a terminal that beat the stop. + /// + /// `dispatch` removes the pending entry before it publishes the terminal, so + /// a cancellation or deadline landing in that window makes `cancel_key` see + /// no entry and report success. Reporting a local error there would discard + /// an authoritative response the host already sent, and send the caller into + /// outcome-unknown recovery for an operation that actually settled. + fn stop_or_take_terminal( + &self, + key: PendingKey, + rx: &mut oneshot::Receiver>, + publish: &AtomicU8, + code: &'static str, + message: &'static str, + ) -> Result { + let stopped = self.cancel_key(key, code); + if stopped.is_ok() { + if let Ok(result) = rx.try_recv() { + return result; + } + } + let outcome = stopped + .err() + .map_or_else(|| classify(publish), |error| error.outcome); + Err(CallError::local(outcome, code, message)) + } + fn cancel_key(&self, key: PendingKey, code: &'static str) -> Result<(), CallError> { let state = lock_unpoisoned(&self.pending).remove(&key); let Some(state) = state else { @@ -1216,7 +1267,14 @@ impl Inner { self.finish_pending( state, Err(CallError::local( - SendOutcome::Terminal, + // Local overflow after the request was sent. + // No `Response`, `Error`, or `StreamEnd` + // was observed and the best-effort `Cancel` + // may not have reached the host, so the run + // may still be committing: `Terminal` would + // claim an authoritative settlement the + // client never saw (§10.1). + SendOutcome::OutcomeUnknown, "stream_saturated", "stream consumer queue saturated", )), @@ -1645,13 +1703,29 @@ fn validate_inbound(header: &EnvelopeHeader) -> Result<(), ()> { return Err(()); } match header.ty { - FrameType::Response | FrameType::Error | FrameType::StreamData | FrameType::StreamEnd => { - // `decode_header` already rejects a mixed zero/nonzero - // channel/epoch pair, so the identity is control (0/0) or routed - // (nonzero/nonzero) by here; only the correlation is left. + // A terminal answers either a control request (0/0) or a routed one. + // `decode_header` already rejects a mixed zero/nonzero channel/epoch + // pair, so the identity is one or the other by here. + FrameType::Response | FrameType::Error => { if header.corr == 0 { return Err(()); } + // §7.1 admits UTF-8 JSON only on channel 0. Without this, a binary + // control response whose bytes happen to parse as JSON would open a + // route and leave the malformed generation live. + if header.channel == 0 && header.flags.is_binary() { + return Err(()); + } + } + // §6.2 requires stream frames to carry an exact pending *routed* + // identity. Grouping them with control-capable terminals accepted `0/0` + // stream frames bearing a pending control correlation, which dispatch + // then reported as `unexpected_stream` while leaving the generation + // usable — a structurally illegal identity has to close it instead. + FrameType::StreamData | FrameType::StreamEnd => { + if header.corr == 0 || header.channel == 0 || header.epoch == 0 { + return Err(()); + } // The direct profile carries stream termination in the header. A // StreamEnd body is structural corruption even though the framing // layer does not classify StreamEnd as pure-header, so the @@ -2406,6 +2480,35 @@ mod tests { )) .is_ok()); + // §6.2 requires a stream frame to name an exact pending ROUTED identity, + // so a control identity is structurally illegal rather than merely + // unmatched — grouping them with terminals accepted it. + assert!(validate_inbound(&header(FrameType::StreamData, 0, 0, 7, 4)).is_err()); + assert!(validate_inbound(&header(FrameType::StreamEnd, 0, 0, 7, 0)).is_err()); + // A terminal may still answer a control request. + assert!(validate_inbound(&header(FrameType::Response, 0, 0, 7, 4)).is_ok()); + assert!(validate_inbound(&header(FrameType::Error, 0, 0, 7, 4)).is_ok()); + + // §7.1 admits UTF-8 JSON only on channel 0, so a binary control + // terminal is malformed even when its bytes happen to parse. + let binary_control = EnvelopeHeader { + len: 4, + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(true, true), + channel: 0, + epoch: 0, + corr: 7, + }; + assert!(validate_inbound(&binary_control).is_err()); + // A routed body stays opaque and may be binary. + let binary_routed = EnvelopeHeader { + channel: 3, + epoch: 9, + ..binary_control + }; + assert!(validate_inbound(&binary_routed).is_ok()); + // Pre-existing rules keep holding. assert!(validate_inbound(&header(FrameType::Response, 3, 9, 0, 4)).is_err()); assert!(validate_inbound(&header(FrameType::Ping, 0, 0, 7, 0)).is_ok()); @@ -2582,6 +2685,48 @@ mod tests { assert!(lock_unpoisoned(&inner.pending).is_empty()); } + #[tokio::test] + async fn a_terminal_that_wins_the_cancellation_race_is_not_discarded() { + // `dispatch` removes the pending entry before it publishes the terminal. + // A stop landing in that window finds nothing to cancel, and reporting a + // local error there would throw away an answer the host already gave and + // send the caller into outcome-unknown recovery for a settled operation. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(60), + ) + .expect("admitted"); + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + + // Reproduce the window exactly: the entry is gone (as `dispatch` leaves + // it) and the terminal is already on the channel. + let state = lock_unpoisoned(&inner.pending) + .remove(&key) + .expect("entry exists"); + match state.kind { + PendingKind::Unary(tx) => tx + .send(Ok(Response { + body: b"authoritative".to_vec(), + binary: false, + })) + .expect("terminal published"), + PendingKind::Stream { .. } => unreachable!("admitted a unary request"), + } + + let mut rx = rx; + let response = inner + .stop_or_take_terminal(key, &mut rx, &publish, "cancelled", "request was cancelled") + .expect("the observed terminal wins over the local stop"); + assert_eq!(response.body, b"authoritative"); + } + #[test] fn absent_admission_facts_are_omitted_rather_than_sent_as_null() { // The host reads any present member as `Some(..)`, so a null would make @@ -2872,6 +3017,12 @@ mod tests { .expect("terminal sender") .expect_err("saturated stream fails"); assert_eq!(error.code(), "stream_saturated"); + // Saturation is a local overflow after the request went out. No + // Response, Error, or StreamEnd was observed, and the best-effort Cancel + // may not have reached the host, so the run may still be committing: + // Terminal would claim an authoritative settlement the client never saw + // and mark a possibly-live operation replay-safe (§10.1). + assert_eq!(error.outcome(), SendOutcome::OutcomeUnknown); let cancel = control_rx.recv().await.expect("stream Cancel"); assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); diff --git a/packages/plugin/src/index.ts b/packages/plugin/src/index.ts index 43eadc671..589336de2 100644 --- a/packages/plugin/src/index.ts +++ b/packages/plugin/src/index.ts @@ -210,6 +210,15 @@ const server: Plugin = async (ctx) => { } const liveSessionState = createLiveSessionState(); + // The transport dials a connection file written by an externally + // launched mc-host daemon; this build ships no host binary, so a + // missing daemon otherwise surfaces only as per-request connect + // errors. One startup line names the dependency. + if (pluginConfig.transform_mode === "rust") { + log( + '[magic-context] transform_mode "rust" requires an externally launched mc-host daemon; requests will retry until its connection file appears', + ); + } const rustModeModuleClient: RustModeModuleClient | undefined = pluginConfig.transform_mode === "rust" ? new McHostModuleTransport() : undefined; diff --git a/packages/plugin/src/shared/mc-host-client/client.test.ts b/packages/plugin/src/shared/mc-host-client/client.test.ts index 6802357f4..a6a8ebbaf 100644 --- a/packages/plugin/src/shared/mc-host-client/client.test.ts +++ b/packages/plugin/src/shared/mc-host-client/client.test.ts @@ -1594,7 +1594,7 @@ describe("transport negotiation", () => { peer, diagnostics: (event) => events.push(event), }); - expectCallError(error, "terminal", "negotiation_failed"); + expectCallError(error, "terminal", "host_negotiation_rejected"); await waitUntil(() => events.some((e) => e.type === "retired" && e.reason === "negotiation_failed"), ); @@ -1836,7 +1836,7 @@ describe("transport negotiation", () => { }); // The host's raw error body is peer-controlled; the caller sees a // bounded negotiation failure, never the wire message (R14). - expectCallError(error, "terminal", "negotiation_failed"); + expectCallError(error, "terminal", "host_negotiation_rejected"); await waitUntil(() => events.some((e) => e.type === "retired" && e.reason === "negotiation_failed"), ); diff --git a/packages/plugin/src/shared/mc-host-client/client.ts b/packages/plugin/src/shared/mc-host-client/client.ts index e6e0f737e..d3d3ed9cb 100644 --- a/packages/plugin/src/shared/mc-host-client/client.ts +++ b/packages/plugin/src/shared/mc-host-client/client.ts @@ -785,11 +785,14 @@ export class McHostClient { // Every Error terminal fails closed with a bounded error: // the raw body is peer-controlled and its message must not // enter caller-visible error graphs (R14). There is no - // legacy `unsupported_operation` continuation. + // `unsupported_operation` continuation. The distinct code + // makes version skew self-describing: a host that does not + // implement `transport.negotiate` answers this request with + // an Error terminal. throw new McHostCallError( "terminal", - "transport negotiation failed: host error terminal", - "negotiation_failed", + "transport negotiation failed: host returned an error terminal (host may predate transport negotiation; restart or upgrade the mc-host daemon)", + "host_negotiation_rejected", ); } throw error; From d7257bab76f432fcd18f4fee7b10eb69070fbf13 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 15:06:57 +0000 Subject: [PATCH 19/37] fix(mc-host,mc-module): stop claiming settlement and stop replaying after cancel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from the codex round on 6307a6e8, each with a test that fails when the fix is reverted. A unary request that receives StreamData reports `OutcomeUnknown` rather than `Terminal`. The host violated the profile, but nothing terminal was observed: StreamData is nonterminal and the scoped `Cancel` this path queues is best-effort, so the run may still be executing or committing. `Terminal` claimed an authoritative settlement the client never saw and suppressed the recovery §10.1 requires. Abandoning the correlation rather than draining to a real terminal is deliberate — a unary correlation has no legal stream frames, so continuing to read them would treat a protocol violation as a supported shape. Cancelling a stream releases its queued items. Each buffered `ChargedItem` holds a `ByteCharge` against the owner-wide retained-response budget, and `next` short-circuits on `finished`, so nothing would ever drain them. A caller that kept a cancelled `ResponseStream` alive pinned those bytes for the value's whole lifetime, and enough of them make a later response fail admission and retire an otherwise healthy generation. `cancel` now closes the receiver before draining it, so the reader cannot refill what it drains. The historian no longer replays a request the caller cancelled. Replay exists for an ambiguous transport loss, where the request may have been dispatched and resending on a fresh generation is how the run gets established. A caller cancellation produces the same `OutcomeUnknown` classification for the same reason, but needs the opposite handling: replaying dials a new connection and re-sends `session.send` after the caller asked to stop, which starts a billable run if the original never went out, and during handler shutdown makes a cancelled task perform connection setup instead of draining. The gate reads the cancellation token rather than the error code, because the token is what the caller actually signalled. The deadline and transport-loss replays are unaffected, and a test asserts they still fire. --- crates/mc-host/src/auth.rs | 43 +-- crates/mc-host/src/client.rs | 303 ++++++++++++++---- crates/mc-host/src/lib.rs | 11 +- crates/mc-host/tests/client.rs | 160 +++++++++ crates/mc-host/tests/support/raw_client.rs | 7 +- crates/mc-module/src/dispatch.rs | 82 +++-- crates/mc-module/src/historian_producer.rs | 103 +++++- crates/mc-module/tests/prepared_output.rs | 6 +- .../src/shared/mc-host-client/client.test.ts | 4 +- 9 files changed, 589 insertions(+), 130 deletions(-) diff --git a/crates/mc-host/src/auth.rs b/crates/mc-host/src/auth.rs index 1dbdb0683..02fd20af5 100644 --- a/crates/mc-host/src/auth.rs +++ b/crates/mc-host/src/auth.rs @@ -19,7 +19,6 @@ pub const MAX_AUTH_MESSAGE_LEN: u32 = 4096; pub const SERVER_PROOF_DOMAIN: &str = "subc-server-v1"; pub const CLIENT_AUTH_DOMAIN: &str = "subc-client-v1"; pub const DEFAULT_CLIENT_ROLE: &str = "client"; -pub const WATCHDOG_CLIENT_ROLE: &str = "watchdog"; type HmacSha256 = Hmac; @@ -47,19 +46,15 @@ pub struct ClientAuth { /// WHAT THIS PROVES: the peer possesses the connection key, and (client side) /// that the daemon does too. Nothing more. /// -/// WHAT `role` IS NOT: it is a string the CLIENT SENT, echoed back unverified. -/// The handshake never checks it against anything, so it carries no authority -- -/// any peer holding the key can claim any role. It exists so a caller can tell -/// self-issued traffic (the daemon's own watchdog probe) from real clients when -/// REPORTING, and it must never decide admission, capacity, or privilege. -/// -/// A type called `Authenticated` invites reading every field as attested. Only -/// the possession of the key is. Module identity, which IS attested, travels a -/// different path entirely (spawn nonces validated at route.open). +/// Deliberately empty: everything else in the handshake transcript is +/// client-asserted and unverified. `ClientHello.role` in particular is parsed +/// and then discarded — any peer holding the key can claim any role, so it +/// must never decide admission, capacity, or privilege. A type called +/// `Authenticated` invites reading its fields as attested, and only key +/// possession is. Module identity, which IS attested, travels a different +/// path entirely (spawn nonces validated at route.open). #[derive(Debug, Clone, PartialEq, Eq)] -pub struct Authenticated { - pub role: String, -} +pub struct Authenticated; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum AuthStage { @@ -237,7 +232,7 @@ where return Err(AuthError::InvalidClientAuth); } - Ok(Authenticated { role: hello.role }) + Ok(Authenticated) } pub async fn authenticate_client( @@ -245,23 +240,11 @@ pub async fn authenticate_client( conn: &ConnectionInfo, deadline: Duration, ) -> Result<(), AuthError> -where - S: AsyncRead + AsyncWrite + Unpin, -{ - authenticate_client_with_role(stream, conn, deadline, DEFAULT_CLIENT_ROLE).await -} - -pub async fn authenticate_client_with_role( - stream: &mut S, - conn: &ConnectionInfo, - deadline: Duration, - role: &str, -) -> Result<(), AuthError> where S: AsyncRead + AsyncWrite + Unpin, { let deadline = Deadline::starting_now(deadline)?; - let result = authenticate_client_inner(stream, conn, deadline, role).await; + let result = authenticate_client_inner(stream, conn, deadline).await; if result.is_err() { let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; } @@ -272,7 +255,6 @@ async fn authenticate_client_inner( stream: &mut S, conn: &ConnectionInfo, deadline: Deadline, - role: &str, ) -> Result<(), AuthError> where S: AsyncRead + AsyncWrite + Unpin, @@ -285,7 +267,7 @@ where AuthStage::ClientHello, &ClientHello { client_nonce, - role: role.to_owned(), + role: DEFAULT_CLIENT_ROLE.to_owned(), }, deadline, ) @@ -832,11 +814,10 @@ mod tests { &server_proof.daemon_id, ); write_auth_json(&mut client, &ClientAuth { client_auth }).await; - let authenticated = server_task + server_task .await .expect("join") .expect("handshake completes"); - assert_eq!(authenticated.role, TEST_ROLE); server_proof } diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 35ae0add7..0e26f0266 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -19,7 +19,7 @@ use std::{ use serde_json::Value; use tokio::{ io::{AsyncRead, AsyncReadExt, AsyncWriteExt}, - net::{tcp::OwnedReadHalf, tcp::OwnedWriteHalf, TcpStream}, + net::{tcp::OwnedWriteHalf, TcpStream}, sync::{mpsc, oneshot}, task::JoinHandle, time::{timeout_at, Instant}, @@ -31,8 +31,8 @@ use crate::{ connection_file::{read_for_client, ConnectionInfo, DAEMON_ID_LEN}, handler::{RouteHandle, RouteIdentity, RouteTarget, TargetKind}, transport_negotiation::{ - decode_negotiate_response, NegotiateResponse, TransportOffer, NEGOTIATION_VERSION, - TRANSPORT_TCP, + decode_negotiate_response, encode_negotiate_request, NegotiateRequest, NegotiateResponse, + TransportOffer, NEGOTIATION_VERSION, TRANSPORT_TCP, }, wire::{ decode_header, encode_owned_frame, pure_header_flags, AdmissionClass, EnvelopeHeader, @@ -70,6 +70,10 @@ const NEGOTIATION_CORRELATION: u64 = 1; const FIRST_APPLICATION_CORRELATION: u64 = 2; const MAX_ERROR_CODE_BYTES: usize = 128; const MAX_ERROR_MESSAGE_BYTES: usize = 512; +/// Read-side socket buffer for the wire read path. Matches the framing +/// layer's `tcp_frame_channel` read buffer so the per-frame header-then-body +/// reads coalesce into large socket reads instead of one syscall per field. +const READ_BUFFER_BYTES: usize = 64 * 1024; /// Exact send-outcome classifications used by recovery policy. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -291,6 +295,10 @@ impl Client { .await .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? .map_err(|_| ClientError::new("dial_failed", "daemon dial failed"))?; + // Interactive request/response traffic; Nagle would add up to one RTT + // of coalescing delay per small frame. Best-effort, as in the server's + // accept path. + let _ = stream.set_nodelay(true); let remaining = deadline.saturating_duration_since(Instant::now()); if remaining.is_zero() { return Err(ClientError::new( @@ -333,7 +341,11 @@ impl Client { }); let reader_inner = Arc::clone(&inner); let reader = tokio::spawn(async move { - reader_loop(reader_inner, read).await; + reader_loop( + reader_inner, + tokio::io::BufReader::with_capacity(READ_BUFFER_BYTES, read), + ) + .await; }); *inner.writer.lock().await = Some(writer); *inner.reader.lock().await = Some(reader); @@ -655,6 +667,14 @@ impl ResponseStream { return Ok(()); } self.finished = true; + // Items already buffered in the channel each hold a `ByteCharge` against + // the owner-wide retained-response budget, and `next` short-circuits on + // `finished`, so nothing would ever drain them. Left in place they stay + // charged for as long as the caller keeps this value, and a later + // response that cannot charge retires an otherwise healthy generation. + // `close` first so the reader task cannot refill what this drains. + self.items.close(); + while self.items.try_recv().is_ok() {} if let Some(inner) = self.inner.upgrade() { inner.cancel_key(self.key, "cancelled")?; } @@ -1134,12 +1154,7 @@ impl Inner { .map_err(|_| ClientError::new("connection_retired", "connection retired")) } - fn dispatch( - self: &Arc, - header: EnvelopeHeader, - body: Vec, - charge: Option, - ) { + fn dispatch(self: &Arc, header: EnvelopeHeader, body: Vec, charge: ByteCharge) { match header.ty { FrameType::Ping => { // V35: the Pong echoes the Ping's flags exactly. @@ -1224,7 +1239,19 @@ impl Inner { self.finish_pending( state, Err(CallError::local( - SendOutcome::Terminal, + // The host violated the profile, but nothing + // terminal was observed: StreamData is + // nonterminal and the Cancel below is + // best-effort, so the run may still be + // executing or committing. `Terminal` would + // claim an authoritative settlement and + // suppress the recovery this needs (§10.1). + // Abandoning here rather than draining to a + // real terminal is deliberate — a unary + // correlation has no legal stream frames, so + // continuing to read them would treat a + // protocol violation as a supported shape. + SendOutcome::OutcomeUnknown, "unexpected_stream", "unary request received stream data", )), @@ -1241,21 +1268,6 @@ impl Inner { ); } PendingKind::Stream { items, .. } => { - // An empty item is never charged, so an absent charge - // means exhaustion only when there were bytes to - // charge for. Reading it as exhaustion either way - // retires the generation over a legal zero-length - // StreamData: `validate_inbound` requires an empty body - // of `StreamEnd` alone. - let charge = match charge { - Some(charge) => Some(charge), - None if header.len == 0 => None, - None => { - drop(pending); - self.retire("response_memory_exhausted"); - return; - } - }; let item = ChargedItem { body, binary: header.flags.is_binary(), @@ -1492,6 +1504,18 @@ struct ByteCharge { bytes: usize, } +impl ByteCharge { + /// A zero-byte charge for bodiless frames. Holding one keeps every + /// inbound frame's accounting uniform: an absent charge never reaches + /// `dispatch`, so "no charge" cannot be misread as an exhausted budget. + const fn none() -> Self { + Self { + owner: Weak::new(), + bytes: 0, + } + } +} + impl Drop for ByteCharge { fn drop(&mut self) { if let Some(owner) = self.owner.upgrade() { @@ -1504,8 +1528,9 @@ impl Drop for ByteCharge { struct ChargedItem { body: Vec, binary: bool, - /// Absent for a zero-length item, which is never charged. - _charge: Option, + /// A zero-length item holds a no-op charge; there were no bytes to + /// account for. + _charge: ByteCharge, } impl ChargedItem { @@ -1575,7 +1600,7 @@ async fn writer_loop( let _ = write.shutdown().await; } -async fn reader_loop(inner: Arc, mut read: OwnedReadHalf) { +async fn reader_loop(inner: Arc, mut read: R) { loop { let frame = match read_active_frame(&mut read, &inner).await { Ok(Some(frame)) => frame, @@ -1598,7 +1623,10 @@ async fn reader_loop(inner: Arc, mut read: OwnedReadHalf) { struct InboundFrame { header: EnvelopeHeader, body: Vec, - charge: Option, + /// Retained-budget accounting for `body`. A bodiless frame carries + /// `ByteCharge::none()`; a refused charge never constructs a frame at all + /// (`read_active_frame` drains and fails the connection instead). + charge: ByteCharge, } async fn read_active_frame( @@ -1618,19 +1646,18 @@ async fn read_active_frame( read_exact_until(read, &mut header_bytes[1..], deadline, &inner.cancel).await?; let header = decode_header(&header_bytes).map_err(|_| ())?; validate_inbound(&header)?; - let charge = if header.len == 0 { - None - } else { - inner.retained_budget.charge(header.len as usize) - }; - let mut body = Vec::new(); - if let Some(_charge) = charge.as_ref() { - body.resize(header.len as usize, 0); - read_exact_until(read, &mut body, deadline, &inner.cancel).await?; - } else if header.len > 0 { + if header.len == 0 { + return Ok(Some(InboundFrame { + header, + body: Vec::new(), + charge: ByteCharge::none(), + })); + } + let Some(charge) = inner.retained_budget.charge(header.len as usize) else { drain_until(read, header.len as usize, deadline, &inner.cancel).await?; return Err(()); - } + }; + let body = read_body_until(read, header.len as usize, deadline, &inner.cancel).await?; Ok(Some(InboundFrame { header, body, @@ -1659,6 +1686,32 @@ async fn read_exact_until( Ok(()) } +/// Reads exactly `len` body bytes under one frame deadline. +/// +/// `read_buf` appends into the vector's spare capacity without +/// zero-initializing it, and `take` caps the read at the frame boundary even +/// when the allocated capacity exceeds `len`. +async fn read_body_until( + read: &mut R, + len: usize, + deadline: Instant, + cancel: &CancellationToken, +) -> Result, ()> { + let mut body = Vec::with_capacity(len); + let mut limited = read.take(len as u64); + while body.len() < len { + let count = tokio::select! { + biased; + () = cancel.cancelled() => return Err(()), + result = timeout_at(deadline, limited.read_buf(&mut body)) => result.map_err(|_| ())?.map_err(|_| ())?, + }; + if count == 0 { + return Err(()); + } + } + Ok(body) +} + async fn drain_until( read: &mut R, mut remaining: usize, @@ -1806,12 +1859,18 @@ fn encode_data_frame( } async fn negotiate_tcp(stream: &mut TcpStream, deadline: Instant) -> Result<(), ClientError> { - let body = serde_json::to_vec(&serde_json::json!({ - "op": "transport.negotiate", - "negotiation_version": NEGOTIATION_VERSION, - "offers": [{"transport": TRANSPORT_TCP, "capability_version": 1}] - })) - .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + // One offers value feeds both the encoded request and response + // validation, so the selection is checked against exactly what was sent. + let request = NegotiateRequest { + negotiation_version: NEGOTIATION_VERSION, + offers: vec![TransportOffer { + transport: TRANSPORT_TCP.to_owned(), + capability_version: 1, + parameters: None, + }], + }; + let body = encode_negotiate_request(&request) + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; let bytes = encode_owned_frame( FrameType::Request, Flags::new(false, Priority::Interactive, false), @@ -1837,12 +1896,7 @@ async fn negotiate_tcp(stream: &mut TcpStream, deadline: Instant) -> Result<(), "transport negotiation failed closed", )); } - let offers = [TransportOffer { - transport: TRANSPORT_TCP.to_owned(), - capability_version: 1, - parameters: None, - }]; - let selection = decode_negotiate_response(&frame.body, &offers).map_err(|_| { + let selection = decode_negotiate_response(&frame.body, &request.offers).map_err(|_| { ClientError::new("negotiation_failed", "transport negotiation failed closed") })?; if !matches!(selection, NegotiateResponse::Tcp { reason: None }) { @@ -1879,7 +1933,7 @@ async fn read_setup_frame( Ok(InboundFrame { header, body, - charge: None, + charge: ByteCharge::none(), }) } @@ -2371,7 +2425,7 @@ mod tests { corr: key.corr, }, Vec::new(), - None, + ByteCharge::none(), ), } @@ -2544,7 +2598,7 @@ mod tests { "{priority:?} is a valid Ping priority, not a reason to retire" ); - inner.dispatch(ping, Vec::new(), None); + inner.dispatch(ping, Vec::new(), ByteCharge::none()); let pong = control_rx.recv().await.expect("Pong queued"); assert_eq!(pong.bytes[5], FrameType::Pong as u8); @@ -2577,8 +2631,9 @@ mod tests { #[tokio::test] async fn a_zero_length_stream_item_is_delivered_without_retiring() { // Only `StreamEnd` must be empty, so a zero-length `StreamData` is - // legal. It carries no charge because there were no bytes to charge - // for, which must not read as an exhausted budget. + // legal. It carries a no-op charge because there were no bytes to + // account for, and it must reach the stream rather than retire the + // generation. let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); let (items_tx, mut items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); let (terminal_tx, _terminal_rx) = oneshot::channel(); @@ -2607,12 +2662,12 @@ mod tests { corr: key.corr, }, Vec::new(), - None, + ByteCharge::none(), ); assert!( !inner.retired.load(Ordering::Acquire), - "an uncharged empty item is not an exhausted budget" + "an empty item carries a no-op charge, not an exhausted budget" ); let item = items_rx.try_recv().expect("the empty item is delivered"); assert!(item.body.is_empty()); @@ -2957,7 +3012,7 @@ mod tests { corr: key.corr, }, Vec::new(), - None, + ByteCharge::none(), ); assert!(matches!( rx.try_recv(), @@ -3009,7 +3064,7 @@ mod tests { corr: stream_key.corr, }, vec![1], - Some(charge), + charge, ); } let error = terminal_rx @@ -3037,13 +3092,135 @@ mod tests { corr: unary_key.corr, }, Vec::new(), - None, + ByteCharge::none(), ); assert!(unary_rx.try_recv().expect("unary settled").is_ok()); assert!(lock_unpoisoned(&inner.pending).is_empty()); inner.retire("test_done"); } + #[tokio::test] + async fn unary_stream_data_is_unknown_not_terminal() { + // StreamData is nonterminal and the Cancel this path queues is + // best-effort, so the host may still be running the request. Reporting + // `Terminal` would claim a settlement the client never observed and + // suppress the recovery §10.1 requires. + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (kind, mut rx) = unary_sender(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(60), + ) + .expect("admitted"); + drop(data_rx.recv().await); + + inner.dispatch( + EnvelopeHeader { + len: 1, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![1], + inner.retained_budget.charge(1).expect("retained byte"), + ); + + let error = rx + .try_recv() + .expect("unary settled") + .expect_err("stream data on a unary is a violation"); + assert_eq!(error.code(), "unexpected_stream"); + assert_eq!(error.outcome(), SendOutcome::OutcomeUnknown); + // The scoped cleanup is unchanged. + let cancel = control_rx.recv().await.expect("scoped Cancel"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + inner.retire("test_done"); + } + + #[tokio::test] + async fn cancelling_a_stream_releases_its_queued_item_charges() { + // Buffered items each hold a charge against the owner-wide retained + // budget, and `next` short-circuits on `finished`, so a cancelled stream + // the caller keeps alive would pin those bytes indefinitely — enough of + // them makes a later response fail admission and retire a healthy + // generation. + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (items_tx, items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + // The host only streams items back after receiving the request, so the + // writer has necessarily claimed it. That also makes the cancel + // OutcomeUnknown, which is the classification that emits the Cancel. + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + + let mut stream = ResponseStream { + inner: Arc::downgrade(&inner), + key, + correlation: key.corr, + items: items_rx, + terminal: Some(terminal_rx), + finished: false, + }; + + // Queue items without draining them, exactly as a slow consumer leaves + // them. + const ITEMS: usize = 4; + for _ in 0..ITEMS { + inner.dispatch( + EnvelopeHeader { + len: 8, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![7; 8], + inner.retained_budget.charge(8).expect("retained bytes"), + ); + } + assert_eq!(inner.retained_budget.used(), ITEMS * 8); + + stream.cancel().expect("cancel succeeds"); + assert_eq!( + inner.retained_budget.used(), + 0, + "cancel released every queued item's charge while the stream is still alive" + ); + // The stream value is deliberately still held here: releasing on drop is + // what this test proves is not sufficient. + assert!(stream + .next() + .await + .expect("cancelled stream ends") + .is_none()); + let cancel = control_rx.recv().await.expect("scoped Cancel"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + inner.retire("test_done"); + drop(stream); + } + #[tokio::test(start_paused = true)] async fn idle_header_is_unbounded_then_partial_frame_has_one_deadline() { let (inner, _data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); diff --git a/crates/mc-host/src/lib.rs b/crates/mc-host/src/lib.rs index e270418a0..f1d3e4824 100644 --- a/crates/mc-host/src/lib.rs +++ b/crates/mc-host/src/lib.rs @@ -33,10 +33,9 @@ mod tcp_frame_channel; mod wire; pub use auth::{ - authenticate_client, authenticate_client_with_role, authenticate_server, compute_proof, - AuthError, AuthStage, Authenticated, ClientAuth, ClientHello, ServerProof, CLIENT_AUTH_DOMAIN, - DEFAULT_CLIENT_ROLE, MAX_AUTH_MESSAGE_LEN, NONCE_LEN, PROOF_LEN, SERVER_PROOF_DOMAIN, - WATCHDOG_CLIENT_ROLE, + authenticate_client, authenticate_server, compute_proof, AuthError, AuthStage, Authenticated, + ClientAuth, ClientHello, ServerProof, CLIENT_AUTH_DOMAIN, DEFAULT_CLIENT_ROLE, + MAX_AUTH_MESSAGE_LEN, NONCE_LEN, PROOF_LEN, SERVER_PROOF_DOMAIN, }; pub use client::{ CallError, Client, ClientError, RequestOptions, Response, ResponseStream, SendOutcome, @@ -67,5 +66,9 @@ pub use runtime::{run, HostError}; /// The version-2 body cap. Published so a consumer preparing an output can /// gate on the same value frame admission enforces, rather than restating it. pub use wire::MAX_FRAME_BODY_LEN; +/// Launch-identity environment variable names. Published so module-side code +/// reads the same names the host injects at spawn, rather than restating the +/// protocol vocabulary as string literals. +pub use wire::{SUBC_LAUNCH_NONCE_ENV, SUBC_MODULE_ID_ENV}; pub use tokio_util::sync::CancellationToken; diff --git a/crates/mc-host/tests/client.rs b/crates/mc-host/tests/client.rs index cda3a54dc..6f8c61cf6 100644 --- a/crates/mc-host/tests/client.rs +++ b/crates/mc-host/tests/client.rs @@ -443,6 +443,166 @@ async fn managed_client_negotiation_failures_retire_socket_without_application_f } } +#[cfg(unix)] +#[tokio::test] +async fn zero_length_stream_item_is_delivered_and_does_not_retire_the_connection() { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + // Only `StreamEnd` must be empty, so a zero-length `StreamData` item is + // legal. It must reach the stream consumer rather than being misread as an + // exhausted retained-byte budget, which would retire the generation and + // fail every unrelated in-flight request. + const STREAM_FLAGS: u8 = raw_client::FLAGS_INTERACTIVE; + + async fn read_frame(socket: &mut tokio::net::TcpStream) -> raw_client::RawFrame { + let mut header = [0u8; raw_client::HEADER_LEN]; + socket.read_exact(&mut header).await.expect("frame header"); + let mut frame = raw_client::decode_header(&header); + if frame.len > 0 { + let mut body = vec![0; frame.len as usize]; + socket.read_exact(&mut body).await.expect("frame body"); + frame.body = body; + } + frame + } + + let root = tempfile::tempdir().unwrap(); + std::fs::set_permissions(root.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + let publication = root.path().join("connection.json"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let key = vec![0x5a; 32]; + let daemon_id = [0x3c; 16]; + let info = ConnectionInfo { + schema: 1, + wire_version: 2, + endpoints: vec![Endpoint { + host: "127.0.0.1".to_owned(), + port: listener.local_addr().unwrap().port(), + }], + key: key.clone(), + daemon_id, + pid: std::process::id(), + daemon_ver: "fake-peer".to_owned(), + }; + std::fs::write(&publication, serde_json::to_vec(&info).unwrap()).unwrap(); + std::fs::set_permissions(&publication, std::fs::Permissions::from_mode(0o600)).unwrap(); + + let peer = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + authenticate_server( + &mut socket, + &key, + &daemon_id, + "fake-peer", + Duration::from_secs(1), + ) + .await + .expect("managed client authenticates"); + + // Negotiation (correlation 1). + let negotiate = read_frame(&mut socket).await; + assert_eq!(negotiate.corr, 1); + let selection = br#"{"op":"transport.negotiate","negotiation_version":1,"selected":{"transport":"tcp","capability_version":1}}"#; + let mut reply = raw_client::header( + selection.len() as u32, + TY_RESPONSE, + FLAGS_RESPONSE_TEXT_LAST, + 0, + 0, + 1, + ); + reply.extend_from_slice(selection); + socket.write_all(&reply).await.expect("selection reply"); + + // route.open (correlation 2) grants channel 7 epoch 1. + let open = read_frame(&mut socket).await; + assert_eq!(open.corr, 2); + let opened = br#"{"op":"route.open","route_channel":7,"route_epoch":1}"#; + let mut reply = raw_client::header( + opened.len() as u32, + TY_RESPONSE, + FLAGS_RESPONSE_TEXT_LAST, + 0, + 0, + 2, + ); + reply.extend_from_slice(opened); + socket.write_all(&reply).await.expect("route.open reply"); + + // Stream request (correlation 3): first item empty, then a payload + // item, then StreamEnd. + let stream_request = read_frame(&mut socket).await; + assert_eq!(stream_request.corr, 3); + assert_eq!(stream_request.channel, 7); + let empty_item = raw_client::header(0, raw_client::TY_STREAM_DATA, STREAM_FLAGS, 7, 1, 3); + socket.write_all(&empty_item).await.expect("empty item"); + let payload = b"payload"; + let mut item = raw_client::header( + payload.len() as u32, + raw_client::TY_STREAM_DATA, + STREAM_FLAGS, + 7, + 1, + 3, + ); + item.extend_from_slice(payload); + socket.write_all(&item).await.expect("payload item"); + let end = raw_client::header(0, raw_client::TY_STREAM_END, STREAM_FLAGS, 7, 1, 3); + socket.write_all(&end).await.expect("stream end"); + + // An unrelated unary (correlation 4) proves the generation survived. + let unary = read_frame(&mut socket).await; + assert_eq!(unary.corr, 4); + let mut reply = raw_client::header( + unary.body.len() as u32, + TY_RESPONSE, + FLAGS_RESPONSE_TEXT_LAST, + 7, + 1, + 4, + ); + reply.extend_from_slice(&unary.body); + socket.write_all(&reply).await.expect("unary echo"); + }); + + let client = Client::connect(&publication) + .await + .expect("managed client connects to fake peer"); + let route = client + .open_route(target(), identity("empty-item")) + .await + .expect("route opens"); + let mut stream = client + .request_stream(route, b"stream".to_vec(), RequestOptions::default()) + .await + .expect("stream starts"); + + let first = stream + .next() + .await + .expect("empty item does not retire the generation") + .expect("first item is delivered"); + assert!(first.body.is_empty(), "the empty item arrives intact"); + let second = stream + .next() + .await + .expect("stream continues past the empty item") + .expect("second item is delivered"); + assert_eq!(second.body, b"payload"); + assert!( + stream.next().await.expect("stream ends cleanly").is_none(), + "StreamEnd terminates the stream" + ); + + let body = b"after-empty-item".to_vec(); + let response = client + .request(route, body.clone(), RequestOptions::default()) + .await + .expect("an unrelated request still succeeds on the same generation"); + assert_eq!(response.body, body); + peer.await.unwrap(); +} + #[tokio::test] async fn close_rejects_new_sends() { let host = TestHost::start().await; diff --git a/crates/mc-host/tests/support/raw_client.rs b/crates/mc-host/tests/support/raw_client.rs index 3c7b030d4..020bc43c2 100644 --- a/crates/mc-host/tests/support/raw_client.rs +++ b/crates/mc-host/tests/support/raw_client.rs @@ -394,10 +394,13 @@ impl RawClient { } async fn negotiate_tcp(&mut self) -> Result<(), String> { + // The body stays hand-rolled (this oracle never calls mc-host's + // encoders), but the version is the shared public protocol constant + // rather than a second copy of the literal. let corr = self .control(&serde_json::json!({ "op": "transport.negotiate", - "negotiation_version": 1, + "negotiation_version": mc_host::transport_negotiation::NEGOTIATION_VERSION, "offers": [{"transport": "tcp", "capability_version": 1}] })) .await @@ -415,7 +418,7 @@ impl RawClient { } let expected = serde_json::json!({ "op": "transport.negotiate", - "negotiation_version": 1, + "negotiation_version": mc_host::transport_negotiation::NEGOTIATION_VERSION, "selected": {"transport": "tcp", "capability_version": 1} }); if frame.json() != expected { diff --git a/crates/mc-module/src/dispatch.rs b/crates/mc-module/src/dispatch.rs index 32a8cfe40..96b5fdcc0 100644 --- a/crates/mc-module/src/dispatch.rs +++ b/crates/mc-module/src/dispatch.rs @@ -88,7 +88,8 @@ impl fmt::Debug for PreparedSegment { } impl PreparedOutput { - /// Retains a JSON value for counting and serialization after reservation. + /// Retains a JSON value; measurement performs the single serialization + /// pass and the write phase reuses the encoded bytes. pub fn json(value: Value) -> Self { Self { source: PreparedSource::Json(Arc::new(value)), @@ -124,13 +125,28 @@ impl PreparedOutput { } /// Measures this immutable source exactly before output reservation. + /// + /// JSON sources are serialized exactly once, here: the encoded bytes are + /// cap-checked as they are produced and carried in the returned + /// measurement, so `write_to` copies them instead of running the + /// serializer a second time. pub fn measure(&self) -> Result, PreparedOutputError> { - let len = match &self.source { - PreparedSource::Json(value) => measure_json(value)?, - PreparedSource::Exact(bytes) => checked_body_len([bytes.len()])?, - PreparedSource::Transform(segments) => measure_transform(segments)?, + let (source, len) = match &self.source { + PreparedSource::Json(value) => { + let encoded = encode_json(value)?; + let len = encoded.len(); + (MeasuredSource::Encoded(encoded), len) + } + PreparedSource::Exact(bytes) => { + let len = checked_body_len([bytes.len()])?; + (MeasuredSource::Exact(bytes), len) + } + PreparedSource::Transform(segments) => { + let len = measure_transform(segments)?; + (MeasuredSource::Transform(segments), len) + } }; - Ok(MeasuredOutput { output: self, len }) + Ok(MeasuredOutput { source, len }) } } @@ -202,10 +218,19 @@ impl fmt::Debug for PreparedOutcome { /// Exact measurement tied to the immutable source that produced it. pub struct MeasuredOutput<'a> { - output: &'a PreparedOutput, + source: MeasuredSource<'a>, len: usize, } +/// Source view captured at measurement time. JSON arrives already encoded +/// (measurement is the single serialization pass); the other variants borrow +/// the prepared source and stream it during the write. +enum MeasuredSource<'a> { + Encoded(Vec), + Exact(&'a [u8]), + Transform(&'a TransformSegments), +} + impl MeasuredOutput<'_> { pub fn len(&self) -> usize { self.len @@ -218,13 +243,10 @@ impl MeasuredOutput<'_> { /// Writes into a caller-reserved destination and verifies exact length. pub fn write_to(&self, destination: &mut W) -> Result { let mut destination = BoundedWriter::new(destination, self.len); - match &self.output.source { - PreparedSource::Json(value) => { - serde_json::to_writer(&mut destination, value) - .map_err(PreparedOutputError::Serialize)?; - } - PreparedSource::Exact(bytes) => destination.write_all(bytes)?, - PreparedSource::Transform(segments) => write_transform(segments, &mut destination)?, + match &self.source { + MeasuredSource::Encoded(bytes) => destination.write_all(bytes)?, + MeasuredSource::Exact(bytes) => destination.write_all(bytes)?, + MeasuredSource::Transform(segments) => write_transform(segments, &mut destination)?, } let written = destination.written(); if written != self.len { @@ -304,16 +326,20 @@ fn checked_body_len( Ok(total) } -fn measure_json(value: &Value) -> Result { - let mut counter = CountingWriter::default(); - let result = serde_json::to_writer(&mut counter, value).map_err(PreparedOutputError::Serialize); - finish_count(counter, result) +/// Serializes a JSON value once, enforcing the wire cap as bytes are +/// produced: each append is cap-checked before the buffer grows, so an +/// over-cap body fails without buffering past the cap. +fn encode_json(value: &Value) -> Result, PreparedOutputError> { + let mut writer = CountingWriter::collecting(); + let result = serde_json::to_writer(&mut writer, value).map_err(PreparedOutputError::Serialize); + let writer = finish_count(writer, result)?; + Ok(writer.collected.unwrap_or_default()) } fn finish_count( counter: CountingWriter, result: Result<(), PreparedOutputError>, -) -> Result { +) -> Result { match counter.failure { Some(CountFailure::Overflow) => Err(PreparedOutputError::LengthOverflow), Some(CountFailure::TooLarge(len)) => Err(PreparedOutputError::BodyTooLarge { @@ -322,7 +348,7 @@ fn finish_count( }), None => { result?; - Ok(counter.len) + Ok(counter) } } } @@ -332,7 +358,7 @@ fn measure_transform(segments: &TransformSegments) -> Result( @@ -381,13 +407,24 @@ enum CountFailure { TooLarge(usize), } +/// Counts (and optionally collects) bytes against the wire cap. The cap check +/// precedes any buffering, so a collecting writer never allocates past the +/// cap before reporting `TooLarge`. #[derive(Default)] struct CountingWriter { len: usize, failure: Option, + collected: Option>, } impl CountingWriter { + fn collecting() -> Self { + Self { + collected: Some(Vec::new()), + ..Self::default() + } + } + fn add_len(&mut self, len: usize) -> Result<(), PreparedOutputError> { let Some(next) = self.len.checked_add(len) else { self.failure = Some(CountFailure::Overflow); @@ -409,6 +446,9 @@ impl Write for CountingWriter { fn write(&mut self, bytes: &[u8]) -> io::Result { self.add_len(bytes.len()) .map_err(|error| io::Error::new(io::ErrorKind::FileTooLarge, error.to_string()))?; + if let Some(buffer) = &mut self.collected { + buffer.extend_from_slice(bytes); + } Ok(bytes.len()) } diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index bca4fdf75..2a704b786 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -17,7 +17,7 @@ use std::{ use async_trait::async_trait; use mc_host::{ CallError, Client, ClientError, RequestOptions, ResponseStream, RouteHandle, RouteIdentity, - RouteTarget, SendOutcome, StreamItem, TargetKind, + RouteTarget, SendOutcome, StreamItem, TargetKind, SUBC_LAUNCH_NONCE_ENV, SUBC_MODULE_ID_ENV, }; use serde_json::{json, Value}; use tokio_util::sync::CancellationToken; @@ -799,7 +799,7 @@ impl HistorianProducer { let frozen_daemon = self.connection.daemon_id(); let response = match self.send_frozen_once(&frozen).await { Ok(response) => response, - Err(error) if is_outcome_unknown(&error) => { + Err(error) if is_outcome_unknown(&error) && !self.stop_requested() => { self.replay_frozen_once(frozen_daemon, frozen_identity, &frozen) .await? } @@ -980,8 +980,8 @@ impl HistorianProducer { project_root: semantic.project_root, harness: semantic.harness, session: semantic.session, - consumer_module_id: nonempty_env("SUBC_MODULE_ID"), - consumer_launch_nonce: nonempty_env("SUBC_LAUNCH_NONCE"), + consumer_module_id: nonempty_env(SUBC_MODULE_ID_ENV), + consumer_launch_nonce: nonempty_env(SUBC_LAUNCH_NONCE_ENV), consumer_capabilities: Vec::new(), admission_facts: None, }, @@ -1086,6 +1086,26 @@ impl HistorianProducer { cancellation: self.config.cancellation.clone(), } } + + /// Whether the caller has asked this producer to stop. + /// + /// Replay exists for an ambiguous *transport* loss: the request may have + /// been dispatched, so resending on a fresh generation is how the run gets + /// established. A caller cancellation produces the same `OutcomeUnknown` + /// classification for the same reason — the bytes may already be gone — but + /// the correct response is the opposite. Replaying there dials a new + /// connection and re-sends `session.send` after the caller explicitly said + /// stop, which starts a billable run if the original never went out, and + /// during handler shutdown makes a cancelled task perform connection setup + /// instead of draining. The token is the authority on that, not the error + /// code: it is what the caller actually signalled. The deadline and + /// transport-loss replays are unaffected because neither cancels it. + fn stop_requested(&self) -> bool { + self.config + .cancellation + .as_ref() + .is_some_and(CancellationToken::is_cancelled) + } } async fn drain_subscribe( @@ -1507,6 +1527,16 @@ mod tests { )) } + /// The shape the managed client reports when the caller's token fires after + /// the writer may already have claimed the request. + fn cancelled_unknown() -> HistorianProducerError { + HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::OutcomeUnknown, + "cancelled", + "request was cancelled", + )) + } + fn terminal(code: &str) -> HistorianProducerError { HistorianProducerError::Call(HistorianCallFailure::untagged( HistorianSendOutcome::Terminal, @@ -1684,6 +1714,71 @@ mod tests { } } + #[tokio::test] + async fn caller_cancellation_prevents_replay_but_transport_loss_still_replays() { + // A caller cancellation and an ambiguous transport loss are both + // OutcomeUnknown for the same reason — the bytes may already be gone — + // but they need opposite handling. Replaying a cancellation dials a new + // connection and re-sends `session.send` after the caller said stop, + // starting a run if the original never went out, and during handler + // shutdown makes a cancelled task do connection setup instead of + // draining. The token is the authority, so the transport replay below + // must survive the gate. + async fn start_with_token( + token: Option, + ) -> ( + Arc, + Arc>, + Result, + ) { + let first = connection(9, [Err(cancelled_unknown())]); + let second = connection(9, [Ok(br#"{"run_id":"replayed"}"#.to_vec())]); + let second_state = Arc::clone(&second.state); + let connector = Arc::new(FakeConnector { + initial: first, + reconnects: Mutex::new(VecDeque::from(vec![(second, None)])), + reconnect_calls: AtomicU64::new(0), + }); + let config = HistorianProducerConfig { + request_timeout: Duration::from_secs(1), + await_timeout: Duration::from_secs(1), + cancellation: token, + ..HistorianProducerConfig::new("/unused", "/project", "opencode") + }; + let mut producer = HistorianProducer::connect_with(config, connector.clone()) + .await + .unwrap(); + let result = producer + .start("session", "", "prompt", "provider/model") + .await; + (connector, second_state, result) + } + + // Cancelled: the failure surfaces to the caller and nothing is resent. + let cancelled = CancellationToken::new(); + cancelled.cancel(); + let (connector, replay_state, result) = start_with_token(Some(cancelled)).await; + let error = result.expect_err("a cancelled start does not succeed by replaying"); + assert!(is_outcome_unknown(&error), "{error:?}"); + assert_eq!( + connector.reconnect_calls.load(Ordering::SeqCst), + 0, + "a cancelled caller must not trigger connection setup" + ); + assert!( + replay_state.lock().unwrap().requests.is_empty(), + "a cancelled caller must not have its request resent" + ); + + // Live token: the intentional transport-loss replay is unaffected. + for token in [None, Some(CancellationToken::new())] { + let (connector, replay_state, result) = start_with_token(token).await; + assert_eq!(result.expect("transport loss replays").run_id, "replayed"); + assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 1); + assert_eq!(replay_state.lock().unwrap().requests.len(), 1); + } + } + #[tokio::test] async fn close_releases_subscription_and_command_routes() { let first = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); diff --git a/crates/mc-module/tests/prepared_output.rs b/crates/mc-module/tests/prepared_output.rs index 62d6212d8..429f94052 100644 --- a/crates/mc-module/tests/prepared_output.rs +++ b/crates/mc-module/tests/prepared_output.rs @@ -231,8 +231,8 @@ impl Write for FailAfter { } #[test] -fn serializer_failure_retains_no_partial_terminal() { - let output = PreparedOutput::json(json!({"value": "serializer must fail after bytes"})); +fn destination_failure_retains_no_partial_terminal() { + let output = PreparedOutput::json(json!({"value": "destination must fail after bytes"})); let measured = output.measure().unwrap(); let mut destination = FailAfter { remaining: 5, @@ -245,7 +245,7 @@ fn serializer_failure_retains_no_partial_terminal() { terminal = Some(destination.accepted); } - assert!(matches!(result, Err(PreparedOutputError::Serialize(_)))); + assert!(matches!(result, Err(PreparedOutputError::Write(_)))); assert!(destination.accepted > 0); assert_eq!(terminal, None); } diff --git a/packages/plugin/src/shared/mc-host-client/client.test.ts b/packages/plugin/src/shared/mc-host-client/client.test.ts index a6a8ebbaf..b539a62a5 100644 --- a/packages/plugin/src/shared/mc-host-client/client.test.ts +++ b/packages/plugin/src/shared/mc-host-client/client.test.ts @@ -1853,7 +1853,7 @@ describe("transport negotiation", () => { negotiate: (frame, conn) => void sendErrorBody(conn, frame.corr, "server_busy"), }); const { error } = await connectRejected({ peer }); - expectCallError(error, "terminal", "negotiation_failed"); + expectCallError(error, "terminal", "host_negotiation_rejected"); const conn = peer.connections[0] as FakePeerConnection; await conn.closed; expect(peer.connections.length).toBe(1); @@ -1876,7 +1876,7 @@ describe("transport negotiation", () => { }), }); const { error } = await connectRejected({ peer }); - expectCallError(error, "terminal", "negotiation_failed"); + expectCallError(error, "terminal", "host_negotiation_rejected"); const conn = peer.connections[0] as FakePeerConnection; await conn.closed; expect(peer.connections.length).toBe(1); From c72b797c7ec26b43690e4a702de7bf716c82f95f Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 15:58:49 +0000 Subject: [PATCH 20/37] fix(mc-module): charge a JSON response body before materializing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `measure` runs before the host's resident-byte reservation, and for a JSON source it was collecting the whole encoded body into an uncharged `Vec` and then copying that into the separately reserved `OutputBuffer`. Peak memory was twice the body, and the first copy sat entirely outside the budget the reservation exists to enforce: concurrent maximum-sized responses could each hold a full 64 MiB encoding and exhaust the process while every one of them respected its charge. The counting pass is restored. `measure` now sizes the reservation without retaining anything, and `write_to` serializes straight into the reserved, length-bounded destination. `MeasuredSource::Json` borrows the value instead of owning bytes, so the property is structural rather than a convention: this path cannot hold an encoded body again without changing the type. The serializer runs twice as a result, once to size and once to fill. That tradeoff is the point — the alternative buys one pass with an uncharged allocation — and `write_to`'s existing length check turns any disagreement between the passes into an error rather than a short body. Destination I/O failures now arrive wrapped in serde's error type, so they are unwrapped back to `PreparedOutputError::Write` and callers keep the write-versus-serialize distinction they had when this path copied a buffer. --- crates/mc-module/src/dispatch.rs | 58 ++++++++++++++++++-------------- crates/mc-module/src/lib.rs | 56 ++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 26 deletions(-) diff --git a/crates/mc-module/src/dispatch.rs b/crates/mc-module/src/dispatch.rs index 96b5fdcc0..59837d3c0 100644 --- a/crates/mc-module/src/dispatch.rs +++ b/crates/mc-module/src/dispatch.rs @@ -126,16 +126,20 @@ impl PreparedOutput { /// Measures this immutable source exactly before output reservation. /// - /// JSON sources are serialized exactly once, here: the encoded bytes are - /// cap-checked as they are produced and carried in the returned - /// measurement, so `write_to` copies them instead of running the - /// serializer a second time. + /// JSON sources are counted, not collected. `measure` runs BEFORE the host's + /// resident-byte reservation, so retaining the encoded body here would hold + /// up to `MAX_WIRE_BODY_BYTES` outside the very budget the reservation + /// exists to enforce: concurrent large responses would each own a full + /// uncharged copy and could exhaust process memory while every one of them + /// respected its charge. The serializer therefore runs twice — once to size + /// the reservation, once to fill it — and `write_to`'s length check turns + /// any disagreement between the two passes into an error rather than a + /// short body. pub fn measure(&self) -> Result, PreparedOutputError> { let (source, len) = match &self.source { PreparedSource::Json(value) => { - let encoded = encode_json(value)?; - let len = encoded.len(); - (MeasuredSource::Encoded(encoded), len) + let len = measure_json(value)?; + (MeasuredSource::Json(value), len) } PreparedSource::Exact(bytes) => { let len = checked_body_len([bytes.len()])?; @@ -226,7 +230,7 @@ pub struct MeasuredOutput<'a> { /// (measurement is the single serialization pass); the other variants borrow /// the prepared source and stream it during the write. enum MeasuredSource<'a> { - Encoded(Vec), + Json(&'a Value), Exact(&'a [u8]), Transform(&'a TransformSegments), } @@ -244,7 +248,19 @@ impl MeasuredOutput<'_> { pub fn write_to(&self, destination: &mut W) -> Result { let mut destination = BoundedWriter::new(destination, self.len); match &self.source { - MeasuredSource::Encoded(bytes) => destination.write_all(bytes)?, + MeasuredSource::Json(value) => { + // Serializing straight into the destination means a destination + // failure reaches us wrapped in serde's error type. Unwrap it so + // callers keep the write-versus-serialize distinction they had + // when this path copied a pre-encoded buffer. + serde_json::to_writer(&mut destination, value).map_err(|error| { + if error.is_io() { + PreparedOutputError::Write(error.into()) + } else { + PreparedOutputError::Serialize(error) + } + })?; + } MeasuredSource::Exact(bytes) => destination.write_all(bytes)?, MeasuredSource::Transform(segments) => write_transform(segments, &mut destination)?, } @@ -326,14 +342,15 @@ fn checked_body_len( Ok(total) } -/// Serializes a JSON value once, enforcing the wire cap as bytes are -/// produced: each append is cap-checked before the buffer grows, so an -/// over-cap body fails without buffering past the cap. -fn encode_json(value: &Value) -> Result, PreparedOutputError> { - let mut writer = CountingWriter::collecting(); +/// Measures a JSON value's exact encoded length without retaining the bytes. +/// +/// The cap is enforced as bytes are produced, so an over-cap body fails during +/// counting rather than after a full encode. +fn measure_json(value: &Value) -> Result { + let mut writer = CountingWriter::default(); let result = serde_json::to_writer(&mut writer, value).map_err(PreparedOutputError::Serialize); let writer = finish_count(writer, result)?; - Ok(writer.collected.unwrap_or_default()) + Ok(writer.len) } fn finish_count( @@ -414,17 +431,9 @@ enum CountFailure { struct CountingWriter { len: usize, failure: Option, - collected: Option>, } impl CountingWriter { - fn collecting() -> Self { - Self { - collected: Some(Vec::new()), - ..Self::default() - } - } - fn add_len(&mut self, len: usize) -> Result<(), PreparedOutputError> { let Some(next) = self.len.checked_add(len) else { self.failure = Some(CountFailure::Overflow); @@ -446,9 +455,6 @@ impl Write for CountingWriter { fn write(&mut self, bytes: &[u8]) -> io::Result { self.add_len(bytes.len()) .map_err(|error| io::Error::new(io::ErrorKind::FileTooLarge, error.to_string()))?; - if let Some(buffer) = &mut self.collected { - buffer.extend_from_slice(bytes); - } Ok(bytes.len()) } diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index f32c4d5de..1ebc50dd7 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -16576,6 +16576,62 @@ mod tests { assert!(events[1..].iter().all(|event| *event == "write")); } + #[tokio::test] + async fn json_settlement_streams_into_the_reservation_without_a_prior_encoding() { + // `measure` runs before the resident-byte reservation, so it must not + // retain the encoded body: a collected copy would sit outside the very + // budget the reservation enforces, and concurrent maximum-sized + // responses could exhaust memory while each one respected its charge. + // A pre-encoded blob would reach the destination as one `write_all`; + // serde streaming into the reserved writer is the observable proof that + // no such blob exists. + let value = serde_json::json!({ + "a": ["one", "two", "three"], + "b": {"c": 1, "d": 2, "e": [true, false, null]}, + "f": "payload", + }); + let events = Arc::new(Mutex::new(Vec::new())); + let reserve_events = Arc::clone(&events); + let reserved = Arc::new(AtomicUsize::new(0)); + let reserved_len = Arc::clone(&reserved); + let settlement = settle_prepared_with( + PreparedOutcome::Response(PreparedOutput::json(value.clone())), + || false, + move |len| { + reserve_events.lock().unwrap().push("reserve"); + reserved_len.store(len, Ordering::SeqCst); + let events = Arc::clone(&reserve_events); + async move { + Ok::<_, ()>(SettlementWriter { + events, + bytes: Vec::with_capacity(len), + }) + } + }, + ) + .await; + let PreparedSettlement::Response(writer) = settlement else { + panic!("successful settlement must return a response"); + }; + let expected = serde_json::to_vec(&value).expect("value serializes"); + assert_eq!(writer.bytes, expected); + // The reservation is sized from the counting pass and must match the + // body the second pass produces exactly, or the two passes disagreed. + assert_eq!(reserved.load(Ordering::SeqCst), expected.len()); + let events = events.lock().unwrap(); + assert_eq!(events.first(), Some(&"reserve")); + let writes = events[1..].len(); + assert!( + events[1..].iter().all(|event| *event == "write"), + "every post-reservation event is a write" + ); + assert!( + writes > 1, + "serde streamed into the reservation ({writes} writes); a single \ + write_all would mean a fully encoded body existed before it" + ); + } + #[tokio::test] async fn production_settlement_error_and_stream_skip_reservation() { let reservations = Arc::new(AtomicUsize::new(0)); From 53318a8736010642d36fdaa2a8449df2c75aec17 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 16:29:15 +0000 Subject: [PATCH 21/37] fix(mc-host,mc-module): stop discarding terminals and stranding host state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six client-side paths gave up something the protocol already guaranteed. `stop_or_take_terminal` polled the reply channel once. `dispatch` removes the pending entry before it sends, so a cancellation or deadline landing in that window found no entry to cancel and an empty channel, and reported `OutcomeUnknown` for an operation the host had already answered. Whoever removed the entry owns the sender and settles it without yielding, so the stop now waits for that owner instead of racing it. A malformed but successful `route.open` response left the connection live. The client never obtained a handle, so it could send no route `Goodbye`, and each repeated open stranded another host-side route and channel permit. Retiring the generation is what obliges the host to settle the unnameable binding. Queue retention and the reader's in-flight frame body shared one byte budget, so a consumer holding a few queued megabytes could make an unrelated maximum-sized terminal unreadable — which the reader could only report by retiring the whole generation. Retention is now charged when an item is queued, against its own budget, and exhaustion cancels the saturating stream rather than the connection. The reader keeps a separate reservation covering the framing maximum. The negotiation setup path checked only the 64 MiB framing maximum, so a malformed peer could impose a 64 MiB allocation on every connect attempt. Negotiation is channel-zero control traffic and is now held to the 65,536-byte cap that `validate_inbound` applies everywhere else. Both header readers waited for all 21 bytes before inspecting the version, even though byte 4 of the frozen prefix already proves an incompatible generation. Both now split the read as the host's reader does, so a peer that sends only the prefix cannot hold a connection to its frame deadline. The historian's replay gate ran once, before a cleanup, a fresh dial, and a route open that none of them observe the cancellation token through. A token firing just after the gate let a stopping handler spend every one of those budgets and bind a host-side route. The replay now rechecks between stages and returns the original ambiguous error, because the frozen request may already have been delivered. --- crates/mc-host/src/client.rs | 555 +++++++++++++++++++-- crates/mc-module/src/historian_producer.rs | 88 +++- 2 files changed, 592 insertions(+), 51 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 0e26f0266..f096d42a4 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -36,8 +36,8 @@ use crate::{ }, wire::{ decode_header, encode_owned_frame, pure_header_flags, AdmissionClass, EnvelopeHeader, - Flags, FrameId, FrameType, Priority, HEADER_LEN, MAX_BODY_LEN, MAX_CONTROL_BODY_LEN, - PROTOCOL_VERSION, + Flags, FrameId, FrameType, Priority, FROZEN_PREFIX_LEN, HEADER_LEN, MAX_BODY_LEN, + MAX_CONTROL_BODY_LEN, PROTOCOL_VERSION, }, }; @@ -63,7 +63,22 @@ pub const CLIENT_DATA_QUEUE_FRAMES: usize = 256; pub const CLIENT_CONTROL_QUEUE_FRAMES: usize = 32; /// Shared queued-byte cap charged by both ordinary and reserved control frames. pub const CLIENT_QUEUED_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; +/// Reservation for the body of the frame the reader is currently decoding. +/// +/// The wire contract obliges an admitted connection to accept any otherwise +/// valid frame, so this reservation must cover the framing maximum and must be +/// separate from `CLIENT_RETAINED_RESPONSE_BYTES`: charging both from one pool +/// let a stream holding a few queued megabytes make an unrelated maximum-sized +/// terminal unreadable, which the reader could only report by retiring the whole +/// generation. One reader task decodes one frame at a time, so this is a +/// per-connection ceiling, not a per-frame multiplier. +pub const CLIENT_INBOUND_FRAME_BYTES: usize = MAX_BODY_LEN as usize; /// Owner-wide bytes retained in pending stream queues. +/// +/// Charged when an item is queued for a consumer, not when it is read, so +/// exhaustion cancels the saturating stream instead of the connection. Sized to +/// admit one maximum-sized item plus headroom; the worst-case resident total for +/// one connection is this plus `CLIENT_INBOUND_FRAME_BYTES`. pub const CLIENT_RETAINED_RESPONSE_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; const NEGOTIATION_CORRELATION: u64 = 1; @@ -328,6 +343,7 @@ impl Client { streams: Mutex::new(0), routes: Mutex::new(HashSet::new()), queue_budget: Arc::new(ByteCounter::new(CLIENT_QUEUED_BYTES)), + read_budget: Arc::new(ByteCounter::new(CLIENT_INBOUND_FRAME_BYTES)), retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), data_tx, control_tx, @@ -401,7 +417,21 @@ impl Client { .await; match response { Ok(response) => { - let handle = parse_route_open(&response.body)?; + // A successful `route.open` whose body has no usable tag, + // channel, or epoch means the host bound a route the client + // cannot name. It can send no route `Goodbye` for it, so + // leaving the connection live lets each repeated open strand + // another host-side route and channel permit until the whole + // generation eventually closes. Retiring here is what obliges + // the host to settle every route on this generation (§11.2), + // including the unnameable one. + let handle = match parse_route_open(&response.body) { + Ok(handle) => handle, + Err(error) => { + self.inner.retire("invalid_route_response"); + return Err(error); + } + }; // Publish under the same lock `close` drains, and recheck // closure while holding it. A close that lands between the // response arriving and this insert would otherwise leave a @@ -717,6 +747,20 @@ const WRITING: u8 = 1; const WRITTEN: u8 = 2; const CANCELLED: u8 = 3; +/// What removing a pending entry for a stop actually found. +/// +/// The distinction is the caller's only evidence about who owns the request's +/// reply channel: `Cancelled` means this stop settled it, while `AlreadyTaken` +/// means a concurrent owner holds the sender and a terminal may still be in +/// flight. +#[derive(Debug)] +enum PendingRemoval { + /// This stop removed the entry and settled the caller. + Cancelled, + /// The entry was gone, so another owner will settle the caller. + AlreadyTaken, +} + struct PendingState { publish: Arc, kind: PendingKind, @@ -748,6 +792,10 @@ struct Inner { streams: Mutex, routes: Mutex>, queue_budget: Arc, + /// Reserved for the body of the one frame the reader is decoding. Separate + /// from `retained_budget` so queue retention can never deny an otherwise + /// valid inbound frame; see `CLIENT_INBOUND_FRAME_BYTES`. + read_budget: Arc, retained_budget: Arc, data_tx: mpsc::Sender, control_tx: mpsc::Sender, @@ -803,20 +851,26 @@ impl Inner { }; let result = match stopped { Stopped::Terminal(result) => result, - Stopped::Cancelled => self.stop_or_take_terminal( - key, - &mut rx, - &publish, - "cancelled", - "request was cancelled", - ), - Stopped::DeadlineExpired => self.stop_or_take_terminal( - key, - &mut rx, - &publish, - "deadline_expired", - "request deadline expired", - ), + Stopped::Cancelled => { + self.stop_or_take_terminal( + key, + &mut rx, + &publish, + "cancelled", + "request was cancelled", + ) + .await + } + Stopped::DeadlineExpired => { + self.stop_or_take_terminal( + key, + &mut rx, + &publish, + "deadline_expired", + "request deadline expired", + ) + .await + } }; guard.disarm(); result @@ -1011,10 +1065,10 @@ impl Inner { /// /// `dispatch` removes the pending entry before it publishes the terminal, so /// a cancellation or deadline landing in that window makes `cancel_key` see - /// no entry and report success. Reporting a local error there would discard - /// an authoritative response the host already sent, and send the caller into - /// outcome-unknown recovery for an operation that actually settled. - fn stop_or_take_terminal( + /// no entry. Reporting a local error there would discard an authoritative + /// response the host already sent, and send the caller into outcome-unknown + /// recovery for an operation that actually settled. + async fn stop_or_take_terminal( &self, key: PendingKey, rx: &mut oneshot::Receiver>, @@ -1022,22 +1076,38 @@ impl Inner { code: &'static str, message: &'static str, ) -> Result { - let stopped = self.cancel_key(key, code); - if stopped.is_ok() { - if let Ok(result) = rx.try_recv() { - return result; + let stopped = match self.cancel_key(key, code) { + // Another owner already took the entry, so it holds this caller's + // sender and is committed to either sending a terminal or dropping + // it. Both resolve this await, and no remover yields between taking + // an entry and settling it, so the wait is bounded by that owner's + // next few instructions. A single `try_recv` here loses the race + // whenever the terminal has not reached the channel yet, which is + // exactly the `remove`-before-`send` window that makes the entry + // absent in the first place. + Ok(PendingRemoval::AlreadyTaken) => { + return rx + .await + .unwrap_or_else(|_| Err(retired_error(classify(publish)))); } - } - let outcome = stopped - .err() - .map_or_else(|| classify(publish), |error| error.outcome); + // This call removed the entry, so `cancel_key` has already settled + // the channel and `try_recv` observes its own result. + Ok(PendingRemoval::Cancelled) => { + if let Ok(result) = rx.try_recv() { + return result; + } + None + } + Err(error) => Some(error.outcome), + }; + let outcome = stopped.unwrap_or_else(|| classify(publish)); Err(CallError::local(outcome, code, message)) } - fn cancel_key(&self, key: PendingKey, code: &'static str) -> Result<(), CallError> { + fn cancel_key(&self, key: PendingKey, code: &'static str) -> Result { let state = lock_unpoisoned(&self.pending).remove(&key); let Some(state) = state else { - return Ok(()); + return Ok(PendingRemoval::AlreadyTaken); }; let outcome = if state .publish @@ -1062,7 +1132,7 @@ impl Inner { // the caller's OutcomeUnknown classification is already correct and // is what actually protects it from replaying. if key.channel == 0 { - return Ok(()); + return Ok(PendingRemoval::Cancelled); } // The Cancel is best-effort cleanup, and the request's bytes may // already be on the wire. Report the failed enqueue, but keep the @@ -1083,7 +1153,7 @@ impl Inner { return Err(CallError::new(outcome, error.code, error.message)); } } - Ok(()) + Ok(PendingRemoval::Cancelled) } /// Queues one pure-header control frame. @@ -1268,12 +1338,22 @@ impl Inner { ); } PendingKind::Stream { items, .. } => { - let item = ChargedItem { + // Retention is charged here, not at read time, so the + // bytes a consumer holds are accounted against the queue + // budget and cannot deny the reader an unrelated frame. + // Exhaustion is the byte-wise form of queue saturation + // and is reported the same way: cancel this stream, keep + // the generation. + let retained = self.retained_budget.charge(body.len()); + let item = retained.map(|retained| ChargedItem { body, binary: header.flags.is_binary(), - _charge: charge, - }; - if items.try_send(item).is_err() { + _charge: retained, + }); + // The read reservation is free the moment the bytes are + // either retained under the queue budget or discarded. + drop(charge); + if item.is_none_or(|item| items.try_send(item).is_err()) { let state = pending.remove(&key).expect("entry exists"); drop(pending); self.finish_pending( @@ -1643,7 +1723,29 @@ async fn read_active_frame( return Ok(None); } let deadline = Instant::now() + CLIENT_FRAME_TIMEOUT; - read_exact_until(read, &mut header_bytes[1..], deadline, &inner.cancel).await?; + // Frozen-prefix discipline (§5): `len` and `ver` live in bytes 0..5, and an + // incompatible version is provable from them alone. Waiting for all 21 bytes + // first lets a peer that sends only the prefix hold this connection — and one + // of the owner's active slots — until the frame deadline, even though byte 4 + // already proved the generation unusable. The host's reader splits the read + // for the same reason; see `tcp_frame_channel::read_frame`. + read_exact_until( + read, + &mut header_bytes[1..FROZEN_PREFIX_LEN], + deadline, + &inner.cancel, + ) + .await?; + if header_bytes[4] != PROTOCOL_VERSION { + return Err(()); + } + read_exact_until( + read, + &mut header_bytes[FROZEN_PREFIX_LEN..], + deadline, + &inner.cancel, + ) + .await?; let header = decode_header(&header_bytes).map_err(|_| ())?; validate_inbound(&header)?; if header.len == 0 { @@ -1653,7 +1755,12 @@ async fn read_active_frame( charge: ByteCharge::none(), })); } - let Some(charge) = inner.retained_budget.charge(header.len as usize) else { + // The reservation covers the framing maximum and belongs to the reader + // alone, so a valid frame is never refused because a consumer is holding + // queued bytes. A refusal here therefore means the header declared more than + // the framing maximum, which `validate_inbound` has already rejected — it + // survives only as the structural guard for that invariant. + let Some(charge) = inner.read_budget.charge(header.len as usize) else { drain_until(read, header.len as usize, deadline, &inner.cancel).await?; return Err(()); }; @@ -1913,23 +2020,33 @@ async fn read_setup_frame( deadline: Instant, ) -> Result { let mut header_bytes = [0u8; HEADER_LEN]; - timeout_at(deadline, stream.read_exact(&mut header_bytes)) - .await - .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? - .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + // Frozen-prefix discipline: bytes 0..5 carry `len` and `ver`, and an + // incompatible version is provable from them alone. Reading all 21 bytes + // first lets a peer that sends 5 bytes and stops hold the handshake open to + // its whole deadline before the version is even inspected. + read_setup_exact(stream, &mut header_bytes[..FROZEN_PREFIX_LEN], deadline).await?; + if header_bytes[4] != PROTOCOL_VERSION { + return Err(ClientError::new( + "negotiation_failed", + "transport negotiation failed", + )); + } + read_setup_exact(stream, &mut header_bytes[FROZEN_PREFIX_LEN..], deadline).await?; let header = decode_header(&header_bytes) .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; - if header.len > MAX_BODY_LEN { + // Negotiation is channel-zero control traffic, so §7.1's 65,536-byte cap + // applies — not the 64 MiB framing maximum. This path never reaches + // `validate_inbound`, so without the tighter check a malformed peer can make + // every connect attempt allocate roughly 64 MiB before the response is + // rejected. + if header.channel != 0 || header.len > MAX_CONTROL_BODY_LEN { return Err(ClientError::new( "negotiation_failed", "transport negotiation failed", )); } let mut body = vec![0u8; header.len as usize]; - timeout_at(deadline, stream.read_exact(&mut body)) - .await - .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? - .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + read_setup_exact(stream, &mut body, deadline).await?; Ok(InboundFrame { header, body, @@ -1937,6 +2054,19 @@ async fn read_setup_frame( }) } +/// Reads exactly `buf.len()` setup bytes under the shared handshake deadline. +async fn read_setup_exact( + stream: &mut TcpStream, + buf: &mut [u8], + deadline: Instant, +) -> Result<(), ClientError> { + timeout_at(deadline, stream.read_exact(buf)) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .map_err(|_| ClientError::new("negotiation_failed", "transport negotiation failed"))?; + Ok(()) +} + fn route_open_body(target: &RouteTarget, identity: &RouteIdentity) -> Result, CallError> { let project_root = identity.project_root.to_str().ok_or_else(|| { CallError::local( @@ -2108,6 +2238,7 @@ mod tests { streams: Mutex::new(0), routes: Mutex::new(HashSet::from([route(1), route(2)])), queue_budget: Arc::new(ByteCounter::new(queued_bytes)), + read_budget: Arc::new(ByteCounter::new(CLIENT_INBOUND_FRAME_BYTES)), retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), data_tx, control_tx, @@ -2413,7 +2544,9 @@ mod tests { drop(data_rx.recv().await); match terminal { - None => inner.cancel_key(key, "cancelled").expect("stream settled"), + None => { + inner.cancel_key(key, "cancelled").expect("stream settled"); + } Some(ty) => inner.dispatch( EnvelopeHeader { len: 0, @@ -2778,10 +2911,98 @@ mod tests { let mut rx = rx; let response = inner .stop_or_take_terminal(key, &mut rx, &publish, "cancelled", "request was cancelled") + .await .expect("the observed terminal wins over the local stop"); assert_eq!(response.body, b"authoritative"); } + #[tokio::test] + async fn a_terminal_still_in_flight_wins_over_the_local_stop() { + // The remove-before-send window, reproduced with the terminal *not yet* + // on the channel: `dispatch` has taken the entry and is about to send. + // A single `try_recv` loses that race every time and would report + // `OutcomeUnknown` for an operation the host already answered, so the + // stop must wait for the owner that holds the sender. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(60), + ) + .expect("admitted"); + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + + let state = lock_unpoisoned(&inner.pending) + .remove(&key) + .expect("entry exists"); + let PendingKind::Unary(tx) = state.kind else { + unreachable!("admitted a unary request") + }; + + let mut rx = rx; + let stop = async { + inner + .stop_or_take_terminal(key, &mut rx, &publish, "cancelled", "request was cancelled") + .await + }; + let publish_terminal = async { + // Let the stop observe the absent entry and start waiting before the + // owner publishes, which is the ordering that makes the race real. + tokio::task::yield_now().await; + tx.send(Ok(Response { + body: b"authoritative".to_vec(), + binary: false, + })) + .expect("terminal published"); + }; + let (result, ()) = tokio::join!(stop, publish_terminal); + let response = result.expect("the in-flight terminal wins over the local stop"); + assert_eq!(response.body, b"authoritative"); + } + + #[tokio::test] + async fn a_dropped_sender_after_an_absent_entry_reports_the_send_outcome() { + // The other way an owner can resolve the channel: it took the entry and + // dropped the sender (generation retirement settles in bulk). The stop + // must still terminate, and must classify from the publish state rather + // than hanging or inventing a terminal. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (kind, rx) = unary_sender(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + kind, + Instant::now() + Duration::from_secs(60), + ) + .expect("admitted"); + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + drop( + lock_unpoisoned(&inner.pending) + .remove(&key) + .expect("entry exists"), + ); + + let mut rx = rx; + let error = inner + .stop_or_take_terminal(key, &mut rx, &publish, "cancelled", "request was cancelled") + .await + .expect_err("a dropped sender publishes no terminal"); + assert_eq!(error.code, "generation_retired"); + assert_eq!( + error.outcome, + SendOutcome::OutcomeUnknown, + "a claimed request whose sender vanished may still have been delivered" + ); + } + #[test] fn absent_admission_facts_are_omitted_rather_than_sent_as_null() { // The host reads any present member as `Some(..)`, so a null would make @@ -3244,6 +3465,242 @@ mod tests { assert!(task.await.expect("reader task").is_err()); } + #[tokio::test] + async fn an_unsupported_version_fails_at_the_frozen_prefix() { + // Byte 4 of the frozen prefix already proves the generation unusable. + // Waiting for the remaining 16 header bytes lets a peer that sends five + // and stops hold this connection for the whole frame deadline, so the + // outer bound below is far shorter than `CLIENT_FRAME_TIMEOUT`: only a + // prefix-first rejection can satisfy it. + let (inner, _data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (mut peer, mut reader) = tokio::io::duplex(64); + let mut prefix = [0u8; FROZEN_PREFIX_LEN]; + prefix[4] = PROTOCOL_VERSION.wrapping_add(1); + peer.write_all(&prefix).await.expect("frozen prefix"); + + let result = tokio::time::timeout( + Duration::from_secs(1), + read_active_frame(&mut reader, &inner), + ) + .await + .expect("an unsupported version must be rejected on the prefix alone"); + assert!( + result.is_err(), + "an unsupported envelope version is not a readable frame" + ); + } + + #[tokio::test] + async fn an_oversize_negotiation_response_is_rejected_on_the_header() { + // Negotiation is channel-zero control traffic, so §7.1's 65,536-byte cap + // applies. This path never reaches `validate_inbound`, so without its own + // check the client accepts the header and allocates the declared body — + // roughly 64 MiB on every connect attempt. The peer below sends no body + // at all, so only a header-only rejection completes inside the bound. + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let peer = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let header = EnvelopeHeader { + len: MAX_CONTROL_BODY_LEN + 1, + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, true), + channel: 0, + epoch: 0, + corr: NEGOTIATION_CORRELATION, + } + .encode(); + socket.write_all(&header).await.unwrap(); + socket + }); + let mut stream = TcpStream::connect(addr).await.unwrap(); + + let outcome = tokio::time::timeout( + Duration::from_secs(1), + read_setup_frame(&mut stream, Instant::now() + CLIENT_HANDSHAKE_TIMEOUT), + ) + .await + .expect("the header alone proves the violation; no body wait"); + let Err(error) = outcome else { + panic!("an oversize control body is rejected"); + }; + assert_eq!(error.code(), "negotiation_failed"); + drop(peer.await); + } + + #[tokio::test] + async fn retained_stream_bytes_never_deny_a_maximum_sized_frame() { + // The wire contract obliges an admitted connection to accept any + // otherwise valid frame. Charging queue retention and the reader's + // in-flight body from one pool let a consumer holding a few megabytes + // make an unrelated maximum-sized terminal unreadable, which the reader + // could report only by retiring the whole generation. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, _terminal_rx) = oneshot::channel(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + drop(data_rx.recv().await); + + let queued = 2 * 1024 * 1024; + let charge = inner.read_budget.charge(queued).expect("read reservation"); + inner.dispatch( + EnvelopeHeader { + len: u32::try_from(queued).expect("fits a frame length"), + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(true, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![0; queued], + charge, + ); + + assert_eq!( + inner.retained_budget.used(), + queued, + "a queued item is accounted against retention, not the read reservation" + ); + assert_eq!( + inner.read_budget.used(), + 0, + "the read reservation is released once the bytes are retained" + ); + assert!( + inner.read_budget.charge(MAX_BODY_LEN as usize).is_some(), + "queued bytes must not deny the reader a maximum-sized frame" + ); + inner.retire("test_done"); + } + + #[tokio::test] + async fn exhausted_retention_cancels_only_the_saturating_stream() { + // Byte-wise retention exhaustion is the same local overflow as item-queue + // saturation and must be reported the same way: cancel this stream, keep + // the generation and every unrelated route on it. + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let (items_tx, _items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let (key, _publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + drop(data_rx.recv().await); + + let hold = inner + .retained_budget + .charge(CLIENT_RETAINED_RESPONSE_BYTES) + .expect("retention fully held by an existing consumer"); + let charge = inner.read_budget.charge(1).expect("read reservation"); + inner.dispatch( + EnvelopeHeader { + len: 1, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![7], + charge, + ); + + let error = terminal_rx + .await + .expect("terminal sender") + .expect_err("the stream that could not retain its item fails"); + assert_eq!(error.code(), "stream_saturated"); + assert_eq!(error.outcome(), SendOutcome::OutcomeUnknown); + let cancel = control_rx.recv().await.expect("stream Cancel"); + assert_eq!(cancel.bytes[5], FrameType::Cancel as u8); + assert!( + !inner.retired.load(Ordering::Acquire), + "a saturated consumer must not retire the generation" + ); + assert_eq!( + inner.read_budget.used(), + 0, + "the discarded item releases the read reservation" + ); + drop(hold); + inner.retire("test_done"); + } + + #[tokio::test] + async fn a_malformed_route_open_success_retires_the_generation() { + // The host bound a route whose success body names no channel or epoch, so + // the client can never send a route `Goodbye` for it. Leaving the + // connection live lets each repeated open strand another host-side route + // and channel permit; retiring is what obliges the host to settle them. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let client = Client { + inner: Arc::clone(&inner), + }; + let open = tokio::spawn(async move { + client + .open_route( + RouteTarget { + kind: TargetKind::ManagementSurface, + module_id: "magic-context".to_owned(), + }, + identity_fixture(), + ) + .await + }); + + let frame = data_rx.recv().await.expect("route.open request"); + let header = decode_header(&frame.bytes).expect("request header"); + inner.dispatch( + EnvelopeHeader { + len: 0, + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, true), + channel: 0, + epoch: 0, + corr: header.corr, + }, + br#"{"ok":true}"#.to_vec(), + ByteCharge::none(), + ); + + let error = open + .await + .expect("open task") + .expect_err("a success body without a route is not a route"); + assert_eq!(error.code(), "invalid_route_response"); + assert!( + inner.retired.load(Ordering::Acquire), + "an unnameable binding must not be left on a live generation" + ); + assert!( + lock_unpoisoned(&inner.routes).is_empty(), + "retirement drops the generation's routes" + ); + } + #[test] fn queue_and_retained_charges_release_exactly() { let budget = Arc::new(ByteCounter::new(10)); diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 2a704b786..32507f525 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -800,7 +800,7 @@ impl HistorianProducer { let response = match self.send_frozen_once(&frozen).await { Ok(response) => response, Err(error) if is_outcome_unknown(&error) && !self.stop_requested() => { - self.replay_frozen_once(frozen_daemon, frozen_identity, &frozen) + self.replay_frozen_once(frozen_daemon, frozen_identity, &frozen, error) .await? } Err(error) => return Err(error), @@ -914,11 +914,19 @@ impl HistorianProducer { Ok(serde_json::from_slice(&response)?) } + /// Replays the frozen request on a fresh generation after an ambiguous send. + /// + /// `ambiguous` is the failure that justified the replay. Every abort inside + /// this function returns it unchanged: the frozen request may already have + /// reached the host, so reporting the cancellation itself — a `NotSent` + /// classification — would tell the caller a possibly-delivered request is + /// safe to send again. async fn replay_frozen_once( &mut self, frozen_daemon: [u8; 16], frozen_identity: SemanticIdentity, frozen: &[u8], + ambiguous: HistorianProducerError, ) -> Result { // Release the ambiguous generation before dialing the replay // connection. Both hold a host connection permit, so overlapping them @@ -933,6 +941,17 @@ impl HistorianProducer { self.command_route = None; self.subscribe_route = None; + // The caller's gate is one check, and the setup below spans three + // separately budgeted awaits — cleanup, a fresh dial with authentication + // and negotiation, and a route open. None of them observe the token, so + // only the final request would notice a cancellation that arrived just + // after the gate; a stopping handler would first spend every one of those + // budgets and bind a host-side route. Rechecking between stages aborts at + // a boundary where nothing is half-done. + if self.stop_requested() { + return Err(ambiguous); + } + let reconnected = self .connector .reconnect(&self.config, &frozen_identity) @@ -947,6 +966,13 @@ impl HistorianProducer { identity_changed, }); } + // `send_frozen_once` opens the command route before it sends, and + // `open_route` carries its own 30-second budget without observing the + // token. The request itself does observe it, so this is the last gate + // that can prevent binding a route for a run nobody is waiting for. + if self.stop_requested() { + return Err(ambiguous); + } self.send_frozen_once(frozen).await } @@ -1404,6 +1430,7 @@ mod tests { close_calls: usize, next_channel: u16, stall_stream: bool, + cancel_on_close: Option, } #[async_trait] @@ -1464,7 +1491,16 @@ mod tests { } async fn close(&self) -> Result<(), HistorianProducerError> { - self.state.lock().unwrap().close_calls += 1; + let cancel = { + let mut state = self.state.lock().unwrap(); + state.close_calls += 1; + state.cancel_on_close.take() + }; + // Lets a test place a cancellation exactly inside the replay's + // cleanup — after the caller's pre-replay gate, before any dial. + if let Some(cancel) = cancel { + cancel.cancel(); + } Ok(()) } } @@ -1779,6 +1815,54 @@ mod tests { } } + #[tokio::test] + async fn a_cancellation_during_replay_setup_stops_before_dialing() { + // The caller's gate is one check, and the replay's setup then spans a + // cleanup, a fresh dial with authentication and negotiation, and a route + // open — none of which observe the token. A cancellation arriving in that + // span must stop the setup, not run to the final request: otherwise a + // stopping handler spends every one of those budgets and binds a + // host-side route for a run nobody will await. + let cancelled = CancellationToken::new(); + let first = connection(9, [Err(cancelled_unknown())]); + first.state.lock().unwrap().cancel_on_close = Some(cancelled.clone()); + let second = connection(9, [Ok(br#"{"run_id":"replayed"}"#.to_vec())]); + let second_state = Arc::clone(&second.state); + let connector = Arc::new(FakeConnector { + initial: first, + reconnects: Mutex::new(VecDeque::from(vec![(second, None)])), + reconnect_calls: AtomicU64::new(0), + }); + let config = HistorianProducerConfig { + request_timeout: Duration::from_secs(1), + await_timeout: Duration::from_secs(1), + cancellation: Some(cancelled), + ..HistorianProducerConfig::new("/unused", "/project", "opencode") + }; + let mut producer = HistorianProducer::connect_with(config, connector.clone()) + .await + .unwrap(); + + let error = producer + .start("session", "", "prompt", "provider/model") + .await + .expect_err("a cancelled replay does not succeed"); + assert!( + is_outcome_unknown(&error), + "the frozen request may already have been delivered, so the abort must \ + stay ambiguous rather than reporting the cancellation as NotSent: {error:?}" + ); + assert_eq!( + connector.reconnect_calls.load(Ordering::SeqCst), + 0, + "a token that fires during replay cleanup must prevent the dial" + ); + assert!( + second_state.lock().unwrap().opened_routes.is_empty(), + "no route may be bound for a cancelled replay" + ); + } + #[tokio::test] async fn close_releases_subscription_and_command_routes() { let first = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); From d56e57cfc4baf6152e63669513a14504ff8f678c Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 16:33:12 +0000 Subject: [PATCH 22/37] docs(mc-module): describe how a JSON output is actually measured The comment on PreparedOutput::json still claimed measurement serializes once and the write phase reuses those bytes. Measurement counts the length without retaining anything, and write_to serializes straight into the destination. --- crates/mc-module/src/dispatch.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/crates/mc-module/src/dispatch.rs b/crates/mc-module/src/dispatch.rs index 59837d3c0..bc5fffb37 100644 --- a/crates/mc-module/src/dispatch.rs +++ b/crates/mc-module/src/dispatch.rs @@ -88,8 +88,11 @@ impl fmt::Debug for PreparedSegment { } impl PreparedOutput { - /// Retains a JSON value; measurement performs the single serialization - /// pass and the write phase reuses the encoded bytes. + /// Retains a JSON value and encodes it only on the write phase. + /// + /// Measurement counts the serialized length without keeping the bytes, so + /// nothing is retained outside the host's reservation; see + /// [`PreparedOutput::measure`] for why the serializer runs twice. pub fn json(value: Value) -> Self { Self { source: PreparedSource::Json(Arc::new(value)), From ae5ccd715fe0d89663f514fdfa1ecf92b7025e7c Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 16:51:18 +0000 Subject: [PATCH 23/37] fix(mc-module): keep a failed replay dial from erasing the ambiguity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The replay exists because the frozen `session.send` may already have committed. When the replacement dial failed, the reconnect error propagated and replaced that classification with `HistorianProducerError::Client`, which reports no send outcome at all — so a consumer asking whether the request went out got no answer instead of the ambiguous one. The dial failure is now logged as context and the original ambiguous error is returned, matching both the cleanup-failure precedent in the same function and the two cancellation aborts beside it. Those aborts already preserved the ambiguity; the dial path was the one exit that did not. --- crates/mc-module/src/historian_producer.rs | 73 ++++++++++++++++++++-- 1 file changed, 69 insertions(+), 4 deletions(-) diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 32507f525..2a4089073 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -952,10 +952,25 @@ impl HistorianProducer { return Err(ambiguous); } - let reconnected = self + let reconnected = match self .connector .reconnect(&self.config, &frozen_identity) - .await?; + .await + { + Ok(reconnected) => reconnected, + Err(error) => { + // The replacement connection is the replay's own machinery, not + // the thing the caller asked about. Surfacing the dial failure + // replaces an `OutcomeUnknown` send with a transport error that + // carries no send classification at all — `send_outcome()` is + // `None` for `Client` — so a consumer can no longer tell that the + // frozen request may already have committed. Keep the ambiguity + // and log the dial failure as context, exactly as the cleanup + // failure above does. + eprintln!("mc-module: historian replay reconnect failed: {error}"); + return Err(ambiguous); + } + }; let daemon_changed = reconnected.connection.daemon_id() != frozen_daemon; let identity_changed = reconnected.identity != frozen_identity; self.connection = reconnected.connection; @@ -1546,8 +1561,15 @@ mod tests { identity: &SemanticIdentity, ) -> Result { self.reconnect_calls.fetch_add(1, Ordering::SeqCst); - let (connection, override_identity) = - self.reconnects.lock().unwrap().pop_front().unwrap(); + // An exhausted queue stands in for a dial that cannot be completed — + // a transient daemon outage during replay setup. + let Some((connection, override_identity)) = self.reconnects.lock().unwrap().pop_front() + else { + return Err(HistorianProducerError::Client(HistorianClientFailure { + code: "connect_failed".to_owned(), + message: "no host to dial".to_owned(), + })); + }; Ok(Reconnected { connection: Box::new(connection), identity: override_identity.unwrap_or_else(|| identity.clone()), @@ -1863,6 +1885,49 @@ mod tests { ); } + #[tokio::test] + async fn a_failed_replay_reconnect_keeps_the_send_ambiguous() { + // The replay exists because the first `session.send` may already have + // committed. If the replacement dial fails — a transient daemon outage — + // reporting that transport error discards the only evidence of that + // ambiguity: `Client` carries no send classification at all, so a + // consumer can no longer tell the request might be live. + let first = connection(9, [Err(cancelled_unknown())]); + let connector = Arc::new(FakeConnector { + initial: first, + // No reconnect available: the dial fails. + reconnects: Mutex::new(VecDeque::new()), + reconnect_calls: AtomicU64::new(0), + }); + let config = HistorianProducerConfig { + request_timeout: Duration::from_secs(1), + await_timeout: Duration::from_secs(1), + ..HistorianProducerConfig::new("/unused", "/project", "opencode") + }; + let mut producer = HistorianProducer::connect_with(config, connector.clone()) + .await + .unwrap(); + + let error = producer + .start("session", "", "prompt", "provider/model") + .await + .expect_err("a replay that cannot dial does not succeed"); + assert_eq!( + connector.reconnect_calls.load(Ordering::SeqCst), + 1, + "the replay did attempt the dial" + ); + assert!( + is_outcome_unknown(&error), + "the dial failure must not replace the ambiguous send classification: {error:?}" + ); + assert_eq!( + error.send_outcome(), + Some(HistorianSendOutcome::OutcomeUnknown), + "a consumer asking whether the request was sent must still get the ambiguous answer" + ); + } + #[tokio::test] async fn close_releases_subscription_and_command_routes() { let first = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); From 9d404a459627aaa30ebadf761dc70185a98e0000 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 17:13:47 +0000 Subject: [PATCH 24/37] fix: bound uncancellable discovery, gate cancelled firings, align symlink rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four findings from the latest review round, all reachable. `Client::connect` wrapped the connection-file snapshot in a timeout, but `spawn_blocking` work is not cancellable: when the timeout fired, dropping the join handle only detached the closure, and a snapshot wedged in a filesystem syscall kept its blocking worker for as long as the mount took. Repeated connect and reconnect attempts each stranded another worker until the pool was gone and unrelated blocking work queued behind mounts nobody was waiting on. Discovery now holds a permit from a small semaphore, and the permit is moved into the closure so a detached worker keeps counting against the cap. Waiting for a permit spends the same handshake budget the snapshot would have, so exhaustion surfaces as the timeout it already is. `start_with_generation` had no cancellation gate at all. `ensure_command_route` runs ahead of the request carrying `open_route`'s own 30-second retry budget, and only the request observes the token, so a handler already shutting down waited through route admission and could bind a route for a firing cancelled before it started. The gate reports `NotSent`, which is exact: no frame has been queued on any route yet. This sharpens the pre-cancelled case, which previously sent the request and reported the ambiguous result instead. The wire protocol dropped the trusted-symlink exception in favour of "host publication, client discovery, and cleanup MUST reject symbolic links", but the TypeScript client still honoured a public `trustedSymlink` option and resolved symlinked connection files. The two clients therefore enforced different credential-authority rules, since the Rust reader always opens with `NOFOLLOW`. The option, its snapshot path, and its now-unreachable error code are removed, so V5 holds for both clients. The direct-host fixture released blocked backend runs through a counting semaphore. A permit added just before cancellation won the biased race stayed in the semaphore, and the next blocked run consumed it immediately and completed with no matching release call — silently rewriting the fault schedule an E2E scenario asserts against. Releases now address a specific blocked invocation, and a run that ends without consuming one withdraws its slot. --- crates/mc-host/src/client.rs | 34 +++++++- .../mc-module/examples/direct_host_fixture.rs | 82 ++++++++++++++++--- crates/mc-module/src/historian_producer.rs | 50 +++++++++-- .../src/shared/mc-host-client/client.ts | 5 -- .../mc-host-client/connection-file.test.ts | 64 +-------------- .../shared/mc-host-client/connection-file.ts | 80 +----------------- 6 files changed, 149 insertions(+), 166 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index f096d42a4..26f7458fd 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -11,7 +11,7 @@ use std::{ path::Path, sync::{ atomic::{AtomicBool, AtomicU8, Ordering}, - Arc, Mutex, MutexGuard, Weak, + Arc, LazyLock, Mutex, MutexGuard, Weak, }, time::Duration, }; @@ -81,6 +81,17 @@ pub const CLIENT_INBOUND_FRAME_BYTES: usize = MAX_BODY_LEN as usize; /// one connection is this plus `CLIENT_INBOUND_FRAME_BYTES`. pub const CLIENT_RETAINED_RESPONSE_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; +/// Concurrent connection-file snapshots allowed across this process. +/// +/// A snapshot runs on the blocking pool and cannot be cancelled, so this caps how +/// many blocking workers a wedged mount can strand; see `Client::connect`. +const CLIENT_DISCOVERY_SLOTS: usize = 4; + +/// Permits for [`CLIENT_DISCOVERY_SLOTS`], held by the blocking closure itself so +/// a detached worker still counts against the cap. +static DISCOVERY_SLOTS: LazyLock> = + LazyLock::new(|| Arc::new(tokio::sync::Semaphore::new(CLIENT_DISCOVERY_SLOTS))); + const NEGOTIATION_CORRELATION: u64 = 1; const FIRST_APPLICATION_CORRELATION: u64 = 2; const MAX_ERROR_CODE_BYTES: usize = 128; @@ -286,9 +297,28 @@ impl Client { // async worker for as long as the mount takes. let deadline = Instant::now() + CLIENT_HANDSHAKE_TIMEOUT; let path = path.as_ref().to_path_buf(); + // Bound how many discovery snapshots can be in flight at once. + // `spawn_blocking` work is not cancellable: when the timeout below fires, + // dropping the join handle only detaches the closure, and a snapshot + // wedged in a filesystem syscall keeps its blocking worker for as long as + // the mount takes. Repeated connect and reconnect attempts would each + // strand another worker until the blocking pool is gone and unrelated + // blocking work queues behind mounts nobody is waiting on. The permit is + // moved into the closure, so a detached worker keeps holding it and the + // cap counts the workers that actually exist rather than the callers that + // gave up. Waiting for a permit spends the same handshake budget the + // snapshot itself would have, so exhaustion surfaces as the timeout it + // already is instead of a new failure mode. + let permit = timeout_at(deadline, Arc::clone(&DISCOVERY_SLOTS).acquire_owned()) + .await + .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? + .expect("discovery semaphore is never closed"); let info = timeout_at( deadline, - tokio::task::spawn_blocking(move || read_for_client(path)), + tokio::task::spawn_blocking(move || { + let _permit = permit; + read_for_client(path) + }), ) .await .map_err(|_| ClientError::new("handshake_timeout", "client handshake timed out"))? diff --git a/crates/mc-module/examples/direct_host_fixture.rs b/crates/mc-module/examples/direct_host_fixture.rs index c2350e1a5..bbd3f83c7 100644 --- a/crates/mc-module/examples/direct_host_fixture.rs +++ b/crates/mc-module/examples/direct_host_fixture.rs @@ -4,6 +4,7 @@ #[cfg(unix)] mod unix { + use std::collections::VecDeque; use std::error::Error; use std::fs; use std::io::{self, Write}; @@ -25,6 +26,7 @@ mod unix { use sha2::Digest; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{UnixListener, UnixStream}; + use tokio::sync::oneshot; const CONTROL_FILE: &str = "direct-host-control.sock"; const STORE_FILE: &str = "mc-store.db"; @@ -73,9 +75,20 @@ mod unix { } } + /// One release channel per blocked invocation, oldest first. + /// + /// A counting semaphore cannot express this: a permit added just before + /// cancellation wins the `biased` race in `execute` stays in the semaphore, + /// and the next `Block` run consumes it immediately and completes without a + /// matching `release-blocked-call`. That silently rewrites the fault schedule + /// an E2E scenario is asserting against. Addressing a specific invocation + /// makes an unconsumed release impossible to mistake for a pending one. + type BlockedQueue = Arc)>>>; + struct ControlledBackend { next: Mutex, - release: Arc, + blocked: BlockedQueue, + next_blocked_id: Arc, shutdown: CancellationToken, counters: Arc, } @@ -84,7 +97,8 @@ mod unix { fn new(shutdown: CancellationToken) -> Arc { Arc::new(Self { next: Mutex::new(NextBehavior::Success), - release: Arc::new(tokio::sync::Semaphore::new(0)), + blocked: Arc::new(Mutex::new(VecDeque::new())), + next_blocked_id: Arc::new(AtomicU64::new(0)), shutdown, counters: Arc::new(BackendCounters::default()), }) @@ -94,13 +108,27 @@ mod unix { *self.next.lock().expect("fixture backend behavior mutex") = behavior; } + /// Hands one release to the oldest still-waiting blocked invocation. + /// + /// Senders whose invocation already lost the race to cancellation are + /// discarded rather than counted, so a release is reported accepted only + /// when a live run actually received it. fn release_blocked(&self) -> bool { - if !take_blocked_slot(&self.counters) { - return false; + loop { + let Some((_id, sender)) = self + .blocked + .lock() + .expect("fixture blocked queue mutex") + .pop_front() + else { + return false; + }; + if sender.send(()).is_ok() { + take_blocked_slot(&self.counters); + self.counters.released.fetch_add(1, Ordering::SeqCst); + return true; + } } - self.counters.released.fetch_add(1, Ordering::SeqCst); - self.release.add_permits(1); - true } fn terminal_error(message: &str) -> BackendTerminal { @@ -122,6 +150,26 @@ mod unix { .is_ok() } + /// Registers one blocked invocation and returns its id and release channel. + fn register_blocked(queue: &BlockedQueue, next_id: &AtomicU64) -> (u64, oneshot::Receiver<()>) { + let id = next_id.fetch_add(1, Ordering::SeqCst); + let (tx, rx) = oneshot::channel(); + queue + .lock() + .expect("fixture blocked queue mutex") + .push_back((id, tx)); + (id, rx) + } + + /// Withdraws a blocked invocation that ended without consuming a release, so + /// its slot cannot be handed a release no run is waiting for. + fn withdraw_blocked(queue: &BlockedQueue, id: u64) { + queue + .lock() + .expect("fixture blocked queue mutex") + .retain(|(queued, _)| *queued != id); + } + impl LlmExecutionBackend for ControlledBackend { fn execute( &self, @@ -134,7 +182,8 @@ mod unix { &mut *self.next.lock().expect("fixture backend behavior mutex"), NextBehavior::Success, ); - let release = Arc::clone(&self.release); + let blocked = Arc::clone(&self.blocked); + let next_blocked_id = Arc::clone(&self.next_blocked_id); let shutdown = self.shutdown.clone(); let counters = Arc::clone(&self.counters); Box::pin(async move { @@ -156,20 +205,33 @@ mod unix { NextBehavior::Block => { counters.blocked.fetch_add(1, Ordering::SeqCst); counters.active_blocked.fetch_add(1, Ordering::SeqCst); + // Registered before the race so a release issued while + // this run is waiting reaches this run and no other. + let (id, release) = register_blocked(&blocked, &next_blocked_id); tokio::select! { biased; () = shutdown.cancelled() => { + withdraw_blocked(&blocked, id); take_blocked_slot(&counters); counters.cancelled.fetch_add(1, Ordering::SeqCst); ControlledBackend::terminal_error("fixture shutting down") } () = cancel.cancelled() => { + withdraw_blocked(&blocked, id); take_blocked_slot(&counters); counters.cancelled.fetch_add(1, Ordering::SeqCst); ControlledBackend::terminal_error("fixture run cancelled") } - permit = release.acquire() => { - permit.expect("fixture release semaphore stays open").forget(); + released = release => { + // `release_blocked` already accounted this run; + // a dropped sender means the fixture is going + // away, which the cancellation branches own. + if released.is_err() { + withdraw_blocked(&blocked, id); + take_blocked_slot(&counters); + counters.cancelled.fetch_add(1, Ordering::SeqCst); + return ControlledBackend::terminal_error("fixture release dropped"); + } events.emit(BackendEvent::AssistantText { text: "fixture-released".to_owned(), finish_reason: None, diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 2a4089073..d885f7a1f 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -766,6 +766,21 @@ impl HistorianProducer { max_output_tokens: u32, temperature: f64, ) -> Result { + // Nothing here observes the token until the request itself, and + // `ensure_command_route` sits in front of it carrying `open_route`'s own + // 30-second retry budget. Without this gate a handler that is already + // shutting down waits through route admission and can bind a route for a + // firing that was cancelled before it started. `NotSent` is exact: no + // frame has been queued on any route yet. + if self.stop_requested() { + return Err(HistorianProducerError::Call( + HistorianCallFailure::untagged( + HistorianSendOutcome::NotSent, + "cancelled", + "historian firing was cancelled before it started".to_owned(), + ), + )); + } self.bind_session(session_id.to_owned()); let (provider, model_name) = model.split_once('/').ok_or_else(|| { HistorianProducerError::Call(HistorianCallFailure::untagged( @@ -1774,22 +1789,23 @@ mod tests { #[tokio::test] async fn caller_cancellation_prevents_replay_but_transport_loss_still_replays() { - // A caller cancellation and an ambiguous transport loss are both - // OutcomeUnknown for the same reason — the bytes may already be gone — + // A cancellation and an ambiguous transport loss can both end a firing, // but they need opposite handling. Replaying a cancellation dials a new // connection and re-sends `session.send` after the caller said stop, // starting a run if the original never went out, and during handler // shutdown makes a cancelled task do connection setup instead of // draining. The token is the authority, so the transport replay below - // must survive the gate. + // must survive the gate while the cancelled firing never reaches it. async fn start_with_token( token: Option, ) -> ( Arc, Arc>, + Arc>, Result, ) { let first = connection(9, [Err(cancelled_unknown())]); + let first_state = Arc::clone(&first.state); let second = connection(9, [Ok(br#"{"run_id":"replayed"}"#.to_vec())]); let second_state = Arc::clone(&second.state); let connector = Arc::new(FakeConnector { @@ -1809,15 +1825,31 @@ mod tests { let result = producer .start("session", "", "prompt", "provider/model") .await; - (connector, second_state, result) + (connector, first_state, second_state, result) } - // Cancelled: the failure surfaces to the caller and nothing is resent. + // Already cancelled before the firing starts: nothing is sent at all, so + // no route is opened and no replay is considered. `NotSent` is the exact + // classification here — unlike a cancellation that lands mid-send, this + // one is provably before any frame was queued. let cancelled = CancellationToken::new(); cancelled.cancel(); - let (connector, replay_state, result) = start_with_token(Some(cancelled)).await; - let error = result.expect_err("a cancelled start does not succeed by replaying"); - assert!(is_outcome_unknown(&error), "{error:?}"); + let (connector, first_state, replay_state, result) = + start_with_token(Some(cancelled)).await; + let error = result.expect_err("a cancelled start does not succeed"); + assert_eq!( + error.send_outcome(), + Some(HistorianSendOutcome::NotSent), + "a firing cancelled before it started queued nothing: {error:?}" + ); + assert!( + first_state.lock().unwrap().requests.is_empty(), + "a pre-cancelled firing must not send session.send at all" + ); + assert!( + first_state.lock().unwrap().opened_routes.is_empty(), + "a pre-cancelled firing must not bind a route" + ); assert_eq!( connector.reconnect_calls.load(Ordering::SeqCst), 0, @@ -1830,7 +1862,7 @@ mod tests { // Live token: the intentional transport-loss replay is unaffected. for token in [None, Some(CancellationToken::new())] { - let (connector, replay_state, result) = start_with_token(token).await; + let (connector, _first_state, replay_state, result) = start_with_token(token).await; assert_eq!(result.expect("transport loss replays").run_id, "replayed"); assert_eq!(connector.reconnect_calls.load(Ordering::SeqCst), 1); assert_eq!(replay_state.lock().unwrap().requests.len(), 1); diff --git a/packages/plugin/src/shared/mc-host-client/client.ts b/packages/plugin/src/shared/mc-host-client/client.ts index 76cea5053..a1fe52525 100644 --- a/packages/plugin/src/shared/mc-host-client/client.ts +++ b/packages/plugin/src/shared/mc-host-client/client.ts @@ -137,8 +137,6 @@ export interface McHostClientOptions extends ConnectOptions { requestTimeoutMs?: number; routeOpenDeadlineMs?: number; shutdownDeadlineMs?: number; - /** Opt-in for the trusted-symlink connection-file form (wire doc 4.2). */ - trustedSymlink?: boolean; /** * Test seam forwarded to the connection-file read's `afterOpen` hook; * lets tests race a snapshot against deadlines deterministically. @@ -390,7 +388,6 @@ export class McHostClient { private readonly defaultTargetKind: ManagedRouteKind; private readonly clock: MonotonicClock | undefined; private readonly sleep: (ms: number) => Promise; - private readonly trustedSymlink: boolean; private readonly connectionFileAfterOpen: (() => void | Promise) | undefined; private readonly generationOptions: McHostClientOptions["generationOptions"]; private readonly diagnostics: McHostDiagnosticsObserver | undefined; @@ -420,7 +417,6 @@ export class McHostClient { this.clock = options.clock; this.sleep = options.sleep ?? ((ms: number) => new Promise((resolve) => setTimeout(resolve, ms))); - this.trustedSymlink = options.trustedSymlink ?? false; this.connectionFileAfterOpen = options.connectionFileAfterOpen; this.generationOptions = options.generationOptions; this.diagnostics = options.diagnostics; @@ -681,7 +677,6 @@ export class McHostClient { try { snapshot = await readConnectionFile(this.connectionFile, { deadline: stage, - trustedSymlink: this.trustedSymlink, afterOpen: this.connectionFileAfterOpen, }); } catch (error) { diff --git a/packages/plugin/src/shared/mc-host-client/connection-file.test.ts b/packages/plugin/src/shared/mc-host-client/connection-file.test.ts index 39c160645..07c5ace25 100644 --- a/packages/plugin/src/shared/mc-host-client/connection-file.test.ts +++ b/packages/plugin/src/shared/mc-host-client/connection-file.test.ts @@ -1,6 +1,6 @@ import { afterAll, beforeAll, describe, expect, test } from "bun:test"; import { execFile } from "node:child_process"; -import { mkdir, mkdtemp, rename, rm, symlink, unlink, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, rename, rm, symlink, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { promisify } from "node:util"; @@ -128,7 +128,7 @@ describe("direct-file snapshot", () => { await expectFailure(fifoPath, "not_regular_file"); }); - test("rejects a symlink outside the trusted exception", async () => { + test("rejects a symlink at the connection-file path", async () => { const target = freshPath("link-target.json"); await writePrivateFile(target, JSON.stringify(validJson())); const linkPath = freshPath("untrusted-link.json"); @@ -196,66 +196,6 @@ describe("direct-file snapshot", () => { }); }); -describe("trusted-symlink snapshot", () => { - async function makeLinkedFile(): Promise<{ linkPath: string; targetPath: string }> { - const targetPath = freshPath("trusted-target.json"); - await writePrivateFile(targetPath, JSON.stringify(validJson())); - const linkPath = freshPath("trusted-link.json"); - await symlink(targetPath, linkPath); - return { linkPath, targetPath }; - } - - test("accepts a stable owner-controlled symlink to a private target", async () => { - const { linkPath } = await makeLinkedFile(); - const snapshot = await readConnectionFile(linkPath, options({ trustedSymlink: true })); - expect(snapshot.endpoint.port).toBe(43_123); - }); - - test("rejects a regular file supplied as the trusted-symlink form", async () => { - const filePath = freshPath("not-a-link.json"); - await writePrivateFile(filePath, JSON.stringify(validJson())); - await expectFailure(filePath, "not_symlink", { trustedSymlink: true }); - }); - - test("rejects a target with group/other permission bits", async () => { - const targetPath = freshPath("loose-target.json"); - await writeFile(targetPath, JSON.stringify(validJson()), { mode: 0o644 }); - const linkPath = freshPath("loose-link.json"); - await symlink(targetPath, linkPath); - await expectFailure(linkPath, "insecure_permissions", { trustedSymlink: true }); - }); - - test("fails closed when the link is replaced mid-validation, with no restart", async () => { - const { linkPath } = await makeLinkedFile(); - let attempts = 0; - const afterOpen = async (): Promise => { - attempts += 1; - const otherTarget = freshPath("other-target.json"); - await writePrivateFile(otherTarget, JSON.stringify(validJson())); - await unlink(linkPath); - await symlink(otherTarget, linkPath); - }; - await expectFailure(linkPath, "replaced_during_read", { - trustedSymlink: true, - afterOpen, - }); - expect(attempts).toBe(1); - }); - - test("fails closed when the target is replaced mid-validation", async () => { - const { linkPath, targetPath } = await makeLinkedFile(); - const afterOpen = async (): Promise => { - const replacement = freshPath("target-replacement.json"); - await writePrivateFile(replacement, JSON.stringify(validJson())); - await rename(replacement, targetPath); - }; - await expectFailure(linkPath, "replaced_during_read", { - trustedSymlink: true, - afterOpen, - }); - }); -}); - describe("snapshot JSON validation", () => { test("rejects invalid UTF-8 bytes", async () => { const filePath = freshPath("bad-utf8.json"); diff --git a/packages/plugin/src/shared/mc-host-client/connection-file.ts b/packages/plugin/src/shared/mc-host-client/connection-file.ts index 02798b751..7aba996b1 100644 --- a/packages/plugin/src/shared/mc-host-client/connection-file.ts +++ b/packages/plugin/src/shared/mc-host-client/connection-file.ts @@ -22,8 +22,7 @@ */ import { constants as fsConstants } from "node:fs"; -import { type FileHandle, lstat, open, readlink } from "node:fs/promises"; -import path from "node:path"; +import { type FileHandle, lstat, open } from "node:fs/promises"; import type { Deadline } from "./deadline"; /** Snapshot cap from wire doc Section 4.1: 65,536 bytes. */ @@ -39,7 +38,6 @@ export type ConnectionFileErrorCode = | "deadline_expired" | "open_failed" | "not_regular_file" - | "not_symlink" | "foreign_owner" | "insecure_permissions" | "oversize" @@ -80,12 +78,6 @@ export interface ConnectionSnapshot { export interface ReadConnectionFileOptions { /** Bounds the whole snapshot; checked between every filesystem step. */ deadline: Deadline; - /** - * Explicit opt-in for the trusted-symlink publication form (wire doc - * Section 4.2). Never auto-detected; a symlink at the path without this - * flag fails closed. - */ - trustedSymlink?: boolean; /** Test seam for the unsupported-platform check. Defaults to the real one. */ platform?: NodeJS.Platform; /** Test seam for the owning UID. Defaults to `process.getuid()`. */ @@ -208,7 +200,7 @@ async function snapshotDirect( const before = await lstat(filePath); if (before.isSymbolicLink()) { throw new ConnectionFileError( - `connection file ${filePath} is a symlink; the trusted-symlink form requires explicit opt-in`, + `connection file ${filePath} is a symlink; client discovery must reject symbolic links`, "not_regular_file", ); } @@ -246,65 +238,6 @@ async function snapshotDirect( } } -/** Trusted-symlink snapshot. Any link or target replacement fails closed. */ -async function snapshotTrustedSymlink( - linkPath: string, - deadline: Deadline, - uid: number, - afterOpen?: () => void | Promise, -): Promise { - checkDeadline(deadline); - const linkBefore = await lstat(linkPath); - if (!linkBefore.isSymbolicLink()) { - throw new ConnectionFileError( - `trusted connection-file path ${linkPath} is not a symlink`, - "not_symlink", - ); - } - if (linkBefore.uid !== uid) { - throw new ConnectionFileError( - `trusted connection-file link ${linkPath} is not owned by the current user`, - "foreign_owner", - ); - } - const linkText = await readlink(linkPath); - const targetPath = path.resolve(path.dirname(linkPath), linkText); - checkDeadline(deadline); - const handle = await openNoFollow(targetPath); - try { - await afterOpen?.(); - const target = await handle.stat(); - validateOpenStat(target, uid, `connection file target ${targetPath}`); - checkDeadline(deadline); - const bytes = await readBounded(handle, deadline); - checkDeadline(deadline); - const linkAfter = await lstat(linkPath); - if (!linkAfter.isSymbolicLink() || !sameIdentity(linkBefore, linkAfter)) { - throw new ConnectionFileError( - `trusted connection-file link ${linkPath} changed during the snapshot`, - "replaced_during_read", - ); - } - const linkTextAfter = await readlink(linkPath); - if (linkTextAfter !== linkText) { - throw new ConnectionFileError( - `trusted connection-file link ${linkPath} was retargeted during the snapshot`, - "replaced_during_read", - ); - } - const targetAfter = await lstat(targetPath); - if (!targetAfter.isFile() || !sameIdentity(target, targetAfter)) { - throw new ConnectionFileError( - `trusted connection-file target ${targetPath} was replaced during the snapshot`, - "replaced_during_read", - ); - } - return bytes; - } finally { - await handle.close().catch(() => {}); - } -} - function invalid(code: ConnectionFileErrorCode, message: string): ConnectionFileError { return new ConnectionFileError(message, code); } @@ -414,15 +347,6 @@ export async function readConnectionFile( ); } const uid = options.uid ?? currentUid(); - if (options.trustedSymlink) { - const bytes = await snapshotTrustedSymlink( - filePath, - options.deadline, - uid, - options.afterOpen, - ); - return decodeAndValidate(bytes); - } let bytes: Uint8Array; try { bytes = await snapshotDirect(filePath, options.deadline, uid, options.afterOpen); From c37b9bc1ad1cb4f06f1c33f743ebb0eee31fefeb Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 17:31:33 +0000 Subject: [PATCH 25/37] fix(e2e): skip Rust-mode suites on every non-Linux platform The prerequisite check excluded only Windows, so a macOS runner reported the prerequisites as met. The fixture always links `BrocaComponent`, whose `initialize` refuses any non-Linux target because crash-ownership records and sweeps read `/proc` process identity, so those runners built the fixture and then failed during startup instead of taking the skip path this check exists to provide. Windows keeps its own reason; every other non-Linux platform now reports the Linux requirement. --- .../e2e-tests/src/rust-runner/hermetic-mc-host.ts | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index e3a58c5a6..50cdec62b 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -229,6 +229,18 @@ export function detectRustModePrereqs(): RustModePrereqs { skipReason: "direct mc-host fixture requires Unix sockets", }; } + // The fixture always links `BrocaComponent`, and its `initialize` refuses + // every non-Linux target because crash-ownership records and sweeps read + // `/proc` process identity. Excluding only Windows let a macOS runner report + // the prerequisites as met, build the fixture, and then fail during startup + // instead of taking the skip path this check exists to provide. + if (process.platform !== "linux") { + return { + ok: false, + skipReason: + "direct mc-host fixture requires Linux: broca crash-ownership records depend on /proc process identity", + }; + } if (!existsSync(join(REPO_ROOT, "Cargo.toml"))) { return { ok: false, From 8e7842e585168e9a59990118051d6ff90b8b4b21 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 17:44:51 +0000 Subject: [PATCH 26/37] fix: abandon cancelled route opens safely and redact auth proofs `open_bound_route` awaited `Client::open_route`, which carries its own 30-second budget and retries retryable terminals without observing the producer's cancellation token. A token firing after the start gate therefore still waited for route admission, and could bind and cache a route for a firing nobody was waiting on. The open now races the token. Abandoning the await is only sound because of the cleanup that goes with it: the host may bind the route after the client stops listening, and the client can never name it, so the connection is closed and its `Goodbye` obliges the host to settle every route on the generation. Without that this would strand a route and a channel permit on each cancelled open. `replay_frozen_once` now also prefers the original ambiguous error whenever its own send fails under a cancelled token. The route-open race can return a local `cancelled`/`NotSent`, which describes the replay attempt and not the first send, and reporting it would tell the caller a possibly-committed request is safe to send again. `ServerProof` and `ClientAuth` derived `Debug`, so any `{:?}` in an error path or panic message printed every byte of the authentication proofs that V24 classifies as sensitive. Both now redact the proof field. The nonces, daemon ID, and daemon version stay visible: they travel in the clear and are what makes a transcript identifiable while debugging. --- crates/mc-host/src/auth.rs | 64 +++++++++- crates/mc-module/src/historian_producer.rs | 135 ++++++++++++++++++--- 2 files changed, 177 insertions(+), 22 deletions(-) diff --git a/crates/mc-host/src/auth.rs b/crates/mc-host/src/auth.rs index 02fd20af5..f50d1858a 100644 --- a/crates/mc-host/src/auth.rs +++ b/crates/mc-host/src/auth.rs @@ -28,7 +28,7 @@ pub struct ClientHello { pub role: String, } -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ServerProof { pub daemon_id: [u8; DAEMON_ID_LEN], pub server_nonce: [u8; NONCE_LEN], @@ -36,11 +36,37 @@ pub struct ServerProof { pub server_proof: [u8; PROOF_LEN], } -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +// V24 classifies proof bytes as sensitive diagnostics. A derived `Debug` prints +// the whole HMAC, so one routine `{:?}` in an error path or panic message +// persists a live authentication transcript secret. The nonces and daemon ID stay +// visible: both travel in the clear and are what makes a transcript identifiable +// while debugging. +impl fmt::Debug for ServerProof { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ServerProof") + .field("daemon_id", &self.daemon_id) + .field("server_nonce", &self.server_nonce) + .field("daemon_ver", &self.daemon_ver) + .field("server_proof", &"[redacted]") + .finish() + } +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ClientAuth { pub client_auth: [u8; PROOF_LEN], } +/// Redacted for the same reason as [`ServerProof`]; this struct is nothing but +/// the proof, so there is no non-secret field to keep. +impl fmt::Debug for ClientAuth { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ClientAuth") + .field("client_auth", &"[redacted]") + .finish() + } +} + /// The outcome of a successful handshake. /// /// WHAT THIS PROVES: the peer possesses the connection key, and (client side) @@ -568,6 +594,40 @@ impl Error for AuthError { mod tests { use super::*; + #[test] + fn proof_debug_output_never_carries_the_proof_bytes() { + // V24 classifies proof bytes as sensitive diagnostics. A derived `Debug` + // prints the whole HMAC, so one `{:?}` in an error path or panic message + // persists a live authentication transcript secret. + let sentinel = 0xAB; + let server = ServerProof { + daemon_id: [1; DAEMON_ID_LEN], + server_nonce: [2; NONCE_LEN], + daemon_ver: "1.2.3".to_owned(), + server_proof: [sentinel; PROOF_LEN], + }; + let rendered = format!("{server:?}"); + let byte = format!("{sentinel}"); + assert!( + !rendered.contains(&byte), + "server_proof bytes leaked into Debug: {rendered}" + ); + assert!(rendered.contains("[redacted]"), "{rendered}"); + // The identifying, non-secret fields stay debuggable. + assert!(rendered.contains("1.2.3"), "{rendered}"); + assert!(rendered.contains("server_nonce"), "{rendered}"); + + let client = ClientAuth { + client_auth: [sentinel; PROOF_LEN], + }; + let rendered = format!("{client:?}"); + assert!( + !rendered.contains(&byte), + "client_auth bytes leaked into Debug: {rendered}" + ); + assert!(rendered.contains("[redacted]"), "{rendered}"); + } + #[test] fn an_unrepresentable_auth_deadline_is_rejected_not_panicked() { // The total is operator configuration, so `Duration::MAX` must report a diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index d885f7a1f..159b9d775 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -997,13 +997,21 @@ impl HistorianProducer { }); } // `send_frozen_once` opens the command route before it sends, and - // `open_route` carries its own 30-second budget without observing the - // token. The request itself does observe it, so this is the last gate + // `open_route` carries its own 30-second budget. This is the last gate // that can prevent binding a route for a run nobody is waiting for. if self.stop_requested() { return Err(ambiguous); } - self.send_frozen_once(frozen).await + let sent = self.send_frozen_once(frozen).await; + // A cancellation landing anywhere inside the replay's own send leaves the + // ORIGINAL send's outcome unresolved, and the replay's local failure — + // `cancelled`/`NotSent` from the route-open race, say — describes only the + // replay. Reporting it would tell the caller a possibly-committed request + // is safe to send again. + if sent.is_err() && self.stop_requested() { + return Err(ambiguous); + } + sent } async fn ensure_command_route(&mut self) -> Result { @@ -1026,23 +1034,50 @@ impl HistorianProducer { async fn open_bound_route(&self) -> Result { let semantic = self.semantic_identity()?; - self.connection - .open_route( - RouteTarget { - module_id: self.config.module_id.clone(), - kind: TargetKind::ManagementSurface, - }, - RouteIdentity { - project_root: semantic.project_root, - harness: semantic.harness, - session: semantic.session, - consumer_module_id: nonempty_env(SUBC_MODULE_ID_ENV), - consumer_launch_nonce: nonempty_env(SUBC_LAUNCH_NONCE_ENV), - consumer_capabilities: Vec::new(), - admission_facts: None, - }, - ) - .await + let open = self.connection.open_route( + RouteTarget { + module_id: self.config.module_id.clone(), + kind: TargetKind::ManagementSurface, + }, + RouteIdentity { + project_root: semantic.project_root, + harness: semantic.harness, + session: semantic.session, + consumer_module_id: nonempty_env(SUBC_MODULE_ID_ENV), + consumer_launch_nonce: nonempty_env(SUBC_LAUNCH_NONCE_ENV), + consumer_capabilities: Vec::new(), + admission_facts: None, + }, + ); + let Some(cancellation) = self.config.cancellation.clone() else { + return open.await; + }; + tokio::select! { + biased; + () = cancellation.cancelled() => { + // `Client::open_route` carries its own 30-second budget and + // retries retryable terminals without observing this token, so + // abandoning the await is the only way a stopping handler does + // not wait for it. + // + // Walking away is safe only because of the cleanup below. The + // host may bind the route after we stop listening, and we can + // never name it, so closing the connection is what settles it: + // its connection `Goodbye` obliges the host to settle every + // route on this generation (§11.2), including the one we + // abandoned. Without that this would strand a route and a + // channel permit on every cancelled open. + if let Err(error) = self.connection.close().await { + eprintln!("mc-module: historian cancelled route-open cleanup failed: {error}"); + } + Err(HistorianProducerError::Call(HistorianCallFailure::untagged( + HistorianSendOutcome::NotSent, + "cancelled", + "historian route open was cancelled".to_owned(), + ))) + } + route = open => route, + } } async fn unary_json( @@ -1461,6 +1496,7 @@ mod tests { next_channel: u16, stall_stream: bool, cancel_on_close: Option, + cancel_on_open_route: Option, } #[async_trait] @@ -1474,6 +1510,16 @@ mod tests { _target: RouteTarget, identity: RouteIdentity, ) -> Result { + // Lets a test place a cancellation inside the route open itself, + // which is the window `Client::open_route`'s own budget does not + // observe. + let cancel = self.state.lock().unwrap().cancel_on_open_route.take(); + if let Some(cancel) = cancel { + cancel.cancel(); + // Yield so the racing `select!` observes the token before this + // open resolves; otherwise the ordering under test never occurs. + tokio::task::yield_now().await; + } let mut state = self.state.lock().unwrap(); state.identities.push(identity); state.next_channel += 1; @@ -1960,6 +2006,55 @@ mod tests { ); } + #[tokio::test] + async fn a_cancellation_during_route_open_abandons_it_and_closes_the_connection() { + // `Client::open_route` carries its own 30-second budget and does not + // observe the token, so a cancellation landing after the start gate must + // abandon the open rather than wait for it. Abandoning is only sound + // because the connection is closed: the host may bind the route after we + // stop listening, and its connection `Goodbye` is what settles a route we + // can never name. + let cancelled = CancellationToken::new(); + let first = connection(9, [Ok(br#"{"run_id":"unreachable"}"#.to_vec())]); + // Cancel while the route open is in flight. + first.state.lock().unwrap().cancel_on_open_route = Some(cancelled.clone()); + let state = Arc::clone(&first.state); + let connector = Arc::new(FakeConnector { + initial: first, + reconnects: Mutex::new(VecDeque::new()), + reconnect_calls: AtomicU64::new(0), + }); + let config = HistorianProducerConfig { + request_timeout: Duration::from_secs(1), + await_timeout: Duration::from_secs(1), + cancellation: Some(cancelled), + ..HistorianProducerConfig::new("/unused", "/project", "opencode") + }; + let mut producer = HistorianProducer::connect_with(config, connector.clone()) + .await + .unwrap(); + + let error = producer + .start("session", "", "prompt", "provider/model") + .await + .expect_err("a cancelled route open does not start a run"); + assert_eq!( + error.send_outcome(), + Some(HistorianSendOutcome::NotSent), + "nothing of the caller's request was queued: {error:?}" + ); + let state = state.lock().unwrap(); + assert!( + state.requests.is_empty(), + "no session.send may follow an abandoned route open" + ); + assert!( + state.close_calls >= 1, + "the abandoned open must be settled by closing the connection, since \ + the host may bind a route this client can never name" + ); + } + #[tokio::test] async fn close_releases_subscription_and_command_routes() { let first = connection(1, [Ok(br#"{"run_id":"run"}"#.to_vec())]); From c5179674f783836bd1990588f50f88237e6bbe2d Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 17:48:46 +0000 Subject: [PATCH 27/37] fix(mc-module): count a fixture release only when the run resumes `release_blocked` treated a successful `oneshot::Sender::send` as consumption, but that only proves the receiver still existed. The blocked run's select is `biased` toward shutdown and cancellation, so a release handed over at that instant is never taken: the run counted itself cancelled while the releaser had already counted it released, leaving one invocation in both buckets with `completed` unchanged and `release-blocked-call` answered `accepted: true` for a run that never resumed. The release branch is now the only place a release is counted, and it acknowledges consumption back to the releaser after updating the counters. A release that lost the race is offered to the next waiting invocation instead of being counted, so `accepted` means the run actually resumed. --- .../mc-module/examples/direct_host_fixture.rs | 55 +++++++++++++------ 1 file changed, 38 insertions(+), 17 deletions(-) diff --git a/crates/mc-module/examples/direct_host_fixture.rs b/crates/mc-module/examples/direct_host_fixture.rs index bbd3f83c7..6ee487c70 100644 --- a/crates/mc-module/examples/direct_host_fixture.rs +++ b/crates/mc-module/examples/direct_host_fixture.rs @@ -83,7 +83,8 @@ mod unix { /// matching `release-blocked-call`. That silently rewrites the fault schedule /// an E2E scenario is asserting against. Addressing a specific invocation /// makes an unconsumed release impossible to mistake for a pending one. - type BlockedQueue = Arc)>>>; + /// The queued value is the ack channel the run uses to confirm it resumed. + type BlockedQueue = Arc>)>>>; struct ControlledBackend { next: Mutex, @@ -108,12 +109,16 @@ mod unix { *self.next.lock().expect("fixture backend behavior mutex") = behavior; } - /// Hands one release to the oldest still-waiting blocked invocation. + /// Hands one release to a blocked invocation and waits for it to resume. /// - /// Senders whose invocation already lost the race to cancellation are - /// discarded rather than counted, so a release is reported accepted only - /// when a live run actually received it. - fn release_blocked(&self) -> bool { + /// `oneshot::Sender::send` only proves the receiver still existed, not + /// that the run selected the release branch: the `biased` select prefers + /// shutdown and cancellation, so a release handed over at that instant is + /// never consumed. Counting it here would report one invocation as both + /// released and cancelled and answer `accepted: true` for a run that never + /// resumed. The run therefore acknowledges consumption, and a release that + /// lost the race moves on to the next waiting invocation. + async fn release_blocked(&self) -> bool { loop { let Some((_id, sender)) = self .blocked @@ -123,11 +128,17 @@ mod unix { else { return false; }; - if sender.send(()).is_ok() { - take_blocked_slot(&self.counters); - self.counters.released.fetch_add(1, Ordering::SeqCst); + let (ack, resumed) = oneshot::channel(); + if sender.send(ack).is_err() { + // The invocation ended before the handoff; it already + // withdrew itself and counted its own outcome. + continue; + } + if resumed.await.is_ok() { return true; } + // Handed over but not consumed: the run lost to cancellation and + // counted itself cancelled. Offer this release to the next one. } } @@ -151,7 +162,10 @@ mod unix { } /// Registers one blocked invocation and returns its id and release channel. - fn register_blocked(queue: &BlockedQueue, next_id: &AtomicU64) -> (u64, oneshot::Receiver<()>) { + fn register_blocked( + queue: &BlockedQueue, + next_id: &AtomicU64, + ) -> (u64, oneshot::Receiver>) { let id = next_id.fetch_add(1, Ordering::SeqCst); let (tx, rx) = oneshot::channel(); queue @@ -222,16 +236,23 @@ mod unix { counters.cancelled.fetch_add(1, Ordering::SeqCst); ControlledBackend::terminal_error("fixture run cancelled") } - released = release => { - // `release_blocked` already accounted this run; - // a dropped sender means the fixture is going - // away, which the cancellation branches own. - if released.is_err() { + granted = release => { + // This branch is the only place a release counts, + // so no invocation can be both released and + // cancelled. A dropped grant means the fixture is + // going away, which the cancellation branches own. + let Ok(ack) = granted else { withdraw_blocked(&blocked, id); take_blocked_slot(&counters); counters.cancelled.fetch_add(1, Ordering::SeqCst); return ControlledBackend::terminal_error("fixture release dropped"); - } + }; + take_blocked_slot(&counters); + counters.released.fetch_add(1, Ordering::SeqCst); + // Acknowledged after the accounting, so a + // releaser that observes the ack also observes + // the counters. + let _ = ack.send(()); events.emit(BackendEvent::AssistantText { text: "fixture-released".to_owned(), finish_reason: None, @@ -432,7 +453,7 @@ mod unix { } ControlCommand::ReleaseBlockedCall => ( ControlResult::Ack { - accepted: backend.release_blocked(), + accepted: backend.release_blocked().await, }, false, ), From 539d73163c4c4e28099a6b557857e8381e6953b0 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 17:57:29 +0000 Subject: [PATCH 28/37] fix(mc-host): reserve queued bytes for control frames MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `send_control` and `encode_data_frame` charged the same 65 MiB queued-byte pool, but they treat exhaustion in opposite ways: a data charge that fails is one caller's local `queued_byte_capacity` error, while a control charge that fails retires the whole generation. Ordinary large request bodies waiting behind writer backpressure could therefore leave fewer than 21 bytes free, and the next keepalive Pong or deadline Cancel would tear down every unrelated route on the connection — a self-inflicted teardown from legitimate traffic rather than any protocol violation. Control frames now draw on their own reservation, sized to exactly the reserved control channel they share a purpose with: every control frame is header-only, so `CLIENT_CONTROL_QUEUE_FRAMES * HEADER_LEN` covers the whole channel and a byte charge can only fail once that channel is already full — the same condition the following `try_send` already retires on. Retiring therefore stays correct while becoming unreachable from data pressure. `data_and_control_charge_one_shared_byte_cap` asserted the shared-pool behaviour that caused this, so it is replaced by `data_saturation_never_starves_a_control_frame`, which saturates the data pool and proves a control frame still queues with the generation intact. --- crates/mc-host/src/client.rs | 70 ++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 18 deletions(-) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 26f7458fd..d46f579be 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -61,8 +61,21 @@ pub const CLIENT_STREAM_QUEUE_ITEMS: usize = 16; pub const CLIENT_DATA_QUEUE_FRAMES: usize = 256; /// Reserved pure-header Pong, Cancel, and Goodbye slots. pub const CLIENT_CONTROL_QUEUE_FRAMES: usize = 32; -/// Shared queued-byte cap charged by both ordinary and reserved control frames. +/// Queued-byte cap for ordinary data frames. +/// +/// Reserved control frames draw on `CLIENT_CONTROL_QUEUED_BYTES` instead, so +/// ordinary traffic cannot starve them: a control charge that failed here retires +/// the whole generation, while a data charge that fails is one caller's local +/// error, and sharing one pool let legitimate large bodies turn into a +/// self-inflicted connection teardown. pub const CLIENT_QUEUED_BYTES: usize = MAX_BODY_LEN as usize + 1_048_576; +/// Queued-byte reservation for the pure-header control frames. +/// +/// Sized to exactly the reserved control channel: every control frame is +/// header-only, so this covers `CLIENT_CONTROL_QUEUE_FRAMES` of them and a byte +/// charge can only fail once that channel is already full — the same condition +/// that retires the generation a few lines later. +pub const CLIENT_CONTROL_QUEUED_BYTES: usize = CLIENT_CONTROL_QUEUE_FRAMES * HEADER_LEN; /// Reservation for the body of the frame the reader is currently decoding. /// /// The wire contract obliges an admitted connection to accept any otherwise @@ -373,6 +386,7 @@ impl Client { streams: Mutex::new(0), routes: Mutex::new(HashSet::new()), queue_budget: Arc::new(ByteCounter::new(CLIENT_QUEUED_BYTES)), + control_budget: Arc::new(ByteCounter::new(CLIENT_CONTROL_QUEUED_BYTES)), read_budget: Arc::new(ByteCounter::new(CLIENT_INBOUND_FRAME_BYTES)), retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), data_tx, @@ -822,6 +836,10 @@ struct Inner { streams: Mutex, routes: Mutex>, queue_budget: Arc, + /// Reserved queued bytes for pure-header control frames, separate from + /// `queue_budget` so ordinary data traffic can never starve a Pong, Cancel, + /// or Goodbye into retiring the generation. + control_budget: Arc, /// Reserved for the body of the one frame the reader is decoding. Separate /// from `retained_budget` so queue retention can never deny an otherwise /// valid inbound frame; see `CLIENT_INBOUND_FRAME_BYTES`. @@ -1208,7 +1226,11 @@ impl Inner { "control encode failed", ) })?; - let charge = self.queue_budget.charge(bytes.len()).ok_or_else(|| { + // The reserved pool, not the shared one: a control charge failing here + // retires the whole generation, so charging it against bytes that ordinary + // requests can legitimately occupy turned a busy connection into a + // self-inflicted teardown. + let charge = self.control_budget.charge(bytes.len()).ok_or_else(|| { self.retire("control_capacity_exhausted"); CallError::local( SendOutcome::Terminal, @@ -2268,6 +2290,7 @@ mod tests { streams: Mutex::new(0), routes: Mutex::new(HashSet::from([route(1), route(2)])), queue_budget: Arc::new(ByteCounter::new(queued_bytes)), + control_budget: Arc::new(ByteCounter::new(CLIENT_CONTROL_QUEUED_BYTES)), read_budget: Arc::new(ByteCounter::new(CLIENT_INBOUND_FRAME_BYTES)), retained_budget: Arc::new(ByteCounter::new(CLIENT_RETAINED_RESPONSE_BYTES)), data_tx, @@ -3206,16 +3229,14 @@ mod tests { } #[tokio::test] - async fn data_and_control_charge_one_shared_byte_cap() { - let (inner, data_rx, control_rx) = test_inner(HEADER_LEN * 2); - inner - .send_control( - FrameType::Pong, - pure_header_flags(), - FrameId::control(1), - None, - ) - .expect("first header"); + async fn data_saturation_never_starves_a_control_frame() { + // Control frames used to charge the same pool as request bodies, and the + // two react to exhaustion in opposite ways: a data charge that fails is + // one caller's local error, while a control charge that fails retires the + // whole generation. Legitimate large bodies sitting in the writer queue + // could therefore make the next keepalive Pong or deadline Cancel tear + // down every unrelated route on the connection. + let (inner, data_rx, mut control_rx) = test_inner(HEADER_LEN); let (kind, _rx) = unary_sender(); inner .admit( @@ -3224,16 +3245,29 @@ mod tests { kind, Instant::now() + Duration::from_secs(1), ) - .expect("data header uses remaining shared bytes"); - assert_eq!(inner.queue_budget.used(), HEADER_LEN * 2); - assert!(inner + .expect("data header fills the whole data budget"); + assert_eq!( + inner.queue_budget.used(), + HEADER_LEN, + "the data pool is now saturated" + ); + + inner .send_control( FrameType::Pong, pure_header_flags(), - FrameId::control(2), - None + FrameId::control(1), + None, ) - .is_err()); + .expect("a reserved control frame does not compete with request bytes"); + assert!( + !inner.retired.load(Ordering::Acquire), + "ordinary traffic must never retire the generation through a starved control frame" + ); + assert_eq!(inner.control_budget.used(), HEADER_LEN); + let queued = control_rx.try_recv().expect("Pong queued"); + assert_eq!(queued.bytes[5], FrameType::Pong as u8); + drop(data_rx); drop(control_rx); assert_eq!(inner.queue_budget.used(), 0); From 248e92c7c7422f70ae8737345dd2db7463a13a6c Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:16:22 +0000 Subject: [PATCH 29/37] fix: require proof a run stopped before fallback, and verify the Cargo workspace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cancellation_confirmed_stopped` treated every terminal cancel error except `teardown_unconfirmed` as proof the provider run had stopped, which is a deny-list where the decision needs positive proof: authorizing fallback starts a second potentially billable run. `Supervisor::cancel` takes its command permit before it calls `run.cancel.cancel()`, so a saturated command semaphore returns terminal `queue_full` while the original run is still executing — and that code passed the deny-list. Only `Ok(())` authorizes fallback now. The supervisor returns it both when it cancelled the run and when the run is already absent, and none of the terminal codes `run.cancel` can actually produce — `queue_full`, `closed`, `teardown_unconfirmed` — prove the run stopped. `terminal_cancel_response_allows_fallback` asserted the opposite using `run_already_terminal`, a code that appears nowhere in the host, so it pinned a contract nothing emits. It is replaced by `a_terminal_cancel_error_never_authorizes_fallback`, which covers the three codes that are actually reachable. `send_outcome` loses its last non-test caller and becomes `pub` alongside the sibling accessors, since whether a request may have reached the host is exactly what a caller needs before retrying. `detectRustModePrereqs` accepted a present `Cargo.toml` plus a working `cargo` as proof the fixture was buildable. The workspace has mandatory `../commons` path dependencies, so a checkout without that sibling passed, bypassed every suite's `skipIf`, and failed inside `buildDirectHostFixture`. It now runs `cargo metadata` and confirms the `direct_host_fixture` example resolves, matching what `scripts/check-rust-prerequisites.ts` already does. --- crates/mc-module/src/historian.rs | 103 ++++++++++-------- crates/mc-module/src/historian_producer.rs | 9 +- .../src/rust-runner/hermetic-mc-host.ts | 50 +++++++++ 3 files changed, 118 insertions(+), 44 deletions(-) diff --git a/crates/mc-module/src/historian.rs b/crates/mc-module/src/historian.rs index 8c22cfb62..3a141cb50 100644 --- a/crates/mc-module/src/historian.rs +++ b/crates/mc-module/src/historian.rs @@ -19,7 +19,7 @@ use mc_store::{ use crate::historian_producer::{ attach_cleanup, ErrorClass, ErrorClassification, HistorianProducer, HistorianProducerError, - HistorianSendOutcome, ProducerOutput, RunHandle, RunState, + ProducerOutput, RunHandle, RunState, }; use crate::historian_validate::{ validate_historian_output, HistorianChunk, HistorianValidationError, StoredCompartmentRange, @@ -1246,14 +1246,20 @@ where log_cleanup_failure(session_id, "close", &producer.close().await); } +/// Whether the cancel attempt proved the provider run is stopped. +/// +/// Authorizing fallback starts a second potentially billable run, so this needs +/// positive proof, not the absence of one known-bad code. `Supervisor::cancel` +/// takes its command permit *before* it calls `run.cancel.cancel()`, so a +/// saturated command semaphore returns a terminal `queue_full` while the provider +/// run is still executing — and treating every terminal code except +/// `teardown_unconfirmed` as proof authorized fallback on exactly that failure. +/// +/// `Ok(())` is the proof: the supervisor returns it when it cancelled the run and +/// when the run is already absent from the index. Every terminal error leaves the +/// run's state unproven, so none of them authorize a second run. fn cancellation_confirmed_stopped(result: &Result<(), HistorianProducerError>) -> bool { - match result { - Ok(()) => true, - Err(error) => { - error.send_outcome() == Some(HistorianSendOutcome::Terminal) - && error.code() != Some("teardown_unconfirmed") - } - } + result.is_ok() } pub async fn run_historian_firing

( @@ -1846,6 +1852,7 @@ mod tests { use mc_store::{ModuleMeta, StoredCompartment}; use crate::ck_wire::{self, CkIngressMessage, CkWireMessage}; + use crate::historian_producer::HistorianSendOutcome; use crate::transform::{transform, ProducerContext, TransformRequest}; fn store(dir: &std::path::Path) -> McStore { @@ -3625,44 +3632,54 @@ mod tests { } } + /// A terminal cancel error never authorizes fallback, because none of the + /// codes `run.cancel` can actually return proves the provider run stopped. + /// `queue_full` is the dangerous one: `Supervisor::cancel` takes its command + /// permit before it cancels, so a saturated command semaphore reports terminal + /// while the run is still executing. Falling back there starts a second + /// billable run beside the first. #[tokio::test] - async fn terminal_cancel_response_allows_fallback() { - let dir = tempfile::tempdir().unwrap(); - let store = store(dir.path()); - seed_prior_compartment(&store); - let chunk = historian_chunk(); - let prior = prior_ranges(); - let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; - let mut producer = ScriptedProducer::default() - .with_start(Ok(run_handle("run-1"))) - .with_output(Err(HistorianProducerError::tagged_call( - "provider_error", - "provider overloaded", - ErrorClass::Transient, - None, - ))) - .with_cancel_result(Err(HistorianProducerError::Call( - crate::historian_producer::HistorianCallFailure::untagged( - HistorianSendOutcome::Terminal, - "run_already_terminal", - "run is already stopped", - ), - ))) - .with_start(Ok(run_handle("run-2"))) - .with_output(Ok(producer_output(historian_xml("fallback output")))); + async fn a_terminal_cancel_error_never_authorizes_fallback() { + for code in ["queue_full", "closed", "teardown_unconfirmed"] { + let dir = tempfile::tempdir().unwrap(); + let store = store(dir.path()); + seed_prior_compartment(&store); + let chunk = historian_chunk(); + let prior = prior_ranges(); + let models = vec!["prov/model-a".to_string(), "prov/model-b".to_string()]; + // Only ONE start is scripted: reaching for model-b would panic the + // scripted queue, so completing at all proves the chain stopped. + let mut producer = ScriptedProducer::default() + .with_start(Ok(run_handle("run-1"))) + .with_output(Err(HistorianProducerError::tagged_call( + "provider_error", + "provider overloaded", + ErrorClass::Transient, + None, + ))) + .with_cancel_result(Err(HistorianProducerError::Call( + crate::historian_producer::HistorianCallFailure::untagged( + HistorianSendOutcome::Terminal, + code, + "cancel did not prove the run stopped", + ), + ))); - let outcome = run_historian_firing( - &mut producer, - fire_request(&store, "placeholder prompt", &models, &chunk, &prior), - ) - .await - .expect("terminal cancel response proves fallback is safe"); + let err = run_historian_firing( + &mut producer, + fire_request(&store, "placeholder prompt", &models, &chunk, &prior), + ) + .await + .unwrap_err(); - let HistorianDriveOutcome::Completed(success) = outcome else { - panic!("expected fallback completion"); - }; - assert_eq!(success.model, "prov/model-b"); - assert_eq!(producer.observed_starts.len(), 2); + assert!(matches!(err, HistorianDriveError::Producer(_))); + assert_eq!(producer.cancels, vec!["run-1"]); + assert_eq!( + producer.observed_starts.len(), + 1, + "a terminal `{code}` cancel must not start a second billable run" + ); + } } #[tokio::test] diff --git a/crates/mc-module/src/historian_producer.rs b/crates/mc-module/src/historian_producer.rs index 159b9d775..bc2ca4882 100644 --- a/crates/mc-module/src/historian_producer.rs +++ b/crates/mc-module/src/historian_producer.rs @@ -348,7 +348,14 @@ impl HistorianProducerError { } } - pub(crate) fn send_outcome(&self) -> Option { + /// The send classification, when this failure carries one. + /// + /// `None` means the failure describes something other than a send attempt — + /// a transport or client error — so a consumer must not read it as "not + /// sent". Public alongside `code`, `classification`, and the other + /// accessors: whether a request may have reached the host is exactly what a + /// caller needs before deciding to retry. + pub fn send_outcome(&self) -> Option { match self { Self::Call(failure) => Some(failure.outcome), Self::CleanupFailed { diff --git a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts index 50cdec62b..28cb35d4a 100644 --- a/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts +++ b/packages/e2e-tests/src/rust-runner/hermetic-mc-host.ts @@ -251,6 +251,56 @@ export function detectRustModePrereqs(): RustModePrereqs { if (cargo.error || cargo.status !== 0) { return { ok: false, skipReason: "cargo is not available on PATH" }; } + // A present `Cargo.toml` and a working `cargo` do not make the fixture + // buildable: the workspace has mandatory `../commons` path dependencies, so a + // checkout without that sibling resolves nothing. Without this, such a + // checkout reported the prerequisites as met, bypassed every suite's `skipIf`, + // and failed inside `buildDirectHostFixture` instead of skipping. `cargo + // metadata` resolves the whole workspace and names the target we build, which + // is what `scripts/check-rust-prerequisites.ts` already does. + const metadata = spawnSync( + "cargo", + [ + "metadata", + "--no-deps", + "--format-version", + "1", + "--manifest-path", + join(REPO_ROOT, "Cargo.toml"), + ], + { encoding: "utf8" }, + ); + if (metadata.error || metadata.status !== 0 || typeof metadata.stdout !== "string") { + return { + ok: false, + skipReason: "cargo workspace does not resolve (missing path dependencies?)", + }; + } + let fixtureAvailable = false; + try { + const parsed = JSON.parse(metadata.stdout) as { + packages?: Array<{ + name?: string; + targets?: Array<{ name?: string; kind?: string[] }>; + }>; + }; + fixtureAvailable = + parsed.packages + ?.find((pkg) => pkg.name === "mc-module") + ?.targets?.some( + (target) => + target.name === "direct_host_fixture" && + target.kind?.includes("example"), + ) === true; + } catch { + fixtureAvailable = false; + } + if (!fixtureAvailable) { + return { + ok: false, + skipReason: "direct_host_fixture example is unavailable in this workspace", + }; + } return { ok: true }; } From a743fb665ede7ddd59955391911480ef8ae7aac9 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:29:28 +0000 Subject: [PATCH 30/37] fix: declare the module's retained bytes and stop leaking launch identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `McHandler::resources()` returned a default declaration, so `retained_resident_bytes` was zero while the component holds the 768 MiB transform-serving cache budget, the 64 MiB snapshot cache, the 16 MiB boundary-token cache, 32 MiB of staged state-import bytes, and two 64 MiB process-global tag caches. The runtime kept offering those same bytes to ingress, so `max_resident_bytes` did not bound the process. It is now declared per retention class, so a cache whose budget changes cannot fall out of the total. Sizing moves to the composition site. The default ceiling used to pre-add Broca's declaration, which only worked because Broca lives inside `mc-host`: a default cannot name an external component's declaration, so any composite linking `McHandler` under-sized itself. Only the composite knows which components are linked, so the default is now the no-retention ingress floor and the fixture sums the declarations of what it actually links. Startup already refuses an under-sized composite, which is now the enforcement rather than an accident. The E2E child-env loop also stopped stripping `SUBC_MODULE_ID` and `SUBC_LAUNCH_NONCE`. Both are still live — `mc_host::wire` defines them and `historian_producer` reads them into every route identity — so a test process launched under a supervisor that sets them made the plugin present that identity to the hermetic host, which rejects it. Also points the transport-overhead doc at the renamed probe script and its current client type. --- crates/mc-host/src/config.rs | 16 +++---- crates/mc-host/tests/handler_contract.rs | 42 ++++++++++++++----- .../mc-module/examples/direct_host_fixture.rs | 12 ++++++ crates/mc-module/src/lib.rs | 26 +++++++++++- crates/mc-module/src/transform.rs | 6 +++ ...rust-mode-transport-overhead-2026-08-10.md | 4 +- .../e2e-tests/src/opencode-runner/spawn.ts | 12 ++++++ 7 files changed, 97 insertions(+), 21 deletions(-) diff --git a/crates/mc-host/src/config.rs b/crates/mc-host/src/config.rs index 5c173cfef..a39d19392 100644 --- a/crates/mc-host/src/config.rs +++ b/crates/mc-host/src/config.rs @@ -114,14 +114,14 @@ impl Default for HostLimits { max_routes: 1024, max_pending_requests: 1024, max_handler_tasks: 256, - // 256 MiB served the two-component profile; the third (Broca) - // component's declared retained reservation is subtracted from - // ingress by the runtime, so the default grows by exactly that - // whole declaration — supervisor budget plus route-map and - // backend-capture headroom — to preserve the former ingress - // headroom. - max_resident_bytes: 256 * 1024 * 1024 - + crate::broca::config::DECLARED_RETAINED_RESIDENT_BYTES, + // Sized for a host whose components declare no retention. The + // runtime subtracts the catalog and every declared + // `retained_resident_bytes` from this figure, and a default cannot + // know which components a composite links — so a composition site + // that links components with real retention must size this itself as + // its own floor plus the sum of their declarations. Startup refuses + // the composite otherwise rather than silently over-offering ingress. + max_resident_bytes: 256 * 1024 * 1024, writer_queue_frames: 64, } } diff --git a/crates/mc-host/tests/handler_contract.rs b/crates/mc-host/tests/handler_contract.rs index 1ec0ea2c8..3615e20eb 100644 --- a/crates/mc-host/tests/handler_contract.rs +++ b/crates/mc-host/tests/handler_contract.rs @@ -485,13 +485,18 @@ async fn retained_declaration_raises_the_resident_floor_exactly() { host.shutdown().await.expect("graceful shutdown"); } -/// The default resident cap absorbs the whole Broca declaration: it is the -/// former two-component 256 MiB default plus exactly the declared retained -/// reservation (the 64 MiB supervisor budget plus the route-map and -/// backend-capture headroom), so ingress headroom is preserved, and the -/// default-limit three-component host starts. +/// The default resident cap is the no-retention ingress floor, and a composite +/// that links components with real retention must size the ceiling itself. +/// +/// The default used to pre-add Broca's declaration, which only worked because +/// Broca lives inside this crate: a default cannot name the declaration of an +/// external component such as `mc_module::McHandler`, so every composite that +/// linked one silently under-sized its ceiling. The knowledge of which +/// components are linked lives at the composition site, so the number does too — +/// and startup refuses an under-sized composite rather than over-offering +/// ingress. #[tokio::test] -async fn the_default_resident_cap_absorbs_the_broca_reservation() { +async fn a_composite_sizes_the_resident_cap_from_its_own_declarations() { const RETAINED: u64 = 64 * 1024 * 1024 + 1024 * (4096 + 256 + 128) + 8 * ((4 * 1024 * 1024 + 64 * 1024) * 5 + 512 * 1024) @@ -500,19 +505,36 @@ async fn the_default_resident_cap_absorbs_the_broca_reservation() { + 3 * 8 * (96 * 1024 + 8 * 1024); let defaults = HostLimits::default(); assert_eq!( - defaults.max_resident_bytes - RETAINED, + defaults.max_resident_bytes, 256 * 1024 * 1024, - "the default grew by exactly the declared retained reservation" + "the default carries no component's retention" ); - let host = CompositeTestHost::start( + // Defaults alone cannot hold a declaring component: startup must refuse it + // rather than hand ingress bytes the component is already holding. + let refused = CompositeTestHost::try_start( three_child_composite(broca_declaration(RETAINED)), |config| { - // Default limits: the production declaration must fit them. config.limits = HostLimits::default(); }, ) .await; + assert!( + refused.is_err(), + "a declaration the ceiling cannot cover must fail startup" + ); + + // Sized at the composition site, the same composite starts. + let host = CompositeTestHost::start( + three_child_composite(broca_declaration(RETAINED)), + |config| { + config.limits = HostLimits { + max_resident_bytes: HostLimits::default().max_resident_bytes + RETAINED, + ..HostLimits::default() + }; + }, + ) + .await; host.shutdown().await.expect("graceful shutdown"); } diff --git a/crates/mc-module/examples/direct_host_fixture.rs b/crates/mc-module/examples/direct_host_fixture.rs index 6ee487c70..3de9643dd 100644 --- a/crates/mc-module/examples/direct_host_fixture.rs +++ b/crates/mc-module/examples/direct_host_fixture.rs @@ -639,6 +639,18 @@ mod unix { data_dir: Some(root.clone()), daemon_ver: "mc-module/direct-host-fixture".to_owned(), init: storage_init(&root), + limits: mc_host::HostLimits { + // This composite is the only place that knows which components + // are linked, so it is the only place that can size the ceiling: + // the default ingress floor plus every linked component's + // declared retention. The runtime subtracts those declarations + // from ingress, so omitting one would either starve ingress or + // fail startup. + max_resident_bytes: mc_host::HostLimits::default().max_resident_bytes + + mc_module::DECLARED_RETAINED_RESIDENT_BYTES + + mc_host::broca::config::DECLARED_RETAINED_RESIDENT_BYTES, + ..mc_host::HostLimits::default() + }, ..Default::default() }; let host_shutdown = shutdown.clone(); diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index 1ebc50dd7..fe5a5dd7e 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -2437,6 +2437,27 @@ const _: () = assert!( <= TRANSFORM_SERVE_CACHE_COMBINED_BUDGET_BYTES ); +/// Every resident byte this component retains for the whole incarnation, +/// declared to the host through [`ResourceDeclaration::retained_resident_bytes`]. +/// +/// The host subtracts this from the ingress admission pool, so +/// `max_resident_bytes` only bounds the process when this number is truthful. +/// Declaring zero left the transform-serving caches, the snapshot cache, the +/// boundary-token cache, and staged state-import bytes outside the accounting +/// entirely: the runtime kept offering those same bytes to ingress while the +/// caches held them. +/// +/// Enumerated per retention class rather than as one total, so a cache whose +/// budget changes cannot silently fall out of the declaration. Broca declares +/// its own retention the same way; each component owns its own number, and the +/// composition site sums them when it sizes `max_resident_bytes`. +pub const DECLARED_RETAINED_RESIDENT_BYTES: u64 = TRANSFORM_SERVE_CACHE_COMBINED_BUDGET_BYTES + as u64 + + TRANSFORM_SNAPSHOT_BUDGET_BYTES as u64 + + BOUNDARY_TOKEN_CACHE_BUDGET_BYTES as u64 + + STATE_IMPORT_MAX_STAGED_BYTES as u64 + + transform::TAG_CACHE_COMBINED_BUDGET_BYTES as u64; + #[derive(Debug, Clone)] struct NativeDeltaFrontier { after: String, @@ -12418,7 +12439,10 @@ impl CompositeComponent for McHandler { } fn resources(&self) -> ResourceDeclaration { - ResourceDeclaration::default() + ResourceDeclaration { + retained_resident_bytes: DECLARED_RETAINED_RESIDENT_BYTES, + ..ResourceDeclaration::default() + } } async fn bind(&self, route: RouteHandle, identity: RouteIdentity) -> BindOutcome { diff --git a/crates/mc-module/src/transform.rs b/crates/mc-module/src/transform.rs index 5d84555ee..03636870b 100644 --- a/crates/mc-module/src/transform.rs +++ b/crates/mc-module/src/transform.rs @@ -141,6 +141,12 @@ pub(crate) const SERIALIZED_OUTPUT_CACHE_BUDGET_BYTES: usize = 256 * 1024 * 1024 const TAG_BASELINE_CACHE_BUDGET_BYTES: usize = 64 * 1024 * 1024; const TAG_MINT_FRONTIER_CACHE_BUDGET_BYTES: usize = 64 * 1024 * 1024; +/// Both tag caches are process-global `OnceLock` singletons, so their retention +/// is per-process rather than per-handler. Summed here so the component's +/// declaration to the host cannot drift from the budgets actually enforced. +pub(crate) const TAG_CACHE_COMBINED_BUDGET_BYTES: usize = + TAG_BASELINE_CACHE_BUDGET_BYTES + TAG_MINT_FRONTIER_CACHE_BUDGET_BYTES; + /// One served CK message plus the canonical bytes used by the module response writer. /// The typed value stays behind an `Arc`, so a cache hit does not clone large tool output trees. #[derive(Debug, Clone)] diff --git a/docs/rust-mode-transport-overhead-2026-08-10.md b/docs/rust-mode-transport-overhead-2026-08-10.md index 86cf9f5e9..47f64085e 100644 --- a/docs/rust-mode-transport-overhead-2026-08-10.md +++ b/docs/rust-mode-transport-overhead-2026-08-10.md @@ -20,7 +20,7 @@ An idle standalone probe puts the complete client → daemon → module echo → ## Measurement design -`packages/plugin/scripts/probe-subc-transport.ts` is deliberately one long-lived Bun process using one established `SubcClient` connection and one reused route. Every sweep arm warms the route five times and then times 50 individual sequential round-trips. It does **not** spawn a process per sample; process startup would add a roughly 60 ms floor with about 90 ms spread and make this discriminator blind. Connection, route-open, and route-close times are reported separately. +`packages/plugin/scripts/probe-mc-host-transport.ts` is deliberately one long-lived Bun process using one established `McHostClient` connection and one reused route. Every sweep arm warms the route five times and then times 50 individual sequential round-trips. It does **not** spawn a process per sample; process startup would add a roughly 60 ms floor with about 90 ms spread and make this discriminator blind. Connection, route-open, and route-close times are reported separately. The sweep uses the production request settings (`Priority.Background`, `AdmissionClass.Normal`) and exact serialized request-body sizes of 1, 4, 8, 16, and 32 KiB. A small interactive health arm is a control. The script also instruments `SubcModuleTransport` only within the probe process to timestamp its global correctness FIFO before enqueue, after dequeue, and after response. No production source or live durable module state is changed; `health`, `echo`, and read-only `session.status` are used. @@ -28,7 +28,7 @@ Command: ```sh cd packages/plugin -bun scripts/probe-subc-transport.ts +bun scripts/probe-mc-host-transport.ts ``` Live run: 2026-08-10T09:42:06Z, daemon 0.3.0 on loopback TCP, 50 samples per arm. diff --git a/packages/e2e-tests/src/opencode-runner/spawn.ts b/packages/e2e-tests/src/opencode-runner/spawn.ts index 95b441849..1d01edde8 100644 --- a/packages/e2e-tests/src/opencode-runner/spawn.ts +++ b/packages/e2e-tests/src/opencode-runner/spawn.ts @@ -447,6 +447,18 @@ async function spawnOpencodeWithProvision( if (key === "OPENCODE_SERVER_PASSWORD") continue; if (key === "OPENCODE_SERVER_USERNAME") continue; if (key === "NODE_ENV") continue; + // Strip any inherited supervised-launch identity. These are still the + // live variable names (`mc_host::wire::SUBC_MODULE_ID_ENV` / + // `SUBC_LAUNCH_NONCE_ENV`), and `historian_producer` reads them into + // `consumer_module_id`/`consumer_launch_nonce` on every route identity. + // When the test process is itself launched under a supervisor that sets + // them, the plugin would present THAT identity to our hermetic host, + // which rejects it as not matching a supervised launch nonce. A real + // install is never launched under a supervised identity, so clearing + // them matches production. Harmless for TS-mode suites, which never + // reach the Rust client. + if (key === "SUBC_MODULE_ID") continue; + if (key === "SUBC_LAUNCH_NONCE") continue; childEnv[key] = value; } childEnv.OPENCODE_CONFIG_DIR = env.configDir; From 0b572dd07a29fd0447975d2aa8045c6a1788f41b Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:34:34 +0000 Subject: [PATCH 31/37] fix(mc-module): charge the request tree before materializing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `handle` parsed every body into a `serde_json::Value` with no reservation while the body's own `InputBuffer` charge was still held. That charge covers wire bytes, not the tree: scalar-dense JSON becomes a node per value, so a body near the 32 MiB transform cap could expand to hundreds of megabytes, and concurrent requests each escaped the resident envelope by their own full expansion. This is the input-side mirror of the output-side reservation already fixed in this branch. The bound is counted from the body rather than assumed as a multiple of it, which is what keeps the gate usable. One string-aware pass counts value separators — only commas and colons outside strings separate values — and adds string bytes separately. A realistic string-heavy transform body therefore reserves close to its true footprint and passes, while a scalar-dense body that genuinely cannot be served inside the envelope is refused rather than silently exceeding it. Exhaustion is classified rather than collapsed: above the ceiling itself is permanent `invalid_params`, since no amount of draining admits it, and a pool currently held by concurrent requests is retryable `queue_full`. `RequestCtx::try_reserve_resident` and `resident_capacity` become public. The scratch pool already existed and Synapse already charged it, but the accessors were crate-private, so no external handler could participate in the accounting the host's envelope depends on. --- crates/mc-host/src/handler.rs | 13 +++- crates/mc-module/src/lib.rs | 142 ++++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+), 2 deletions(-) diff --git a/crates/mc-host/src/handler.rs b/crates/mc-host/src/handler.rs index 0075e0224..7eb7a5f66 100644 --- a/crates/mc-host/src/handler.rs +++ b/crates/mc-host/src/handler.rs @@ -426,14 +426,23 @@ impl RequestCtx { self.stream.reserve_direct(exact_len, serializer).await } - pub(crate) fn try_reserve_resident(&self, bytes: usize) -> Option { + /// Reserves resident bytes for request-derived state the handler is about to + /// allocate — parse scratch, an owned tree decoded from `body`, anything whose + /// lifetime is the request rather than the response. + /// + /// `None` means the pool cannot cover it right now. Charge BEFORE allocating: + /// a reservation taken afterwards has already let the allocation escape the + /// envelope, and concurrent requests each escape by their own full amount. + /// The returned charge releases on drop, so hold it for as long as the bytes + /// are live. + pub fn try_reserve_resident(&self, bytes: usize) -> Option { self.scratch.try_charge(bytes) } /// The resident ceiling `try_reserve_resident` is measured against. A /// reservation above this can never be acquired, so callers report it /// as a permanent rejection instead of retryable backpressure. - pub(crate) fn resident_capacity(&self) -> usize { + pub fn resident_capacity(&self) -> usize { self.scratch.capacity() } diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index fe5a5dd7e..3a1942899 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -12465,6 +12465,30 @@ impl CompositeComponent for McHandler { if let Err(outcome) = enforce_request_byte_cap(ctx.body.as_slice()) { return settle_prepared(&ctx, outcome).await; } + // Charge the tree BEFORE materializing it. `from_slice::` turns a + // body into a node per value plus an owned `String` per string, which for + // scalar-dense JSON is many times the wire bytes — and the body's own + // ingress charge does not cover any of it. Reserved afterwards, every + // concurrent request would already have escaped the resident envelope by + // its own full expansion. + // + // The bound is counted from the body rather than assumed as a multiple of + // it, mirroring what the response path does with `measure`: a realistic + // string-heavy transform body reserves close to its true footprint, while + // a scalar-dense body that genuinely cannot be served inside the envelope + // is refused instead of silently exceeding it. + let Some(footprint) = value_footprint_bound(ctx.body.as_slice()) else { + return settle_prepared(&ctx, request_too_large_error()).await; + }; + let _parse_charge = match ctx.try_reserve_resident(footprint) { + Some(charge) => charge, + None if footprint > ctx.resident_capacity() => { + // Above the ceiling itself: no amount of draining admits it, so + // this is permanent rather than backpressure. + return settle_prepared(&ctx, request_too_large_error()).await; + } + None => return settle_prepared(&ctx, resident_capacity_error()).await, + }; let request = serde_json::from_slice::(ctx.body.as_slice()).unwrap_or(Value::Null); let inbound_bytes = ctx.body.len(); let outcome = self @@ -14747,6 +14771,70 @@ impl RequestMethodProbe { /// legitimately carry a session's full message array (multi-MiB on large /// sessions), so they get the wider cap. Method sniffing on raw bytes avoids /// parsing multi-MiB JSON just to reject it. +/// Slack on the counted node storage, covering `Vec` and map growth: both double +/// as they fill, so the live allocation can reach twice the storage the final +/// element count needs. +const VALUE_NODE_SLACK: usize = 2; + +/// Fixed headroom for the root value, the deserializer's own scratch, and the +/// small allocations that do not scale with the body. +const VALUE_ENVELOPE_BYTES: usize = 4096; + +/// Upper bound on the heap a `serde_json::Value` tree decoded from `body` can +/// occupy. `None` on arithmetic overflow, which callers treat as unsatisfiable. +/// +/// Counted, not assumed: one pass classifies every byte as inside or outside a +/// string, because only outside-string `,` and `:` separate values. A document +/// holds at most one root value, plus one per outside-string comma (array +/// elements and object members), plus one per outside-string colon (member +/// values) — which bounds the node count without building anything. String bytes +/// are added separately, since each string becomes an owned `String`. +fn value_footprint_bound(body: &[u8]) -> Option { + let mut nodes: usize = 1; + let mut string_bytes: usize = 0; + let mut in_string = false; + let mut escaped = false; + for &byte in body { + if in_string { + string_bytes += 1; + if escaped { + escaped = false; + } else if byte == b'\\' { + escaped = true; + } else if byte == b'"' { + in_string = false; + } + continue; + } + match byte { + b'"' => in_string = true, + b',' | b':' => nodes += 1, + _ => {} + } + } + nodes + .checked_mul(std::mem::size_of::())? + .checked_mul(VALUE_NODE_SLACK)? + .checked_add(string_bytes)? + .checked_add(VALUE_ENVELOPE_BYTES) +} + +/// Permanent: the tree cannot fit this host's resident ceiling at any load. +fn request_too_large_error() -> PreparedOutcome { + PreparedOutcome::Error { + code: "invalid_params".to_string(), + message: "request body needs more resident bytes than this host can hold".to_string(), + } +} + +/// Retryable: the ceiling could hold it, but concurrent requests hold it now. +fn resident_capacity_error() -> PreparedOutcome { + PreparedOutcome::Error { + code: "queue_full".to_string(), + message: "resident capacity for request parsing is exhausted".to_string(), + } +} + fn enforce_request_byte_cap(body: &[u8]) -> Result<(), PreparedOutcome> { if body.len() <= MAX_FACADE_FRAME_BYTES { return Ok(()); @@ -18098,6 +18186,60 @@ mod tests { ); } + #[test] + fn value_footprint_counts_nodes_outside_strings_only() { + // The string-awareness is the load-bearing part. Punctuation inside a + // string separates no values, so counting it would inflate the bound + // until ordinary string-heavy transform bodies were refused. + let quoted = br#"{"a":"x,y,z:w,,,::"}"#; + let bare = br#"{"a":1,"b":2,"c":3}"#; + assert!( + value_footprint_bound(quoted).unwrap() < value_footprint_bound(bare).unwrap(), + "commas and colons inside a string must not count as value separators" + ); + + // An escaped quote does not end the string, so the rest stays inside it. + let escaped = br#"{"a":"he said \"x,y,z\" ok"}"#; + let node_cost = std::mem::size_of::() * VALUE_NODE_SLACK; + assert!( + value_footprint_bound(escaped).unwrap() + < VALUE_ENVELOPE_BYTES + escaped.len() + 4 * node_cost, + "an escaped quote must not drop the scan out of the string" + ); + } + + #[test] + fn scalar_dense_bodies_bound_far_above_their_wire_size() { + // This is the case the charge exists for: every two wire bytes become a + // whole `Value` node, so the tree dwarfs the body that the ingress + // charge covered. + let dense: Vec = { + let mut body = Vec::from(b"[1".as_slice()); + for _ in 0..10_000 { + body.extend_from_slice(b",1"); + } + body.push(b']'); + body + }; + let bound = value_footprint_bound(&dense).expect("bound fits usize"); + assert!( + bound > dense.len() * 8, + "a scalar-dense body must bound far above its wire size, got {bound} for {} bytes", + dense.len() + ); + + // A string-heavy body of the same length bounds near its wire size, so + // realistic transform traffic is not refused by this gate. + let mut stringy = Vec::from(b"[\"".as_slice()); + stringy.extend(std::iter::repeat_n(b'x', dense.len())); + stringy.extend_from_slice(b"\"]"); + let stringy_bound = value_footprint_bound(&stringy).expect("bound fits usize"); + assert!( + stringy_bound < stringy.len() * 2, + "string bytes must not be charged as nodes, got {stringy_bound}" + ); + } + #[test] fn transform_snapshot_cache_is_generation_safe_and_lru_bounded() { let request = |session_id: &str| { From b4f3f323f4752cbe02b6033a356157e9c3cc5759 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:39:00 +0000 Subject: [PATCH 32/37] refactor(mc-host): single-source path hardening and frame charge admission MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two copies of the directory-traversal hardening existed: `connection_file`'s read-only `open_parent` and `instance`'s create-and-tighten `secure_runtime_dir`. The walks themselves are genuinely different jobs, so merging them would have meant flag parameters for "create or not" and "tighten or not". What was actually duplicated is the hardening itself — the open flags, the anchor open plus its ancestor-safety proof, and the rule for which path components may be walked — and that is the part where a missed variant reopens a traversal hole. Those three now live once beside `is_safe_ancestor`, which both already shared, and both callers use them. `normal_components` also removes a spelling difference: one copy matched all four `Component` variants explicitly while the other rejected non-`Normal` components by falling through a guard. Same effective rule, two encodings, no way for the compiler to notice if one changed. The charge-or-cancel interaction in `dispatch` was written out five times — unary responses, error terminals, stream reservations, direct stream sends, and the shutdown ack — each deciding by hand that a cancellation loses the frame and an admission timeout cancels the generation. `charge_frame_or_cancel` states that once, including why the two outcomes are not the same event: a cancellation means the generation is already going, while a timeout proves the writer is not draining and so must tear it down instead of accumulating waiters behind a stalled socket. --- crates/mc-host/src/connection_file.rs | 46 +++++----- crates/mc-host/src/dispatch.rs | 117 ++++++++++++++------------ crates/mc-host/src/instance.rs | 85 ++++++++++++------- 3 files changed, 141 insertions(+), 107 deletions(-) diff --git a/crates/mc-host/src/connection_file.rs b/crates/mc-host/src/connection_file.rs index b12afbb62..a034f71fe 100644 --- a/crates/mc-host/src/connection_file.rs +++ b/crates/mc-host/src/connection_file.rs @@ -9,12 +9,12 @@ use std::{ error::Error, ffi::OsString, fmt, io, - path::{Component, Path, PathBuf}, + path::{Path, PathBuf}, }; use rustix::{ fd::OwnedFd, - fs::{openat, Mode, OFlags, CWD}, + fs::{openat, Mode, OFlags}, }; use serde::{Deserialize, Serialize}; @@ -243,27 +243,31 @@ fn open_parent(path: &Path) -> Result<(OwnedFd, OsString), ConnectionFileError> .ok_or_else(|| ConnectionFileError::InvalidPath { path: path.to_path_buf(), })?; - let flags = OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::RDONLY | OFlags::CLOEXEC; - let mut current = openat( - CWD, - if parent.is_absolute() { "/" } else { "." }, - flags, - Mode::empty(), - ) - .map_err(|source| io_error("open_anchor", path, source.into()))?; + // Anchor open, ancestor-safety proof, and component classification are shared + // with `instance::secure_runtime_dir`. The walks themselves are not: that one + // creates and tightens the host's own runtime directory, while this one is + // read-only discovery. Only the hardening rules are common, and those are the + // part that must never drift. + let mut current = crate::instance::open_safe_anchor(path) + .map_err(|source| io_error("open_anchor", path, source.into()))? + .ok_or_else(|| ConnectionFileError::Insecure { + path: path.to_path_buf(), + })?; validate_directory(¤t, path, false)?; - for component in parent.components() { - let Component::Normal(component) = component else { - if matches!(component, Component::RootDir | Component::CurDir) { - continue; - } - return Err(ConnectionFileError::InvalidPath { - path: path.to_path_buf(), - }); - }; - current = openat(¤t, component, flags, Mode::empty()) - .map_err(|source| io_error("open_parent", path, source.into()))?; + let components = crate::instance::normal_components(parent).ok_or_else(|| { + ConnectionFileError::InvalidPath { + path: path.to_path_buf(), + } + })?; + for component in components { + current = openat( + ¤t, + component, + crate::instance::HARDENED_DIR_FLAGS, + Mode::empty(), + ) + .map_err(|source| io_error("open_parent", path, source.into()))?; validate_directory(¤t, path, false)?; } validate_directory(¤t, path, true)?; diff --git a/crates/mc-host/src/dispatch.rs b/crates/mc-host/src/dispatch.rs index da97f6538..a97340bd7 100644 --- a/crates/mc-host/src/dispatch.rs +++ b/crates/mc-host/src/dispatch.rs @@ -97,6 +97,49 @@ fn escaped_json_len(s: &str) -> usize { } /// Builds an error terminal body under a pre-acquired egress reservation. +/// Charges `bytes` of frame budget, or gives up and tears the generation down. +/// +/// `None` means abandon the frame. Two ways to get there, and they are not the +/// same event: a cancellation won the race, in which case the generation is +/// already going away; or admission outlived the writer's deadline, which proves +/// the writer is not draining, so the generation is cancelled here rather than +/// left accumulating waiters behind a stalled socket. +/// +/// `also_cancelled` is the request-scoped token where one exists (stream paths +/// watch both it and the generation's). +/// +/// Single-sourced because five call sites — unary responses, error terminals, +/// stream reservations, direct stream sends, and the shutdown ack — each encoded +/// this interaction by hand. A correctness fix applied to one and missed in +/// another would leave those five paths with different cancellation semantics, +/// which is exactly the kind of divergence nothing in the type system catches. +async fn charge_frame_or_cancel( + budget: &crate::wire::ByteBudget, + generation: &GenerationCore, + bytes: u32, + deadline: Instant, + also_cancelled: Option<&CancellationToken>, +) -> Option { + let request_cancelled = async { + match also_cancelled { + Some(token) => token.cancelled().await, + None => std::future::pending().await, + } + }; + tokio::select! { + biased; + () = request_cancelled => None, + () = generation.token.cancelled() => None, + charge = timeout_at(deadline, budget.charge(bytes)) => match charge { + Ok(charge) => Some(charge), + Err(_) => { + generation.token.cancel(); + None + } + }, + } +} + /// The encoded size is computed exactly from the escaped field lengths, so /// the charge exists BEFORE the body materializes — concurrent handler /// errors wait on the budget as bytes, not as retained encoded buffers. @@ -127,17 +170,9 @@ async fn charged_error_body( } let frame_bytes = u32::try_from(body_len + HEADER_LEN).map_err(|_| ())?; let deadline = gen.writer.admission_deadline(); - let charge = tokio::select! { - biased; - () = gen.token.cancelled() => return Err(()), - charge = tokio::time::timeout_at(deadline, budget.charge(frame_bytes)) => match charge { - Ok(charge) => charge, - Err(_) => { - gen.token.cancel(); - return Err(()); - } - }, - }; + let charge = charge_frame_or_cancel(budget, gen, frame_bytes, deadline, None) + .await + .ok_or(())?; let body = error_body_json_into( // Header spare capacity up front: an exactly sized buffer would force // `encode_owned_frame`'s reserve to reallocate, transiently retaining @@ -200,17 +235,9 @@ pub async fn emit_frame( crate::wire::ByteCharge::none() } else { let frame_bytes = u32::try_from(body.len() + HEADER_LEN).map_err(|_| ())?; - tokio::select! { - biased; - () = gen.token.cancelled() => return Err(()), - charge = timeout_at(deadline, budget.charge(frame_bytes)) => match charge { - Ok(charge) => charge, - Err(_) => { - gen.token.cancel(); - return Err(()); - } - }, - } + charge_frame_or_cancel(budget, gen, frame_bytes, deadline, None) + .await + .ok_or(())? }; let (bytes, tail) = crate::wire::encode_split_frame(ty, flags, id, body).map_err(|_| ())?; gen.writer @@ -441,18 +468,10 @@ impl StreamSink { } let bytes = u32::try_from(max_len + HEADER_LEN).map_err(|_| StreamClosed)?; let deadline = self.gen.writer.admission_deadline(); - let charge = tokio::select! { - biased; - () = self.cancel.cancelled() => return Err(StreamClosed), - () = self.gen.token.cancelled() => return Err(StreamClosed), - charge = timeout_at(deadline, self.budget.charge(bytes)) => match charge { - Ok(charge) => charge, - Err(_) => { - self.gen.token.cancel(); - return Err(StreamClosed); - } - }, - }; + let charge = + charge_frame_or_cancel(&self.budget, &self.gen, bytes, deadline, Some(&self.cancel)) + .await + .ok_or(StreamClosed)?; if self.cancel.is_cancelled() || self.gen.token.is_cancelled() || self.settlement.won.load(Ordering::SeqCst) @@ -480,18 +499,10 @@ impl StreamSink { } let bytes = u32::try_from(exact_len + crate::wire::HEADER_LEN).map_err(|_| StreamClosed)?; let deadline = self.gen.writer.admission_deadline(); - let charge = tokio::select! { - biased; - () = self.cancel.cancelled() => return Err(StreamClosed), - () = self.gen.token.cancelled() => return Err(StreamClosed), - charge = timeout_at(deadline, self.budget.charge(bytes)) => match charge { - Ok(charge) => charge, - Err(_) => { - self.gen.token.cancel(); - return Err(StreamClosed); - } - }, - }; + let charge = + charge_frame_or_cancel(&self.budget, &self.gen, bytes, deadline, Some(&self.cancel)) + .await + .ok_or(StreamClosed)?; if self.cancel.is_cancelled() || self.gen.token.is_cancelled() || self.settlement.won.load(Ordering::SeqCst) @@ -637,16 +648,10 @@ pub async fn handle_host_shutdown( } let deadline = gen.writer.admission_deadline(); let frame_bytes = u32::try_from(body.len() + HEADER_LEN).expect("fixed-size body"); - let charge = tokio::select! { - biased; - () = gen.token.cancelled() => return, - charge = timeout_at(deadline, shared.egress_budget.charge(frame_bytes)) => match charge { - Ok(charge) => charge, - Err(_) => { - gen.token.cancel(); - return; - } - }, + let Some(charge) = + charge_frame_or_cancel(&shared.egress_budget, gen, frame_bytes, deadline, None).await + else { + return; }; let Ok(bytes) = encode_owned_frame( FrameType::Response, diff --git a/crates/mc-host/src/instance.rs b/crates/mc-host/src/instance.rs index ae503ffec..a009bc7d9 100644 --- a/crates/mc-host/src/instance.rs +++ b/crates/mc-host/src/instance.rs @@ -311,25 +311,13 @@ impl Drop for InstanceGuard { /// normalizing newly created components to 0700. Returns a pinned descriptor /// for the final directory after validating its ownership and mode. pub(crate) fn secure_runtime_dir(dir_path: &Path) -> Result { - let flags = OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::RDONLY | OFlags::CLOEXEC; - let mut current = openat( - CWD, - if dir_path.is_absolute() { "/" } else { "." }, - flags, - Mode::empty(), - ) - .map_err(|e| io_err("open_anchor", dir_path, e))?; - // The anchor is part of the resolved path: for a relative root it is the - // process working directory, which another principal may control and - // could use to replace an otherwise secure first component. - let anchor_stat = - rustix::fs::fstat(¤t).map_err(|e| io_err("fstat_anchor", dir_path, e))?; - if !is_safe_ancestor(&anchor_stat) { - return Err(InstanceError::Insecure { + let flags = HARDENED_DIR_FLAGS; + let mut current = open_safe_anchor(dir_path) + .map_err(|e| io_err("open_anchor", dir_path, e))? + .ok_or_else(|| InstanceError::Insecure { what: "runtime directory ancestor", path: dir_path.to_path_buf(), - }); - } + })?; let mut walked = if dir_path.is_absolute() { PathBuf::from("/") } else { @@ -340,19 +328,10 @@ pub(crate) fn secure_runtime_dir(dir_path: &Path) -> Result continue, - Component::Normal(name) => names.push(name), - Component::ParentDir | Component::Prefix(_) => { - return Err(InstanceError::Insecure { - what: "runtime directory path", - path: dir_path.to_path_buf(), - }); - } - } - } + let names = normal_components(dir_path).ok_or_else(|| InstanceError::Insecure { + what: "runtime directory path", + path: dir_path.to_path_buf(), + })?; let saw_component = !names.is_empty(); let last = names.len().saturating_sub(1); @@ -587,6 +566,52 @@ pub(crate) fn mode_bits(stat: &rustix::fs::Stat) -> u32 { stat.st_mode } +/// Open flags for every hardened directory traversal: a directory, never +/// following a link, read-only, close-on-exec. +/// +/// Shared so a traversal cannot be hardened in one caller and not the other. +pub(crate) const HARDENED_DIR_FLAGS: OFlags = OFlags::DIRECTORY + .union(OFlags::NOFOLLOW) + .union(OFlags::RDONLY) + .union(OFlags::CLOEXEC); + +/// The path components a hardened traversal may walk. +/// +/// `RootDir` and `CurDir` are the anchor, already opened, so they are skipped. +/// Everything else — `ParentDir`, `Prefix` — is refused rather than resolved: +/// walking `..` would let a pathname climb out of the tree the anchor pinned. +/// Single-sourced because a missed variant here is a path-traversal hole, and +/// two copies of this rule can drift apart silently. +pub(crate) fn normal_components(path: &Path) -> Option> { + let mut names = Vec::new(); + for component in path.components() { + match component { + Component::RootDir | Component::CurDir => continue, + Component::Normal(name) => names.push(name), + Component::ParentDir | Component::Prefix(_) => return None, + } + } + Some(names) +} + +/// Opens the anchor a hardened traversal starts from and proves it is not +/// replaceable. +/// +/// The anchor is part of the resolved path: for a relative path it is the +/// process working directory, which another principal may control and could use +/// to replace an otherwise secure first component. `Ok(None)` means the anchor +/// opened but is unsafe, which each caller reports in its own error type. +pub(crate) fn open_safe_anchor(path: &Path) -> Result, rustix::io::Errno> { + let anchor = openat( + CWD, + if path.is_absolute() { "/" } else { "." }, + HARDENED_DIR_FLAGS, + Mode::empty(), + )?; + let stat = rustix::fs::fstat(&anchor)?; + Ok(is_safe_ancestor(&stat).then_some(anchor)) +} + /// A directory no other principal can replace: owned by us or by root, and /// not group/other-writable unless sticky (a sticky directory forbids /// renaming entries you do not own, which is what `/tmp` relies on). From 066debe105911b1c4ddd6ef08a6b08d2482f4fb6 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:44:18 +0000 Subject: [PATCH 33/37] refactor: derive shutdown admission from the token, single-source auth teardown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shutdown was tracked in two primitives that had to be flipped together: `task_admission_open: Mutex` and `cancel: CancellationToken`. Both existing paths happened to agree, but nothing enforced it, so a third path — or an edit to either — could admit a task after cancellation or cancel while new tasks still passed the gate. The token is now the only state. What remains is a gate mutex holding no state, whose sole job is to stop a spawn that already passed the check from landing in a tracker `wait` has stopped watching; that ordering was the real reason a lock was there. `Drop` needs no gate at all, since `&mut self` already excludes a concurrent spawn. `CLIENT_DISCOVERY_SLOTS` rises from 4 to 64. The cap exists to bound how many blocking workers a wedged mount can strand, not to throttle healthy discovery, and `mc-module` now links this crate and dials through the same process-wide pool for its own reconnects — so 4 could serialize a reconnect burst against the handshake deadline where separate processes previously had independent capacity. 64 stays far below Tokio's default 512-thread blocking pool. The auth wrappers keep their four lines of scaffolding, but the error-path teardown policy — bounding shutdown by the same absolute deadline as the handshake — moves into one function both call. Attempting to share the whole wrapper needed a higher-ranked closure bound that Rust cannot infer here, and the scaffolding is shape rather than policy; how a failed handshake tears down is the part that must not diverge between server and client. --- crates/mc-host/src/auth.rs | 23 ++++++++++--- crates/mc-host/src/client.rs | 11 +++++-- crates/mc-module/src/lib.rs | 45 ++++++++++++-------------- crates/mc-module/tests/host_adapter.rs | 9 +++++- 4 files changed, 55 insertions(+), 33 deletions(-) diff --git a/crates/mc-host/src/auth.rs b/crates/mc-host/src/auth.rs index f50d1858a..b53e4d974 100644 --- a/crates/mc-host/src/auth.rs +++ b/crates/mc-host/src/auth.rs @@ -190,6 +190,22 @@ impl Deadline { } } +/// Error-path teardown for either handshake side. +/// +/// Bounded by the SAME absolute deadline as the handshake itself, so a failed +/// attempt — and the unauthenticated-handshake slot it holds — is released +/// promptly instead of waiting out another full budget. +/// +/// The policy lives here rather than in both wrappers: the surrounding four lines +/// of scaffolding are shape, but *how* a failed handshake tears down is the part +/// that must not diverge between server and client. +async fn teardown_failed_handshake(stream: &mut S, deadline: Deadline) +where + S: AsyncWrite + Unpin, +{ + let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; +} + pub async fn authenticate_server( stream: &mut S, key: &[u8], @@ -203,10 +219,7 @@ where let deadline = Deadline::starting_now(deadline)?; let result = authenticate_server_inner(stream, key, daemon_id, daemon_ver, deadline).await; if result.is_err() { - // Bound teardown by the SAME absolute deadline so a failed handshake (and - // the unauthenticated-handshake slot it holds) is released promptly instead - // of waiting out another full budget. - let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; + teardown_failed_handshake(stream, deadline).await; } result } @@ -272,7 +285,7 @@ where let deadline = Deadline::starting_now(deadline)?; let result = authenticate_client_inner(stream, conn, deadline).await; if result.is_err() { - let _ = time::timeout(deadline.remaining_or_zero(), stream.shutdown()).await; + teardown_failed_handshake(stream, deadline).await; } result } diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index d46f579be..2e2aa44e9 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -96,9 +96,14 @@ pub const CLIENT_RETAINED_RESPONSE_BYTES: usize = MAX_BODY_LEN as usize + 1_048_ /// Concurrent connection-file snapshots allowed across this process. /// -/// A snapshot runs on the blocking pool and cannot be cancelled, so this caps how -/// many blocking workers a wedged mount can strand; see `Client::connect`. -const CLIENT_DISCOVERY_SLOTS: usize = 4; +/// The cap exists to bound how many blocking workers a wedged mount can strand, +/// not to throttle healthy discovery: a snapshot on a responsive filesystem +/// completes in microseconds, so contention only appears when the mount is +/// already the problem. Sized well above the connects a process makes at once — +/// `mc-module` links this crate and dials through the same pool for its own +/// reconnects — while staying far below Tokio's default 512-thread blocking pool, +/// so a wedged mount cannot starve unrelated blocking work. +const CLIENT_DISCOVERY_SLOTS: usize = 64; /// Permits for [`CLIENT_DISCOVERY_SLOTS`], held by the blocking closure itself so /// a detached worker still counts against the cap. diff --git a/crates/mc-module/src/lib.rs b/crates/mc-module/src/lib.rs index 3a1942899..f4bd4af75 100644 --- a/crates/mc-module/src/lib.rs +++ b/crates/mc-module/src/lib.rs @@ -2997,7 +2997,14 @@ impl ProjectionCache { pub struct McHandler { store: Arc>>>, store_open: Arc, - task_admission_open: Mutex, + /// Serializes "is the module still accepting tasks?" against shutdown. + /// + /// Holds no state: `cancel` is the single source of truth for whether + /// admission is open. This exists only so the check and the `tasks.spawn` + /// that follows it cannot straddle a shutdown that closes the tracker in + /// between. A second boolean here would be state that must be flipped in + /// lockstep with the token, and nothing would enforce that. + spawn_gate: Mutex<()>, cancel: CancellationToken, tasks: TaskTracker, producer_factory: Arc, @@ -3524,7 +3531,7 @@ impl McHandler { McHandler { store: Arc::new(Mutex::new(None)), store_open: Arc::new(StoreOpenCoordinator::new()), - task_admission_open: Mutex::new(true), + spawn_gate: Mutex::new(()), cancel, tasks: TaskTracker::new(), producer_factory, @@ -3593,11 +3600,8 @@ impl McHandler { F: Future + Send + 'static, T: Send + 'static, { - let admission = self - .task_admission_open - .lock() - .expect("module task admission mutex"); - if !*admission { + let _gate = self.spawn_gate.lock().expect("module spawn gate mutex"); + if self.cancel.is_cancelled() { return None; } Some(self.tasks.spawn(future)) @@ -3612,11 +3616,7 @@ impl McHandler { } fn begin_store_open(&self, descriptor: StorageDescriptor) -> Result<(), InitError> { - if !*self - .task_admission_open - .lock() - .expect("module task admission mutex") - { + if self.cancel.is_cancelled() { return Err(InitError("module task admission is closed".to_owned())); } if self.store().is_some() @@ -3839,7 +3839,7 @@ impl McHandler { McHandler { store: Arc::new(Mutex::new(None)), store_open: Arc::new(StoreOpenCoordinator::new()), - task_admission_open: Mutex::new(true), + spawn_gate: Mutex::new(()), cancel: CancellationToken::new(), tasks: TaskTracker::new(), producer_factory: factory, @@ -12419,11 +12419,10 @@ impl McHandler { impl Drop for McHandler { fn drop(&mut self) { - if let Ok(admission) = self.task_admission_open.get_mut() { - *admission = false; - } - self.tasks.close(); + // `&mut self` excludes any concurrent spawn, so the gate is unnecessary + // here. Cancel first, matching `shutdown`: the token closes admission. self.cancel.cancel(); + self.tasks.close(); } } @@ -12510,14 +12509,13 @@ impl CompositeComponent for McHandler { async fn shutdown(&self) -> Result<(), ShutdownError> { { - let mut admission = self - .task_admission_open - .lock() - .expect("module task admission mutex"); - *admission = false; + // Closing admission and closing the tracker under the gate is what + // stops a spawn that already passed the check from landing in a + // tracker `wait` has stopped watching. + let _gate = self.spawn_gate.lock().expect("module spawn gate mutex"); + self.cancel.cancel(); self.tasks.close(); } - self.cancel.cancel(); self.tasks.wait().await; self.bindings.lock().expect("bindings mutex").clear(); @@ -17623,7 +17621,6 @@ mod tests { assert!(state.exited.load(Ordering::SeqCst)); assert!(handler.cancel.is_cancelled()); assert!(handler.tasks.is_empty()); - assert!(!*handler.task_admission_open.lock().unwrap()); assert!(handler.spawn_module_task(async {}).is_none()); } diff --git a/crates/mc-module/tests/host_adapter.rs b/crates/mc-module/tests/host_adapter.rs index 1dd93ec83..73928ebc5 100644 --- a/crates/mc-module/tests/host_adapter.rs +++ b/crates/mc-module/tests/host_adapter.rs @@ -139,7 +139,14 @@ fn adapter_source_has_one_prepared_outcome_and_tracked_spawn_boundary() { assert!(!production.contains(&removed_client_api)); assert!(!production.contains("tokio::spawn(")); assert!(production.contains("spawn_module_task")); - assert!(production.contains("task_admission_open")); + // Shutdown state has one source of truth: the cancellation token. The gate is + // a critical section, not a second flag — an admission boolean here would have + // to be flipped in lockstep with the token, with nothing enforcing it. + assert!(production.contains("spawn_gate")); + assert!( + !production.contains("task_admission_open"), + "admission must be derived from the cancellation token, not tracked separately" + ); assert!(production.contains("self.tasks.close()")); assert!(production.contains("self.cancel.cancel()")); assert!(production.contains("self.tasks.wait().await")); From 1dbeb523d2c35e13b7130ee5d9377cee2d6848b9 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 18:50:47 +0000 Subject: [PATCH 34/37] refactor(mc-host): share the bounded frame-read mechanics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client reader and the host framing layer each carried their own read-exact/read-body/drain loops, identical apart from the error type they produced. That put the subtle parts in two places: the `biased` select that prefers cancellation over another read, treating a zero-length read as end-of-stream instead of looping, and capping the body read at the frame boundary so a pipelined next header is never consumed as body. `frame_read` now owns those three loops and reports why a read stopped — cancelled, EOF, deadline, or I/O. Each caller keeps its own classification, which is the part that genuinely differs: the host distinguishes a stop inside a frame from one while draining an oversize body so the close reason names the phase, while the client treats every stop as fatal to the generation because it resynchronizes by reconnecting rather than by guessing where the next header begins. The higher-level readers stay separate. They disagree on protocol policy — what lengths are legal, what an oversize control body means, which budget the body is charged against — and folding those together would have meant parameterizing policy instead of sharing mechanics. --- crates/mc-host/src/client.rs | 51 +++------- crates/mc-host/src/frame_read.rs | 125 ++++++++++++++++++++++++ crates/mc-host/src/lib.rs | 1 + crates/mc-host/src/tcp_frame_channel.rs | 89 +++++++---------- 4 files changed, 176 insertions(+), 90 deletions(-) create mode 100644 crates/mc-host/src/frame_read.rs diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 2e2aa44e9..9194e85dd 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -1829,32 +1829,21 @@ async fn read_active_frame( })) } +/// Fills `buf` under the frame deadline. Every stop is fatal to this generation: +/// the client resynchronizes by reconnecting, never by guessing where the next +/// header begins. async fn read_exact_until( read: &mut R, buf: &mut [u8], deadline: Instant, cancel: &CancellationToken, ) -> Result<(), ()> { - let mut offset = 0; - while offset < buf.len() { - let count = tokio::select! { - biased; - () = cancel.cancelled() => return Err(()), - result = timeout_at(deadline, read.read(&mut buf[offset..])) => result.map_err(|_| ())?.map_err(|_| ())?, - }; - if count == 0 { - return Err(()); - } - offset += count; - } - Ok(()) + crate::frame_read::read_exact(read, buf, deadline, cancel) + .await + .map_err(|_| ()) } /// Reads exactly `len` body bytes under one frame deadline. -/// -/// `read_buf` appends into the vector's spare capacity without -/// zero-initializing it, and `take` caps the read at the frame boundary even -/// when the allocated capacity exceeds `len`. async fn read_body_until( read: &mut R, len: usize, @@ -1862,33 +1851,23 @@ async fn read_body_until( cancel: &CancellationToken, ) -> Result, ()> { let mut body = Vec::with_capacity(len); - let mut limited = read.take(len as u64); - while body.len() < len { - let count = tokio::select! { - biased; - () = cancel.cancelled() => return Err(()), - result = timeout_at(deadline, limited.read_buf(&mut body)) => result.map_err(|_| ())?.map_err(|_| ())?, - }; - if count == 0 { - return Err(()); - } - } + crate::frame_read::read_body(read, &mut body, len, deadline, cancel) + .await + .map_err(|_| ())?; Ok(body) } +/// Discards a body this client refused to retain, so the failure is reported +/// against a stream still aligned on a header boundary. async fn drain_until( read: &mut R, - mut remaining: usize, + remaining: usize, deadline: Instant, cancel: &CancellationToken, ) -> Result<(), ()> { - let mut scratch = [0u8; 8192]; - while remaining > 0 { - let take = remaining.min(scratch.len()); - read_exact_until(read, &mut scratch[..take], deadline, cancel).await?; - remaining -= take; - } - Ok(()) + crate::frame_read::drain(read, remaining, deadline, cancel) + .await + .map_err(|_| ()) } /// Turns a caller-supplied timeout into an absolute deadline. diff --git a/crates/mc-host/src/frame_read.rs b/crates/mc-host/src/frame_read.rs new file mode 100644 index 000000000..cc635a71f --- /dev/null +++ b/crates/mc-host/src/frame_read.rs @@ -0,0 +1,125 @@ +//! Deadline- and cancellation-bounded frame reads, shared by the host's framing +//! layer and the client's reader. +//! +//! These are the byte-moving mechanics only: fill a buffer, fill a body, discard a +//! declared body. The protocol policy around them — which lengths are legal, what +//! an oversize control body means, whether a short read is corruption or an +//! orderly close — stays with each caller, because the two answer those questions +//! differently. +//! +//! Single-sourced because the mechanics are where the subtle parts live: the +//! `biased` select that prefers cancellation over another read, treating a +//! zero-length read as end-of-stream rather than looping, and capping the body +//! read at the frame boundary so a pipelined next header is never consumed as +//! body. Two copies of that drifting apart reintroduces exactly the bugs the +//! comments around them describe. + +use tokio::io::{AsyncRead, AsyncReadExt}; +use tokio::time::{timeout_at, Instant}; +use tokio_util::sync::CancellationToken; + +/// Why a bounded read stopped short. Callers map this onto their own error type, +/// since the same stop means different things to a host framing layer and to a +/// client reader. +#[derive(Debug)] +pub(crate) enum ReadStop { + /// The cancellation token fired; no further read was attempted. + Cancelled, + /// A read returned zero bytes with the buffer unfilled. + Eof, + /// The deadline passed before the buffer filled. + DeadlineExpired, + Io(std::io::Error), +} + +/// Fills `buf` completely, or stops. +pub(crate) async fn read_exact( + reader: &mut R, + buf: &mut [u8], + deadline: Instant, + cancel: &CancellationToken, +) -> Result<(), ReadStop> +where + R: AsyncRead + Unpin, +{ + let mut filled = 0; + while filled < buf.len() { + let read = tokio::select! { + biased; + () = cancel.cancelled() => return Err(ReadStop::Cancelled), + result = timeout_at(deadline, reader.read(&mut buf[filled..])) => match result { + Ok(read) => read.map_err(ReadStop::Io)?, + Err(_) => return Err(ReadStop::DeadlineExpired), + }, + }; + if read == 0 { + return Err(ReadStop::Eof); + } + filled += read; + } + Ok(()) +} + +/// Appends exactly `len` body bytes into `buf`. +/// +/// `read_buf` appends into spare capacity without zero-initializing it, and +/// `take` caps the read at the frame boundary even when the allocated capacity +/// exceeds `len` — without that cap a pipelined next header would be read as this +/// frame's body. +pub(crate) async fn read_body( + reader: &mut R, + buf: &mut Vec, + len: usize, + deadline: Instant, + cancel: &CancellationToken, +) -> Result<(), ReadStop> +where + R: AsyncRead + Unpin, +{ + let mut limited = reader.take(len as u64); + while buf.len() < len { + let read = tokio::select! { + biased; + () = cancel.cancelled() => return Err(ReadStop::Cancelled), + result = timeout_at(deadline, limited.read_buf(buf)) => match result { + Ok(read) => read.map_err(ReadStop::Io)?, + Err(_) => return Err(ReadStop::DeadlineExpired), + }, + }; + if read == 0 { + return Err(ReadStop::Eof); + } + } + Ok(()) +} + +/// Discards exactly `declared` bytes, realigning the stream on a body the caller +/// refused to buffer. +pub(crate) async fn drain( + reader: &mut R, + declared: usize, + deadline: Instant, + cancel: &CancellationToken, +) -> Result<(), ReadStop> +where + R: AsyncRead + Unpin, +{ + let mut scratch = [0u8; 8192]; + let mut remaining = declared; + while remaining > 0 { + let want = remaining.min(scratch.len()); + let read = tokio::select! { + biased; + () = cancel.cancelled() => return Err(ReadStop::Cancelled), + result = timeout_at(deadline, reader.read(&mut scratch[..want])) => match result { + Ok(read) => read.map_err(ReadStop::Io)?, + Err(_) => return Err(ReadStop::DeadlineExpired), + }, + }; + if read == 0 { + return Err(ReadStop::Eof); + } + remaining -= read; + } + Ok(()) +} diff --git a/crates/mc-host/src/lib.rs b/crates/mc-host/src/lib.rs index c4428b9e4..0ddab2316 100644 --- a/crates/mc-host/src/lib.rs +++ b/crates/mc-host/src/lib.rs @@ -28,6 +28,7 @@ mod control; mod dispatch; #[doc(hidden)] pub mod frame_channel; +mod frame_read; mod instance; mod panic_boundary; mod routing; diff --git a/crates/mc-host/src/tcp_frame_channel.rs b/crates/mc-host/src/tcp_frame_channel.rs index 7cfce0460..f1a8c9b7c 100644 --- a/crates/mc-host/src/tcp_frame_channel.rs +++ b/crates/mc-host/src/tcp_frame_channel.rs @@ -19,7 +19,7 @@ use crate::wire::{ PROTOCOL_VERSION, }; use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader}; -use tokio::time::{timeout_at, Duration, Instant}; +use tokio::time::{Duration, Instant}; use tokio_util::sync::CancellationToken; use crate::frame_channel::{ @@ -230,6 +230,7 @@ where /// Discards the declared bytes of an early-rejected oversize control body /// without allocating it, preserving stream alignment. Failure here closes the /// generation as usual; the already-queued terminal stays authoritative. +/// Discards a declared body the caller refused to buffer, realigning the stream. async fn drain_declared_body( reader: &mut R, declared: u32, @@ -239,29 +240,12 @@ async fn drain_declared_body( where R: AsyncRead + Unpin, { - let mut scratch = [0u8; 8192]; - let mut remaining = declared as usize; - while remaining > 0 { - let want = remaining.min(scratch.len()); - let read = tokio::select! { - biased; - () = cancel.cancelled() => return Err(ReadClose::Cancelled), - result = timeout_at(deadline, reader.read(&mut scratch[..want])) => match result { - Ok(read) => read.map_err(ReadClose::Io)?, - Err(_) => return Err(ReadClose::Corrupt("drain deadline expired")), - }, - }; - if read == 0 { - return Err(ReadClose::Corrupt("EOF while draining oversize body")); - } - remaining -= read; - } - Ok(()) + crate::frame_read::drain(reader, declared as usize, deadline, cancel) + .await + .map_err(drain_close) } -/// Offset-tracked exact read under an absolute deadline. Cancellation-safe by -/// construction: partial progress retires the generation, so a torn read never -/// resumes on the same stream. +/// Fills `buf`, classifying a short read as this layer sees it. async fn read_exact_deadline( reader: &mut R, buf: &mut [u8], @@ -271,27 +255,12 @@ async fn read_exact_deadline( where R: AsyncRead + Unpin, { - let mut filled = 0; - while filled < buf.len() { - let read = tokio::select! { - biased; - () = cancel.cancelled() => return Err(ReadClose::Cancelled), - result = timeout_at(deadline, reader.read(&mut buf[filled..])) => match result { - Ok(read) => read.map_err(ReadClose::Io)?, - Err(_) => return Err(ReadClose::Corrupt("frame deadline expired")), - }, - }; - if read == 0 { - return Err(ReadClose::Corrupt("EOF inside frame")); - } - filled += read; - } - Ok(()) + crate::frame_read::read_exact(reader, buf, deadline, cancel) + .await + .map_err(frame_close) } -/// `read_buf` appends into `buf`'s spare capacity without zero-initializing -/// it. `take` caps the read at the frame boundary even when the vector's -/// allocated capacity exceeds `len`. +/// Reads exactly `len` body bytes under the frame deadline. async fn read_body_deadline( reader: &mut R, buf: &mut Vec, @@ -302,21 +271,33 @@ async fn read_body_deadline( where R: AsyncRead + Unpin, { - let mut limited = reader.take(len as u64); - while buf.len() < len { - let read = tokio::select! { - biased; - () = cancel.cancelled() => return Err(ReadClose::Cancelled), - result = timeout_at(deadline, limited.read_buf(buf)) => match result { - Ok(read) => read.map_err(ReadClose::Io)?, - Err(_) => return Err(ReadClose::Corrupt("frame deadline expired")), - }, - }; - if read == 0 { - return Err(ReadClose::Corrupt("EOF inside frame")); + crate::frame_read::read_body(reader, buf, len, deadline, cancel) + .await + .map_err(frame_close) +} + +/// A stop inside a frame: EOF and deadline both mean stream alignment is lost, so +/// the generation closes without resynchronization (protocol section 6.3). +fn frame_close(stop: crate::frame_read::ReadStop) -> ReadClose { + match stop { + crate::frame_read::ReadStop::Cancelled => ReadClose::Cancelled, + crate::frame_read::ReadStop::Eof => ReadClose::Corrupt("EOF inside frame"), + crate::frame_read::ReadStop::DeadlineExpired => { + ReadClose::Corrupt("frame deadline expired") + } + crate::frame_read::ReadStop::Io(error) => ReadClose::Io(error), + } +} + +/// Same classes, named for the drain so a failure says which phase lost alignment. +fn drain_close(stop: crate::frame_read::ReadStop) -> ReadClose { + match stop { + crate::frame_read::ReadStop::Eof => ReadClose::Corrupt("EOF while draining oversize body"), + crate::frame_read::ReadStop::DeadlineExpired => { + ReadClose::Corrupt("drain deadline expired") } + other => frame_close(other), } - Ok(()) } /// The single serialized write task for one connection. From ff2923a69bdcbb479bbb1c28068a2a2ebf762176 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 19:07:00 +0000 Subject: [PATCH 35/37] fix: repair the macOS build and refresh the stale crash-claims evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `validate_directory` read `stat.st_mode` directly. That field is `u16` on Darwin and `u32` on Linux, so mixing it with this crate's `u32` type constants compiles on Linux and fails on macOS with "no implementation for `u16 & u32`", "mismatched types", and "can't compare `u16` with `u32`" — the three errors the macOS job was reporting. `instance::mode_bits` is the cfg-gated widening that exists for exactly this, and every other mode check already went through it. Verified rather than reasoned: a scratch crate checked against `aarch64-apple-darwin` reproduces all three errors with the old expression and compiles clean with the new one. A source-shape test now fails the suite if any production line in either file reads `st_mode` without the accessor, since no Linux build can catch that regression. The `v84-process-crash` evidence digest covers `IMPLEMENTATION_FILES`, which includes `packages/e2e-tests/src/opencode-runner/spawn.ts` — restoring the supervised-identity strip there legitimately invalidated it. Regenerated via the test's own `UPDATE_CLAIMS_CRASH_EVIDENCE` path; the new digest matches the value CI computed, confirming the committed file was stale from `ef7980c9` rather than the computation having drifted. --- crates/mc-host/src/connection_file.rs | 44 +++++++++++++++++-- .../claims-backfill/v84-process-crash.json | 4 +- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/crates/mc-host/src/connection_file.rs b/crates/mc-host/src/connection_file.rs index a034f71fe..5ed03befc 100644 --- a/crates/mc-host/src/connection_file.rs +++ b/crates/mc-host/src/connection_file.rs @@ -19,7 +19,7 @@ use rustix::{ use serde::{Deserialize, Serialize}; use crate::{ - instance::{is_safe_ancestor, is_secure_regular, read_all_fd, S_IFDIR, S_IFMT}, + instance::{is_safe_ancestor, is_secure_regular, mode_bits, read_all_fd, S_IFDIR, S_IFMT}, wire::PROTOCOL_VERSION, }; @@ -281,8 +281,13 @@ fn validate_directory( ) -> Result<(), ConnectionFileError> { let stat = rustix::fs::fstat(fd).map_err(|source| io_error("fstat_parent", path, source.into()))?; - let directory = (stat.st_mode & S_IFMT) == S_IFDIR; - let private = stat.st_uid == rustix::process::geteuid().as_raw() && stat.st_mode & 0o077 == 0; + // `mode_bits` and not `stat.st_mode`: `st_mode` is `u16` on Darwin and `u32` + // on Linux, so mixing it with the `u32` type constants compiles on one target + // and not the other. Every other mode check in this crate already goes + // through that helper. + let mode = mode_bits(&stat); + let directory = (mode & S_IFMT) == S_IFDIR; + let private = stat.st_uid == rustix::process::geteuid().as_raw() && mode & 0o077 == 0; if !directory || !is_safe_ancestor(&stat) || (require_private && !private) { return Err(ConnectionFileError::Insecure { path: path.to_path_buf(), @@ -343,6 +348,39 @@ fn io_error(op: &'static str, path: &Path, source: io::Error) -> ConnectionFileE mod tests { use super::*; + /// `st_mode` is `u16` on Darwin and `u32` on Linux, so mode arithmetic against + /// this crate's `u32` type constants compiles on one target and not the other. + /// `mode_bits` is the cfg-gated widening that makes it portable, and reading + /// `st_mode` directly bypassed it — a break no Linux build could catch, which + /// is why it reached CI as a macOS-only failure. + #[test] + fn mode_arithmetic_goes_through_the_portable_accessor() { + for source in [ + include_str!("connection_file.rs"), + include_str!("instance.rs"), + ] { + let production = source + .split("#[cfg(test)]\nmod tests {") + .next() + .expect("production source"); + for (number, line) in production.lines().enumerate() { + // Prose may name the field while explaining why not to use it. + let code = line.split("//").next().unwrap_or(""); + // The accessor's own two cfg branches are the one place that may + // touch the field. + if code.contains("u32::from(stat.st_mode)") || code.trim() == "stat.st_mode" { + continue; + } + assert!( + !code.contains(".st_mode"), + "line {} reads st_mode directly instead of mode_bits(): {}", + number + 1, + line.trim() + ); + } + } + } + fn info() -> ConnectionInfo { ConnectionInfo { schema: SCHEMA_VERSION, diff --git a/docs/evidence/claims-backfill/v84-process-crash.json b/docs/evidence/claims-backfill/v84-process-crash.json index 59dd7bbb0..5e8ebcbe8 100644 --- a/docs/evidence/claims-backfill/v84-process-crash.json +++ b/docs/evidence/claims-backfill/v84-process-crash.json @@ -1,8 +1,8 @@ { "schemaVersion": "claims-process-crash-evidence/v1", - "commitUnderTest": "ef7980c9566a191f1b2eaa7c4f87c90a51ed7e80", + "commitUnderTest": "1dbeb523d2c35e13b7130ee5d9377cee2d6848b9", "dirtyDiffDigestPolicy": "sha256(sorted U6 implementation path + NUL + full file bytes + NUL); evidence file excluded", - "dirtyDiffDigest": "59861ab8048726d2efd2905cb57d6b2064cf6cebfdf920c07f49a4aefa0ea34d", + "dirtyDiffDigest": "43379ae38c04afa252e4ddff1e86413dfdeaf5f06355813ce8cd42e1ddb97bc7", "implementationFiles": [ "ARCHITECTURE.md", "STRUCTURE.md", From afe7ab002283a430a85fbb3fd2731173b7c245ff Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 19:19:04 +0000 Subject: [PATCH 36/37] fix(mc-host): create the test FIFO through a portable interface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixing the mode-width break let the macOS job reach the test-compile step it had never got to, which failed on `rustix::fs::mknodat` — rustix excludes both `mknodat` and `mkfifoat` on Apple targets, so the FIFO-rejection test compiled only on Linux. `cargo test --lib ` builds the whole test target, so one Linux-only helper broke the macOS run regardless of which test was selected. It now shells out to the POSIX `mkfifo` utility. Calling `mkfifo(2)` directly would need `unsafe`, and this crate is `deny(unsafe_code)`; weakening that for a test fixture is not a trade worth making. The test stays compiled on every platform rather than being cfg'd out on the one whose absence hid the break — a blocking open on a FIFO wedges `Client::connect` on macOS exactly as it would on Linux, so that is the last platform to stop checking it. --- crates/mc-host/src/connection_file.rs | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/crates/mc-host/src/connection_file.rs b/crates/mc-host/src/connection_file.rs index 5ed03befc..0f4c807b0 100644 --- a/crates/mc-host/src/connection_file.rs +++ b/crates/mc-host/src/connection_file.rs @@ -414,14 +414,19 @@ mod tests { .expect("owner-only scratch dir"); let path = dir.join("subc-connection.json"); let _ = std::fs::remove_file(&path); - rustix::fs::mknodat( - rustix::fs::CWD, - &path, - rustix::fs::FileType::Fifo, - Mode::from_bits_truncate(0o600), - 0, - ) - .expect("mkfifo"); + // The POSIX `mkfifo` utility, not rustix: rustix gates both `mknodat` and + // `mkfifoat` away from Apple targets, and this crate is + // `deny(unsafe_code)`, so calling `mkfifo(2)` directly is not available + // either. The rejection matters just as much on macOS, so the test stays + // compiled on every platform rather than being cfg'd out on the one whose + // absence let a build break reach CI unnoticed. + let made = std::process::Command::new("mkfifo") + .arg(&path) + .status() + .expect("mkfifo is a POSIX utility present on every supported platform"); + assert!(made.success(), "mkfifo failed: {made:?}"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)) + .expect("owner-only fifo"); // No writer is ever opened, so a blocking open can never complete. // Bounded on a worker thread rather than called directly: a regression From 527866fe77dfc1353d85a45e864f0859379617c6 Mon Sep 17 00:00:00 2001 From: AhravDutta Date: Tue, 25 Aug 2026 19:38:57 +0000 Subject: [PATCH 37/37] fix(mc-host): release a route bound for an abandoned control open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A caller that drops or times out `open_route` after the request reaches the writer leaves a correlation the client will not settle. Identity 0/0 has no legal `Cancel` to withdraw the operation, so the host may still bind the route and answer. That `Response` arrived unmatched and was dropped as an ordinary stale terminal, which stranded the binding: the client never learned the handle, so it could send no route `Goodbye`, and every repeated abandon consumed another host route and channel permit until the generation ended. Wire protocol section 8.2 already fixes the remedy for a successful bind the client cannot cache — best-effort route `Goodbye`, and close the connection only when that cleanup cannot be queued. Apply it to the unmatched control `Response` as well, so the reclaim costs one frame instead of the whole generation and unrelated routes stay live. A bind already in the route cache belongs to a caller that received it and is left alone; releasing it would close a route still in use. Also record the token- and deadline-driven stream cancellation contract. That path settles through `cancel_key`, which cannot reach the receiver the caller holds, so queued items stay charged against the retained-response budget. That is correct because the bytes stay reachable: `finished` remains false, so `next` still drains them in order before the terminal, and `Drop` drains whatever is left. `ResponseStream::cancel` has to drain by hand only because it sets `finished` and makes the same bytes unreadable forever. --- crates/mc-host/src/client.rs | 315 ++++++++++++++++++++++++++++++++++ docs/mc-host-wire-protocol.md | 2 + 2 files changed, 317 insertions(+) diff --git a/crates/mc-host/src/client.rs b/crates/mc-host/src/client.rs index 9194e85dd..0e10c4398 100644 --- a/crates/mc-host/src/client.rs +++ b/crates/mc-host/src/client.rs @@ -1309,6 +1309,16 @@ impl Inner { }; let state = lock_unpoisoned(&self.pending).remove(&key); let Some(state) = state else { + // An unmatched terminal is normally dropped, but a `Response` + // on identity 0/0 can carry a route the host bound for an + // `open_route` whose caller has since dropped or timed out. + // Abandoning that request cannot withdraw it - identity 0/0 + // has no legal `Cancel` (Section 6.2) - so the bind lands with + // no caller to name it, and dropping it here strands a host + // route and channel permit until the generation ends. + if header.ty == FrameType::Response && header.channel == 0 { + self.release_stranded_route(&body); + } return; }; drop(charge); @@ -1446,6 +1456,37 @@ impl Inner { } } + /// Returns a late route bind that no caller can ever own. + /// + /// Section 8.2 fixes the remedy for a successful bind the client cannot + /// cache: send a best-effort route `Goodbye`, and close the connection only + /// when that cleanup cannot be queued. A body that names no route is left + /// alone; there is nothing to release and nothing to report to a caller that + /// is already gone. + /// + /// A bind already in the route cache belongs to a caller that received it, + /// so a duplicate terminal for it must not be treated as stranded - the + /// `Goodbye` would close a route still in use. + fn release_stranded_route(&self, body: &[u8]) { + let Ok(route) = parse_route_open(body) else { + return; + }; + if lock_unpoisoned(&self.routes).contains(&route) { + return; + } + if self + .send_control( + FrameType::Goodbye, + pure_header_flags(), + FrameId::routed(route, 0), + None, + ) + .is_err() + { + self.retire("stranded_route_cleanup_failed"); + } + } + fn finish_pending(&self, state: PendingState, result: Result) { match state.kind { PendingKind::Unary(tx) => { @@ -3577,6 +3618,280 @@ mod tests { drop(peer.await); } + #[tokio::test] + async fn an_abandoned_control_open_releases_a_late_bound_route() { + // A dropped or timed-out `open_route` leaves the request written, so the + // host may still bind the route and answer. That terminal arrives with no + // pending entry, and dropping it silently strands the binding: the caller + // never learns the handle, so it can send no route `Goodbye`, and each + // repeated abandon burns another host-side route and channel permit for + // the life of the generation. Section 8.2 fixes the remedy for a late bind + // the client cannot own - best-effort route `Goodbye`, and close the + // connection only when that cleanup cannot be queued. + let (inner, mut data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let control = RouteHandle { + channel: 0, + epoch: 0, + }; + let (tx, _rx) = oneshot::channel(); + let (key, publish) = inner + .admit( + control, + Vec::new(), + PendingKind::Unary(tx), + Instant::now() + Duration::from_secs(60), + ) + .expect("control request admitted"); + // The host answers only a request it received, so the writer claimed it. + // That is also what makes the abandonment `OutcomeUnknown` - the branch + // that has no legal `Cancel` to send on identity 0/0. + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + + // Exactly what `UnaryAdmissionGuard::drop` does for a dropped caller. + let removal = inner + .cancel_key(key, "caller_dropped") + .expect("abandoning a control request never fails"); + assert!(matches!(removal, PendingRemoval::Cancelled)); + assert!( + control_rx.try_recv().is_err(), + "identity 0/0 has no legal Cancel, so abandoning emits no control frame" + ); + + // The host bound the route anyway and answers the abandoned correlation. + let bound = RouteHandle { + channel: 9, + epoch: 3, + }; + let body = serde_json::to_vec(&serde_json::json!({ + "op": "route.open", + "route_channel": bound.channel, + "route_epoch": bound.epoch, + })) + .expect("body encodes"); + inner.dispatch( + EnvelopeHeader { + len: u32::try_from(body.len()).expect("fits a frame length"), + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, false), + channel: control.channel, + epoch: control.epoch, + corr: key.corr, + }, + body, + ByteCharge::none(), + ); + + let goodbye = control_rx + .try_recv() + .expect("a stranded bind is released with a route Goodbye"); + assert_eq!(goodbye.bytes[5], FrameType::Goodbye as u8); + let header = decode_header(&goodbye.bytes).expect("the Goodbye decodes"); + assert_eq!(header.channel, bound.channel, "the exact stranded channel"); + assert_eq!(header.epoch, bound.epoch, "the exact stranded epoch"); + assert_eq!(header.corr, 0, "a route Goodbye carries correlation 0"); + assert!( + !inner.retired.load(Ordering::Acquire), + "reclaiming one route must not take unrelated routes with it" + ); + assert!( + !lock_unpoisoned(&inner.routes).contains(&bound), + "a late bind never enters the client cache" + ); + inner.retire("test_done"); + } + + #[tokio::test] + async fn a_duplicate_bind_terminal_never_closes_an_owned_route() { + // Reclaiming a stranded bind reads an unmatched control `Response` for a + // route handle, and a duplicate terminal for a route the caller already + // received is unmatched too. Treating that as stranded would send a + // `Goodbye` for a route still in use, so the route cache is what + // separates "nobody owns this" from "somebody does". + let (inner, _data_rx, mut control_rx) = test_inner(CLIENT_QUEUED_BYTES); + let owned = route(1); + assert!( + lock_unpoisoned(&inner.routes).contains(&owned), + "the fixture owns this route" + ); + let body = serde_json::to_vec(&serde_json::json!({ + "op": "route.open", + "route_channel": owned.channel, + "route_epoch": owned.epoch, + })) + .expect("body encodes"); + inner.dispatch( + EnvelopeHeader { + len: u32::try_from(body.len()).expect("fits a frame length"), + ver: PROTOCOL_VERSION, + ty: FrameType::Response, + flags: response_flags(false, false), + channel: 0, + epoch: 0, + corr: FIRST_APPLICATION_CORRELATION, + }, + body, + ByteCharge::none(), + ); + + assert!( + control_rx.try_recv().is_err(), + "an owned route is never released by a duplicate bind terminal" + ); + assert!( + lock_unpoisoned(&inner.routes).contains(&owned), + "the owned route stays live" + ); + inner.retire("test_done"); + } + + #[tokio::test] + async fn token_cancelling_a_stream_leaves_its_queued_items_reachable() { + // The token and deadline watcher settles a stream through `cancel_key`, + // which cannot reach the receiver the caller holds, so queued items stay + // charged against the owner-wide retained budget. That is correct only + // because the bytes remain reachable: `finished` stays false, so `next` + // still drains them and `Drop` drains whatever is left. `cancel` has to + // drain by hand precisely because it sets `finished` and makes the same + // bytes unreadable forever. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (items_tx, items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + let mut stream = ResponseStream { + inner: Arc::downgrade(&inner), + key, + correlation: key.corr, + items: items_rx, + terminal: Some(terminal_rx), + finished: false, + }; + + const ITEMS: usize = 4; + for _ in 0..ITEMS { + inner.dispatch( + EnvelopeHeader { + len: 8, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![7; 8], + inner.retained_budget.charge(8).expect("retained bytes"), + ); + } + assert_eq!(inner.retained_budget.used(), ITEMS * 8); + + // Exactly what the watcher spawned by `start_stream` does. + let _ = inner.cancel_key(key, "cancelled"); + assert!( + !stream.finished, + "a watcher cancellation does not short-circuit the consumer" + ); + + // Every queued item is still delivered, in order, before the terminal. + let mut drained = 0; + loop { + match stream.next().await { + Ok(Some(item)) => { + assert_eq!(item.body, vec![7; 8]); + drained += 1; + } + Ok(None) => panic!("a cancelled stream reports its cancellation"), + Err(error) => { + assert_eq!(error.code(), "cancelled"); + break; + } + } + } + assert_eq!(drained, ITEMS, "the queued items survive the cancellation"); + assert_eq!( + inner.retained_budget.used(), + 0, + "draining the cancelled stream releases every charge" + ); + inner.retire("test_done"); + drop(stream); + } + + #[tokio::test] + async fn dropping_a_token_cancelled_stream_releases_its_queued_charges() { + // The other half of the reachability contract: a caller that never polls + // a watcher-cancelled stream still releases the retained bytes when it + // drops the value, so no charge outlives the consumer that holds it. + let (inner, mut data_rx, _control_rx) = test_inner(CLIENT_QUEUED_BYTES); + lock_unpoisoned(&inner.routes).insert(route(1)); + let (items_tx, items_rx) = mpsc::channel(CLIENT_STREAM_QUEUE_ITEMS); + let (terminal_tx, terminal_rx) = oneshot::channel(); + let (key, publish) = inner + .admit( + route(1), + Vec::new(), + PendingKind::Stream { + items: items_tx, + terminal: terminal_tx, + _settled: CancellationToken::new().drop_guard(), + }, + Instant::now() + Duration::from_secs(60), + ) + .expect("stream admitted"); + assert!(claim_for_write(&publish), "the writer claimed the request"); + drop(data_rx.recv().await); + let stream = ResponseStream { + inner: Arc::downgrade(&inner), + key, + correlation: key.corr, + items: items_rx, + terminal: Some(terminal_rx), + finished: false, + }; + + const ITEMS: usize = 4; + for _ in 0..ITEMS { + inner.dispatch( + EnvelopeHeader { + len: 8, + ver: PROTOCOL_VERSION, + ty: FrameType::StreamData, + flags: response_flags(false, false), + channel: key.channel, + epoch: key.epoch, + corr: key.corr, + }, + vec![7; 8], + inner.retained_budget.charge(8).expect("retained bytes"), + ); + } + let _ = inner.cancel_key(key, "cancelled"); + assert_eq!(inner.retained_budget.used(), ITEMS * 8); + + drop(stream); + assert_eq!( + inner.retained_budget.used(), + 0, + "dropping the consumer releases every queued charge" + ); + inner.retire("test_done"); + } + #[tokio::test] async fn retained_stream_bytes_never_deny_a_maximum_sized_frame() { // The wire contract obliges an admitted connection to accept any diff --git a/docs/mc-host-wire-protocol.md b/docs/mc-host-wire-protocol.md index f1fc35020..dca69512d 100644 --- a/docs/mc-host-wire-protocol.md +++ b/docs/mc-host-wire-protocol.md @@ -723,6 +723,8 @@ sequenceDiagram Local close racing `route.open` wins. A late successful bind MUST NOT enter client cache. Client sends best-effort route `Goodbye`; if it cannot queue cleanup safely, it closes the connection. Host still invokes route-gone exactly once. +An abandoned `route.open` reaches the same state by a different route: a caller that drops or times out after the request is written leaves a correlation the client will not settle, and identity `0/0` has no legal `Cancel` to withdraw the operation (Section 6.2), so the host may still bind and answer. That `Response` arrives unmatched. Dropping it as an ordinary unmatched terminal would strand the binding for the life of the generation, because the client never learns the handle and can therefore send no route `Goodbye` for it; each repeated abandon would consume another host route and channel permit. The client MUST therefore treat an unmatched control `Response` that names a route as a late bind it cannot own and apply the same remedy. A bind already present in the client route cache belongs to a caller that received it and MUST NOT be released this way. + ### 8.3 Request identity and allocation Each sender allocates correlations monotonically from 1 within a connection generation. The two directions are independent namespaces: a host `Ping` correlation MAY be numerically equal to a pending consumer correlation on the same connection, and neither affects the other. Matching is direction-scoped by frame type — `Response`, `Error`, `StreamData`, and `StreamEnd` settle only consumer-originated requests; `Pong` settles only host-originated `Ping`. The no-reuse rule applies within one sender's namespace. A correlation MUST NOT be reused, even after terminal completion. `u64::MAX` may identify one final request; before another request, sender MUST retire the generation and reconnect. Published `HistorianProducer` currently saturates at `u64::MAX`; compatibility work in `magic-context-c50.4` MUST replace saturation with checked exhaustion and generation retirement.