From 5535fdd42a2282e0659fa29149cdb0720c62e638 Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 09:02:20 +0300 Subject: [PATCH 1/9] =?UTF-8?q?feat(daemon):=20benchd=20M0=20=E2=80=94=20t?= =?UTF-8?q?he=20bench=20daemon=20skeleton,=20isolated=20and=20testable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The roadmap's M0 (docs/future-planning/bench-roadmap.md), started on the operator's word: daemon/ is a self-contained cargo workspace beside the Swift tree with three crates — benchd (the daemon), bench (the CLI), bench-wire (every wire type and shared rule, spelled once, so the two binaries cannot drift). What works: a suite-aware record root (BENCH_SUITE isolates socket and state, BENCH_DIR overrides outright for tests; a name that cannot isolate refuses the launch rather than falling back to ~/.bench — the #86/#285 posture as a newtype), an append-only events.jsonl where every mutation is logged before the response that reports it, one unix socket answering status/events/stop, and helm's spool exit codes kept on the CLI (0 ok, 2 no daemon, 3 refused, 4 failed). Gate: daemon/test.sh — fmt, clippy -D warnings, build, then tests; the conformance suite runs the real binaries as subprocesses across every status case, and proves M0's line: a live and a suite instance side by side share nothing, with the negative control that the shared root never appears. direction.md carries the spine (bench-visible means logged; one door; one spelling) and what was deliberately adopted or refused from the DeepSeek Harness study. Not yet wired: the root AGENTS.md daemon section and a CI job on daemon/** — both ride with the PR. --- daemon/.gitignore | 1 + daemon/AGENTS.md | 42 +++ daemon/Cargo.lock | 172 ++++++++++ daemon/Cargo.toml | 14 + daemon/crates/bench-wire/Cargo.toml | 9 + daemon/crates/bench-wire/src/lib.rs | 323 ++++++++++++++++++ daemon/crates/bench/Cargo.toml | 9 + daemon/crates/bench/src/main.rs | 158 +++++++++ daemon/crates/bench/tests/conformance.rs | 346 +++++++++++++++++++ daemon/crates/benchd/Cargo.toml | 10 + daemon/crates/benchd/src/main.rs | 415 +++++++++++++++++++++++ daemon/direction.md | 98 ++++++ daemon/test.sh | 14 + 13 files changed, 1611 insertions(+) create mode 100644 daemon/.gitignore create mode 100644 daemon/AGENTS.md create mode 100644 daemon/Cargo.lock create mode 100644 daemon/Cargo.toml create mode 100644 daemon/crates/bench-wire/Cargo.toml create mode 100644 daemon/crates/bench-wire/src/lib.rs create mode 100644 daemon/crates/bench/Cargo.toml create mode 100644 daemon/crates/bench/src/main.rs create mode 100644 daemon/crates/bench/tests/conformance.rs create mode 100644 daemon/crates/benchd/Cargo.toml create mode 100644 daemon/crates/benchd/src/main.rs create mode 100644 daemon/direction.md create mode 100755 daemon/test.sh diff --git a/daemon/.gitignore b/daemon/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/daemon/.gitignore @@ -0,0 +1 @@ +/target diff --git a/daemon/AGENTS.md b/daemon/AGENTS.md new file mode 100644 index 0000000..4aaac51 --- /dev/null +++ b/daemon/AGENTS.md @@ -0,0 +1,42 @@ +# AGENTS.md — daemon/ + +The bench daemon: a self-contained cargo workspace, the `pi/`-style carve-out. Read +`direction.md` first; the milestone sequence and invariants are +`../docs/future-planning/bench-roadmap.md`. Vocabulary stays canonical in `../CONTEXT.md`. + +## Gate + +``` +bash daemon/test.sh +``` + +fmt-check, clippy `-D warnings`, build, then tests — **build before test is load-bearing**: +the conformance suite runs the real `benchd` binary as a subprocess and locates it beside +its own. Run this gate when `daemon/` changed; the Swift gate at the repo root neither +knows nor needs the Rust toolchain, in either direction. + +## Layout + +- `crates/bench-wire` — every wire type and shared resolution rule, spelled once. If + `benchd` and `bench` could disagree about a value, its rule belongs here. +- `crates/benchd` — the daemon. Foreground, one unix socket, serial request handling. +- `crates/bench` — the CLI, the one agent-facing surface (and the future skill surface). +- There is deliberately **no root `Cargo.toml`** in the repo: `cargo` at the repo root + fails loudly instead of half-working. + +## Rules + +- **Bench-visible means logged.** A mutation appends its event before the response that + reports it. A new capability is new event kinds + new verbs over the same socket — + never a second channel (no files-as-IPC, no extra sockets, no notification side paths). +- **Tests never touch the operator's estate.** Claim a disposable `HOME` (or `BENCH_DIR`) + under the OS tempdir — the OS tempdir specifically: unix socket paths cap near 104 + bytes and long scratch paths fail at bind. Include the negative control: assert the + shared root shape was never created (`hooks/test.sh`'s pattern). +- **Bounded children.** A test that spawns a daemon owns exactly that pid, kills it in a + Drop guard, and waits. Never kill by pattern (repo root AGENTS.md; #291 is why). +- **Exit codes are the contract**: 0 ok · 2 no daemon · 3 refused · 4 daemon failed. + A refusal names the rule it applied and the route to use instead. +- **Wire changes ride with their conformance test** in `crates/bench/tests/` — real + binaries, both directions, every status case, same as `SpoolWireConformanceTests`. +- Conventional commits, written as a human — no AI attribution (repo rule). diff --git a/daemon/Cargo.lock b/daemon/Cargo.lock new file mode 100644 index 0000000..99c4aa2 --- /dev/null +++ b/daemon/Cargo.lock @@ -0,0 +1,172 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bench" +version = "0.0.1" +dependencies = [ + "bench-wire", + "serde_json", +] + +[[package]] +name = "bench-wire" +version = "0.0.1" +dependencies = [ + "serde", + "serde_json", +] + +[[package]] +name = "benchd" +version = "0.0.1" +dependencies = [ + "bench-wire", + "serde_json", + "time", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "time" +version = "0.3.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/daemon/Cargo.toml b/daemon/Cargo.toml new file mode 100644 index 0000000..72e6da2 --- /dev/null +++ b/daemon/Cargo.toml @@ -0,0 +1,14 @@ +# The bench daemon workspace. Deliberately NOT reachable from a repo-root Cargo.toml — +# `cargo` at the repo root should fail loudly, not half-work (bench-roadmap.md, M0). +[workspace] +resolver = "2" +members = ["crates/bench-wire", "crates/benchd", "crates/bench"] + +[workspace.package] +version = "0.0.1" +edition = "2024" + +[workspace.dependencies] +serde = { version = "1", features = ["derive"] } +serde_json = "1" +time = { version = "0.3", features = ["formatting"] } diff --git a/daemon/crates/bench-wire/Cargo.toml b/daemon/crates/bench-wire/Cargo.toml new file mode 100644 index 0000000..cdaeda0 --- /dev/null +++ b/daemon/crates/bench-wire/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "bench-wire" +version.workspace = true +edition.workspace = true +description = "Every bench wire type and shared resolution rule, spelled once." + +[dependencies] +serde.workspace = true +serde_json.workspace = true diff --git a/daemon/crates/bench-wire/src/lib.rs b/daemon/crates/bench-wire/src/lib.rs new file mode 100644 index 0000000..fd94c8a --- /dev/null +++ b/daemon/crates/bench-wire/src/lib.rs @@ -0,0 +1,323 @@ +//! Every wire type and every shared resolution rule, spelled once. +//! +//! This crate exists so that `benchd` and `bench` can never disagree about what travels +//! on the socket or where a suite's state lives. helm spent real incidents on the other +//! arrangement — three copies of the mailbox rule in three languages, held together by a +//! conformance harness (`hooks/mailbox-conformance.mjs`). Rust on both ends of this socket +//! means the single spelling is finally free; anything that later reads these types from +//! Swift gets a generated or conformance-pinned copy, never a hand-written one +//! (bench-roadmap.md, invariant 9). +//! +//! The protocol itself is deliberately small: one connection carries one JSON request line +//! and one JSON response line, then closes. No framing, no multiplexing, no versioned +//! handshake — those arrive when a milestone needs them, and `Request`/`Response` carry +//! nothing a later field cannot extend compatibly. + +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use std::path::{Path, PathBuf}; + +/// A request line larger than this is refused, not read. The cap is about the reader: +/// every accepted byte can end up in an event log an agent later pulls into context. +/// Same argument as helm's 64 KB canvas-state cap. +pub const MAX_REQUEST_BYTES: usize = 64 * 1024; + +// --------------------------------------------------------------------------- +// Suite names +// --------------------------------------------------------------------------- + +/// A validated suite name — the isolation primitive, ported from helm's +/// `HELM_DEFAULTS_SUITE` (#86) with the same posture: **refuse loudly rather than fall +/// back to the live instance.** A test that launches under a suite it believes isolates +/// it, and silently lands in the operator's `~/.bench`, is exactly the disaster helm +/// #285 documents. So a name that cannot isolate is an error at the edge, never a +/// fallback to the shared root. +/// +/// The unchecked value is unrepresentable: the only route in is `validate`, and every +/// path builder below takes `&SuiteName`, not `&str`. (helm's `RequestID`/#260 argument, +/// applied on day one instead of after the incident.) +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(transparent)] +pub struct SuiteName(String); + +impl SuiteName { + /// Lowercase ASCII letters, digits and `-`; must start alphanumeric; at most 32 + /// bytes. Everything else is refused with a reason that names the rule, because the + /// refusal is read by an agent that has to fix its call. + pub fn validate(raw: &str) -> Result { + if raw.is_empty() { + return Err( + "a suite name cannot be empty — unset BENCH_SUITE for the live instance".into(), + ); + } + if raw.contains('/') || raw.contains('\\') || raw.contains("..") { + return Err(format!( + "a path is not a suite name: {raw:?} — the suite decides the directory, never names it" + )); + } + if raw.len() > 32 { + return Err(format!( + "suite name too long ({} bytes, max 32): {raw:?}", + raw.len() + )); + } + let mut chars = raw.chars(); + let first_ok = chars + .next() + .is_some_and(|c| c.is_ascii_lowercase() || c.is_ascii_digit()); + let rest_ok = raw + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'); + if !first_ok || !rest_ok { + return Err(format!( + "a suite name is lowercase ASCII letters, digits and '-', starting alphanumeric: {raw:?}" + )); + } + Ok(SuiteName(raw.to_string())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +// --------------------------------------------------------------------------- +// Request ids +// --------------------------------------------------------------------------- + +/// A validated request id. Port of helm's `RequestID` (#260), not a reinvention: the +/// pattern is the filename-safe one, because M3's socketless drop-box will use ids as +/// filenames and an ungated id writes wherever the caller likes. Gating it now costs one +/// type; gating it at M3 would be a migration. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(transparent)] +pub struct RequestId(String); + +impl RequestId { + /// First char ASCII alphanumeric; the rest alphanumeric, `.`, `_` or `-`; 1–64 bytes. + pub fn validate(raw: &str) -> Result { + let mut chars = raw.chars(); + let first_ok = chars.next().is_some_and(|c| c.is_ascii_alphanumeric()); + let rest_ok = raw + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '_' || c == '-'); + if raw.is_empty() || raw.len() > 64 || !first_ok || !rest_ok { + return Err(format!( + "a request id is 1-64 bytes of [A-Za-z0-9._-], starting alphanumeric: {raw:?}" + )); + } + Ok(RequestId(raw.to_string())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +// --------------------------------------------------------------------------- +// The envelope +// --------------------------------------------------------------------------- + +/// What a caller sends. `id` and `verb` stay raw `String`s **on purpose**: a request is +/// decoded permissively in shape and judged strictly afterwards, so a malformed id or an +/// unknown verb is a `refused` response naming the reason rather than unreadable JSON +/// with no reply. That is helm's standing carve-out (`CloseRequest.terminal`, +/// `SpawnRequest.cwd`) and it is load-bearing here for the same reason. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Request { + pub id: String, + pub verb: String, + #[serde(default)] + pub args: Value, +} + +/// Every response carries a status, and the status is the exit code: a caller never +/// parses prose to learn what happened. `reason` is for humans and agents; `data` is the +/// verb's payload. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Response { + pub id: String, + pub status: Status, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub data: Option, +} + +/// The three outcomes, mapped onto helm's spool exit-code discipline. `2` (no daemon) is +/// deliberately absent: it is the *transport's* failure, decided by the caller when the +/// socket cannot be reached, never something a daemon could say about itself. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Status { + Ok, + Refused, + Error, +} + +impl Status { + /// 0 ok / 3 refused / 4 daemon failed — helm's spool codes, kept (bench-roadmap M0). + pub fn exit_code(self) -> i32 { + match self { + Status::Ok => 0, + Status::Refused => 3, + Status::Error => 4, + } + } +} + +/// The caller-side exit for "the socket could not be reached at all". +pub const EXIT_NO_DAEMON: i32 = 2; + +// --------------------------------------------------------------------------- +// Events +// --------------------------------------------------------------------------- + +/// One line of the append-only record. **Bench-visible means logged**: anything a +/// projection, a snapshot, or a later reader is allowed to know happened must be +/// reconstructable from this stream — the file is the record, the socket is only +/// transport (bench-roadmap, invariants 7 and the dsh lesson in direction.md). +/// +/// `kind` is namespaced `domain/what` (`daemon/started`). A reader that meets a kind it +/// does not know must refuse or skip *visibly*, never misread it — which is why the +/// envelope stays this small and flat. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Event { + pub seq: u64, + /// RFC 3339 UTC. A `cat` of the log is meant to be readable by a human having a bad + /// morning; epoch integers are not that. + pub at: String, + pub kind: String, + #[serde(default, skip_serializing_if = "Value::is_null")] + pub data: Value, +} + +// --------------------------------------------------------------------------- +// Where a bench lives on disk +// --------------------------------------------------------------------------- + +/// Resolve the record root. One rule, spelled once, used by both binaries: +/// +/// 1. `BENCH_DIR` names the root outright and wins over everything — it is what a test +/// claims into instead of the operator's estate (helm's `HELM_MAIL_DIR` rule). +/// 2. else `/.bench-` when a suite is set, +/// 3. else the shared `/.bench`. +/// +/// `home` is a parameter, not a `$HOME` read, so the rule is testable and the caller is +/// forced to say whose home it means. +pub fn resolve_root(bench_dir: Option<&str>, suite: Option<&SuiteName>, home: &Path) -> PathBuf { + if let Some(dir) = bench_dir { + return PathBuf::from(dir); + } + match suite { + Some(s) => home.join(format!(".bench-{}", s.as_str())), + None => home.join(".bench"), + } +} + +pub fn socket_path(root: &Path) -> PathBuf { + root.join("benchd.sock") +} + +pub fn events_path(root: &Path) -> PathBuf { + root.join("events.jsonl") +} + +/// A `sockaddr_un` path is capped (~104 bytes on macOS), and exceeding it fails at bind +/// with an error that names none of this. Check it where the path is decided and say +/// what to do about it. +pub fn check_socket_path(path: &Path) -> Result<(), String> { + let len = path.as_os_str().len(); + if len > 100 { + return Err(format!( + "socket path is {len} bytes; unix sockets cap near 104 — point BENCH_DIR at a shorter path: {}", + path.display() + )); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn suite_names_that_isolate_are_accepted() { + for ok in ["x", "bench-dev", "a1", "m0-smoke"] { + assert!( + SuiteName::validate(ok).is_ok(), + "{ok} should be a valid suite" + ); + } + } + + #[test] + fn suite_names_that_cannot_isolate_are_refused_not_defaulted() { + for bad in [ + "", + "has/slash", + "..", + "a..b", + "UPPER", + "with space", + "-leading", + "x".repeat(33).as_str(), + ] { + assert!( + SuiteName::validate(bad).is_err(), + "{bad:?} should be refused" + ); + } + } + + #[test] + fn request_ids_follow_the_filename_safe_pattern() { + assert!(RequestId::validate("bench-123-9f").is_ok()); + assert!(RequestId::validate("a.b_c-d").is_ok()); + for bad in [ + "", + "../../etc", + "-lead", + "id with space", + "x".repeat(65).as_str(), + ] { + assert!( + RequestId::validate(bad).is_err(), + "{bad:?} should be refused" + ); + } + } + + #[test] + fn bench_dir_wins_suite_decorates_shared_is_default() { + let home = Path::new("/home/op"); + let suite = SuiteName::validate("dev").unwrap(); + assert_eq!( + resolve_root(Some("/claimed/root"), Some(&suite), home), + PathBuf::from("/claimed/root") + ); + assert_eq!( + resolve_root(None, Some(&suite), home), + PathBuf::from("/home/op/.bench-dev") + ); + assert_eq!( + resolve_root(None, None, home), + PathBuf::from("/home/op/.bench") + ); + } + + #[test] + fn status_maps_to_helm_exit_codes() { + assert_eq!(Status::Ok.exit_code(), 0); + assert_eq!(Status::Refused.exit_code(), 3); + assert_eq!(Status::Error.exit_code(), 4); + assert_eq!(EXIT_NO_DAEMON, 2); + } + + #[test] + fn overlong_socket_paths_are_named_before_bind() { + let long = PathBuf::from(format!("/{}", "d".repeat(120))); + assert!(check_socket_path(&long).is_err()); + assert!(check_socket_path(Path::new("/tmp/b/benchd.sock")).is_ok()); + } +} diff --git a/daemon/crates/bench/Cargo.toml b/daemon/crates/bench/Cargo.toml new file mode 100644 index 0000000..5e08ccb --- /dev/null +++ b/daemon/crates/bench/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "bench" +version.workspace = true +edition.workspace = true +description = "The bench CLI: the one agent-facing surface over the benchd socket." + +[dependencies] +bench-wire = { path = "../bench-wire" } +serde_json.workspace = true diff --git a/daemon/crates/bench/src/main.rs b/daemon/crates/bench/src/main.rs new file mode 100644 index 0000000..9b6ce96 --- /dev/null +++ b/daemon/crates/bench/src/main.rs @@ -0,0 +1,158 @@ +//! bench — the CLI, which is also the future agent skill surface (bench-roadmap M0/M3). +//! +//! One connection, one JSON request line, one JSON response line. The exit code IS the +//! outcome — an agent reads `$?`, not prose: +//! +//! 0 ok (the verb's data, pretty JSON, on stdout) +//! 2 no daemon (the socket could not be reached — transport, not a daemon answer) +//! 3 refused (the daemon said no and named why, on stderr) +//! 4 daemon failed (the daemon tried and could not, named why, on stderr) +//! +//! These are helm's spool codes, kept on purpose: every agent skill in this repo already +//! knows them, and a code that changes meaning across tools is worse than no code. + +use bench_wire::{ + EXIT_NO_DAEMON, Request, RequestId, Response, SuiteName, resolve_root, socket_path, +}; +use serde_json::{Value, json}; +use std::io::{BufRead, BufReader, Write}; +use std::os::unix::net::UnixStream; +use std::path::PathBuf; +use std::process; +use std::time::{SystemTime, UNIX_EPOCH}; + +fn main() { + process::exit(run()); +} + +fn usage() -> &'static str { + "usage: bench [--suite ] \n\ + verbs: status daemon identity, root, uptime, event count\n\ + \x20 events [--since N] read the record back from seq N\n\ + \x20 stop ask the daemon to log its stop and exit\n\ + env: BENCH_SUITE (flag wins) · BENCH_DIR (root override, wins over suite)\n\ + exit: 0 ok · 2 no daemon · 3 refused · 4 daemon failed" +} + +fn run() -> i32 { + let mut args = std::env::args().skip(1).peekable(); + let mut suite_flag: Option = None; + let mut verb: Option = None; + let mut since: u64 = 0; + + while let Some(arg) = args.next() { + match arg.as_str() { + "--suite" => match args.next() { + Some(v) => suite_flag = Some(v), + None => return refuse("--suite needs a name"), + }, + "--since" => match args.next().and_then(|v| v.parse::().ok()) { + Some(n) => since = n, + None => return refuse("--since needs a sequence number"), + }, + "--help" | "-h" => { + println!("{}", usage()); + return 0; + } + other if verb.is_none() && !other.starts_with('-') => verb = Some(other.to_string()), + other => return refuse(&format!("unknown argument {other:?}\n{}", usage())), + } + } + + let Some(verb) = verb else { + return refuse(usage()); + }; + + // The suite is validated here, before any socket is touched: a name that cannot + // isolate must never resolve to the shared root by accident (#86 semantics). The + // daemon applies the identical rule from the same crate — one spelling, two edges. + let suite_raw = suite_flag.or_else(|| std::env::var("BENCH_SUITE").ok()); + let suite = match suite_raw.as_deref() { + Some(raw) => match SuiteName::validate(raw) { + Ok(s) => Some(s), + Err(why) => return refuse(&why), + }, + None => None, + }; + let home = match std::env::var("HOME") { + Ok(h) => PathBuf::from(h), + Err(_) => return refuse("HOME is not set; bench cannot resolve a record root"), + }; + let bench_dir = std::env::var("BENCH_DIR").ok(); + let root = resolve_root(bench_dir.as_deref(), suite.as_ref(), &home); + let sock = socket_path(&root); + + let args_value = match verb.as_str() { + "events" if since > 0 => json!({ "since": since }), + _ => Value::Null, + }; + + let request = Request { + id: request_id(), + verb, + args: args_value, + }; + + let stream = match UnixStream::connect(&sock) { + Ok(s) => s, + Err(e) => { + eprintln!("bench: no daemon at {} ({e})", sock.display()); + return EXIT_NO_DAEMON; + } + }; + + let mut line = match serde_json::to_string(&request) { + Ok(l) => l, + Err(e) => return fail(&format!("cannot encode request: {e}")), + }; + line.push('\n'); + if let Err(e) = (&stream).write_all(line.as_bytes()) { + eprintln!("bench: write to {} failed ({e})", sock.display()); + return EXIT_NO_DAEMON; + } + + let mut reply = String::new(); + if BufReader::new(&stream).read_line(&mut reply).is_err() || reply.is_empty() { + eprintln!("bench: no answer from {}", sock.display()); + return EXIT_NO_DAEMON; + } + + let response: Response = match serde_json::from_str(&reply) { + Ok(r) => r, + Err(e) => return fail(&format!("unreadable response ({e}): {}", reply.trim())), + }; + + if let Some(reason) = &response.reason { + eprintln!("bench: {reason}"); + } + if let Some(data) = &response.data { + match serde_json::to_string_pretty(data) { + Ok(pretty) => println!("{pretty}"), + Err(_) => println!("{data}"), + } + } + response.status.exit_code() +} + +fn refuse(why: &str) -> i32 { + eprintln!("bench: {why}"); + 3 +} + +fn fail(why: &str) -> i32 { + eprintln!("bench: {why}"); + 4 +} + +/// A fresh id per invocation, inside `RequestId`'s own pattern — validated, not assumed, +/// so the client can never send an id the daemon-side rule would refuse. +fn request_id() -> String { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.subsec_nanos()) + .unwrap_or(0); + let candidate = format!("bench-{}-{nanos:08x}", process::id()); + RequestId::validate(&candidate) + .map(|id| id.as_str().to_string()) + .unwrap_or_else(|_| "bench-fallback".to_string()) +} diff --git a/daemon/crates/bench/tests/conformance.rs b/daemon/crates/bench/tests/conformance.rs new file mode 100644 index 0000000..a24b8fc --- /dev/null +++ b/daemon/crates/bench/tests/conformance.rs @@ -0,0 +1,346 @@ +//! Conformance: run the REAL binaries as subprocesses and check both directions — +//! the `SpoolWireConformanceTests` shape, ported (bench-roadmap, working discipline). +//! Nothing here mocks the socket, the daemon, or the CLI; what these tests pass is what +//! an agent's shell gets. +//! +//! Ground rules carried from helm's incidents: +//! - **Never the operator's estate.** Every test claims its own `HOME` under the OS +//! tempdir, and the negative control asserts the shared `~/.bench` shape was never +//! created there (#285's lesson: isolation is proven, not assumed). +//! - **Bounded children.** Every daemon is killed by the guard's Drop by the pid we +//! spawned — never a pattern — and waited on (#291's lesson). +//! - Requires a prior `cargo build --workspace` (daemon/test.sh does this): the benchd +//! binary is located beside our own CARGO_BIN_EXE path. + +use std::fs; +use std::io::{BufRead, BufReader, Write}; +use std::os::unix::net::UnixStream; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::sync::atomic::{AtomicU32, Ordering}; +use std::time::{Duration, Instant}; + +fn bench_bin() -> PathBuf { + PathBuf::from(env!("CARGO_BIN_EXE_bench")) +} + +fn benchd_bin() -> PathBuf { + let p = bench_bin().parent().unwrap().join("benchd"); + assert!( + p.exists(), + "benchd binary not built — run daemon/test.sh (or cargo build --workspace) first" + ); + p +} + +/// A disposable HOME under the OS tempdir. The OS tempdir, not the repo and not a long +/// scratch path: unix socket paths cap near 104 bytes, and bench-wire refuses overlong +/// ones — a test home has to stay short enough to bind in. +struct TestHome { + dir: PathBuf, +} + +impl TestHome { + /// The name is SHORT on purpose (`bcf--`): the macOS per-user tempdir is + /// already ~50 bytes, and `/.bench-/benchd.sock` has to stay under the + /// ~104-byte `sun_path` cap. A descriptive label here once pushed every socket past + /// it and all eight daemons refused to start — correctly, and uselessly. The `label` + /// is kept only for the panic message. + fn claim(label: &str) -> TestHome { + static NEXT: AtomicU32 = AtomicU32::new(0); + let n = NEXT.fetch_add(1, Ordering::Relaxed); + let dir = std::env::temp_dir().join(format!("bcf-{}-{n}", std::process::id())); + fs::create_dir_all(&dir).unwrap_or_else(|e| panic!("claim home for {label}: {e}")); + TestHome { dir } + } +} + +impl Drop for TestHome { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.dir); + } +} + +/// Owns exactly the daemon it spawned; Drop kills that pid and waits. +struct DaemonGuard { + child: Child, + socket: PathBuf, +} + +impl DaemonGuard { + fn start(home: &Path, suite: Option<&str>) -> DaemonGuard { + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", home) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let root = match suite { + Some(s) => { + cmd.arg("--suite").arg(s); + home.join(format!(".bench-{s}")) + } + None => home.join(".bench"), + }; + let child = cmd.spawn().expect("spawn benchd"); + let socket = root.join("benchd.sock"); + let guard = DaemonGuard { child, socket }; + guard.await_socket(); + guard + } + + fn await_socket(&self) { + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline { + if UnixStream::connect(&self.socket).is_ok() { + return; + } + std::thread::sleep(Duration::from_millis(20)); + } + panic!("benchd never answered at {}", self.socket.display()); + } +} + +impl Drop for DaemonGuard { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + +struct CliRun { + code: i32, + stdout: String, + stderr: String, +} + +fn bench(home: &Path, args: &[&str]) -> CliRun { + let out = Command::new(bench_bin()) + .env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", home) + .args(args) + .output() + .expect("run bench"); + CliRun { + code: out.status.code().unwrap_or(-1), + stdout: String::from_utf8_lossy(&out.stdout).into_owned(), + stderr: String::from_utf8_lossy(&out.stderr).into_owned(), + } +} + +// --------------------------------------------------------------------------- +// The exit-code contract, status case by status case +// --------------------------------------------------------------------------- + +#[test] +fn no_daemon_is_exit_2_and_names_the_socket() { + let home = TestHome::claim("nodaemon"); + let run = bench(&home.dir, &["status"]); + assert_eq!(run.code, 2, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("benchd.sock"), + "refusal must name the socket: {}", + run.stderr + ); +} + +#[test] +fn ok_is_exit_0_with_the_verbs_data() { + let home = TestHome::claim("ok"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["status"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + let data: serde_json::Value = serde_json::from_str(&run.stdout).expect("status prints JSON"); + assert!(data["pid"].as_u64().is_some()); + assert_eq!(data["suite"], serde_json::Value::Null); + assert!( + data["events"].as_u64().unwrap() >= 1, + "daemon/started must be logged" + ); +} + +#[test] +fn an_unknown_verb_is_refused_with_exit_3_naming_the_known_verbs() { + let home = TestHome::claim("refused"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["definitely-not-a-verb"]); + assert_eq!(run.code, 3, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("status, events, stop"), + "a refusal names the route to use instead: {}", + run.stderr + ); +} + +#[test] +fn a_suite_that_cannot_isolate_is_refused_client_side_before_any_socket() { + let home = TestHome::claim("badsuite"); + let run = bench(&home.dir, &["--suite", "has/slash", "status"]); + assert_eq!(run.code, 3, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("path"), + "the refusal names the rule: {}", + run.stderr + ); +} + +#[test] +fn an_oversized_request_line_is_refused_not_read() { + let home = TestHome::claim("oversize"); + let daemon = DaemonGuard::start(&home.dir, None); + let stream = UnixStream::connect(&daemon.socket).unwrap(); + let huge = format!( + "{{\"id\":\"big\",\"verb\":\"status\",\"pad\":\"{}\"}}\n", + "x".repeat(70_000) + ); + (&stream).write_all(huge.as_bytes()).unwrap(); + let mut reply = String::new(); + BufReader::new(&stream).read_line(&mut reply).unwrap(); + let response: serde_json::Value = serde_json::from_str(&reply).unwrap(); + assert_eq!(response["status"], "refused", "reply: {reply}"); +} + +// --------------------------------------------------------------------------- +// The record +// --------------------------------------------------------------------------- + +#[test] +fn stop_is_logged_before_it_is_answered_and_the_file_outlives_the_daemon() { + let home = TestHome::claim("stop"); + let daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["stop"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + + // The daemon exits on its own after answering; wait for it rather than killing it, + // so what we then read is a *completed* record. + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&daemon.socket).is_ok() { + std::thread::sleep(Duration::from_millis(20)); + } + let after = bench(&home.dir, &["status"]); + assert_eq!(after.code, 2, "a stopped daemon is exit 2, not an error"); + + // Files are the record: read the log with cat-level tooling, no daemon involved. + let log = fs::read_to_string(home.dir.join(".bench/events.jsonl")).unwrap(); + let kinds: Vec = log + .lines() + .map(|l| { + serde_json::from_str::(l).unwrap()["kind"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert_eq!(kinds.first().map(String::as_str), Some("daemon/started")); + assert_eq!(kinds.last().map(String::as_str), Some("daemon/stopped")); +} + +#[test] +fn events_reads_back_exactly_what_was_logged() { + let home = TestHome::claim("events"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["events"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + let data: serde_json::Value = serde_json::from_str(&run.stdout).unwrap(); + assert_eq!(data["truncated"], false); + let events = data["events"].as_array().unwrap(); + assert_eq!(events[0]["kind"], "daemon/started"); + assert_eq!(events[0]["seq"], 0); +} + +// --------------------------------------------------------------------------- +// M0's prove line: two instances, zero shared state +// --------------------------------------------------------------------------- + +#[test] +fn a_live_and_a_suite_instance_share_nothing() { + let home = TestHome::claim("isolation"); + let live = DaemonGuard::start(&home.dir, None); + let suite = DaemonGuard::start(&home.dir, Some("m0")); + assert_ne!(live.socket, suite.socket); + + let live_status = bench(&home.dir, &["status"]); + let suite_status = bench(&home.dir, &["--suite", "m0", "status"]); + assert_eq!(live_status.code, 0); + assert_eq!(suite_status.code, 0); + + let live_data: serde_json::Value = serde_json::from_str(&live_status.stdout).unwrap(); + let suite_data: serde_json::Value = serde_json::from_str(&suite_status.stdout).unwrap(); + assert_ne!(live_data["pid"], suite_data["pid"], "two daemons, not one"); + assert_ne!(live_data["root"], suite_data["root"], "two roots, not one"); + assert_eq!(suite_data["suite"], "m0"); + + // Stopping the suite instance must not touch the live one — the whole point. + let stop = bench(&home.dir, &["--suite", "m0", "stop"]); + assert_eq!(stop.code, 0); + let live_after = bench(&home.dir, &["status"]); + assert_eq!( + live_after.code, 0, + "the live instance survived the suite's stop" + ); + + // Two records on disk, each with its own history. + assert!(home.dir.join(".bench/events.jsonl").exists()); + assert!(home.dir.join(".bench-m0/events.jsonl").exists()); +} + +#[test] +fn bench_dir_overrides_everything_which_is_what_a_test_claims_into() { + let home = TestHome::claim("benchdir"); + let claimed = home.dir.join("claimed"); + fs::create_dir_all(&claimed).unwrap(); + + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_SUITE") + .env("HOME", home.dir.join("unused-home")) + .env("BENCH_DIR", &claimed) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let mut child = cmd.spawn().unwrap(); + let socket = claimed.join("benchd.sock"); + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&socket).is_err() { + std::thread::sleep(Duration::from_millis(20)); + } + + let out = Command::new(bench_bin()) + .env("HOME", home.dir.join("unused-home")) + .env("BENCH_DIR", &claimed) + .arg("status") + .output() + .unwrap(); + let _ = child.kill(); + let _ = child.wait(); + assert_eq!(out.status.code(), Some(0)); + + // Negative control (#285's shape): the claimed dir got the state, the un-claimed + // home shape was never created. + assert!(claimed.join("events.jsonl").exists()); + assert!( + !home.dir.join("unused-home/.bench").exists(), + "shared root must not appear" + ); +} + +#[test] +fn a_second_daemon_on_a_claimed_root_is_refused_loudly() { + let home = TestHome::claim("double"); + let _first = DaemonGuard::start(&home.dir, None); + let out = Command::new(benchd_bin()) + .env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .output() + .expect("run second benchd"); + assert_eq!( + out.status.code(), + Some(3), + "one daemon per root is a refusal, not a race" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("already answers"), + "the refusal says who holds the root: {stderr}" + ); +} diff --git a/daemon/crates/benchd/Cargo.toml b/daemon/crates/benchd/Cargo.toml new file mode 100644 index 0000000..0a8fce9 --- /dev/null +++ b/daemon/crates/benchd/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "benchd" +version.workspace = true +edition.workspace = true +description = "The bench daemon: owns the record, answers on the socket." + +[dependencies] +bench-wire = { path = "../bench-wire" } +serde_json.workspace = true +time.workspace = true diff --git a/daemon/crates/benchd/src/main.rs b/daemon/crates/benchd/src/main.rs new file mode 100644 index 0000000..80a2365 --- /dev/null +++ b/daemon/crates/benchd/src/main.rs @@ -0,0 +1,415 @@ +//! benchd — the bench daemon. M0: skeleton and isolation. +//! +//! What exists at this milestone: a suite-aware record root, an append-only event log +//! that is the single source of truth, and one unix socket answering three verbs +//! (`status`, `events`, `stop`). What deliberately does not exist yet: panes, mail, +//! attention, taps — those are M1+ (docs/future-planning/bench-roadmap.md) and each +//! arrives as new event kinds plus new verbs over this same spine, never as a second +//! channel beside it. +//! +//! Design rules this file carries (argued in ../../direction.md): +//! - **Bench-visible means logged.** Every mutation appends an event before the response +//! that reports it; readers project from the log, never from daemon memory alone. +//! - **One door.** The socket is the only way in; the CLI, the face, and every agent use +//! the same verbs. There is no privileged in-process path to grow attached to. +//! - **Refuse loudly.** Unknown verbs, malformed requests, oversized lines, a corrupt +//! log, an already-claimed socket: each is a named refusal, never a silent default. +//! +//! The daemon runs in the foreground and takes one request per connection, serially. +//! That is enough for M0's callers by construction, and a bounded, inspectable behavior +//! beats a concurrency story nothing needs yet. + +use bench_wire::{ + Event, MAX_REQUEST_BYTES, Request, Response, Status, SuiteName, check_socket_path, events_path, + resolve_root, socket_path, +}; +use serde_json::{Value, json}; +use std::fs::{self, File, OpenOptions}; +use std::io::{BufRead, BufReader, Read, Write}; +use std::os::unix::fs::{DirBuilderExt, PermissionsExt}; +use std::os::unix::net::{UnixListener, UnixStream}; +use std::path::PathBuf; +use std::process; +use std::time::Instant; + +fn main() { + process::exit(run()); +} + +fn usage() -> &'static str { + "usage: benchd [--suite ]\n\ + env: BENCH_SUITE suite name (the --suite flag wins)\n\ + \x20 BENCH_DIR record root override (wins over suite; what tests claim into)\n\ + exit: 0 clean stop · 3 refused to start · 4 failed" +} + +fn run() -> i32 { + // Flag wins over environment — the caller's explicit word over the inherited one, + // helm's PaneEnvironment convention. + let mut args = std::env::args().skip(1); + let mut suite_flag: Option = None; + while let Some(arg) = args.next() { + match arg.as_str() { + "--suite" => match args.next() { + Some(v) => suite_flag = Some(v), + None => return refuse_start("--suite needs a name"), + }, + "--help" | "-h" => { + println!("{}", usage()); + return 0; + } + other => return refuse_start(&format!("unknown argument {other:?}\n{}", usage())), + } + } + + let suite_raw = suite_flag.or_else(|| std::env::var("BENCH_SUITE").ok()); + let suite = match suite_raw.as_deref() { + Some(raw) => match SuiteName::validate(raw) { + Ok(s) => Some(s), + // The whole point of the suite is isolation; a name that cannot isolate + // must stop the launch, never fall back to the operator's live root (#86). + Err(why) => return refuse_start(&why), + }, + None => None, + }; + + let home = match std::env::var("HOME") { + Ok(h) => PathBuf::from(h), + Err(_) => return fail_start("HOME is not set; benchd cannot resolve a record root"), + }; + let bench_dir = std::env::var("BENCH_DIR").ok(); + let root = resolve_root(bench_dir.as_deref(), suite.as_ref(), &home); + + match Daemon::start(root, suite) { + Ok(mut daemon) => daemon.serve(), + Err(StartError::Refused(why)) => refuse_start(&why), + Err(StartError::Failed(why)) => fail_start(&why), + } +} + +fn refuse_start(why: &str) -> i32 { + eprintln!("benchd: refusing to start: {why}"); + 3 +} + +fn fail_start(why: &str) -> i32 { + eprintln!("benchd: {why}"); + 4 +} + +enum StartError { + Refused(String), + Failed(String), +} + +struct Daemon { + root: PathBuf, + suite: Option, + listener: UnixListener, + log: File, + next_seq: u64, + started_at: String, + booted: Instant, +} + +impl Daemon { + fn start(root: PathBuf, suite: Option) -> Result { + // 0700, like every helm record directory: single-user machine, but the record + // is still nobody else's to read. + let mut builder = fs::DirBuilder::new(); + builder.recursive(true).mode(0o700); + builder.create(&root).map_err(|e| { + StartError::Failed(format!("cannot create record root {}: {e}", root.display())) + })?; + + let sock = socket_path(&root); + check_socket_path(&sock).map_err(StartError::Refused)?; + + // A socket file can outlive its daemon (SIGKILL leaves it behind). Connectable + // means a live daemon owns this root — refuse, because two writers on one log is + // corruption with extra steps. Dead means stale — say so and reclaim. + if sock.exists() { + match UnixStream::connect(&sock) { + Ok(_) => { + return Err(StartError::Refused(format!( + "a live benchd already answers at {} — one daemon per root", + sock.display() + ))); + } + Err(_) => { + eprintln!("benchd: removing stale socket {}", sock.display()); + fs::remove_file(&sock).map_err(|e| { + StartError::Failed(format!("cannot remove stale socket: {e}")) + })?; + } + } + } + + // Boot-time integrity scan. The log is the record; a daemon that appends after + // a line it cannot read would be writing history it does not understand. Refuse + // with the line number rather than guessing (direction.md: refuse loudly). + let events = events_path(&root); + let next_seq = match File::open(&events) { + Ok(f) => { + let mut seq = 0u64; + for (i, line) in BufReader::new(f).lines().enumerate() { + let line = line.map_err(|e| { + StartError::Failed(format!("cannot read {}: {e}", events.display())) + })?; + if line.trim().is_empty() { + continue; + } + let ev: Event = serde_json::from_str(&line).map_err(|e| { + StartError::Refused(format!( + "event log {} line {} is not a readable event ({e}) — refusing to append after history this daemon cannot read", + events.display(), + i + 1 + )) + })?; + seq = ev.seq + 1; + } + seq + } + Err(_) => 0, + }; + + let log = OpenOptions::new() + .create(true) + .append(true) + .open(&events) + .map_err(|e| { + StartError::Failed(format!("cannot open event log {}: {e}", events.display())) + })?; + let _ = fs::set_permissions(&events, fs::Permissions::from_mode(0o600)); + + let listener = UnixListener::bind(&sock) + .map_err(|e| StartError::Failed(format!("cannot bind {}: {e}", sock.display())))?; + + let mut daemon = Daemon { + root, + suite, + listener, + log, + next_seq, + started_at: now_rfc3339(), + booted: Instant::now(), + }; + daemon + .append( + "daemon/started", + json!({ + "pid": process::id(), + "version": env!("CARGO_PKG_VERSION"), + "suite": daemon.suite.as_ref().map(|s| s.as_str()), + }), + ) + .map_err(StartError::Failed)?; + eprintln!( + "benchd {} listening at {} (root {})", + env!("CARGO_PKG_VERSION"), + socket_path(&daemon.root).display(), + daemon.root.display() + ); + Ok(daemon) + } + + fn serve(&mut self) -> i32 { + loop { + let stream = match self.listener.accept() { + Ok((s, _)) => s, + Err(e) => { + eprintln!("benchd: accept failed: {e}"); + continue; + } + }; + match self.handle(stream) { + Handled::Continue => {} + Handled::Stop => break, + } + } + // The stop event was appended before the response that promised it; all that is + // left is to stop answering. + let _ = fs::remove_file(socket_path(&self.root)); + 0 + } + + fn handle(&mut self, stream: UnixStream) -> Handled { + let mut reader = BufReader::new(match stream.try_clone() { + Ok(s) => s, + Err(_) => return Handled::Continue, + }); + let mut line = String::new(); + // Bounded read: a line that never ends must not become memory nobody asked for. + let mut limited = (&mut reader).take(MAX_REQUEST_BYTES as u64 + 1); + if limited.read_line(&mut line).is_err() { + return Handled::Continue; + } + if line.len() > MAX_REQUEST_BYTES { + respond( + &stream, + &Response { + id: "oversized".into(), + status: Status::Refused, + reason: Some(format!("request exceeds {MAX_REQUEST_BYTES} bytes")), + data: None, + }, + ); + return Handled::Continue; + } + + // Permissive in shape, strict in judgment: a body that is not a Request still + // gets a refusal naming the parse failure, under the only id we have. + let request: Request = match serde_json::from_str(&line) { + Ok(r) => r, + Err(e) => { + respond( + &stream, + &Response { + id: "unparseable".into(), + status: Status::Refused, + reason: Some(format!("not a request: {e}")), + data: None, + }, + ); + return Handled::Continue; + } + }; + + let (response, outcome) = self.dispatch(&request); + respond(&stream, &response); + outcome + } + + fn dispatch(&mut self, req: &Request) -> (Response, Handled) { + match req.verb.as_str() { + "status" => (self.ok(req, self.status_data()), Handled::Continue), + "events" => { + let since = req.args.get("since").and_then(Value::as_u64).unwrap_or(0); + match self.read_events(since) { + Ok(data) => (self.ok(req, data), Handled::Continue), + Err(why) => (self.error(req, why), Handled::Continue), + } + } + "stop" => { + // Logged before answered: the record must already say "stopped" when the + // caller is told it worked (bench-visible means logged). + match self.append("daemon/stopped", json!({ "pid": process::id() })) { + Ok(()) => (self.ok(req, json!({ "stopping": true })), Handled::Stop), + Err(why) => (self.error(req, why), Handled::Continue), + } + } + other => ( + Response { + id: req.id.clone(), + status: Status::Refused, + reason: Some(format!( + "unknown verb {other:?} — this daemon answers: status, events, stop" + )), + data: None, + }, + Handled::Continue, + ), + } + } + + fn status_data(&self) -> Value { + json!({ + "pid": process::id(), + "version": env!("CARGO_PKG_VERSION"), + "suite": self.suite.as_ref().map(|s| s.as_str()), + "root": self.root.display().to_string(), + "socket": socket_path(&self.root).display().to_string(), + "started_at": self.started_at, + "uptime_secs": self.booted.elapsed().as_secs(), + "events": self.next_seq, + }) + } + + /// Read back the log — from the file, not from memory, because the file is the + /// record and this verb is how a reader checks that claim. Caps are reported, never + /// silent: `returned < total` plus `truncated` says exactly what was left out. + fn read_events(&self, since: u64) -> Result { + const MAX_RETURNED: usize = 1000; + let path = events_path(&self.root); + let file = File::open(&path).map_err(|e| format!("cannot open {}: {e}", path.display()))?; + let mut events: Vec = Vec::new(); + let mut total = 0u64; + for line in BufReader::new(file).lines() { + let line = line.map_err(|e| format!("cannot read {}: {e}", path.display()))?; + if line.trim().is_empty() { + continue; + } + let ev: Event = serde_json::from_str(&line) + .map_err(|e| format!("corrupt event in {}: {e}", path.display()))?; + if ev.seq < since { + continue; + } + total += 1; + if events.len() < MAX_RETURNED { + events.push(ev); + } + } + let returned = events.len(); + Ok(json!({ + "events": events, + "total": total, + "returned": returned, + "truncated": (returned as u64) < total, + })) + } + + fn append(&mut self, kind: &str, data: Value) -> Result<(), String> { + let event = Event { + seq: self.next_seq, + at: now_rfc3339(), + kind: kind.to_string(), + data, + }; + let mut line = + serde_json::to_string(&event).map_err(|e| format!("cannot encode event: {e}"))?; + line.push('\n'); + self.log + .write_all(line.as_bytes()) + .and_then(|()| self.log.flush()) + .map_err(|e| format!("cannot append to event log: {e}"))?; + self.next_seq += 1; + Ok(()) + } + + fn ok(&self, req: &Request, data: Value) -> Response { + Response { + id: req.id.clone(), + status: Status::Ok, + reason: None, + data: Some(data), + } + } + + fn error(&self, req: &Request, why: String) -> Response { + Response { + id: req.id.clone(), + status: Status::Error, + reason: Some(why), + data: None, + } + } +} + +enum Handled { + Continue, + Stop, +} + +fn respond(mut stream: &UnixStream, response: &Response) { + if let Ok(mut line) = serde_json::to_string(response) { + line.push('\n'); + let _ = stream.write_all(line.as_bytes()); + } + let _ = stream.shutdown(std::net::Shutdown::Both); +} + +fn now_rfc3339() -> String { + time::OffsetDateTime::now_utc() + .format(&time::format_description::well_known::Rfc3339) + .unwrap_or_else(|_| "unknown".into()) +} diff --git a/daemon/direction.md b/daemon/direction.md new file mode 100644 index 0000000..dc7bc38 --- /dev/null +++ b/daemon/direction.md @@ -0,0 +1,98 @@ +# direction — the bench daemon + +An entry point, not a spec — the same posture as helm's `docs/direction.md`. The full +argument lives in `../docs/future-planning/workbench-audit-2026-08.md`; the milestone +sequence in `../docs/future-planning/bench-roadmap.md`. This file is what a session that +just opened `daemon/` needs to hold in its head. + +## What this is + +`benchd` is the headless Rust daemon helm grows into: it will own everything that must +survive — ptys, VT state, the workbench document, mail, tasks, the attention queue, the +event log — while the SwiftUI app thins into a face that renders daemon state. The +operator and the agents are equal owners; every verb exists in an addressed, non-seizing +form; both parties go through the same socket. Migration is strangler-style inside this +repo: one vertical at a time, old code unwired only when the new is proven. + +**Where it stands: M0.** A suite-aware record root, an append-only event log, one unix +socket, three verbs (`status`, `events`, `stop`), a CLI speaking helm's exit-code +discipline, and a conformance gate that runs the real binaries. Nothing helm does today +is owned here yet — M1 (attention queue and taps) is the first capability helm never had. + +## The spine + +Three commitments, made now, that every later milestone builds on rather than beside: + +**1. Bench-visible means logged.** The append-only event log (`events.jsonl` under the +record root) is the single source of truth. Every mutation appends its event *before* +the response that reports it; snapshots, queues, and "what happened overnight" are +projections of the stream, never a second store. helm's `snapshot.json` becomes a +projection at M4; the silent parked-workspace canvas drop (found 2026-08-17) is the +class of bug this rule deletes — an offer in the log with no consumer is a visible fact, +not a `return` in a guard. + +**2. One door.** The socket is the only way in. The CLI, the face, every agent, and the +operator's own keystrokes (from M4) use the same verbs through the same gateway — so +"equal owners" is mechanically true, not aspirational, and there is no privileged +in-process path for a capability to grow attached to. helm accreted six spool kinds, an +OSC push channel, mail hooks in two runtimes, and a snapshot file — each individually +argued, collectively sixteen ropes and no spine. The seventh capability here is a verb +and an event kind, not a new channel. + +**3. One spelling of every shared rule.** `bench-wire` holds the wire types AND the +resolution rules (suite names, record roots, request ids, caps). Both binaries compile +against it, so they cannot drift; anything outside the workspace that later needs these +types — the Swift face — gets a generated copy or a conformance-pinned duplicate under +the repo's honest-duplicate rule, never a hand-written one. + +## Learned from the field, on purpose + +DeepSeek Harness (dsh, studied 2026-08-17 — canvas in the helm prp store) is the most +complete existing implementation of a log-first agent daemon, and three of its ideas are +adopted here deliberately: + +- **The logged-envelope invariant.** dsh logs the full model request before dispatch, so + every request is a pure function of the log, and an independent checker rebuilds and + compares. Ours is the bench-shaped analog: every response's claim must be derivable + from the record — `stop` is logged before it is answered, and the conformance suite + reads the file, not the daemon, to verify history. +- **Caps are reported, never silent.** dsh documents every truncation; `events` returns + `total`/`returned`/`truncated` from day one, because a capped read that looks complete + is how "covered everything" gets believed. +- **Refusals name the route.** helm's spool refusals already point at the tool to use + instead; dsh's tool errors are typed and self-describing. Every refusal here names the + rule it applied and, where one exists, what to do about it. + +And one of dsh's ideas is **rejected** with equal deliberation: the plugin kernel. +benchd serves one operator whose gate is the pull request; composability-as-product +(profiles, patch layers, realm isolation) is generality this estate does not need and a +complexity bill dsh's own five-day-old ecosystem is already paying. Capabilities land as +code in this workspace, reviewed, behind the one door. + +## Rules that bind every milestone + +The checklist form is bench-roadmap.md's invariants; the ones already load-bearing in +this workspace: + +- **Suites isolate or refuse.** `BENCH_SUITE=` moves socket, root, and every byte + of state; a name that cannot isolate stops the launch — never a fallback to the + operator's live `~/.bench` (helm #86/#285, ported as `SuiteName`). +- **Files are the record.** Everything persistent is a file a plain `cat` can read; + sockets are transport, never the only copy. +- **Exit codes are the contract**: 0 ok · 2 no daemon · 3 refused · 4 daemon failed — + helm's spool codes, kept, because every agent skill in this repo already reads them. +- **Validated newtypes at the edges** (`SuiteName`, `RequestId`), with the standing + carve-out: a request is decoded permissively in shape and judged strictly afterwards, + so malformed input earns a refusal naming the reason, not a dropped connection. +- **No orchestrator concept, ever** — no role, rank, or team field in the daemon, the + wire, or the CLI (roadmap invariant 2). Hierarchy is prompts and skills, run *on* the + bench. +- **Conformance over trust**: every wire contract gets a test that runs the real binary + as a subprocess and checks both directions, every status case. + +## How it grows + +M1 attention queue + taps (first new capability, purely additive) → M2 mail authority → +M3 the CLI replaces the spool scripts → M4 the workbench document → M5 ptys → M6 reach +over the tailnet → M7 the forge. Each milestone: new event kinds, new verbs, same spine. +The roadmap is the sequence; the operator names the milestone that starts. diff --git a/daemon/test.sh b/daemon/test.sh new file mode 100755 index 0000000..38202a3 --- /dev/null +++ b/daemon/test.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# The daemon gate — the pi/-style carve-out: run when daemon/ changed, needs only the +# Rust toolchain, and the Swift gate never learns about it (bench-roadmap, invariant 9). +# +# Order matters: build the whole workspace BEFORE testing, because the conformance suite +# runs the real benchd binary as a subprocess and locates it beside its own — a test run +# without the build finds nothing and says so. +set -euo pipefail +cd "$(dirname "$0")" + +cargo fmt --check +cargo clippy --workspace --all-targets -- -D warnings +cargo build --workspace +cargo test --workspace From 010daac227a6a61ba655d06348cb51d568f72053 Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 09:45:15 +0300 Subject: [PATCH 2/9] feat(daemon): the bench surface as an executable spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A justfile of the operations the bench is for, written before they exist: recipes for spawn/close/attn/mail/watch/attach call the CLI with the argument shapes we want, and a verb the daemon does not answer yet fails with the daemon's own refusal. 'just spec' probes every wanted verb against a throwaway daemon and prints the scoreboard — 3 answered (M0), 14 not yet — so a milestone landing flips rows without the file changing shape. Wire verbs are namespaced like event kinds (attn/post, mail/send): one vocabulary for what happened and what is asked. Prompt-by-file in the spawn recipe, never argv (#93). Pattern borrowed from disler's herdr justfile; this file graduates into the bench skill at M3. --- daemon/justfile | 104 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 daemon/justfile diff --git a/daemon/justfile b/daemon/justfile new file mode 100644 index 0000000..bb7ed76 --- /dev/null +++ b/daemon/justfile @@ -0,0 +1,104 @@ +# The bench surface as executable spec. +# +# Recipes here are the operations the bench is FOR, written down before they exist. +# A recipe whose verb the daemon does not answer yet fails with the daemon's own +# refusal naming what it does answer — so `just spec` is a live scoreboard of the +# surface, and a milestone landing flips rows without this file changing shape. +# (The idea is stolen from disler's fixing-smartass-opus-5 justfile, which drives +# herdr's CLI the same way; this file graduates into the bench skill at M3.) +# +# Wire verbs are namespaced like event kinds: `attn/post`, `mail/send` — one +# vocabulary for what happened and what is asked. + +BENCH := "cargo run -q -p bench --" + +default: + @just --list + +# ---- today (M0) ------------------------------------------------------------- + +# Run the live daemon in the foreground. +daemon: + cargo run -q -p benchd + +# Daemon identity, root, uptime, event count. +status: + {{BENCH}} status + +# Read the record back. +events: + {{BENCH}} events + +# Log the stop, then exit. +stop: + {{BENCH}} stop + +# The full M0 loop against a throwaway root — green today, stays green. +smoke: + #!/usr/bin/env bash + set -euo pipefail + export BENCH_DIR=$(mktemp -d "${TMPDIR:-/tmp}/bsmk.XXXX")/r + (timeout 60 cargo run -q -p benchd &) + sleep 1 + cargo run -q -p bench -- status + cargo run -q -p bench -- stop + echo "smoke: ok (root $BENCH_DIR)" + +# ---- wanted (failing on purpose until their milestone lands) ---------------- + +# M1 — post an attention item. +attn-post kind="blocked" detail="": + {{BENCH}} attn/post --kind {{kind}} --detail "{{detail}}" + +# M1 — the queue, oldest blocked first. +attn-list: + {{BENCH}} attn/list + +# M1 — block until a tenant is genuinely blocked or done. Zero-token supervision. +watch handle: + {{BENCH}} watch --handle {{handle}} + +# M2 — mail through benchd's authority. +mail-send to subject body-file: + {{BENCH}} mail/send --to {{to}} --subject "{{subject}}" --body-file {{body-file}} + +# M2 — the mailbox listing. +mail-list: + {{BENCH}} mail/list + +# M3 — spawn an interactive agent into a bench pty. Prompt by FILE, never argv (#93). +spawn agent="claude" cwd="." prompt-file="": + {{BENCH}} spawn --agent {{agent}} --cwd {{cwd}} --prompt-file {{prompt-file}} + +# M3 — close a pane; the busy refusal and no-force-over-keyboard rules ride along. +close pane: + {{BENCH}} close --pane {{pane}} + +# M5 — attach this terminal to a bench pty. +attach pane: + {{BENCH}} attach --pane {{pane}} + +# ---- the scoreboard --------------------------------------------------------- + +# Probe every wanted verb against a throwaway daemon: ✓ answered / ✗ not yet. +spec: + #!/usr/bin/env bash + set -uo pipefail + export BENCH_DIR=$(mktemp -d "${TMPDIR:-/tmp}/bspc.XXXX")/r + (timeout 60 cargo run -q -p benchd &) + sleep 1 + echo "bench surface, $(date -u +%Y-%m-%dT%H:%MZ):" + for verb in status events stop:probe spawn close select name capture cmd \ + attn/post attn/list attn/ack watch mail/send mail/list mail/read attach; do + v=${verb%:probe} + if [ "$verb" = "stop:probe" ]; then + # probed last for real; listed here so the row exists + printf ' ✓ %-12s (M0)\n' "stop"; continue + fi + if cargo run -q -p bench -- "$v" >/dev/null 2>&1; then + printf ' ✓ %-12s\n' "$v" + else + printf ' ✗ %-12s not yet\n' "$v" + fi + done + cargo run -q -p bench -- stop >/dev/null 2>&1 || true From 3f0d829e282997f93c141af086ef1ed7a200e41f Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 12:11:06 +0300 Subject: [PATCH 3/9] =?UTF-8?q?docs(daemon):=20spike=20verdicts=20?= =?UTF-8?q?=E2=80=94=20pty=20ownership=20and=20the=20mail=20ring=20are=20p?= =?UTF-8?q?roven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two measurements that retire the design's riskiest bets before any milestone builds on them. pty-ownership: a headless process with no controlling terminal hosts full interactive claude/codex/pi TUIs, paste-then-submit prompt delivery, computed replies in 8-10s, live resize survived — invariant 3's core holds. mail-wake: agent-to-agent mail as files + an event log + a reactor answering mail/received with a pty-paste wake; a number passed around the claude→codex→pi→claude ring gained +1 per hop (100→103, 31.5s cycle), so every runtime read, computed and sent for real. The pty is the universal wake transport; claude's session socket and --bg stay optional optimizations. --- daemon/spikes/mail-wake.md | 63 ++++++++++++++++++++++++++++++++++ daemon/spikes/pty-ownership.md | 39 +++++++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 daemon/spikes/mail-wake.md create mode 100644 daemon/spikes/pty-ownership.md diff --git a/daemon/spikes/mail-wake.md b/daemon/spikes/mail-wake.md new file mode 100644 index 0000000..202ec96 --- /dev/null +++ b/daemon/spikes/mail-wake.md @@ -0,0 +1,63 @@ +# Spike: agent-to-agent mail with wakes, across runtimes, as log reactions + +**Question**: can agent↔agent mail — pi⇄claude⇄codex, all directions — work as nothing +but (a) files as the record, (b) an event log, (c) a reactor that answers +`mail/received` with `agent/wake`, using the pty-paste wake the pty spike proved? +This is the operator's own decomposition — *"mailbox is mailbox; if the daemon can +`agent/spawn`, then `mail/received ⇒ agent/wake` is just a reaction"* — put under test. + +**Verdict: PROVEN.** 2026-08-18, same throwaway harness family as the pty spike +(`portable-pty`, headless, no controlling terminal). Three agents in harness-owned +ptys; a number seeded as mail and passed around the ring, **each hop adds 1** — so a +correct final value cannot come from echoes; every runtime must read real mail, compute, +and send real mail. + +## The measurement + +Ring `claude → codex → pi → claude`, seed 100, expected 103. Result: **103, cycle +31.5s**, hops at 0.8s / 9.0s / 20.5s / 31.5s (~10s per hop: wake + read + compute + +write). Ready-to-brief time for all three agents: 10.5s. Postures: helm's +`SpoolUnattendedPolicy` (claude `--dangerously-skip-permissions`, codex +`--dangerously-bypass-approvals-and-sandbox`, pi `--approve`). + +The harness's own `events.jsonl` is the deliverable's shape as well as its evidence: + +``` +agent/spawned ×3 → agent/ready ×3 → mail/sent (harness seed) +→ mail/received(post-claude) → agent/woken(post-claude) +→ mail/received(post-codex) → agent/woken(post-codex) +→ mail/received(post-pi) → agent/woken(post-pi) +→ mail/received(post-claude, body 103) → cycle/complete +``` + +## Design consequences + +- **The pty is the universal wake transport.** One mechanism woke all three runtimes — + paste the notice (path, never body) into an idle composer, submit separately. No + per-runtime sockets were needed. Claude's session socket (#320, 0.12s) and `claude + --bg` remain available as *optimizations* for one runtime, to be evaluated at M3 as + the roadmap already says — they are not load-bearing for the design. +- **Wake policy is a reactor over the log, not part of mail.** The mailbox stayed pure + (directories + files + `mail/*` events); waking was a separate loop subscribing to + `mail/received`. Spawn-on-mail is the same reactor with one more rule — + `mail/received` for a handle with no live pane ⇒ `agent/spawn` — and loop caps on + agent↔agent chains belong in this reactor too (bench-side, roadmap M2). +- **Idle detection was the crude form and it sufficed** (pty quiet ≥2s). benchd's taps + (M1) make this judgement properly — which is the dependency that put attention before + mail in the roadmap. The spike confirms the *rest* of M2 carries no comparable risk. +- **Crate shape**: the mailbox rules (roots, handles, notice format) want to be a + `bench-mail` crate beside `bench-wire` — usable by benchd's verbs and by a standalone + CLI, which covers the "portable, maybe its own app" instinct without a second daemon. + +## Honest limits + +- The ring covered 3 of 6 directed pairs (each runtime sent once and received once); + the mechanism is pair-agnostic, but the other three pairs were not run. +- Recipients were sequenced idle; concurrent cross-traffic, a busy recipient, and the + wake-while-thinking case were not exercised — that is tap territory. +- One cosmetic harness bug: the `cycle/complete` log line prints Rust's `Some(103)` + rather than plain JSON. The measurement stands; the line would not pass benchd's own + boot-time log scan, which is a nice accidental proof of why that scan exists. + +Harness: scratchpad `pty-spike/src/bin/mail_spike.rs` (throwaway; this file is the +deliverable). Companion: `pty-ownership-verdict.md`. diff --git a/daemon/spikes/pty-ownership.md b/daemon/spikes/pty-ownership.md new file mode 100644 index 0000000..3bc49a9 --- /dev/null +++ b/daemon/spikes/pty-ownership.md @@ -0,0 +1,39 @@ +# Spike: can a headless daemon own the pty under a full interactive agent TUI? + +**Question** (bench-roadmap invariant 3, the design's riskiest bet): can a process with no +window and no controlling terminal — what benchd will be — own a pty hosting a full +interactive agent, deliver a prompt paste-then-submit, and get real work back? + +**Verdict: PROVEN, for all three runtimes.** 2026-08-18, throwaway Rust harness +(`portable-pty` 0.9), spawned from a Claude Code tool call (tty `??`, no controlling +terminal — the headless condition, not a simulation of it). Postures were helm's +`SpoolUnattendedPolicy` verbatim; prompt delivered as paste, then `\r` separately +(helm's launch-line rule); the required reply was **computed** (`6*7` → `BENCH-42-DONE`) +so the echo of our own typed bytes could not fake a pass. + +| runtime | posture flag | TUI drew | interactive | computed reply | resize mid-session | +|---|---|---|---|---|---| +| claude 2.1.234 | `--dangerously-skip-permissions` | yes | 3.9s | 10.0s | survived | +| codex 0.147.0 | `--dangerously-bypass-approvals-and-sandbox` | yes | 4.9s | 9.3s | survived | +| pi 0.83.0 | `--approve` | yes | 3.3s | 8.1s | survived | + +Notes with design weight: + +- **The pty owner's first duty is draining the master.** The harness reads continuously + on a thread; an undrained master blocks the agent on write. benchd's session loop must + never stop reading, whatever the attach state is. +- **Marker matching needed ANSI stripping and whitespace-tolerant search** — replies + arrive interleaved with redraw sequences. The attach protocol should ship grid state, + not raw byte scrollback, which is what the roadmap already says (M5). +- **codex's posture flag**: helm documents `-p yolo` (a profile); the spike used the + explicit `--dangerously-bypass-approvals-and-sandbox`, which worked on 0.147.0. The + M3 posture table should record both spellings and pick one. + +**Not proven here, still owed to M5**: the attach protocol itself (grid snapshot + +diffs, keys up), long-lived survival (hours, sleep/wake), reboot resume (typing +`--resume` into a fresh pty), and backpressure behavior when a reader stalls. The core +bet those all sit on — full TUIs run correctly under a headless pty owner — is the thing +this spike retires. + +Harness source: scratchpad `pty-spike/` (throwaway, session-local; this file is the +deliverable). From 2bb607196e5e411a90bbd68f4c92c8e40f4c7bae Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 12:15:21 +0300 Subject: [PATCH 4/9] docs(daemon): the group room is proven, and the posture is written down MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit group-room spike: a shared room directory with fan-out wakes generalizes the mailbox with no new mechanism — three runtimes, one question, three correct answer files, exactly the 9 wakes the fan-out arithmetic predicts, six NOOP turns observed, and the room settled on its own without hitting the cap. The recipient-set is the only concept a group adds; the loop brake (a wake cap per room) belongs in the reactor, never the mailbox. direction.md gains the posture as a rule: agents are smart — expose capabilities, never parse prose. Interpretation belongs to the model, determinism to the tool boundary; a capability that seems to need output-scraping is a missing tap or a missing verb. The roadmap's codex classification fallback is named as the one deliberate, dying exception. --- daemon/direction.md | 16 ++++++++++++++ daemon/spikes/group-room.md | 44 +++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 daemon/spikes/group-room.md diff --git a/daemon/direction.md b/daemon/direction.md index dc7bc38..c02d9af 100644 --- a/daemon/direction.md +++ b/daemon/direction.md @@ -69,6 +69,22 @@ benchd serves one operator whose gate is the pull request; composability-as-prod complexity bill dsh's own five-day-old ecosystem is already paying. Capabilities land as code in this workspace, reviewed, behind the one door. +## Posture: the agents are smart + +Adopted 2026-08-18, the operator's words made a rule. **We expose capabilities; we do +not parse prose.** Nothing in the bench regexes, keyword-matches, or otherwise +reconstructs meaning from human- or agent-written text — a reply is read by an agent, +not by a parser. Interpretation belongs to the model; determinism belongs at the tool +boundary (validated verbs, newtypes, exit codes), which is where `SuiteName` and +`RequestId` already sit. Concretely: the daemon never parses a mail body; notices point +rather than quote; status comes from taps — the runtimes' own typed hook and event +channels — and a capability that seems to need output-scraping is a missing tap or a +missing verb, not a regex waiting to be written. The one deliberate exception the +roadmap names: a last-resort output-classification fallback for a runtime with no tap +at all (M1, codex) — status inference only, never intent, retired the day the tap +exists. The spike harnesses' READY/NOOP markers were test instrumentation, not a +pattern to copy into the product. + ## Rules that bind every milestone The checklist form is bench-roadmap.md's invariants; the ones already load-bearing in diff --git a/daemon/spikes/group-room.md b/daemon/spikes/group-room.md new file mode 100644 index 0000000..55ea792 --- /dev/null +++ b/daemon/spikes/group-room.md @@ -0,0 +1,44 @@ +# Spike: a group mailbox — one room, three runtimes, fan-out wakes + +**Question**: does the mailbox generalize to a GROUP — a shared room where a message +from any member reaches every other member — on the same primitives the ring spike +proved (files, an event log, pty-paste wakes)? The genuinely new risks: fan-out (one +message wakes N−1 agents), loop amplification (every answer is itself a message that +wakes everyone again), and the assignment discipline (members hold full file tools but +are pointed at exactly one message per wake). + +**Verdict: PROVEN.** 2026-08-18. A room is a directory; a message is a file; the +reactor wakes every member except the author. Three members (claude, codex, pi, helm +postures), one seeded question (9×9), protocol: answer once as a file, NOOP every other +wake, read only the file the notice names. + +## The measurement + +- **All three answer files correct** (`member-x: 81`), no double-posts. +- **Exactly 9 wakes** — the fan-out arithmetic to the wake (3 for the seed + 3 answers + × 2 non-authors), against a cap of 12 that was never hit. +- **6 NOOP turns observed** — every answer-notice was received, read, and correctly + declined by members who had already posted. The room **settled on its own** in 46.9s: + traffic stopped because the protocol said stop, not because the cap fired. +- Concurrency was real: wakes about one message landed while other members were + mid-turn on the previous one; the per-agent idle wait serialized delivery per member + while members ran concurrently. + +## Design consequences + +- **A group is not a new mechanism.** Room = shared inbox directory; fan-out = the same + `mail/received ⇒ agent/wake` reactor with "everyone but the author" as the recipient + set. `bench-mail`'s model needs one concept (a recipient set on a mailbox), not a + second subsystem. +- **The amplification math is the thing to respect**: N members turn one message into + N−1 wakes, and every reply compounds it. A wake cap per room per window (the spike + carried 12) is the loop brake, and it belongs in the reactor — the mailbox stays pure. +- **The assignment discipline held under temptation.** Members had unrestricted file + tools and a directory full of siblings; pointed notices ("read only that file") were + followed. Capability open, policy in the brief — the posture, observed working. +- Not exercised: rooms at 5+, cross-room traffic, a member joining mid-conversation, + and adversarial protocol violation. The first busy real room will say more than + another synthetic run would. + +Harness: scratchpad `pty-spike/src/bin/group_spike.rs` (throwaway; this file is the +deliverable). Companions: `pty-ownership-verdict.md`, `mail-wake-verdict.md`. From 5f90ee92906d8b5ce59b9e1b8ef06cd20fe693e3 Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 12:15:21 +0300 Subject: [PATCH 5/9] chore(daemon): wire the daemon gate into AGENTS.md and CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The M0 loose ends: AGENTS.md gains the daemon/ gate block beside the pi/ and hooks/ carve-outs it mirrors, and CI gains a separate job triggered only by daemon/** — fmt, clippy -D warnings, build, then the conformance tests, needing only the Rust toolchain. The Swift jobs are untouched in both directions. --- .github/workflows/daemon.yml | 21 +++++++++++++++++++++ AGENTS.md | 13 +++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 .github/workflows/daemon.yml diff --git a/.github/workflows/daemon.yml b/.github/workflows/daemon.yml new file mode 100644 index 0000000..4b41848 --- /dev/null +++ b/.github/workflows/daemon.yml @@ -0,0 +1,21 @@ +# The daemon gate as its own CI job, per the pi/-and-hooks/ pattern: triggered only by +# daemon/** changes, needs only the Rust toolchain, and the Swift jobs never learn +# about it (bench-roadmap invariant 9). +name: daemon + +on: + pull_request: + paths: + - "daemon/**" + - ".github/workflows/daemon.yml" + +jobs: + gate: + name: fmt · clippy · build · test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + - run: bash daemon/test.sh diff --git a/AGENTS.md b/AGENTS.md index 1b6756d..caac4b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -169,6 +169,19 @@ the Swift gate must not. bash .claude/skills/pi-extensions/scripts/test.sh ``` +**If you touched `daemon/`, run its gate:** + +``` +bash daemon/test.sh +``` + +`daemon/` is the bench daemon (`benchd`) — a self-contained Rust cargo workspace, the +same carve-out as `pi/` and `hooks/`: its gate needs only the Rust toolchain, its CI job +triggers only on `daemon/**`, and the Swift gate never learns about it. Read +`daemon/direction.md` before working there; the milestone sequence is +`docs/future-planning/bench-roadmap.md`, and M0 (skeleton and suite isolation) is the +part that exists. + **If you touched `hooks/`, run its gate:** ``` From 5c0ac88976c507a713898d1af88ca6f61f96102c Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 12:18:37 +0300 Subject: [PATCH 6/9] docs(daemon): model and effort selection at spawn is proven for all three runtimes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven chrome-verified cases, zero model turns: claude --model/--effort, codex -m plus -c model_reasoning_effort, pi provider/model with the :thinking suffix — every selection accepted and declared by the runtime's own UI. The spawn verb gains {agent, model?, effort?} as one more column in the posture table; pi's usable set is its own catalog filtered by auth.json, so the bench asks the runtimes rather than keeping a catalog of its own. --- daemon/spikes/model-selection.md | 47 ++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 daemon/spikes/model-selection.md diff --git a/daemon/spikes/model-selection.md b/daemon/spikes/model-selection.md new file mode 100644 index 0000000..926c31f --- /dev/null +++ b/daemon/spikes/model-selection.md @@ -0,0 +1,47 @@ +# Spike: provider, model, and effort are controllable at spawn — and verifiable + +**Question**: when the bench spawns an agent, can the caller pick the model and the +reasoning effort per runtime — and can the selection be *verified* rather than trusted? + +**Verdict: PROVEN, 7/7 cases.** 2026-08-18. Zero model turns: each case spawns the TUI +into a harness-owned pty with selection flags and reads the chrome — the runtime's own +declaration of what it is running — then kills it. ~5s per case. + +| case | flags | chrome showed | +|---|---|---| +| claude sonnet | `--model sonnet` | sonnet ✓ | +| claude opus | `--model opus` | opus ✓ | +| claude opus, effort high | `--model opus --effort high` | opus + high ✓ | +| codex model | `-m gpt-5.3-codex` | 5.3 ✓ | +| codex effort high | `-c model_reasoning_effort=high` | 5.3 + high ✓ | +| pi sonnet | `--model anthropic/claude-sonnet-5` | sonnet ✓ | +| pi opus, thinking high | `--model anthropic/claude-opus-4-5:high` | opus + high ✓ | + +## The selection surface, per runtime + +- **claude**: `--model ` + `--effort ` — both first-class flags. +- **codex**: `-m/--model ` + `-c model_reasoning_effort=` (the generic + `-c key=value` config override). +- **pi**: `--provider ` / `--model ` supporting `provider/id` **with an + optional `:` suffix**, plus `--thinking off|minimal|low|medium|high|xhigh|max` + and `--models` for a cycling set. The catalog is `~/.pi/agent/models.json` (+ + `models-store.json`), listed by `pi --list-models`; **which entries are usable is + decided by `auth.json`** — the authenticated subset the operator cares about, not the + whole library. + +## Design consequences for `bench spawn` (M3) + +- The spawn verb carries `{agent, model?, effort?}` and maps them per runtime exactly as + the posture table already maps permission flags — one more column in the same + `SpoolUnattendedPolicy`-shaped table, not a new mechanism. Omitted means the runtime's + own default, never a bench-invented one. +- **Chrome-reading is spike instrumentation, not the product's verification.** benchd + should record what was *requested* in the spawn event (`agent/spawned` carries model + + effort) and leave verification to the runtime's own typed channels where they exist — + never a regex over the pane (posture: capabilities, not parsing). +- pi's authenticated-subset question suggests a later `bench spawn --list` passthrough + (ask each runtime what it can run) rather than bench maintaining its own catalog — + the runtimes already own that truth. + +Harness: scratchpad `pty-spike/src/bin/model_spike.rs` (throwaway; this file is the +deliverable). Companions: pty-ownership, mail-wake, group-room. From 0e03b3a15fb86666ce61ec5a33241723c71ccf14 Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 13:58:38 +0300 Subject: [PATCH 7/9] docs(daemon): resume and fork proven on all three runtimes; mail moves ahead of attention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit session-state spike: resume and fork work from a fresh pty for claude, codex and pi — with session ids minted by the spawner, which is the rule the spike also paid for the hard way (a recency-based pick grabbed a live session that was not ours; record the id at spawn, never infer it later). Two more findings with design weight: SIGKILL races the transcript write, so the resume-storm needs a drain-then-die grace; and a quiet heuristic is not a ready signal — actions should key off grid content, which is M5's layer. Rewind exists headless in claude but blind selection is guesswork; fork-from-id is the practical older-state tool everywhere. The Claude socket is deliberately not re-spiked: helm #320 already measured it, and its numbers say the pty wake stays primary. bench-roadmap.md records the operator's reordering: mail (M2) lands before the attention queue (M1), since the spikes proved the wake path without taps. --- daemon/spikes/session-state.md | 55 +++++++++++++++++++++++++++ docs/future-planning/bench-roadmap.md | 7 ++++ 2 files changed, 62 insertions(+) create mode 100644 daemon/spikes/session-state.md diff --git a/daemon/spikes/session-state.md b/daemon/spikes/session-state.md new file mode 100644 index 0000000..4a89686 --- /dev/null +++ b/daemon/spikes/session-state.md @@ -0,0 +1,55 @@ +# Spike: resume, fork, and rewind — re-entering session state from outside + +**Question**: the bench's recovery story (reboot resume, benchd restart as a +resume-storm) and its branching story assume the runtimes can re-enter and branch +session state from a fresh pty. Per runtime: RESUME (kill the pty, come back, context +intact), FORK (branch; the original stays), REWIND (attach to an older point). + +**Verdict: resume and fork PROVEN on all three runtimes; rewind PARTIAL (claude only, +UI proven drivable, programmatic selection needs a grid).** 2026-08-18, pty harness, +anti-echo probes (`CODEWORD=`/`FORKED=`/`ROLLED=` wrappers, since a resumed TUI redraws +old conversation into the grid and a bare match would pass on scrollback). + +| runtime | resume | fork | rewind-to-point | +|---|---|---|---| +| claude 2.1.234 | ✓ `--resume ` (id minted at spawn via `--session-id`) | ✓ `--resume --fork-session` | `/rewind` UI opens headless; blind selection landed on newest — needs grid-aware navigation | +| codex 0.147.0 | ✓ `codex resume ` | ✓ `codex fork ` — native | not surfaced via CLI | +| pi 0.83.0 | ✓ `-c` (continue in cwd) | ✓ `--fork ` — native | not surfaced via CLI | + +## Findings with design weight + +- **SIGKILL races the transcript write.** Killing a TUI immediately after its reply + reached the grid lost the session's last turn on every runtime until a 2–3s grace was + added. benchd's resume-storm and any pane close must give the child a drain-then-die + window, not a bare kill. +- **A settle heuristic is not a ready signal.** Resumed TUIs draw in bursts while + loading history; pasting on "quiet for 2s" intermittently landed before the composer + existed. The fix was a *content* ready-gate (claude's footer text). The general + lesson is the pty spike's again: benchd should key actions off grid state, which is + M5's layer. +- **Session identity should be minted by the spawner.** `claude --session-id ` + and pi's `--session-id` let benchd *choose* the id at spawn — no discovery, no + newest-file race. codex names its own; selection must be anchored to content or a + recorded id, never to recency (a recency pick during this spike grabbed the + operator's live codex session and forked it — harmless by fork semantics, but the + lesson is permanent: **record the id at spawn, never infer it later**). +- Rewind: the capability exists interactively in claude (checkpoint list renders fine + headless); driving it blind is guesswork. Either wait for M5's grid layer or use + fork-from-id as the practical "older state" tool — fork is proven everywhere. + +## The socket question (not re-run, and deliberately) + +helm #320 already measured the Claude socket end to end: idle wake 0.12s (default) / +1.01s (bypass + `crossSessionInbound: "accept"`), the bypass-without-setting case +**silently holds** with a modal nobody sees, platform dedup coalesces only identical +messages (a real ping-pong is never identical), the hook cannot distinguish a wake +from typing (so a cap in a hook counts nothing), and the wire format is unpublished — +the debug-line `{"type":"user","message":{...}}` shape. Its recommendation lands +exactly where benchd already stands: **the courier owns the wake and the cap**, +poke-as-doorbell if used at all, file mailbox canonical. For the bench: the pty wake +is proven, runtime-neutral, and ours; the socket buys ~a second of latency on one +runtime at the price of an unpublished format plus a settings precondition. Not +load-bearing — revisit only if idle-gating proves costly in practice. + +Harness: scratchpad `pty-spike/src/bin/session_spike.rs` (throwaway). Companions: +pty-ownership, mail-wake, group-room, model-selection; helm #320 for the socket. diff --git a/docs/future-planning/bench-roadmap.md b/docs/future-planning/bench-roadmap.md index daad8a7..70f8d3e 100644 --- a/docs/future-planning/bench-roadmap.md +++ b/docs/future-planning/bench-roadmap.md @@ -150,6 +150,13 @@ against the real CLI. **Prove:** two instances (live + suite) run side by side with zero shared state. **Unwire:** nothing. +> **Reordering, operator-ruled 2026-08-18: mail lands before the attention queue.** +> *"Attention can go later because we don't know where we want attention right now."* +> The tap dependency that originally ordered them is softened by measurement — the +> spike verdicts in `daemon/spikes/` proved the pty-paste wake on all three runtimes +> with a plain pty-quiet idle gate, so mail's wake path does not wait for taps. +> Milestone numbers below are left as written; start with M2, then return here. + ## M1 — Taps and the attention queue (purely additive) **Goal:** the first new capability helm never had, proving daemon + socket + CLI + taps From aec13ca93c1c39dfda27edceb9f9b69451cd196f Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 14:30:49 +0300 Subject: [PATCH 8/9] fix(daemon): a torn log tail heals, a stalled client cannot park the daemon MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings R1-R5 and R7 from PR #340, each fixed with its test written first and watched red against the old binaries. R1: a torn LAST line in events.jsonl no longer bricks the root — the tail is quarantined to events.jsonl.torn-, the log truncates to its last whole event, and the repair is itself logged (log/repaired), because bench-visible means logged applies to the daemon's own surgery too. A bad line in the middle still refuses, naming the line: that one is unexplained rather than an interrupted append. Appends now also sync_data. R2: I/O bounds on both sides, spelled once in bench-wire — the daemon gives a connection 5s and answers a refusal on timeout; the client gives an answer 15s (strictly longer, so the daemon's refusal outruns the client giving up) and maps a timeout to exit 2, because no exit code at all is the one failure an unattended agent cannot act on. R3: pre-socket refusals in both binaries now derive from Status::exit_code(). R4: a fresh log opens with a log/format marker event. R5: the verb set is KNOWN_VERBS in bench-wire — the dispatcher matches the parsed enum, the refusal derives its list, and a conformance test pins the justfile's probe line to it. R7: future-planning's README and roadmap now say M0 landed; the rest stays a proposal. Gate: 16 conformance + 7 wire tests green; the six new tests were red first (torn tail bricked, stalled client hung 12s+, mute daemon hung the client 25s+, no format marker) and the two that guard unchanged behavior stayed green. --- daemon/crates/bench-wire/src/lib.rs | 63 +++++++ daemon/crates/bench/src/main.rs | 19 +- daemon/crates/bench/tests/conformance.rs | 224 ++++++++++++++++++++++- daemon/crates/benchd/src/main.rs | 173 +++++++++++++---- docs/future-planning/README.md | 3 +- docs/future-planning/bench-roadmap.md | 2 + 6 files changed, 443 insertions(+), 41 deletions(-) diff --git a/daemon/crates/bench-wire/src/lib.rs b/daemon/crates/bench-wire/src/lib.rs index fd94c8a..0885500 100644 --- a/daemon/crates/bench-wire/src/lib.rs +++ b/daemon/crates/bench-wire/src/lib.rs @@ -114,6 +114,49 @@ impl RequestId { } } +// --------------------------------------------------------------------------- +// Verbs +// --------------------------------------------------------------------------- + +/// Every verb this daemon answers, spelled once (PR #340 review, R5). The refusal +/// string derives from this list, the dispatcher matches on the parsed enum so the +/// compiler forces a verdict when a verb is added, and the justfile's probe list is +/// pinned to it by a conformance test that reads the justfile's own source. +pub const KNOWN_VERBS: &[&str] = &["status", "events", "stop"]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Verb { + Status, + Events, + Stop, +} + +impl Verb { + /// `None` is an unknown verb — the caller owes a refusal naming `KNOWN_VERBS`. + pub fn parse(raw: &str) -> Option { + match raw { + "status" => Some(Verb::Status), + "events" => Some(Verb::Events), + "stop" => Some(Verb::Stop), + _ => None, + } + } +} + +// --------------------------------------------------------------------------- +// I/O bounds +// --------------------------------------------------------------------------- + +/// One connection may hold the daemon's serial loop for at most this long (R2): a +/// client that connects and never finishes its line gets a refusal, not the daemon. +pub const DAEMON_IO_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); + +/// A caller waits at most this long for an answer — strictly longer than the daemon's +/// own bound, so a daemon-side refusal always outruns the client giving up. A timeout +/// maps to `EXIT_NO_DAEMON`: no exit code at all is the one failure an unattended +/// agent cannot act on. +pub const CLIENT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15); + // --------------------------------------------------------------------------- // The envelope // --------------------------------------------------------------------------- @@ -173,6 +216,13 @@ pub const EXIT_NO_DAEMON: i32 = 2; // Events // --------------------------------------------------------------------------- +/// The first event of every fresh log declares the file's format (R4): the record is +/// read outside the process, and a reader that predates a change must fail loudly on +/// the marker instead of misreading history — `BenchSnapshot.format`'s rule, applied to +/// the file that matters most. +pub const EVENTS_LOG_FORMAT: &str = "bench.events-log"; +pub const EVENTS_LOG_VERSION: u64 = 0; + /// One line of the append-only record. **Bench-visible means logged**: anything a /// projection, a snapshot, or a later reader is allowed to know happened must be /// reconstructable from this stream — the file is the record, the socket is only @@ -306,6 +356,19 @@ mod tests { ); } + #[test] + fn every_known_verb_parses_and_nothing_else_does() { + for v in KNOWN_VERBS { + assert!(Verb::parse(v).is_some(), "{v} is listed but does not parse"); + } + assert_eq!( + KNOWN_VERBS.len(), + 3, + "a new verb joins KNOWN_VERBS and this count together" + ); + assert!(Verb::parse("frobnicate").is_none()); + } + #[test] fn status_maps_to_helm_exit_codes() { assert_eq!(Status::Ok.exit_code(), 0); diff --git a/daemon/crates/bench/src/main.rs b/daemon/crates/bench/src/main.rs index 9b6ce96..0ce9dc8 100644 --- a/daemon/crates/bench/src/main.rs +++ b/daemon/crates/bench/src/main.rs @@ -12,7 +12,8 @@ //! knows them, and a code that changes meaning across tools is worse than no code. use bench_wire::{ - EXIT_NO_DAEMON, Request, RequestId, Response, SuiteName, resolve_root, socket_path, + CLIENT_READ_TIMEOUT, DAEMON_IO_TIMEOUT, EXIT_NO_DAEMON, Request, RequestId, Response, Status, + SuiteName, resolve_root, socket_path, }; use serde_json::{Value, json}; use std::io::{BufRead, BufReader, Write}; @@ -100,6 +101,11 @@ fn run() -> i32 { return EXIT_NO_DAEMON; } }; + // Bounded on both directions (R2): a caller that hangs produces no exit code, + // which is the one failure an unattended agent cannot act on. A timeout is exit 2 + // — from the caller's seat, a daemon that never answers IS no daemon. + let _ = stream.set_write_timeout(Some(DAEMON_IO_TIMEOUT)); + let _ = stream.set_read_timeout(Some(CLIENT_READ_TIMEOUT)); let mut line = match serde_json::to_string(&request) { Ok(l) => l, @@ -113,7 +119,11 @@ fn run() -> i32 { let mut reply = String::new(); if BufReader::new(&stream).read_line(&mut reply).is_err() || reply.is_empty() { - eprintln!("bench: no answer from {}", sock.display()); + eprintln!( + "bench: no answer from {} within {}s", + sock.display(), + CLIENT_READ_TIMEOUT.as_secs() + ); return EXIT_NO_DAEMON; } @@ -134,14 +144,15 @@ fn run() -> i32 { response.status.exit_code() } +// Pre-socket exits derive from the same enum as socket-answered ones (R3). fn refuse(why: &str) -> i32 { eprintln!("bench: {why}"); - 3 + Status::Refused.exit_code() } fn fail(why: &str) -> i32 { eprintln!("bench: {why}"); - 4 + Status::Error.exit_code() } /// A fresh id per invocation, inside `RequestId`'s own pattern — validated, not assumed, diff --git a/daemon/crates/bench/tests/conformance.rs b/daemon/crates/bench/tests/conformance.rs index a24b8fc..0769a9b 100644 --- a/daemon/crates/bench/tests/conformance.rs +++ b/daemon/crates/bench/tests/conformance.rs @@ -232,7 +232,8 @@ fn stop_is_logged_before_it_is_answered_and_the_file_outlives_the_daemon() { .to_string() }) .collect(); - assert_eq!(kinds.first().map(String::as_str), Some("daemon/started")); + assert_eq!(kinds.first().map(String::as_str), Some("log/format")); + assert_eq!(kinds.get(1).map(String::as_str), Some("daemon/started")); assert_eq!(kinds.last().map(String::as_str), Some("daemon/stopped")); } @@ -245,8 +246,227 @@ fn events_reads_back_exactly_what_was_logged() { let data: serde_json::Value = serde_json::from_str(&run.stdout).unwrap(); assert_eq!(data["truncated"], false); let events = data["events"].as_array().unwrap(); - assert_eq!(events[0]["kind"], "daemon/started"); + assert_eq!(events[0]["kind"], "log/format"); assert_eq!(events[0]["seq"], 0); + assert_eq!(events[1]["kind"], "daemon/started"); +} + +// --------------------------------------------------------------------------- +// The record survives what interrupts it (PR #340 review, R1/R2/R4/R5) +// --------------------------------------------------------------------------- + +/// Run a command with a hard deadline; a hang is a FAILING outcome with its own name, +/// never a stuck test run. +fn run_bounded(cmd: &mut Command, deadline: Duration) -> Option { + let mut child = cmd + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("spawn bounded command"); + let start = Instant::now(); + loop { + if let Ok(Some(status)) = child.try_wait() { + let mut stdout = String::new(); + let mut stderr = String::new(); + use std::io::Read as _; + child + .stdout + .take() + .map(|mut s| s.read_to_string(&mut stdout)); + child + .stderr + .take() + .map(|mut s| s.read_to_string(&mut stderr)); + return Some(CliRun { + code: status.code().unwrap_or(-1), + stdout, + stderr, + }); + } + if start.elapsed() > deadline { + let _ = child.kill(); + let _ = child.wait(); + return None; + } + std::thread::sleep(Duration::from_millis(50)); + } +} + +fn write_seed_log(root: &Path, torn_tail: Option<&str>, garbage_middle: bool) { + fs::create_dir_all(root).unwrap(); + let mut log = String::new(); + log.push_str("{\"seq\":0,\"at\":\"2026-08-18T00:00:00Z\",\"kind\":\"log/format\",\"data\":{\"format\":\"bench.events-log\",\"version\":0}}\n"); + log.push_str("{\"seq\":1,\"at\":\"2026-08-18T00:00:01Z\",\"kind\":\"daemon/started\",\"data\":{\"pid\":1}}\n"); + if garbage_middle { + log.push_str("this line was never an event\n"); + } + log.push_str("{\"seq\":2,\"at\":\"2026-08-18T00:00:02Z\",\"kind\":\"daemon/stopped\",\"data\":{\"pid\":1}}\n"); + if let Some(tail) = torn_tail { + log.push_str(tail); // no newline: an interrupted append + } + fs::write(root.join("events.jsonl"), log).unwrap(); +} + +#[test] +fn a_torn_last_line_is_quarantined_and_the_daemon_starts() { + let home = TestHome::claim("torn"); + let root = home.dir.join("r"); + write_seed_log(&root, Some("{\"seq\":3,\"at\":\"2026-08-18T00:0"), false); + + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .env("BENCH_DIR", &root); + cmd.stdout(Stdio::null()).stderr(Stdio::null()); + let mut child = cmd.spawn().unwrap(); + let socket = root.join("benchd.sock"); + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&socket).is_err() { + std::thread::sleep(Duration::from_millis(20)); + } + + let status = Command::new(bench_bin()) + .env("HOME", &home.dir) + .env("BENCH_DIR", &root) + .arg("status") + .output() + .unwrap(); + let _ = child.kill(); + let _ = child.wait(); + assert_eq!( + status.status.code(), + Some(0), + "a torn LAST line must be forgiven, not brick the root: {}", + String::from_utf8_lossy(&status.stderr) + ); + + // The tail is quarantined beside the log — dropped bytes are named, never vanished. + let quarantined = fs::read_dir(&root) + .unwrap() + .filter_map(|e| e.ok()) + .any(|e| e.file_name().to_string_lossy().contains("torn")); + assert!( + quarantined, + "the torn tail must be quarantined, not silently discarded" + ); + + // Bench-visible means logged: the repair itself is an event in the record. + let log = fs::read_to_string(root.join("events.jsonl")).unwrap(); + assert!( + log.contains("log/repaired"), + "the repair must be logged: {log}" + ); +} + +#[test] +fn a_bad_line_in_the_middle_still_refuses_naming_the_line() { + let home = TestHome::claim("midbad"); + let root = home.dir.join("r"); + write_seed_log(&root, None, true); + + let out = Command::new(benchd_bin()) + .env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .env("BENCH_DIR", &root) + .output() + .unwrap(); + assert_eq!( + out.status.code(), + Some(3), + "unexplained middle corruption keeps refusing" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("line 3"), + "the refusal names the line: {stderr}" + ); +} + +#[test] +fn a_stalled_client_does_not_park_the_daemon() { + let home = TestHome::claim("stall"); + let daemon = DaemonGuard::start(&home.dir, None); + + // A client that connects, sends half a line, and just sits there. + let stalled = UnixStream::connect(&daemon.socket).unwrap(); + (&stalled) + .write_all(b"{\"id\":\"stall\",\"verb\":\"stat") + .unwrap(); + + let t0 = Instant::now(); + let run = run_bounded( + Command::new(bench_bin()) + .env("HOME", &home.dir) + .args(["status"]), + Duration::from_secs(12), + ); + drop(stalled); + let run = run.expect("bench status hung past 12s behind one stalled client — R2 unfixed"); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + assert!( + t0.elapsed() < Duration::from_secs(10), + "service must resume within the daemon's own I/O bound" + ); +} + +#[test] +fn a_daemon_that_never_answers_is_exit_2_not_a_hang() { + let home = TestHome::claim("mute"); + let root = home.dir.join(".bench"); + fs::create_dir_all(&root).unwrap(); + // A listener that accepts and never answers — the worst-behaved daemon possible. + let listener = std::os::unix::net::UnixListener::bind(root.join("benchd.sock")).unwrap(); + let _keep = std::thread::spawn(move || { + let mut held = Vec::new(); + while let Ok((s, _)) = listener.accept() { + held.push(s); + } + }); + + let run = run_bounded( + Command::new(bench_bin()) + .env("HOME", &home.dir) + .args(["status"]), + Duration::from_secs(25), + ); + let run = run.expect("bench hung past 25s on a mute daemon — the client has no read bound"); + assert_eq!( + run.code, 2, + "a timeout is EXIT_NO_DAEMON, never silence: {}", + run.stderr + ); +} + +#[test] +fn a_fresh_log_opens_with_its_format_marker() { + let home = TestHome::claim("fmt"); + let daemon = DaemonGuard::start(&home.dir, None); + let _ = &daemon; + let log = fs::read_to_string(home.dir.join(".bench/events.jsonl")).unwrap(); + let first: serde_json::Value = serde_json::from_str(log.lines().next().unwrap()).unwrap(); + assert_eq!(first["kind"], "log/format"); + assert_eq!(first["data"]["format"], "bench.events-log"); +} + +#[test] +fn the_justfile_probes_every_known_verb() { + // The probe list is hand-typed in the justfile (a `just` recipe cannot import a + // crate), so it is pinned here the way helm pins the spool scripts' id pattern: + // read the literal out of the source and compare (R5). + let justfile = fs::read_to_string(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../justfile")) + .expect("daemon/justfile readable from the bench crate"); + let probe_line: String = justfile + .lines() + .skip_while(|l| !l.contains("for verb in")) + .take_while(|l| !l.contains("; do")) + .collect::>() + .join(" "); + for verb in bench_wire::KNOWN_VERBS { + assert!( + probe_line.contains(verb), + "justfile spec probe list is missing known verb {verb:?}: {probe_line}" + ); + } } // --------------------------------------------------------------------------- diff --git a/daemon/crates/benchd/src/main.rs b/daemon/crates/benchd/src/main.rs index 80a2365..4f1427d 100644 --- a/daemon/crates/benchd/src/main.rs +++ b/daemon/crates/benchd/src/main.rs @@ -20,7 +20,8 @@ //! beats a concurrency story nothing needs yet. use bench_wire::{ - Event, MAX_REQUEST_BYTES, Request, Response, Status, SuiteName, check_socket_path, events_path, + DAEMON_IO_TIMEOUT, EVENTS_LOG_FORMAT, EVENTS_LOG_VERSION, Event, KNOWN_VERBS, + MAX_REQUEST_BYTES, Request, Response, Status, SuiteName, Verb, check_socket_path, events_path, resolve_root, socket_path, }; use serde_json::{Value, json}; @@ -87,14 +88,16 @@ fn run() -> i32 { } } +// Pre-socket exits derive from the same enum as socket-answered ones (R3): one +// spelling of the contract, no hand-typed twin to drift. fn refuse_start(why: &str) -> i32 { eprintln!("benchd: refusing to start: {why}"); - 3 + Status::Refused.exit_code() } fn fail_start(why: &str) -> i32 { eprintln!("benchd: {why}"); - 4 + Status::Error.exit_code() } enum StartError { @@ -102,6 +105,79 @@ enum StartError { Failed(String), } +struct RepairNote { + quarantine: PathBuf, + dropped_bytes: usize, +} + +/// Read the log with byte offsets. A clean log returns the next seq. An unreadable +/// line refuses — unless it is the LAST non-empty line, which is an interrupted append: +/// quarantine the tail to a named sibling, truncate the log back to its last good byte, +/// and report the repair so the caller can log it (R1). +fn scan_log(events: &PathBuf) -> Result<(u64, Option), StartError> { + let bytes = match fs::read(events) { + Ok(b) => b, + Err(_) => return Ok((0, None)), + }; + let text = String::from_utf8_lossy(&bytes); + let mut seq = 0u64; + let mut offset = 0usize; + let chunks: Vec<&str> = text.split_inclusive('\n').collect(); + for (i, chunk) in chunks.iter().enumerate() { + let line = chunk.trim_end_matches('\n'); + if line.trim().is_empty() { + offset += chunk.len(); + continue; + } + match serde_json::from_str::(line) { + Ok(ev) => { + seq = ev.seq + 1; + offset += chunk.len(); + } + Err(e) => { + let rest_is_empty = chunks[i + 1..].iter().all(|c| c.trim().is_empty()); + if !rest_is_empty { + return Err(StartError::Refused(format!( + "event log {} line {} is not a readable event ({e}) — refusing to append after history this daemon cannot read", + events.display(), + i + 1 + ))); + } + // Torn tail: quarantine, truncate, and say so loudly. + let epoch = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + let quarantine = events.with_file_name(format!("events.jsonl.torn-{epoch}")); + let dropped = &bytes[offset..]; + fs::write(&quarantine, dropped).map_err(|err| { + StartError::Failed(format!("cannot quarantine torn tail: {err}")) + })?; + let file = OpenOptions::new().write(true).open(events).map_err(|err| { + StartError::Failed(format!("cannot open log for repair: {err}")) + })?; + file.set_len(offset as u64).map_err(|err| { + StartError::Failed(format!("cannot truncate torn tail: {err}")) + })?; + eprintln!( + "benchd: event log {} ended in a torn line ({e}); {} byte(s) quarantined to {} and the log truncated to its last whole event", + events.display(), + dropped.len(), + quarantine.display() + ); + return Ok(( + seq, + Some(RepairNote { + quarantine, + dropped_bytes: dropped.len(), + }), + )); + } + } + } + Ok((seq, None)) +} + struct Daemon { root: PathBuf, suite: Option, @@ -146,32 +222,14 @@ impl Daemon { } // Boot-time integrity scan. The log is the record; a daemon that appends after - // a line it cannot read would be writing history it does not understand. Refuse - // with the line number rather than guessing (direction.md: refuse loudly). + // history it cannot read would be writing history it does not understand. One + // exception, from PR #340's review (R1): a torn LAST line is an interrupted + // append — the daemon's own crash mid-write, or ENOSPC part-way through a line + // — and refusing it forever bricks the root with no route out. The tail is + // quarantined beside the log and the repair is itself logged. A bad line in + // the MIDDLE stays a refusal naming the line: that one is unexplained. let events = events_path(&root); - let next_seq = match File::open(&events) { - Ok(f) => { - let mut seq = 0u64; - for (i, line) in BufReader::new(f).lines().enumerate() { - let line = line.map_err(|e| { - StartError::Failed(format!("cannot read {}: {e}", events.display())) - })?; - if line.trim().is_empty() { - continue; - } - let ev: Event = serde_json::from_str(&line).map_err(|e| { - StartError::Refused(format!( - "event log {} line {} is not a readable event ({e}) — refusing to append after history this daemon cannot read", - events.display(), - i + 1 - )) - })?; - seq = ev.seq + 1; - } - seq - } - Err(_) => 0, - }; + let (next_seq, repair) = scan_log(&events)?; let log = OpenOptions::new() .create(true) @@ -194,6 +252,28 @@ impl Daemon { started_at: now_rfc3339(), booted: Instant::now(), }; + // A fresh log opens with its format marker (R4): the file is read outside the + // process, and a reader that predates a change must fail on the marker rather + // than misread history. + if daemon.next_seq == 0 { + daemon + .append( + "log/format", + json!({ "format": EVENTS_LOG_FORMAT, "version": EVENTS_LOG_VERSION }), + ) + .map_err(StartError::Failed)?; + } + if let Some(note) = repair { + daemon + .append( + "log/repaired", + json!({ + "quarantine": note.quarantine.display().to_string(), + "dropped_bytes": note.dropped_bytes, + }), + ) + .map_err(StartError::Failed)?; + } daemon .append( "daemon/started", @@ -234,6 +314,11 @@ impl Daemon { } fn handle(&mut self, stream: UnixStream) -> Handled { + // Bounded in TIME as well as bytes (R2): this is the daemon's only loop, and a + // client that connects and never finishes its line must get the refusal, not + // the daemon. Timeouts are set before the reader clone so both share them. + let _ = stream.set_read_timeout(Some(DAEMON_IO_TIMEOUT)); + let _ = stream.set_write_timeout(Some(DAEMON_IO_TIMEOUT)); let mut reader = BufReader::new(match stream.try_clone() { Ok(s) => s, Err(_) => return Handled::Continue, @@ -242,6 +327,18 @@ impl Daemon { // Bounded read: a line that never ends must not become memory nobody asked for. let mut limited = (&mut reader).take(MAX_REQUEST_BYTES as u64 + 1); if limited.read_line(&mut line).is_err() { + respond( + &stream, + &Response { + id: "timed-out".into(), + status: Status::Refused, + reason: Some(format!( + "request not completed within {}s — one line, newline-terminated", + DAEMON_IO_TIMEOUT.as_secs() + )), + data: None, + }, + ); return Handled::Continue; } if line.len() > MAX_REQUEST_BYTES { @@ -281,16 +378,19 @@ impl Daemon { } fn dispatch(&mut self, req: &Request) -> (Response, Handled) { - match req.verb.as_str() { - "status" => (self.ok(req, self.status_data()), Handled::Continue), - "events" => { + // The verb is parsed, not string-matched (R5): the enum makes a new verb a + // compile-forced decision here, and the refusal derives its list from the same + // spelling the parser uses. + match Verb::parse(&req.verb) { + Some(Verb::Status) => (self.ok(req, self.status_data()), Handled::Continue), + Some(Verb::Events) => { let since = req.args.get("since").and_then(Value::as_u64).unwrap_or(0); match self.read_events(since) { Ok(data) => (self.ok(req, data), Handled::Continue), Err(why) => (self.error(req, why), Handled::Continue), } } - "stop" => { + Some(Verb::Stop) => { // Logged before answered: the record must already say "stopped" when the // caller is told it worked (bench-visible means logged). match self.append("daemon/stopped", json!({ "pid": process::id() })) { @@ -298,12 +398,14 @@ impl Daemon { Err(why) => (self.error(req, why), Handled::Continue), } } - other => ( + None => ( Response { id: req.id.clone(), status: Status::Refused, reason: Some(format!( - "unknown verb {other:?} — this daemon answers: status, events, stop" + "unknown verb {:?} — this daemon answers: {}", + req.verb, + KNOWN_VERBS.join(", ") )), data: None, }, @@ -372,6 +474,9 @@ impl Daemon { .write_all(line.as_bytes()) .and_then(|()| self.log.flush()) .map_err(|e| format!("cannot append to event log: {e}"))?; + // Best-effort durability: the record is the point of this process. A failed + // sync is not a failed append — the bytes are handed off either way. + let _ = self.log.sync_data(); self.next_seq += 1; Ok(()) } diff --git a/docs/future-planning/README.md b/docs/future-planning/README.md index 3e65e9c..2c59dff 100644 --- a/docs/future-planning/README.md +++ b/docs/future-planning/README.md @@ -1,6 +1,7 @@ # future-planning -**Nothing in this directory is built, and nothing in it authorises building.** +**M0 — the daemon skeleton at `daemon/` — is built (PR #340); everything else in this +directory remains unbuilt, and nothing in it authorises building.** These documents describe a proposed successor to helm — a headless daemon owning ptys and state, with the app as a thin face. They are here so the reasoning survives and can be argued diff --git a/docs/future-planning/bench-roadmap.md b/docs/future-planning/bench-roadmap.md index 70f8d3e..695959b 100644 --- a/docs/future-planning/bench-roadmap.md +++ b/docs/future-planning/bench-roadmap.md @@ -96,6 +96,8 @@ Each is argued in the audit doc; this is the checklist form. ## M0 — Skeleton and isolation +**Landed 2026-08-18, PR #340** — with the spike verdicts in `daemon/spikes/`. + **Goal:** `daemon/` exists, runs, and is safe to develop against on the machine that hosts the developers. From 0ecd931c8c800937615158403c88421de6fa5f81 Mon Sep 17 00:00:00 2001 From: Rasmus Widing Date: Tue, 18 Aug 2026 14:54:06 +0300 Subject: [PATCH 9/9] =?UTF-8?q?docs(daemon):=20split=20M5=20=E2=80=94=20th?= =?UTF-8?q?e=20daemon-session=20core=20lands=20before=20mail,=20the=20pain?= =?UTF-8?q?ter=20stays?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pty spike changed the price of half of M5, and the operator split it: M5a (benchd-owned ptys for new spawns, a dtach-grade raw attach relay, and the spawn/attach/close/resume verbs — no painter, no migration) is pulled forward ahead of mail, so the wake becomes one mechanism on a pty the daemon owns; M5b (painter, VT grid, migrating existing panes, the libghostty unwire) keeps its place. Interim costs are recorded in the note rather than discovered later. The daemon CI job now runs on every PR and exits early when daemon/ is untouched — it is about to become a required check, and a required check whose path filter keeps it from reporting blocks every Swift-only PR forever. --- .github/workflows/daemon.yml | 29 +++++++++++++++++++-------- docs/future-planning/bench-roadmap.md | 18 ++++++++++++++++- 2 files changed, 38 insertions(+), 9 deletions(-) diff --git a/.github/workflows/daemon.yml b/.github/workflows/daemon.yml index 4b41848..dbf59ff 100644 --- a/.github/workflows/daemon.yml +++ b/.github/workflows/daemon.yml @@ -1,13 +1,12 @@ -# The daemon gate as its own CI job, per the pi/-and-hooks/ pattern: triggered only by -# daemon/** changes, needs only the Rust toolchain, and the Swift jobs never learn -# about it (bench-roadmap invariant 9). +# The daemon gate as its own CI job, per the pi/-and-hooks/ pattern. It runs on EVERY +# pull request and exits early when daemon/ is untouched — deliberately, because this +# check is a required status: a job that only triggers on daemon/** paths never reports +# on a Swift-only PR, and a required check that never reports blocks the merge forever. +# Always report, spend nothing when there is nothing to gate. name: daemon on: pull_request: - paths: - - "daemon/**" - - ".github/workflows/daemon.yml" jobs: gate: @@ -15,7 +14,21 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: dtolnay/rust-toolchain@stable + with: + fetch-depth: 0 + - name: did daemon/ change? + id: changed + run: | + if git diff --name-only "origin/${{ github.base_ref }}...HEAD" \ + | grep -qE '^(daemon/|\.github/workflows/daemon\.yml)'; then + echo "run=true" >> "$GITHUB_OUTPUT" + else + echo "run=false" >> "$GITHUB_OUTPUT" + echo "daemon/ untouched — gate satisfied by construction" + fi + - if: steps.changed.outputs.run == 'true' + uses: dtolnay/rust-toolchain@stable with: components: rustfmt, clippy - - run: bash daemon/test.sh + - if: steps.changed.outputs.run == 'true' + run: bash daemon/test.sh diff --git a/docs/future-planning/bench-roadmap.md b/docs/future-planning/bench-roadmap.md index 695959b..06385a4 100644 --- a/docs/future-planning/bench-roadmap.md +++ b/docs/future-planning/bench-roadmap.md @@ -157,7 +157,8 @@ against the real CLI. > The tap dependency that originally ordered them is softened by measurement — the > spike verdicts in `daemon/spikes/` proved the pty-paste wake on all three runtimes > with a plain pty-quiet idle gate, so mail's wake path does not wait for taps. -> Milestone numbers below are left as written; start with M2, then return here. +> Milestone numbers below are left as written; the running order is **M5a (split out +> below) → M2 (mail) → M1 (attention)**. ## M1 — Taps and the attention queue (purely additive) @@ -272,6 +273,21 @@ from local disk. **Unwire:** `WorkspacePersistence`'s workbench half, `BenchSnapshotModel` (the snapshot becomes a benchd projection of the event stream, same file shape for readers). +> **Split, operator-ruled 2026-08-18: M5a is pulled forward to land before mail; M5b +> stays here.** The pty spike proved daemon-owned ptys hosting full TUIs, which makes +> the cheap half cheap and the wake story clean: +> +> - **M5a — the daemon-session core, before mail.** benchd owns ptys for **new spawns +> only**: `spawn`/`attach`/`close`/minimal `resume` verbs, a dtach-grade raw byte +> relay for attach (ring-buffer replay, resize; no VT grid, no painter), viewed by +> running `bench attach` inside an ordinary helm pane. Mail's wake then pastes into +> a pty benchd owns — uniform across claude, codex and pi, no per-runtime transports. +> Accepted interim costs, chosen knowingly: attach-hosted agents lose helm's pane +> `agent` records and mark-routing identity until M4/M5b; canvas-push passthrough +> through the relay is assumed and must be spiked before relying on it. +> - **M5b — everything below this note as written**: the painter, VT-grid state, +> migrating existing libghostty panes, and the unwiring list. Unchanged. + ## M5 — Ptys move, per-pane **Goal:** benchd owns the processes; the face becomes a painter; sessions survive the