diff --git a/.github/workflows/daemon.yml b/.github/workflows/daemon.yml new file mode 100644 index 0000000..dbf59ff --- /dev/null +++ b/.github/workflows/daemon.yml @@ -0,0 +1,34 @@ +# The daemon gate as its own CI job, per the pi/-and-hooks/ pattern. It runs on EVERY +# pull request and exits early when daemon/ is untouched — deliberately, because this +# check is a required status: a job that only triggers on daemon/** paths never reports +# on a Swift-only PR, and a required check that never reports blocks the merge forever. +# Always report, spend nothing when there is nothing to gate. +name: daemon + +on: + pull_request: + +jobs: + gate: + name: fmt · clippy · build · test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: did daemon/ change? + id: changed + run: | + if git diff --name-only "origin/${{ github.base_ref }}...HEAD" \ + | grep -qE '^(daemon/|\.github/workflows/daemon\.yml)'; then + echo "run=true" >> "$GITHUB_OUTPUT" + else + echo "run=false" >> "$GITHUB_OUTPUT" + echo "daemon/ untouched — gate satisfied by construction" + fi + - if: steps.changed.outputs.run == 'true' + uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + - if: steps.changed.outputs.run == 'true' + run: bash daemon/test.sh diff --git a/AGENTS.md b/AGENTS.md index 1b6756d..caac4b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -169,6 +169,19 @@ the Swift gate must not. bash .claude/skills/pi-extensions/scripts/test.sh ``` +**If you touched `daemon/`, run its gate:** + +``` +bash daemon/test.sh +``` + +`daemon/` is the bench daemon (`benchd`) — a self-contained Rust cargo workspace, the +same carve-out as `pi/` and `hooks/`: its gate needs only the Rust toolchain, its CI job +triggers only on `daemon/**`, and the Swift gate never learns about it. Read +`daemon/direction.md` before working there; the milestone sequence is +`docs/future-planning/bench-roadmap.md`, and M0 (skeleton and suite isolation) is the +part that exists. + **If you touched `hooks/`, run its gate:** ``` diff --git a/daemon/.gitignore b/daemon/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/daemon/.gitignore @@ -0,0 +1 @@ +/target diff --git a/daemon/AGENTS.md b/daemon/AGENTS.md new file mode 100644 index 0000000..4aaac51 --- /dev/null +++ b/daemon/AGENTS.md @@ -0,0 +1,42 @@ +# AGENTS.md — daemon/ + +The bench daemon: a self-contained cargo workspace, the `pi/`-style carve-out. Read +`direction.md` first; the milestone sequence and invariants are +`../docs/future-planning/bench-roadmap.md`. Vocabulary stays canonical in `../CONTEXT.md`. + +## Gate + +``` +bash daemon/test.sh +``` + +fmt-check, clippy `-D warnings`, build, then tests — **build before test is load-bearing**: +the conformance suite runs the real `benchd` binary as a subprocess and locates it beside +its own. Run this gate when `daemon/` changed; the Swift gate at the repo root neither +knows nor needs the Rust toolchain, in either direction. + +## Layout + +- `crates/bench-wire` — every wire type and shared resolution rule, spelled once. If + `benchd` and `bench` could disagree about a value, its rule belongs here. +- `crates/benchd` — the daemon. Foreground, one unix socket, serial request handling. +- `crates/bench` — the CLI, the one agent-facing surface (and the future skill surface). +- There is deliberately **no root `Cargo.toml`** in the repo: `cargo` at the repo root + fails loudly instead of half-working. + +## Rules + +- **Bench-visible means logged.** A mutation appends its event before the response that + reports it. A new capability is new event kinds + new verbs over the same socket — + never a second channel (no files-as-IPC, no extra sockets, no notification side paths). +- **Tests never touch the operator's estate.** Claim a disposable `HOME` (or `BENCH_DIR`) + under the OS tempdir — the OS tempdir specifically: unix socket paths cap near 104 + bytes and long scratch paths fail at bind. Include the negative control: assert the + shared root shape was never created (`hooks/test.sh`'s pattern). +- **Bounded children.** A test that spawns a daemon owns exactly that pid, kills it in a + Drop guard, and waits. Never kill by pattern (repo root AGENTS.md; #291 is why). +- **Exit codes are the contract**: 0 ok · 2 no daemon · 3 refused · 4 daemon failed. + A refusal names the rule it applied and the route to use instead. +- **Wire changes ride with their conformance test** in `crates/bench/tests/` — real + binaries, both directions, every status case, same as `SpoolWireConformanceTests`. +- Conventional commits, written as a human — no AI attribution (repo rule). diff --git a/daemon/Cargo.lock b/daemon/Cargo.lock new file mode 100644 index 0000000..99c4aa2 --- /dev/null +++ b/daemon/Cargo.lock @@ -0,0 +1,172 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bench" +version = "0.0.1" +dependencies = [ + "bench-wire", + "serde_json", +] + +[[package]] +name = "bench-wire" +version = "0.0.1" +dependencies = [ + "serde", + "serde_json", +] + +[[package]] +name = "benchd" +version = "0.0.1" +dependencies = [ + "bench-wire", + "serde_json", + "time", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "time" +version = "0.3.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/daemon/Cargo.toml b/daemon/Cargo.toml new file mode 100644 index 0000000..72e6da2 --- /dev/null +++ b/daemon/Cargo.toml @@ -0,0 +1,14 @@ +# The bench daemon workspace. Deliberately NOT reachable from a repo-root Cargo.toml — +# `cargo` at the repo root should fail loudly, not half-work (bench-roadmap.md, M0). +[workspace] +resolver = "2" +members = ["crates/bench-wire", "crates/benchd", "crates/bench"] + +[workspace.package] +version = "0.0.1" +edition = "2024" + +[workspace.dependencies] +serde = { version = "1", features = ["derive"] } +serde_json = "1" +time = { version = "0.3", features = ["formatting"] } diff --git a/daemon/crates/bench-wire/Cargo.toml b/daemon/crates/bench-wire/Cargo.toml new file mode 100644 index 0000000..cdaeda0 --- /dev/null +++ b/daemon/crates/bench-wire/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "bench-wire" +version.workspace = true +edition.workspace = true +description = "Every bench wire type and shared resolution rule, spelled once." + +[dependencies] +serde.workspace = true +serde_json.workspace = true diff --git a/daemon/crates/bench-wire/src/lib.rs b/daemon/crates/bench-wire/src/lib.rs new file mode 100644 index 0000000..0885500 --- /dev/null +++ b/daemon/crates/bench-wire/src/lib.rs @@ -0,0 +1,386 @@ +//! Every wire type and every shared resolution rule, spelled once. +//! +//! This crate exists so that `benchd` and `bench` can never disagree about what travels +//! on the socket or where a suite's state lives. helm spent real incidents on the other +//! arrangement — three copies of the mailbox rule in three languages, held together by a +//! conformance harness (`hooks/mailbox-conformance.mjs`). Rust on both ends of this socket +//! means the single spelling is finally free; anything that later reads these types from +//! Swift gets a generated or conformance-pinned copy, never a hand-written one +//! (bench-roadmap.md, invariant 9). +//! +//! The protocol itself is deliberately small: one connection carries one JSON request line +//! and one JSON response line, then closes. No framing, no multiplexing, no versioned +//! handshake — those arrive when a milestone needs them, and `Request`/`Response` carry +//! nothing a later field cannot extend compatibly. + +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use std::path::{Path, PathBuf}; + +/// A request line larger than this is refused, not read. The cap is about the reader: +/// every accepted byte can end up in an event log an agent later pulls into context. +/// Same argument as helm's 64 KB canvas-state cap. +pub const MAX_REQUEST_BYTES: usize = 64 * 1024; + +// --------------------------------------------------------------------------- +// Suite names +// --------------------------------------------------------------------------- + +/// A validated suite name — the isolation primitive, ported from helm's +/// `HELM_DEFAULTS_SUITE` (#86) with the same posture: **refuse loudly rather than fall +/// back to the live instance.** A test that launches under a suite it believes isolates +/// it, and silently lands in the operator's `~/.bench`, is exactly the disaster helm +/// #285 documents. So a name that cannot isolate is an error at the edge, never a +/// fallback to the shared root. +/// +/// The unchecked value is unrepresentable: the only route in is `validate`, and every +/// path builder below takes `&SuiteName`, not `&str`. (helm's `RequestID`/#260 argument, +/// applied on day one instead of after the incident.) +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(transparent)] +pub struct SuiteName(String); + +impl SuiteName { + /// Lowercase ASCII letters, digits and `-`; must start alphanumeric; at most 32 + /// bytes. Everything else is refused with a reason that names the rule, because the + /// refusal is read by an agent that has to fix its call. + pub fn validate(raw: &str) -> Result { + if raw.is_empty() { + return Err( + "a suite name cannot be empty — unset BENCH_SUITE for the live instance".into(), + ); + } + if raw.contains('/') || raw.contains('\\') || raw.contains("..") { + return Err(format!( + "a path is not a suite name: {raw:?} — the suite decides the directory, never names it" + )); + } + if raw.len() > 32 { + return Err(format!( + "suite name too long ({} bytes, max 32): {raw:?}", + raw.len() + )); + } + let mut chars = raw.chars(); + let first_ok = chars + .next() + .is_some_and(|c| c.is_ascii_lowercase() || c.is_ascii_digit()); + let rest_ok = raw + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'); + if !first_ok || !rest_ok { + return Err(format!( + "a suite name is lowercase ASCII letters, digits and '-', starting alphanumeric: {raw:?}" + )); + } + Ok(SuiteName(raw.to_string())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +// --------------------------------------------------------------------------- +// Request ids +// --------------------------------------------------------------------------- + +/// A validated request id. Port of helm's `RequestID` (#260), not a reinvention: the +/// pattern is the filename-safe one, because M3's socketless drop-box will use ids as +/// filenames and an ungated id writes wherever the caller likes. Gating it now costs one +/// type; gating it at M3 would be a migration. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(transparent)] +pub struct RequestId(String); + +impl RequestId { + /// First char ASCII alphanumeric; the rest alphanumeric, `.`, `_` or `-`; 1–64 bytes. + pub fn validate(raw: &str) -> Result { + let mut chars = raw.chars(); + let first_ok = chars.next().is_some_and(|c| c.is_ascii_alphanumeric()); + let rest_ok = raw + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '_' || c == '-'); + if raw.is_empty() || raw.len() > 64 || !first_ok || !rest_ok { + return Err(format!( + "a request id is 1-64 bytes of [A-Za-z0-9._-], starting alphanumeric: {raw:?}" + )); + } + Ok(RequestId(raw.to_string())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +// --------------------------------------------------------------------------- +// Verbs +// --------------------------------------------------------------------------- + +/// Every verb this daemon answers, spelled once (PR #340 review, R5). The refusal +/// string derives from this list, the dispatcher matches on the parsed enum so the +/// compiler forces a verdict when a verb is added, and the justfile's probe list is +/// pinned to it by a conformance test that reads the justfile's own source. +pub const KNOWN_VERBS: &[&str] = &["status", "events", "stop"]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Verb { + Status, + Events, + Stop, +} + +impl Verb { + /// `None` is an unknown verb — the caller owes a refusal naming `KNOWN_VERBS`. + pub fn parse(raw: &str) -> Option { + match raw { + "status" => Some(Verb::Status), + "events" => Some(Verb::Events), + "stop" => Some(Verb::Stop), + _ => None, + } + } +} + +// --------------------------------------------------------------------------- +// I/O bounds +// --------------------------------------------------------------------------- + +/// One connection may hold the daemon's serial loop for at most this long (R2): a +/// client that connects and never finishes its line gets a refusal, not the daemon. +pub const DAEMON_IO_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); + +/// A caller waits at most this long for an answer — strictly longer than the daemon's +/// own bound, so a daemon-side refusal always outruns the client giving up. A timeout +/// maps to `EXIT_NO_DAEMON`: no exit code at all is the one failure an unattended +/// agent cannot act on. +pub const CLIENT_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15); + +// --------------------------------------------------------------------------- +// The envelope +// --------------------------------------------------------------------------- + +/// What a caller sends. `id` and `verb` stay raw `String`s **on purpose**: a request is +/// decoded permissively in shape and judged strictly afterwards, so a malformed id or an +/// unknown verb is a `refused` response naming the reason rather than unreadable JSON +/// with no reply. That is helm's standing carve-out (`CloseRequest.terminal`, +/// `SpawnRequest.cwd`) and it is load-bearing here for the same reason. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Request { + pub id: String, + pub verb: String, + #[serde(default)] + pub args: Value, +} + +/// Every response carries a status, and the status is the exit code: a caller never +/// parses prose to learn what happened. `reason` is for humans and agents; `data` is the +/// verb's payload. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Response { + pub id: String, + pub status: Status, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reason: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub data: Option, +} + +/// The three outcomes, mapped onto helm's spool exit-code discipline. `2` (no daemon) is +/// deliberately absent: it is the *transport's* failure, decided by the caller when the +/// socket cannot be reached, never something a daemon could say about itself. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Status { + Ok, + Refused, + Error, +} + +impl Status { + /// 0 ok / 3 refused / 4 daemon failed — helm's spool codes, kept (bench-roadmap M0). + pub fn exit_code(self) -> i32 { + match self { + Status::Ok => 0, + Status::Refused => 3, + Status::Error => 4, + } + } +} + +/// The caller-side exit for "the socket could not be reached at all". +pub const EXIT_NO_DAEMON: i32 = 2; + +// --------------------------------------------------------------------------- +// Events +// --------------------------------------------------------------------------- + +/// The first event of every fresh log declares the file's format (R4): the record is +/// read outside the process, and a reader that predates a change must fail loudly on +/// the marker instead of misreading history — `BenchSnapshot.format`'s rule, applied to +/// the file that matters most. +pub const EVENTS_LOG_FORMAT: &str = "bench.events-log"; +pub const EVENTS_LOG_VERSION: u64 = 0; + +/// One line of the append-only record. **Bench-visible means logged**: anything a +/// projection, a snapshot, or a later reader is allowed to know happened must be +/// reconstructable from this stream — the file is the record, the socket is only +/// transport (bench-roadmap, invariants 7 and the dsh lesson in direction.md). +/// +/// `kind` is namespaced `domain/what` (`daemon/started`). A reader that meets a kind it +/// does not know must refuse or skip *visibly*, never misread it — which is why the +/// envelope stays this small and flat. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Event { + pub seq: u64, + /// RFC 3339 UTC. A `cat` of the log is meant to be readable by a human having a bad + /// morning; epoch integers are not that. + pub at: String, + pub kind: String, + #[serde(default, skip_serializing_if = "Value::is_null")] + pub data: Value, +} + +// --------------------------------------------------------------------------- +// Where a bench lives on disk +// --------------------------------------------------------------------------- + +/// Resolve the record root. One rule, spelled once, used by both binaries: +/// +/// 1. `BENCH_DIR` names the root outright and wins over everything — it is what a test +/// claims into instead of the operator's estate (helm's `HELM_MAIL_DIR` rule). +/// 2. else `/.bench-` when a suite is set, +/// 3. else the shared `/.bench`. +/// +/// `home` is a parameter, not a `$HOME` read, so the rule is testable and the caller is +/// forced to say whose home it means. +pub fn resolve_root(bench_dir: Option<&str>, suite: Option<&SuiteName>, home: &Path) -> PathBuf { + if let Some(dir) = bench_dir { + return PathBuf::from(dir); + } + match suite { + Some(s) => home.join(format!(".bench-{}", s.as_str())), + None => home.join(".bench"), + } +} + +pub fn socket_path(root: &Path) -> PathBuf { + root.join("benchd.sock") +} + +pub fn events_path(root: &Path) -> PathBuf { + root.join("events.jsonl") +} + +/// A `sockaddr_un` path is capped (~104 bytes on macOS), and exceeding it fails at bind +/// with an error that names none of this. Check it where the path is decided and say +/// what to do about it. +pub fn check_socket_path(path: &Path) -> Result<(), String> { + let len = path.as_os_str().len(); + if len > 100 { + return Err(format!( + "socket path is {len} bytes; unix sockets cap near 104 — point BENCH_DIR at a shorter path: {}", + path.display() + )); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn suite_names_that_isolate_are_accepted() { + for ok in ["x", "bench-dev", "a1", "m0-smoke"] { + assert!( + SuiteName::validate(ok).is_ok(), + "{ok} should be a valid suite" + ); + } + } + + #[test] + fn suite_names_that_cannot_isolate_are_refused_not_defaulted() { + for bad in [ + "", + "has/slash", + "..", + "a..b", + "UPPER", + "with space", + "-leading", + "x".repeat(33).as_str(), + ] { + assert!( + SuiteName::validate(bad).is_err(), + "{bad:?} should be refused" + ); + } + } + + #[test] + fn request_ids_follow_the_filename_safe_pattern() { + assert!(RequestId::validate("bench-123-9f").is_ok()); + assert!(RequestId::validate("a.b_c-d").is_ok()); + for bad in [ + "", + "../../etc", + "-lead", + "id with space", + "x".repeat(65).as_str(), + ] { + assert!( + RequestId::validate(bad).is_err(), + "{bad:?} should be refused" + ); + } + } + + #[test] + fn bench_dir_wins_suite_decorates_shared_is_default() { + let home = Path::new("/home/op"); + let suite = SuiteName::validate("dev").unwrap(); + assert_eq!( + resolve_root(Some("/claimed/root"), Some(&suite), home), + PathBuf::from("/claimed/root") + ); + assert_eq!( + resolve_root(None, Some(&suite), home), + PathBuf::from("/home/op/.bench-dev") + ); + assert_eq!( + resolve_root(None, None, home), + PathBuf::from("/home/op/.bench") + ); + } + + #[test] + fn every_known_verb_parses_and_nothing_else_does() { + for v in KNOWN_VERBS { + assert!(Verb::parse(v).is_some(), "{v} is listed but does not parse"); + } + assert_eq!( + KNOWN_VERBS.len(), + 3, + "a new verb joins KNOWN_VERBS and this count together" + ); + assert!(Verb::parse("frobnicate").is_none()); + } + + #[test] + fn status_maps_to_helm_exit_codes() { + assert_eq!(Status::Ok.exit_code(), 0); + assert_eq!(Status::Refused.exit_code(), 3); + assert_eq!(Status::Error.exit_code(), 4); + assert_eq!(EXIT_NO_DAEMON, 2); + } + + #[test] + fn overlong_socket_paths_are_named_before_bind() { + let long = PathBuf::from(format!("/{}", "d".repeat(120))); + assert!(check_socket_path(&long).is_err()); + assert!(check_socket_path(Path::new("/tmp/b/benchd.sock")).is_ok()); + } +} diff --git a/daemon/crates/bench/Cargo.toml b/daemon/crates/bench/Cargo.toml new file mode 100644 index 0000000..5e08ccb --- /dev/null +++ b/daemon/crates/bench/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "bench" +version.workspace = true +edition.workspace = true +description = "The bench CLI: the one agent-facing surface over the benchd socket." + +[dependencies] +bench-wire = { path = "../bench-wire" } +serde_json.workspace = true diff --git a/daemon/crates/bench/src/main.rs b/daemon/crates/bench/src/main.rs new file mode 100644 index 0000000..0ce9dc8 --- /dev/null +++ b/daemon/crates/bench/src/main.rs @@ -0,0 +1,169 @@ +//! bench — the CLI, which is also the future agent skill surface (bench-roadmap M0/M3). +//! +//! One connection, one JSON request line, one JSON response line. The exit code IS the +//! outcome — an agent reads `$?`, not prose: +//! +//! 0 ok (the verb's data, pretty JSON, on stdout) +//! 2 no daemon (the socket could not be reached — transport, not a daemon answer) +//! 3 refused (the daemon said no and named why, on stderr) +//! 4 daemon failed (the daemon tried and could not, named why, on stderr) +//! +//! These are helm's spool codes, kept on purpose: every agent skill in this repo already +//! knows them, and a code that changes meaning across tools is worse than no code. + +use bench_wire::{ + CLIENT_READ_TIMEOUT, DAEMON_IO_TIMEOUT, EXIT_NO_DAEMON, Request, RequestId, Response, Status, + SuiteName, resolve_root, socket_path, +}; +use serde_json::{Value, json}; +use std::io::{BufRead, BufReader, Write}; +use std::os::unix::net::UnixStream; +use std::path::PathBuf; +use std::process; +use std::time::{SystemTime, UNIX_EPOCH}; + +fn main() { + process::exit(run()); +} + +fn usage() -> &'static str { + "usage: bench [--suite ] \n\ + verbs: status daemon identity, root, uptime, event count\n\ + \x20 events [--since N] read the record back from seq N\n\ + \x20 stop ask the daemon to log its stop and exit\n\ + env: BENCH_SUITE (flag wins) · BENCH_DIR (root override, wins over suite)\n\ + exit: 0 ok · 2 no daemon · 3 refused · 4 daemon failed" +} + +fn run() -> i32 { + let mut args = std::env::args().skip(1).peekable(); + let mut suite_flag: Option = None; + let mut verb: Option = None; + let mut since: u64 = 0; + + while let Some(arg) = args.next() { + match arg.as_str() { + "--suite" => match args.next() { + Some(v) => suite_flag = Some(v), + None => return refuse("--suite needs a name"), + }, + "--since" => match args.next().and_then(|v| v.parse::().ok()) { + Some(n) => since = n, + None => return refuse("--since needs a sequence number"), + }, + "--help" | "-h" => { + println!("{}", usage()); + return 0; + } + other if verb.is_none() && !other.starts_with('-') => verb = Some(other.to_string()), + other => return refuse(&format!("unknown argument {other:?}\n{}", usage())), + } + } + + let Some(verb) = verb else { + return refuse(usage()); + }; + + // The suite is validated here, before any socket is touched: a name that cannot + // isolate must never resolve to the shared root by accident (#86 semantics). The + // daemon applies the identical rule from the same crate — one spelling, two edges. + let suite_raw = suite_flag.or_else(|| std::env::var("BENCH_SUITE").ok()); + let suite = match suite_raw.as_deref() { + Some(raw) => match SuiteName::validate(raw) { + Ok(s) => Some(s), + Err(why) => return refuse(&why), + }, + None => None, + }; + let home = match std::env::var("HOME") { + Ok(h) => PathBuf::from(h), + Err(_) => return refuse("HOME is not set; bench cannot resolve a record root"), + }; + let bench_dir = std::env::var("BENCH_DIR").ok(); + let root = resolve_root(bench_dir.as_deref(), suite.as_ref(), &home); + let sock = socket_path(&root); + + let args_value = match verb.as_str() { + "events" if since > 0 => json!({ "since": since }), + _ => Value::Null, + }; + + let request = Request { + id: request_id(), + verb, + args: args_value, + }; + + let stream = match UnixStream::connect(&sock) { + Ok(s) => s, + Err(e) => { + eprintln!("bench: no daemon at {} ({e})", sock.display()); + return EXIT_NO_DAEMON; + } + }; + // Bounded on both directions (R2): a caller that hangs produces no exit code, + // which is the one failure an unattended agent cannot act on. A timeout is exit 2 + // — from the caller's seat, a daemon that never answers IS no daemon. + let _ = stream.set_write_timeout(Some(DAEMON_IO_TIMEOUT)); + let _ = stream.set_read_timeout(Some(CLIENT_READ_TIMEOUT)); + + let mut line = match serde_json::to_string(&request) { + Ok(l) => l, + Err(e) => return fail(&format!("cannot encode request: {e}")), + }; + line.push('\n'); + if let Err(e) = (&stream).write_all(line.as_bytes()) { + eprintln!("bench: write to {} failed ({e})", sock.display()); + return EXIT_NO_DAEMON; + } + + let mut reply = String::new(); + if BufReader::new(&stream).read_line(&mut reply).is_err() || reply.is_empty() { + eprintln!( + "bench: no answer from {} within {}s", + sock.display(), + CLIENT_READ_TIMEOUT.as_secs() + ); + return EXIT_NO_DAEMON; + } + + let response: Response = match serde_json::from_str(&reply) { + Ok(r) => r, + Err(e) => return fail(&format!("unreadable response ({e}): {}", reply.trim())), + }; + + if let Some(reason) = &response.reason { + eprintln!("bench: {reason}"); + } + if let Some(data) = &response.data { + match serde_json::to_string_pretty(data) { + Ok(pretty) => println!("{pretty}"), + Err(_) => println!("{data}"), + } + } + response.status.exit_code() +} + +// Pre-socket exits derive from the same enum as socket-answered ones (R3). +fn refuse(why: &str) -> i32 { + eprintln!("bench: {why}"); + Status::Refused.exit_code() +} + +fn fail(why: &str) -> i32 { + eprintln!("bench: {why}"); + Status::Error.exit_code() +} + +/// A fresh id per invocation, inside `RequestId`'s own pattern — validated, not assumed, +/// so the client can never send an id the daemon-side rule would refuse. +fn request_id() -> String { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.subsec_nanos()) + .unwrap_or(0); + let candidate = format!("bench-{}-{nanos:08x}", process::id()); + RequestId::validate(&candidate) + .map(|id| id.as_str().to_string()) + .unwrap_or_else(|_| "bench-fallback".to_string()) +} diff --git a/daemon/crates/bench/tests/conformance.rs b/daemon/crates/bench/tests/conformance.rs new file mode 100644 index 0000000..0769a9b --- /dev/null +++ b/daemon/crates/bench/tests/conformance.rs @@ -0,0 +1,566 @@ +//! Conformance: run the REAL binaries as subprocesses and check both directions — +//! the `SpoolWireConformanceTests` shape, ported (bench-roadmap, working discipline). +//! Nothing here mocks the socket, the daemon, or the CLI; what these tests pass is what +//! an agent's shell gets. +//! +//! Ground rules carried from helm's incidents: +//! - **Never the operator's estate.** Every test claims its own `HOME` under the OS +//! tempdir, and the negative control asserts the shared `~/.bench` shape was never +//! created there (#285's lesson: isolation is proven, not assumed). +//! - **Bounded children.** Every daemon is killed by the guard's Drop by the pid we +//! spawned — never a pattern — and waited on (#291's lesson). +//! - Requires a prior `cargo build --workspace` (daemon/test.sh does this): the benchd +//! binary is located beside our own CARGO_BIN_EXE path. + +use std::fs; +use std::io::{BufRead, BufReader, Write}; +use std::os::unix::net::UnixStream; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::sync::atomic::{AtomicU32, Ordering}; +use std::time::{Duration, Instant}; + +fn bench_bin() -> PathBuf { + PathBuf::from(env!("CARGO_BIN_EXE_bench")) +} + +fn benchd_bin() -> PathBuf { + let p = bench_bin().parent().unwrap().join("benchd"); + assert!( + p.exists(), + "benchd binary not built — run daemon/test.sh (or cargo build --workspace) first" + ); + p +} + +/// A disposable HOME under the OS tempdir. The OS tempdir, not the repo and not a long +/// scratch path: unix socket paths cap near 104 bytes, and bench-wire refuses overlong +/// ones — a test home has to stay short enough to bind in. +struct TestHome { + dir: PathBuf, +} + +impl TestHome { + /// The name is SHORT on purpose (`bcf--`): the macOS per-user tempdir is + /// already ~50 bytes, and `/.bench-/benchd.sock` has to stay under the + /// ~104-byte `sun_path` cap. A descriptive label here once pushed every socket past + /// it and all eight daemons refused to start — correctly, and uselessly. The `label` + /// is kept only for the panic message. + fn claim(label: &str) -> TestHome { + static NEXT: AtomicU32 = AtomicU32::new(0); + let n = NEXT.fetch_add(1, Ordering::Relaxed); + let dir = std::env::temp_dir().join(format!("bcf-{}-{n}", std::process::id())); + fs::create_dir_all(&dir).unwrap_or_else(|e| panic!("claim home for {label}: {e}")); + TestHome { dir } + } +} + +impl Drop for TestHome { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.dir); + } +} + +/// Owns exactly the daemon it spawned; Drop kills that pid and waits. +struct DaemonGuard { + child: Child, + socket: PathBuf, +} + +impl DaemonGuard { + fn start(home: &Path, suite: Option<&str>) -> DaemonGuard { + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", home) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let root = match suite { + Some(s) => { + cmd.arg("--suite").arg(s); + home.join(format!(".bench-{s}")) + } + None => home.join(".bench"), + }; + let child = cmd.spawn().expect("spawn benchd"); + let socket = root.join("benchd.sock"); + let guard = DaemonGuard { child, socket }; + guard.await_socket(); + guard + } + + fn await_socket(&self) { + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline { + if UnixStream::connect(&self.socket).is_ok() { + return; + } + std::thread::sleep(Duration::from_millis(20)); + } + panic!("benchd never answered at {}", self.socket.display()); + } +} + +impl Drop for DaemonGuard { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + +struct CliRun { + code: i32, + stdout: String, + stderr: String, +} + +fn bench(home: &Path, args: &[&str]) -> CliRun { + let out = Command::new(bench_bin()) + .env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", home) + .args(args) + .output() + .expect("run bench"); + CliRun { + code: out.status.code().unwrap_or(-1), + stdout: String::from_utf8_lossy(&out.stdout).into_owned(), + stderr: String::from_utf8_lossy(&out.stderr).into_owned(), + } +} + +// --------------------------------------------------------------------------- +// The exit-code contract, status case by status case +// --------------------------------------------------------------------------- + +#[test] +fn no_daemon_is_exit_2_and_names_the_socket() { + let home = TestHome::claim("nodaemon"); + let run = bench(&home.dir, &["status"]); + assert_eq!(run.code, 2, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("benchd.sock"), + "refusal must name the socket: {}", + run.stderr + ); +} + +#[test] +fn ok_is_exit_0_with_the_verbs_data() { + let home = TestHome::claim("ok"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["status"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + let data: serde_json::Value = serde_json::from_str(&run.stdout).expect("status prints JSON"); + assert!(data["pid"].as_u64().is_some()); + assert_eq!(data["suite"], serde_json::Value::Null); + assert!( + data["events"].as_u64().unwrap() >= 1, + "daemon/started must be logged" + ); +} + +#[test] +fn an_unknown_verb_is_refused_with_exit_3_naming_the_known_verbs() { + let home = TestHome::claim("refused"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["definitely-not-a-verb"]); + assert_eq!(run.code, 3, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("status, events, stop"), + "a refusal names the route to use instead: {}", + run.stderr + ); +} + +#[test] +fn a_suite_that_cannot_isolate_is_refused_client_side_before_any_socket() { + let home = TestHome::claim("badsuite"); + let run = bench(&home.dir, &["--suite", "has/slash", "status"]); + assert_eq!(run.code, 3, "stderr: {}", run.stderr); + assert!( + run.stderr.contains("path"), + "the refusal names the rule: {}", + run.stderr + ); +} + +#[test] +fn an_oversized_request_line_is_refused_not_read() { + let home = TestHome::claim("oversize"); + let daemon = DaemonGuard::start(&home.dir, None); + let stream = UnixStream::connect(&daemon.socket).unwrap(); + let huge = format!( + "{{\"id\":\"big\",\"verb\":\"status\",\"pad\":\"{}\"}}\n", + "x".repeat(70_000) + ); + (&stream).write_all(huge.as_bytes()).unwrap(); + let mut reply = String::new(); + BufReader::new(&stream).read_line(&mut reply).unwrap(); + let response: serde_json::Value = serde_json::from_str(&reply).unwrap(); + assert_eq!(response["status"], "refused", "reply: {reply}"); +} + +// --------------------------------------------------------------------------- +// The record +// --------------------------------------------------------------------------- + +#[test] +fn stop_is_logged_before_it_is_answered_and_the_file_outlives_the_daemon() { + let home = TestHome::claim("stop"); + let daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["stop"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + + // The daemon exits on its own after answering; wait for it rather than killing it, + // so what we then read is a *completed* record. + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&daemon.socket).is_ok() { + std::thread::sleep(Duration::from_millis(20)); + } + let after = bench(&home.dir, &["status"]); + assert_eq!(after.code, 2, "a stopped daemon is exit 2, not an error"); + + // Files are the record: read the log with cat-level tooling, no daemon involved. + let log = fs::read_to_string(home.dir.join(".bench/events.jsonl")).unwrap(); + let kinds: Vec = log + .lines() + .map(|l| { + serde_json::from_str::(l).unwrap()["kind"] + .as_str() + .unwrap() + .to_string() + }) + .collect(); + assert_eq!(kinds.first().map(String::as_str), Some("log/format")); + assert_eq!(kinds.get(1).map(String::as_str), Some("daemon/started")); + assert_eq!(kinds.last().map(String::as_str), Some("daemon/stopped")); +} + +#[test] +fn events_reads_back_exactly_what_was_logged() { + let home = TestHome::claim("events"); + let _daemon = DaemonGuard::start(&home.dir, None); + let run = bench(&home.dir, &["events"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + let data: serde_json::Value = serde_json::from_str(&run.stdout).unwrap(); + assert_eq!(data["truncated"], false); + let events = data["events"].as_array().unwrap(); + assert_eq!(events[0]["kind"], "log/format"); + assert_eq!(events[0]["seq"], 0); + assert_eq!(events[1]["kind"], "daemon/started"); +} + +// --------------------------------------------------------------------------- +// The record survives what interrupts it (PR #340 review, R1/R2/R4/R5) +// --------------------------------------------------------------------------- + +/// Run a command with a hard deadline; a hang is a FAILING outcome with its own name, +/// never a stuck test run. +fn run_bounded(cmd: &mut Command, deadline: Duration) -> Option { + let mut child = cmd + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("spawn bounded command"); + let start = Instant::now(); + loop { + if let Ok(Some(status)) = child.try_wait() { + let mut stdout = String::new(); + let mut stderr = String::new(); + use std::io::Read as _; + child + .stdout + .take() + .map(|mut s| s.read_to_string(&mut stdout)); + child + .stderr + .take() + .map(|mut s| s.read_to_string(&mut stderr)); + return Some(CliRun { + code: status.code().unwrap_or(-1), + stdout, + stderr, + }); + } + if start.elapsed() > deadline { + let _ = child.kill(); + let _ = child.wait(); + return None; + } + std::thread::sleep(Duration::from_millis(50)); + } +} + +fn write_seed_log(root: &Path, torn_tail: Option<&str>, garbage_middle: bool) { + fs::create_dir_all(root).unwrap(); + let mut log = String::new(); + log.push_str("{\"seq\":0,\"at\":\"2026-08-18T00:00:00Z\",\"kind\":\"log/format\",\"data\":{\"format\":\"bench.events-log\",\"version\":0}}\n"); + log.push_str("{\"seq\":1,\"at\":\"2026-08-18T00:00:01Z\",\"kind\":\"daemon/started\",\"data\":{\"pid\":1}}\n"); + if garbage_middle { + log.push_str("this line was never an event\n"); + } + log.push_str("{\"seq\":2,\"at\":\"2026-08-18T00:00:02Z\",\"kind\":\"daemon/stopped\",\"data\":{\"pid\":1}}\n"); + if let Some(tail) = torn_tail { + log.push_str(tail); // no newline: an interrupted append + } + fs::write(root.join("events.jsonl"), log).unwrap(); +} + +#[test] +fn a_torn_last_line_is_quarantined_and_the_daemon_starts() { + let home = TestHome::claim("torn"); + let root = home.dir.join("r"); + write_seed_log(&root, Some("{\"seq\":3,\"at\":\"2026-08-18T00:0"), false); + + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .env("BENCH_DIR", &root); + cmd.stdout(Stdio::null()).stderr(Stdio::null()); + let mut child = cmd.spawn().unwrap(); + let socket = root.join("benchd.sock"); + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&socket).is_err() { + std::thread::sleep(Duration::from_millis(20)); + } + + let status = Command::new(bench_bin()) + .env("HOME", &home.dir) + .env("BENCH_DIR", &root) + .arg("status") + .output() + .unwrap(); + let _ = child.kill(); + let _ = child.wait(); + assert_eq!( + status.status.code(), + Some(0), + "a torn LAST line must be forgiven, not brick the root: {}", + String::from_utf8_lossy(&status.stderr) + ); + + // The tail is quarantined beside the log — dropped bytes are named, never vanished. + let quarantined = fs::read_dir(&root) + .unwrap() + .filter_map(|e| e.ok()) + .any(|e| e.file_name().to_string_lossy().contains("torn")); + assert!( + quarantined, + "the torn tail must be quarantined, not silently discarded" + ); + + // Bench-visible means logged: the repair itself is an event in the record. + let log = fs::read_to_string(root.join("events.jsonl")).unwrap(); + assert!( + log.contains("log/repaired"), + "the repair must be logged: {log}" + ); +} + +#[test] +fn a_bad_line_in_the_middle_still_refuses_naming_the_line() { + let home = TestHome::claim("midbad"); + let root = home.dir.join("r"); + write_seed_log(&root, None, true); + + let out = Command::new(benchd_bin()) + .env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .env("BENCH_DIR", &root) + .output() + .unwrap(); + assert_eq!( + out.status.code(), + Some(3), + "unexplained middle corruption keeps refusing" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("line 3"), + "the refusal names the line: {stderr}" + ); +} + +#[test] +fn a_stalled_client_does_not_park_the_daemon() { + let home = TestHome::claim("stall"); + let daemon = DaemonGuard::start(&home.dir, None); + + // A client that connects, sends half a line, and just sits there. + let stalled = UnixStream::connect(&daemon.socket).unwrap(); + (&stalled) + .write_all(b"{\"id\":\"stall\",\"verb\":\"stat") + .unwrap(); + + let t0 = Instant::now(); + let run = run_bounded( + Command::new(bench_bin()) + .env("HOME", &home.dir) + .args(["status"]), + Duration::from_secs(12), + ); + drop(stalled); + let run = run.expect("bench status hung past 12s behind one stalled client — R2 unfixed"); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + assert!( + t0.elapsed() < Duration::from_secs(10), + "service must resume within the daemon's own I/O bound" + ); +} + +#[test] +fn a_daemon_that_never_answers_is_exit_2_not_a_hang() { + let home = TestHome::claim("mute"); + let root = home.dir.join(".bench"); + fs::create_dir_all(&root).unwrap(); + // A listener that accepts and never answers — the worst-behaved daemon possible. + let listener = std::os::unix::net::UnixListener::bind(root.join("benchd.sock")).unwrap(); + let _keep = std::thread::spawn(move || { + let mut held = Vec::new(); + while let Ok((s, _)) = listener.accept() { + held.push(s); + } + }); + + let run = run_bounded( + Command::new(bench_bin()) + .env("HOME", &home.dir) + .args(["status"]), + Duration::from_secs(25), + ); + let run = run.expect("bench hung past 25s on a mute daemon — the client has no read bound"); + assert_eq!( + run.code, 2, + "a timeout is EXIT_NO_DAEMON, never silence: {}", + run.stderr + ); +} + +#[test] +fn a_fresh_log_opens_with_its_format_marker() { + let home = TestHome::claim("fmt"); + let daemon = DaemonGuard::start(&home.dir, None); + let _ = &daemon; + let log = fs::read_to_string(home.dir.join(".bench/events.jsonl")).unwrap(); + let first: serde_json::Value = serde_json::from_str(log.lines().next().unwrap()).unwrap(); + assert_eq!(first["kind"], "log/format"); + assert_eq!(first["data"]["format"], "bench.events-log"); +} + +#[test] +fn the_justfile_probes_every_known_verb() { + // The probe list is hand-typed in the justfile (a `just` recipe cannot import a + // crate), so it is pinned here the way helm pins the spool scripts' id pattern: + // read the literal out of the source and compare (R5). + let justfile = fs::read_to_string(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../justfile")) + .expect("daemon/justfile readable from the bench crate"); + let probe_line: String = justfile + .lines() + .skip_while(|l| !l.contains("for verb in")) + .take_while(|l| !l.contains("; do")) + .collect::>() + .join(" "); + for verb in bench_wire::KNOWN_VERBS { + assert!( + probe_line.contains(verb), + "justfile spec probe list is missing known verb {verb:?}: {probe_line}" + ); + } +} + +// --------------------------------------------------------------------------- +// M0's prove line: two instances, zero shared state +// --------------------------------------------------------------------------- + +#[test] +fn a_live_and_a_suite_instance_share_nothing() { + let home = TestHome::claim("isolation"); + let live = DaemonGuard::start(&home.dir, None); + let suite = DaemonGuard::start(&home.dir, Some("m0")); + assert_ne!(live.socket, suite.socket); + + let live_status = bench(&home.dir, &["status"]); + let suite_status = bench(&home.dir, &["--suite", "m0", "status"]); + assert_eq!(live_status.code, 0); + assert_eq!(suite_status.code, 0); + + let live_data: serde_json::Value = serde_json::from_str(&live_status.stdout).unwrap(); + let suite_data: serde_json::Value = serde_json::from_str(&suite_status.stdout).unwrap(); + assert_ne!(live_data["pid"], suite_data["pid"], "two daemons, not one"); + assert_ne!(live_data["root"], suite_data["root"], "two roots, not one"); + assert_eq!(suite_data["suite"], "m0"); + + // Stopping the suite instance must not touch the live one — the whole point. + let stop = bench(&home.dir, &["--suite", "m0", "stop"]); + assert_eq!(stop.code, 0); + let live_after = bench(&home.dir, &["status"]); + assert_eq!( + live_after.code, 0, + "the live instance survived the suite's stop" + ); + + // Two records on disk, each with its own history. + assert!(home.dir.join(".bench/events.jsonl").exists()); + assert!(home.dir.join(".bench-m0/events.jsonl").exists()); +} + +#[test] +fn bench_dir_overrides_everything_which_is_what_a_test_claims_into() { + let home = TestHome::claim("benchdir"); + let claimed = home.dir.join("claimed"); + fs::create_dir_all(&claimed).unwrap(); + + let mut cmd = Command::new(benchd_bin()); + cmd.env_remove("BENCH_SUITE") + .env("HOME", home.dir.join("unused-home")) + .env("BENCH_DIR", &claimed) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let mut child = cmd.spawn().unwrap(); + let socket = claimed.join("benchd.sock"); + let deadline = Instant::now() + Duration::from_secs(5); + while Instant::now() < deadline && UnixStream::connect(&socket).is_err() { + std::thread::sleep(Duration::from_millis(20)); + } + + let out = Command::new(bench_bin()) + .env("HOME", home.dir.join("unused-home")) + .env("BENCH_DIR", &claimed) + .arg("status") + .output() + .unwrap(); + let _ = child.kill(); + let _ = child.wait(); + assert_eq!(out.status.code(), Some(0)); + + // Negative control (#285's shape): the claimed dir got the state, the un-claimed + // home shape was never created. + assert!(claimed.join("events.jsonl").exists()); + assert!( + !home.dir.join("unused-home/.bench").exists(), + "shared root must not appear" + ); +} + +#[test] +fn a_second_daemon_on_a_claimed_root_is_refused_loudly() { + let home = TestHome::claim("double"); + let _first = DaemonGuard::start(&home.dir, None); + let out = Command::new(benchd_bin()) + .env_remove("BENCH_DIR") + .env_remove("BENCH_SUITE") + .env("HOME", &home.dir) + .output() + .expect("run second benchd"); + assert_eq!( + out.status.code(), + Some(3), + "one daemon per root is a refusal, not a race" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("already answers"), + "the refusal says who holds the root: {stderr}" + ); +} diff --git a/daemon/crates/benchd/Cargo.toml b/daemon/crates/benchd/Cargo.toml new file mode 100644 index 0000000..0a8fce9 --- /dev/null +++ b/daemon/crates/benchd/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "benchd" +version.workspace = true +edition.workspace = true +description = "The bench daemon: owns the record, answers on the socket." + +[dependencies] +bench-wire = { path = "../bench-wire" } +serde_json.workspace = true +time.workspace = true diff --git a/daemon/crates/benchd/src/main.rs b/daemon/crates/benchd/src/main.rs new file mode 100644 index 0000000..4f1427d --- /dev/null +++ b/daemon/crates/benchd/src/main.rs @@ -0,0 +1,520 @@ +//! benchd — the bench daemon. M0: skeleton and isolation. +//! +//! What exists at this milestone: a suite-aware record root, an append-only event log +//! that is the single source of truth, and one unix socket answering three verbs +//! (`status`, `events`, `stop`). What deliberately does not exist yet: panes, mail, +//! attention, taps — those are M1+ (docs/future-planning/bench-roadmap.md) and each +//! arrives as new event kinds plus new verbs over this same spine, never as a second +//! channel beside it. +//! +//! Design rules this file carries (argued in ../../direction.md): +//! - **Bench-visible means logged.** Every mutation appends an event before the response +//! that reports it; readers project from the log, never from daemon memory alone. +//! - **One door.** The socket is the only way in; the CLI, the face, and every agent use +//! the same verbs. There is no privileged in-process path to grow attached to. +//! - **Refuse loudly.** Unknown verbs, malformed requests, oversized lines, a corrupt +//! log, an already-claimed socket: each is a named refusal, never a silent default. +//! +//! The daemon runs in the foreground and takes one request per connection, serially. +//! That is enough for M0's callers by construction, and a bounded, inspectable behavior +//! beats a concurrency story nothing needs yet. + +use bench_wire::{ + DAEMON_IO_TIMEOUT, EVENTS_LOG_FORMAT, EVENTS_LOG_VERSION, Event, KNOWN_VERBS, + MAX_REQUEST_BYTES, Request, Response, Status, SuiteName, Verb, check_socket_path, events_path, + resolve_root, socket_path, +}; +use serde_json::{Value, json}; +use std::fs::{self, File, OpenOptions}; +use std::io::{BufRead, BufReader, Read, Write}; +use std::os::unix::fs::{DirBuilderExt, PermissionsExt}; +use std::os::unix::net::{UnixListener, UnixStream}; +use std::path::PathBuf; +use std::process; +use std::time::Instant; + +fn main() { + process::exit(run()); +} + +fn usage() -> &'static str { + "usage: benchd [--suite ]\n\ + env: BENCH_SUITE suite name (the --suite flag wins)\n\ + \x20 BENCH_DIR record root override (wins over suite; what tests claim into)\n\ + exit: 0 clean stop · 3 refused to start · 4 failed" +} + +fn run() -> i32 { + // Flag wins over environment — the caller's explicit word over the inherited one, + // helm's PaneEnvironment convention. + let mut args = std::env::args().skip(1); + let mut suite_flag: Option = None; + while let Some(arg) = args.next() { + match arg.as_str() { + "--suite" => match args.next() { + Some(v) => suite_flag = Some(v), + None => return refuse_start("--suite needs a name"), + }, + "--help" | "-h" => { + println!("{}", usage()); + return 0; + } + other => return refuse_start(&format!("unknown argument {other:?}\n{}", usage())), + } + } + + let suite_raw = suite_flag.or_else(|| std::env::var("BENCH_SUITE").ok()); + let suite = match suite_raw.as_deref() { + Some(raw) => match SuiteName::validate(raw) { + Ok(s) => Some(s), + // The whole point of the suite is isolation; a name that cannot isolate + // must stop the launch, never fall back to the operator's live root (#86). + Err(why) => return refuse_start(&why), + }, + None => None, + }; + + let home = match std::env::var("HOME") { + Ok(h) => PathBuf::from(h), + Err(_) => return fail_start("HOME is not set; benchd cannot resolve a record root"), + }; + let bench_dir = std::env::var("BENCH_DIR").ok(); + let root = resolve_root(bench_dir.as_deref(), suite.as_ref(), &home); + + match Daemon::start(root, suite) { + Ok(mut daemon) => daemon.serve(), + Err(StartError::Refused(why)) => refuse_start(&why), + Err(StartError::Failed(why)) => fail_start(&why), + } +} + +// Pre-socket exits derive from the same enum as socket-answered ones (R3): one +// spelling of the contract, no hand-typed twin to drift. +fn refuse_start(why: &str) -> i32 { + eprintln!("benchd: refusing to start: {why}"); + Status::Refused.exit_code() +} + +fn fail_start(why: &str) -> i32 { + eprintln!("benchd: {why}"); + Status::Error.exit_code() +} + +enum StartError { + Refused(String), + Failed(String), +} + +struct RepairNote { + quarantine: PathBuf, + dropped_bytes: usize, +} + +/// Read the log with byte offsets. A clean log returns the next seq. An unreadable +/// line refuses — unless it is the LAST non-empty line, which is an interrupted append: +/// quarantine the tail to a named sibling, truncate the log back to its last good byte, +/// and report the repair so the caller can log it (R1). +fn scan_log(events: &PathBuf) -> Result<(u64, Option), StartError> { + let bytes = match fs::read(events) { + Ok(b) => b, + Err(_) => return Ok((0, None)), + }; + let text = String::from_utf8_lossy(&bytes); + let mut seq = 0u64; + let mut offset = 0usize; + let chunks: Vec<&str> = text.split_inclusive('\n').collect(); + for (i, chunk) in chunks.iter().enumerate() { + let line = chunk.trim_end_matches('\n'); + if line.trim().is_empty() { + offset += chunk.len(); + continue; + } + match serde_json::from_str::(line) { + Ok(ev) => { + seq = ev.seq + 1; + offset += chunk.len(); + } + Err(e) => { + let rest_is_empty = chunks[i + 1..].iter().all(|c| c.trim().is_empty()); + if !rest_is_empty { + return Err(StartError::Refused(format!( + "event log {} line {} is not a readable event ({e}) — refusing to append after history this daemon cannot read", + events.display(), + i + 1 + ))); + } + // Torn tail: quarantine, truncate, and say so loudly. + let epoch = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + let quarantine = events.with_file_name(format!("events.jsonl.torn-{epoch}")); + let dropped = &bytes[offset..]; + fs::write(&quarantine, dropped).map_err(|err| { + StartError::Failed(format!("cannot quarantine torn tail: {err}")) + })?; + let file = OpenOptions::new().write(true).open(events).map_err(|err| { + StartError::Failed(format!("cannot open log for repair: {err}")) + })?; + file.set_len(offset as u64).map_err(|err| { + StartError::Failed(format!("cannot truncate torn tail: {err}")) + })?; + eprintln!( + "benchd: event log {} ended in a torn line ({e}); {} byte(s) quarantined to {} and the log truncated to its last whole event", + events.display(), + dropped.len(), + quarantine.display() + ); + return Ok(( + seq, + Some(RepairNote { + quarantine, + dropped_bytes: dropped.len(), + }), + )); + } + } + } + Ok((seq, None)) +} + +struct Daemon { + root: PathBuf, + suite: Option, + listener: UnixListener, + log: File, + next_seq: u64, + started_at: String, + booted: Instant, +} + +impl Daemon { + fn start(root: PathBuf, suite: Option) -> Result { + // 0700, like every helm record directory: single-user machine, but the record + // is still nobody else's to read. + let mut builder = fs::DirBuilder::new(); + builder.recursive(true).mode(0o700); + builder.create(&root).map_err(|e| { + StartError::Failed(format!("cannot create record root {}: {e}", root.display())) + })?; + + let sock = socket_path(&root); + check_socket_path(&sock).map_err(StartError::Refused)?; + + // A socket file can outlive its daemon (SIGKILL leaves it behind). Connectable + // means a live daemon owns this root — refuse, because two writers on one log is + // corruption with extra steps. Dead means stale — say so and reclaim. + if sock.exists() { + match UnixStream::connect(&sock) { + Ok(_) => { + return Err(StartError::Refused(format!( + "a live benchd already answers at {} — one daemon per root", + sock.display() + ))); + } + Err(_) => { + eprintln!("benchd: removing stale socket {}", sock.display()); + fs::remove_file(&sock).map_err(|e| { + StartError::Failed(format!("cannot remove stale socket: {e}")) + })?; + } + } + } + + // Boot-time integrity scan. The log is the record; a daemon that appends after + // history it cannot read would be writing history it does not understand. One + // exception, from PR #340's review (R1): a torn LAST line is an interrupted + // append — the daemon's own crash mid-write, or ENOSPC part-way through a line + // — and refusing it forever bricks the root with no route out. The tail is + // quarantined beside the log and the repair is itself logged. A bad line in + // the MIDDLE stays a refusal naming the line: that one is unexplained. + let events = events_path(&root); + let (next_seq, repair) = scan_log(&events)?; + + let log = OpenOptions::new() + .create(true) + .append(true) + .open(&events) + .map_err(|e| { + StartError::Failed(format!("cannot open event log {}: {e}", events.display())) + })?; + let _ = fs::set_permissions(&events, fs::Permissions::from_mode(0o600)); + + let listener = UnixListener::bind(&sock) + .map_err(|e| StartError::Failed(format!("cannot bind {}: {e}", sock.display())))?; + + let mut daemon = Daemon { + root, + suite, + listener, + log, + next_seq, + started_at: now_rfc3339(), + booted: Instant::now(), + }; + // A fresh log opens with its format marker (R4): the file is read outside the + // process, and a reader that predates a change must fail on the marker rather + // than misread history. + if daemon.next_seq == 0 { + daemon + .append( + "log/format", + json!({ "format": EVENTS_LOG_FORMAT, "version": EVENTS_LOG_VERSION }), + ) + .map_err(StartError::Failed)?; + } + if let Some(note) = repair { + daemon + .append( + "log/repaired", + json!({ + "quarantine": note.quarantine.display().to_string(), + "dropped_bytes": note.dropped_bytes, + }), + ) + .map_err(StartError::Failed)?; + } + daemon + .append( + "daemon/started", + json!({ + "pid": process::id(), + "version": env!("CARGO_PKG_VERSION"), + "suite": daemon.suite.as_ref().map(|s| s.as_str()), + }), + ) + .map_err(StartError::Failed)?; + eprintln!( + "benchd {} listening at {} (root {})", + env!("CARGO_PKG_VERSION"), + socket_path(&daemon.root).display(), + daemon.root.display() + ); + Ok(daemon) + } + + fn serve(&mut self) -> i32 { + loop { + let stream = match self.listener.accept() { + Ok((s, _)) => s, + Err(e) => { + eprintln!("benchd: accept failed: {e}"); + continue; + } + }; + match self.handle(stream) { + Handled::Continue => {} + Handled::Stop => break, + } + } + // The stop event was appended before the response that promised it; all that is + // left is to stop answering. + let _ = fs::remove_file(socket_path(&self.root)); + 0 + } + + fn handle(&mut self, stream: UnixStream) -> Handled { + // Bounded in TIME as well as bytes (R2): this is the daemon's only loop, and a + // client that connects and never finishes its line must get the refusal, not + // the daemon. Timeouts are set before the reader clone so both share them. + let _ = stream.set_read_timeout(Some(DAEMON_IO_TIMEOUT)); + let _ = stream.set_write_timeout(Some(DAEMON_IO_TIMEOUT)); + let mut reader = BufReader::new(match stream.try_clone() { + Ok(s) => s, + Err(_) => return Handled::Continue, + }); + let mut line = String::new(); + // Bounded read: a line that never ends must not become memory nobody asked for. + let mut limited = (&mut reader).take(MAX_REQUEST_BYTES as u64 + 1); + if limited.read_line(&mut line).is_err() { + respond( + &stream, + &Response { + id: "timed-out".into(), + status: Status::Refused, + reason: Some(format!( + "request not completed within {}s — one line, newline-terminated", + DAEMON_IO_TIMEOUT.as_secs() + )), + data: None, + }, + ); + return Handled::Continue; + } + if line.len() > MAX_REQUEST_BYTES { + respond( + &stream, + &Response { + id: "oversized".into(), + status: Status::Refused, + reason: Some(format!("request exceeds {MAX_REQUEST_BYTES} bytes")), + data: None, + }, + ); + return Handled::Continue; + } + + // Permissive in shape, strict in judgment: a body that is not a Request still + // gets a refusal naming the parse failure, under the only id we have. + let request: Request = match serde_json::from_str(&line) { + Ok(r) => r, + Err(e) => { + respond( + &stream, + &Response { + id: "unparseable".into(), + status: Status::Refused, + reason: Some(format!("not a request: {e}")), + data: None, + }, + ); + return Handled::Continue; + } + }; + + let (response, outcome) = self.dispatch(&request); + respond(&stream, &response); + outcome + } + + fn dispatch(&mut self, req: &Request) -> (Response, Handled) { + // The verb is parsed, not string-matched (R5): the enum makes a new verb a + // compile-forced decision here, and the refusal derives its list from the same + // spelling the parser uses. + match Verb::parse(&req.verb) { + Some(Verb::Status) => (self.ok(req, self.status_data()), Handled::Continue), + Some(Verb::Events) => { + let since = req.args.get("since").and_then(Value::as_u64).unwrap_or(0); + match self.read_events(since) { + Ok(data) => (self.ok(req, data), Handled::Continue), + Err(why) => (self.error(req, why), Handled::Continue), + } + } + Some(Verb::Stop) => { + // Logged before answered: the record must already say "stopped" when the + // caller is told it worked (bench-visible means logged). + match self.append("daemon/stopped", json!({ "pid": process::id() })) { + Ok(()) => (self.ok(req, json!({ "stopping": true })), Handled::Stop), + Err(why) => (self.error(req, why), Handled::Continue), + } + } + None => ( + Response { + id: req.id.clone(), + status: Status::Refused, + reason: Some(format!( + "unknown verb {:?} — this daemon answers: {}", + req.verb, + KNOWN_VERBS.join(", ") + )), + data: None, + }, + Handled::Continue, + ), + } + } + + fn status_data(&self) -> Value { + json!({ + "pid": process::id(), + "version": env!("CARGO_PKG_VERSION"), + "suite": self.suite.as_ref().map(|s| s.as_str()), + "root": self.root.display().to_string(), + "socket": socket_path(&self.root).display().to_string(), + "started_at": self.started_at, + "uptime_secs": self.booted.elapsed().as_secs(), + "events": self.next_seq, + }) + } + + /// Read back the log — from the file, not from memory, because the file is the + /// record and this verb is how a reader checks that claim. Caps are reported, never + /// silent: `returned < total` plus `truncated` says exactly what was left out. + fn read_events(&self, since: u64) -> Result { + const MAX_RETURNED: usize = 1000; + let path = events_path(&self.root); + let file = File::open(&path).map_err(|e| format!("cannot open {}: {e}", path.display()))?; + let mut events: Vec = Vec::new(); + let mut total = 0u64; + for line in BufReader::new(file).lines() { + let line = line.map_err(|e| format!("cannot read {}: {e}", path.display()))?; + if line.trim().is_empty() { + continue; + } + let ev: Event = serde_json::from_str(&line) + .map_err(|e| format!("corrupt event in {}: {e}", path.display()))?; + if ev.seq < since { + continue; + } + total += 1; + if events.len() < MAX_RETURNED { + events.push(ev); + } + } + let returned = events.len(); + Ok(json!({ + "events": events, + "total": total, + "returned": returned, + "truncated": (returned as u64) < total, + })) + } + + fn append(&mut self, kind: &str, data: Value) -> Result<(), String> { + let event = Event { + seq: self.next_seq, + at: now_rfc3339(), + kind: kind.to_string(), + data, + }; + let mut line = + serde_json::to_string(&event).map_err(|e| format!("cannot encode event: {e}"))?; + line.push('\n'); + self.log + .write_all(line.as_bytes()) + .and_then(|()| self.log.flush()) + .map_err(|e| format!("cannot append to event log: {e}"))?; + // Best-effort durability: the record is the point of this process. A failed + // sync is not a failed append — the bytes are handed off either way. + let _ = self.log.sync_data(); + self.next_seq += 1; + Ok(()) + } + + fn ok(&self, req: &Request, data: Value) -> Response { + Response { + id: req.id.clone(), + status: Status::Ok, + reason: None, + data: Some(data), + } + } + + fn error(&self, req: &Request, why: String) -> Response { + Response { + id: req.id.clone(), + status: Status::Error, + reason: Some(why), + data: None, + } + } +} + +enum Handled { + Continue, + Stop, +} + +fn respond(mut stream: &UnixStream, response: &Response) { + if let Ok(mut line) = serde_json::to_string(response) { + line.push('\n'); + let _ = stream.write_all(line.as_bytes()); + } + let _ = stream.shutdown(std::net::Shutdown::Both); +} + +fn now_rfc3339() -> String { + time::OffsetDateTime::now_utc() + .format(&time::format_description::well_known::Rfc3339) + .unwrap_or_else(|_| "unknown".into()) +} diff --git a/daemon/direction.md b/daemon/direction.md new file mode 100644 index 0000000..c02d9af --- /dev/null +++ b/daemon/direction.md @@ -0,0 +1,114 @@ +# direction — the bench daemon + +An entry point, not a spec — the same posture as helm's `docs/direction.md`. The full +argument lives in `../docs/future-planning/workbench-audit-2026-08.md`; the milestone +sequence in `../docs/future-planning/bench-roadmap.md`. This file is what a session that +just opened `daemon/` needs to hold in its head. + +## What this is + +`benchd` is the headless Rust daemon helm grows into: it will own everything that must +survive — ptys, VT state, the workbench document, mail, tasks, the attention queue, the +event log — while the SwiftUI app thins into a face that renders daemon state. The +operator and the agents are equal owners; every verb exists in an addressed, non-seizing +form; both parties go through the same socket. Migration is strangler-style inside this +repo: one vertical at a time, old code unwired only when the new is proven. + +**Where it stands: M0.** A suite-aware record root, an append-only event log, one unix +socket, three verbs (`status`, `events`, `stop`), a CLI speaking helm's exit-code +discipline, and a conformance gate that runs the real binaries. Nothing helm does today +is owned here yet — M1 (attention queue and taps) is the first capability helm never had. + +## The spine + +Three commitments, made now, that every later milestone builds on rather than beside: + +**1. Bench-visible means logged.** The append-only event log (`events.jsonl` under the +record root) is the single source of truth. Every mutation appends its event *before* +the response that reports it; snapshots, queues, and "what happened overnight" are +projections of the stream, never a second store. helm's `snapshot.json` becomes a +projection at M4; the silent parked-workspace canvas drop (found 2026-08-17) is the +class of bug this rule deletes — an offer in the log with no consumer is a visible fact, +not a `return` in a guard. + +**2. One door.** The socket is the only way in. The CLI, the face, every agent, and the +operator's own keystrokes (from M4) use the same verbs through the same gateway — so +"equal owners" is mechanically true, not aspirational, and there is no privileged +in-process path for a capability to grow attached to. helm accreted six spool kinds, an +OSC push channel, mail hooks in two runtimes, and a snapshot file — each individually +argued, collectively sixteen ropes and no spine. The seventh capability here is a verb +and an event kind, not a new channel. + +**3. One spelling of every shared rule.** `bench-wire` holds the wire types AND the +resolution rules (suite names, record roots, request ids, caps). Both binaries compile +against it, so they cannot drift; anything outside the workspace that later needs these +types — the Swift face — gets a generated copy or a conformance-pinned duplicate under +the repo's honest-duplicate rule, never a hand-written one. + +## Learned from the field, on purpose + +DeepSeek Harness (dsh, studied 2026-08-17 — canvas in the helm prp store) is the most +complete existing implementation of a log-first agent daemon, and three of its ideas are +adopted here deliberately: + +- **The logged-envelope invariant.** dsh logs the full model request before dispatch, so + every request is a pure function of the log, and an independent checker rebuilds and + compares. Ours is the bench-shaped analog: every response's claim must be derivable + from the record — `stop` is logged before it is answered, and the conformance suite + reads the file, not the daemon, to verify history. +- **Caps are reported, never silent.** dsh documents every truncation; `events` returns + `total`/`returned`/`truncated` from day one, because a capped read that looks complete + is how "covered everything" gets believed. +- **Refusals name the route.** helm's spool refusals already point at the tool to use + instead; dsh's tool errors are typed and self-describing. Every refusal here names the + rule it applied and, where one exists, what to do about it. + +And one of dsh's ideas is **rejected** with equal deliberation: the plugin kernel. +benchd serves one operator whose gate is the pull request; composability-as-product +(profiles, patch layers, realm isolation) is generality this estate does not need and a +complexity bill dsh's own five-day-old ecosystem is already paying. Capabilities land as +code in this workspace, reviewed, behind the one door. + +## Posture: the agents are smart + +Adopted 2026-08-18, the operator's words made a rule. **We expose capabilities; we do +not parse prose.** Nothing in the bench regexes, keyword-matches, or otherwise +reconstructs meaning from human- or agent-written text — a reply is read by an agent, +not by a parser. Interpretation belongs to the model; determinism belongs at the tool +boundary (validated verbs, newtypes, exit codes), which is where `SuiteName` and +`RequestId` already sit. Concretely: the daemon never parses a mail body; notices point +rather than quote; status comes from taps — the runtimes' own typed hook and event +channels — and a capability that seems to need output-scraping is a missing tap or a +missing verb, not a regex waiting to be written. The one deliberate exception the +roadmap names: a last-resort output-classification fallback for a runtime with no tap +at all (M1, codex) — status inference only, never intent, retired the day the tap +exists. The spike harnesses' READY/NOOP markers were test instrumentation, not a +pattern to copy into the product. + +## Rules that bind every milestone + +The checklist form is bench-roadmap.md's invariants; the ones already load-bearing in +this workspace: + +- **Suites isolate or refuse.** `BENCH_SUITE=` moves socket, root, and every byte + of state; a name that cannot isolate stops the launch — never a fallback to the + operator's live `~/.bench` (helm #86/#285, ported as `SuiteName`). +- **Files are the record.** Everything persistent is a file a plain `cat` can read; + sockets are transport, never the only copy. +- **Exit codes are the contract**: 0 ok · 2 no daemon · 3 refused · 4 daemon failed — + helm's spool codes, kept, because every agent skill in this repo already reads them. +- **Validated newtypes at the edges** (`SuiteName`, `RequestId`), with the standing + carve-out: a request is decoded permissively in shape and judged strictly afterwards, + so malformed input earns a refusal naming the reason, not a dropped connection. +- **No orchestrator concept, ever** — no role, rank, or team field in the daemon, the + wire, or the CLI (roadmap invariant 2). Hierarchy is prompts and skills, run *on* the + bench. +- **Conformance over trust**: every wire contract gets a test that runs the real binary + as a subprocess and checks both directions, every status case. + +## How it grows + +M1 attention queue + taps (first new capability, purely additive) → M2 mail authority → +M3 the CLI replaces the spool scripts → M4 the workbench document → M5 ptys → M6 reach +over the tailnet → M7 the forge. Each milestone: new event kinds, new verbs, same spine. +The roadmap is the sequence; the operator names the milestone that starts. diff --git a/daemon/justfile b/daemon/justfile new file mode 100644 index 0000000..bb7ed76 --- /dev/null +++ b/daemon/justfile @@ -0,0 +1,104 @@ +# The bench surface as executable spec. +# +# Recipes here are the operations the bench is FOR, written down before they exist. +# A recipe whose verb the daemon does not answer yet fails with the daemon's own +# refusal naming what it does answer — so `just spec` is a live scoreboard of the +# surface, and a milestone landing flips rows without this file changing shape. +# (The idea is stolen from disler's fixing-smartass-opus-5 justfile, which drives +# herdr's CLI the same way; this file graduates into the bench skill at M3.) +# +# Wire verbs are namespaced like event kinds: `attn/post`, `mail/send` — one +# vocabulary for what happened and what is asked. + +BENCH := "cargo run -q -p bench --" + +default: + @just --list + +# ---- today (M0) ------------------------------------------------------------- + +# Run the live daemon in the foreground. +daemon: + cargo run -q -p benchd + +# Daemon identity, root, uptime, event count. +status: + {{BENCH}} status + +# Read the record back. +events: + {{BENCH}} events + +# Log the stop, then exit. +stop: + {{BENCH}} stop + +# The full M0 loop against a throwaway root — green today, stays green. +smoke: + #!/usr/bin/env bash + set -euo pipefail + export BENCH_DIR=$(mktemp -d "${TMPDIR:-/tmp}/bsmk.XXXX")/r + (timeout 60 cargo run -q -p benchd &) + sleep 1 + cargo run -q -p bench -- status + cargo run -q -p bench -- stop + echo "smoke: ok (root $BENCH_DIR)" + +# ---- wanted (failing on purpose until their milestone lands) ---------------- + +# M1 — post an attention item. +attn-post kind="blocked" detail="": + {{BENCH}} attn/post --kind {{kind}} --detail "{{detail}}" + +# M1 — the queue, oldest blocked first. +attn-list: + {{BENCH}} attn/list + +# M1 — block until a tenant is genuinely blocked or done. Zero-token supervision. +watch handle: + {{BENCH}} watch --handle {{handle}} + +# M2 — mail through benchd's authority. +mail-send to subject body-file: + {{BENCH}} mail/send --to {{to}} --subject "{{subject}}" --body-file {{body-file}} + +# M2 — the mailbox listing. +mail-list: + {{BENCH}} mail/list + +# M3 — spawn an interactive agent into a bench pty. Prompt by FILE, never argv (#93). +spawn agent="claude" cwd="." prompt-file="": + {{BENCH}} spawn --agent {{agent}} --cwd {{cwd}} --prompt-file {{prompt-file}} + +# M3 — close a pane; the busy refusal and no-force-over-keyboard rules ride along. +close pane: + {{BENCH}} close --pane {{pane}} + +# M5 — attach this terminal to a bench pty. +attach pane: + {{BENCH}} attach --pane {{pane}} + +# ---- the scoreboard --------------------------------------------------------- + +# Probe every wanted verb against a throwaway daemon: ✓ answered / ✗ not yet. +spec: + #!/usr/bin/env bash + set -uo pipefail + export BENCH_DIR=$(mktemp -d "${TMPDIR:-/tmp}/bspc.XXXX")/r + (timeout 60 cargo run -q -p benchd &) + sleep 1 + echo "bench surface, $(date -u +%Y-%m-%dT%H:%MZ):" + for verb in status events stop:probe spawn close select name capture cmd \ + attn/post attn/list attn/ack watch mail/send mail/list mail/read attach; do + v=${verb%:probe} + if [ "$verb" = "stop:probe" ]; then + # probed last for real; listed here so the row exists + printf ' ✓ %-12s (M0)\n' "stop"; continue + fi + if cargo run -q -p bench -- "$v" >/dev/null 2>&1; then + printf ' ✓ %-12s\n' "$v" + else + printf ' ✗ %-12s not yet\n' "$v" + fi + done + cargo run -q -p bench -- stop >/dev/null 2>&1 || true diff --git a/daemon/spikes/group-room.md b/daemon/spikes/group-room.md new file mode 100644 index 0000000..55ea792 --- /dev/null +++ b/daemon/spikes/group-room.md @@ -0,0 +1,44 @@ +# Spike: a group mailbox — one room, three runtimes, fan-out wakes + +**Question**: does the mailbox generalize to a GROUP — a shared room where a message +from any member reaches every other member — on the same primitives the ring spike +proved (files, an event log, pty-paste wakes)? The genuinely new risks: fan-out (one +message wakes N−1 agents), loop amplification (every answer is itself a message that +wakes everyone again), and the assignment discipline (members hold full file tools but +are pointed at exactly one message per wake). + +**Verdict: PROVEN.** 2026-08-18. A room is a directory; a message is a file; the +reactor wakes every member except the author. Three members (claude, codex, pi, helm +postures), one seeded question (9×9), protocol: answer once as a file, NOOP every other +wake, read only the file the notice names. + +## The measurement + +- **All three answer files correct** (`member-x: 81`), no double-posts. +- **Exactly 9 wakes** — the fan-out arithmetic to the wake (3 for the seed + 3 answers + × 2 non-authors), against a cap of 12 that was never hit. +- **6 NOOP turns observed** — every answer-notice was received, read, and correctly + declined by members who had already posted. The room **settled on its own** in 46.9s: + traffic stopped because the protocol said stop, not because the cap fired. +- Concurrency was real: wakes about one message landed while other members were + mid-turn on the previous one; the per-agent idle wait serialized delivery per member + while members ran concurrently. + +## Design consequences + +- **A group is not a new mechanism.** Room = shared inbox directory; fan-out = the same + `mail/received ⇒ agent/wake` reactor with "everyone but the author" as the recipient + set. `bench-mail`'s model needs one concept (a recipient set on a mailbox), not a + second subsystem. +- **The amplification math is the thing to respect**: N members turn one message into + N−1 wakes, and every reply compounds it. A wake cap per room per window (the spike + carried 12) is the loop brake, and it belongs in the reactor — the mailbox stays pure. +- **The assignment discipline held under temptation.** Members had unrestricted file + tools and a directory full of siblings; pointed notices ("read only that file") were + followed. Capability open, policy in the brief — the posture, observed working. +- Not exercised: rooms at 5+, cross-room traffic, a member joining mid-conversation, + and adversarial protocol violation. The first busy real room will say more than + another synthetic run would. + +Harness: scratchpad `pty-spike/src/bin/group_spike.rs` (throwaway; this file is the +deliverable). Companions: `pty-ownership-verdict.md`, `mail-wake-verdict.md`. diff --git a/daemon/spikes/mail-wake.md b/daemon/spikes/mail-wake.md new file mode 100644 index 0000000..202ec96 --- /dev/null +++ b/daemon/spikes/mail-wake.md @@ -0,0 +1,63 @@ +# Spike: agent-to-agent mail with wakes, across runtimes, as log reactions + +**Question**: can agent↔agent mail — pi⇄claude⇄codex, all directions — work as nothing +but (a) files as the record, (b) an event log, (c) a reactor that answers +`mail/received` with `agent/wake`, using the pty-paste wake the pty spike proved? +This is the operator's own decomposition — *"mailbox is mailbox; if the daemon can +`agent/spawn`, then `mail/received ⇒ agent/wake` is just a reaction"* — put under test. + +**Verdict: PROVEN.** 2026-08-18, same throwaway harness family as the pty spike +(`portable-pty`, headless, no controlling terminal). Three agents in harness-owned +ptys; a number seeded as mail and passed around the ring, **each hop adds 1** — so a +correct final value cannot come from echoes; every runtime must read real mail, compute, +and send real mail. + +## The measurement + +Ring `claude → codex → pi → claude`, seed 100, expected 103. Result: **103, cycle +31.5s**, hops at 0.8s / 9.0s / 20.5s / 31.5s (~10s per hop: wake + read + compute + +write). Ready-to-brief time for all three agents: 10.5s. Postures: helm's +`SpoolUnattendedPolicy` (claude `--dangerously-skip-permissions`, codex +`--dangerously-bypass-approvals-and-sandbox`, pi `--approve`). + +The harness's own `events.jsonl` is the deliverable's shape as well as its evidence: + +``` +agent/spawned ×3 → agent/ready ×3 → mail/sent (harness seed) +→ mail/received(post-claude) → agent/woken(post-claude) +→ mail/received(post-codex) → agent/woken(post-codex) +→ mail/received(post-pi) → agent/woken(post-pi) +→ mail/received(post-claude, body 103) → cycle/complete +``` + +## Design consequences + +- **The pty is the universal wake transport.** One mechanism woke all three runtimes — + paste the notice (path, never body) into an idle composer, submit separately. No + per-runtime sockets were needed. Claude's session socket (#320, 0.12s) and `claude + --bg` remain available as *optimizations* for one runtime, to be evaluated at M3 as + the roadmap already says — they are not load-bearing for the design. +- **Wake policy is a reactor over the log, not part of mail.** The mailbox stayed pure + (directories + files + `mail/*` events); waking was a separate loop subscribing to + `mail/received`. Spawn-on-mail is the same reactor with one more rule — + `mail/received` for a handle with no live pane ⇒ `agent/spawn` — and loop caps on + agent↔agent chains belong in this reactor too (bench-side, roadmap M2). +- **Idle detection was the crude form and it sufficed** (pty quiet ≥2s). benchd's taps + (M1) make this judgement properly — which is the dependency that put attention before + mail in the roadmap. The spike confirms the *rest* of M2 carries no comparable risk. +- **Crate shape**: the mailbox rules (roots, handles, notice format) want to be a + `bench-mail` crate beside `bench-wire` — usable by benchd's verbs and by a standalone + CLI, which covers the "portable, maybe its own app" instinct without a second daemon. + +## Honest limits + +- The ring covered 3 of 6 directed pairs (each runtime sent once and received once); + the mechanism is pair-agnostic, but the other three pairs were not run. +- Recipients were sequenced idle; concurrent cross-traffic, a busy recipient, and the + wake-while-thinking case were not exercised — that is tap territory. +- One cosmetic harness bug: the `cycle/complete` log line prints Rust's `Some(103)` + rather than plain JSON. The measurement stands; the line would not pass benchd's own + boot-time log scan, which is a nice accidental proof of why that scan exists. + +Harness: scratchpad `pty-spike/src/bin/mail_spike.rs` (throwaway; this file is the +deliverable). Companion: `pty-ownership-verdict.md`. diff --git a/daemon/spikes/model-selection.md b/daemon/spikes/model-selection.md new file mode 100644 index 0000000..926c31f --- /dev/null +++ b/daemon/spikes/model-selection.md @@ -0,0 +1,47 @@ +# Spike: provider, model, and effort are controllable at spawn — and verifiable + +**Question**: when the bench spawns an agent, can the caller pick the model and the +reasoning effort per runtime — and can the selection be *verified* rather than trusted? + +**Verdict: PROVEN, 7/7 cases.** 2026-08-18. Zero model turns: each case spawns the TUI +into a harness-owned pty with selection flags and reads the chrome — the runtime's own +declaration of what it is running — then kills it. ~5s per case. + +| case | flags | chrome showed | +|---|---|---| +| claude sonnet | `--model sonnet` | sonnet ✓ | +| claude opus | `--model opus` | opus ✓ | +| claude opus, effort high | `--model opus --effort high` | opus + high ✓ | +| codex model | `-m gpt-5.3-codex` | 5.3 ✓ | +| codex effort high | `-c model_reasoning_effort=high` | 5.3 + high ✓ | +| pi sonnet | `--model anthropic/claude-sonnet-5` | sonnet ✓ | +| pi opus, thinking high | `--model anthropic/claude-opus-4-5:high` | opus + high ✓ | + +## The selection surface, per runtime + +- **claude**: `--model ` + `--effort ` — both first-class flags. +- **codex**: `-m/--model ` + `-c model_reasoning_effort=` (the generic + `-c key=value` config override). +- **pi**: `--provider ` / `--model ` supporting `provider/id` **with an + optional `:` suffix**, plus `--thinking off|minimal|low|medium|high|xhigh|max` + and `--models` for a cycling set. The catalog is `~/.pi/agent/models.json` (+ + `models-store.json`), listed by `pi --list-models`; **which entries are usable is + decided by `auth.json`** — the authenticated subset the operator cares about, not the + whole library. + +## Design consequences for `bench spawn` (M3) + +- The spawn verb carries `{agent, model?, effort?}` and maps them per runtime exactly as + the posture table already maps permission flags — one more column in the same + `SpoolUnattendedPolicy`-shaped table, not a new mechanism. Omitted means the runtime's + own default, never a bench-invented one. +- **Chrome-reading is spike instrumentation, not the product's verification.** benchd + should record what was *requested* in the spawn event (`agent/spawned` carries model + + effort) and leave verification to the runtime's own typed channels where they exist — + never a regex over the pane (posture: capabilities, not parsing). +- pi's authenticated-subset question suggests a later `bench spawn --list` passthrough + (ask each runtime what it can run) rather than bench maintaining its own catalog — + the runtimes already own that truth. + +Harness: scratchpad `pty-spike/src/bin/model_spike.rs` (throwaway; this file is the +deliverable). Companions: pty-ownership, mail-wake, group-room. diff --git a/daemon/spikes/pty-ownership.md b/daemon/spikes/pty-ownership.md new file mode 100644 index 0000000..3bc49a9 --- /dev/null +++ b/daemon/spikes/pty-ownership.md @@ -0,0 +1,39 @@ +# Spike: can a headless daemon own the pty under a full interactive agent TUI? + +**Question** (bench-roadmap invariant 3, the design's riskiest bet): can a process with no +window and no controlling terminal — what benchd will be — own a pty hosting a full +interactive agent, deliver a prompt paste-then-submit, and get real work back? + +**Verdict: PROVEN, for all three runtimes.** 2026-08-18, throwaway Rust harness +(`portable-pty` 0.9), spawned from a Claude Code tool call (tty `??`, no controlling +terminal — the headless condition, not a simulation of it). Postures were helm's +`SpoolUnattendedPolicy` verbatim; prompt delivered as paste, then `\r` separately +(helm's launch-line rule); the required reply was **computed** (`6*7` → `BENCH-42-DONE`) +so the echo of our own typed bytes could not fake a pass. + +| runtime | posture flag | TUI drew | interactive | computed reply | resize mid-session | +|---|---|---|---|---|---| +| claude 2.1.234 | `--dangerously-skip-permissions` | yes | 3.9s | 10.0s | survived | +| codex 0.147.0 | `--dangerously-bypass-approvals-and-sandbox` | yes | 4.9s | 9.3s | survived | +| pi 0.83.0 | `--approve` | yes | 3.3s | 8.1s | survived | + +Notes with design weight: + +- **The pty owner's first duty is draining the master.** The harness reads continuously + on a thread; an undrained master blocks the agent on write. benchd's session loop must + never stop reading, whatever the attach state is. +- **Marker matching needed ANSI stripping and whitespace-tolerant search** — replies + arrive interleaved with redraw sequences. The attach protocol should ship grid state, + not raw byte scrollback, which is what the roadmap already says (M5). +- **codex's posture flag**: helm documents `-p yolo` (a profile); the spike used the + explicit `--dangerously-bypass-approvals-and-sandbox`, which worked on 0.147.0. The + M3 posture table should record both spellings and pick one. + +**Not proven here, still owed to M5**: the attach protocol itself (grid snapshot + +diffs, keys up), long-lived survival (hours, sleep/wake), reboot resume (typing +`--resume` into a fresh pty), and backpressure behavior when a reader stalls. The core +bet those all sit on — full TUIs run correctly under a headless pty owner — is the thing +this spike retires. + +Harness source: scratchpad `pty-spike/` (throwaway, session-local; this file is the +deliverable). diff --git a/daemon/spikes/session-state.md b/daemon/spikes/session-state.md new file mode 100644 index 0000000..4a89686 --- /dev/null +++ b/daemon/spikes/session-state.md @@ -0,0 +1,55 @@ +# Spike: resume, fork, and rewind — re-entering session state from outside + +**Question**: the bench's recovery story (reboot resume, benchd restart as a +resume-storm) and its branching story assume the runtimes can re-enter and branch +session state from a fresh pty. Per runtime: RESUME (kill the pty, come back, context +intact), FORK (branch; the original stays), REWIND (attach to an older point). + +**Verdict: resume and fork PROVEN on all three runtimes; rewind PARTIAL (claude only, +UI proven drivable, programmatic selection needs a grid).** 2026-08-18, pty harness, +anti-echo probes (`CODEWORD=`/`FORKED=`/`ROLLED=` wrappers, since a resumed TUI redraws +old conversation into the grid and a bare match would pass on scrollback). + +| runtime | resume | fork | rewind-to-point | +|---|---|---|---| +| claude 2.1.234 | ✓ `--resume ` (id minted at spawn via `--session-id`) | ✓ `--resume --fork-session` | `/rewind` UI opens headless; blind selection landed on newest — needs grid-aware navigation | +| codex 0.147.0 | ✓ `codex resume ` | ✓ `codex fork ` — native | not surfaced via CLI | +| pi 0.83.0 | ✓ `-c` (continue in cwd) | ✓ `--fork ` — native | not surfaced via CLI | + +## Findings with design weight + +- **SIGKILL races the transcript write.** Killing a TUI immediately after its reply + reached the grid lost the session's last turn on every runtime until a 2–3s grace was + added. benchd's resume-storm and any pane close must give the child a drain-then-die + window, not a bare kill. +- **A settle heuristic is not a ready signal.** Resumed TUIs draw in bursts while + loading history; pasting on "quiet for 2s" intermittently landed before the composer + existed. The fix was a *content* ready-gate (claude's footer text). The general + lesson is the pty spike's again: benchd should key actions off grid state, which is + M5's layer. +- **Session identity should be minted by the spawner.** `claude --session-id ` + and pi's `--session-id` let benchd *choose* the id at spawn — no discovery, no + newest-file race. codex names its own; selection must be anchored to content or a + recorded id, never to recency (a recency pick during this spike grabbed the + operator's live codex session and forked it — harmless by fork semantics, but the + lesson is permanent: **record the id at spawn, never infer it later**). +- Rewind: the capability exists interactively in claude (checkpoint list renders fine + headless); driving it blind is guesswork. Either wait for M5's grid layer or use + fork-from-id as the practical "older state" tool — fork is proven everywhere. + +## The socket question (not re-run, and deliberately) + +helm #320 already measured the Claude socket end to end: idle wake 0.12s (default) / +1.01s (bypass + `crossSessionInbound: "accept"`), the bypass-without-setting case +**silently holds** with a modal nobody sees, platform dedup coalesces only identical +messages (a real ping-pong is never identical), the hook cannot distinguish a wake +from typing (so a cap in a hook counts nothing), and the wire format is unpublished — +the debug-line `{"type":"user","message":{...}}` shape. Its recommendation lands +exactly where benchd already stands: **the courier owns the wake and the cap**, +poke-as-doorbell if used at all, file mailbox canonical. For the bench: the pty wake +is proven, runtime-neutral, and ours; the socket buys ~a second of latency on one +runtime at the price of an unpublished format plus a settings precondition. Not +load-bearing — revisit only if idle-gating proves costly in practice. + +Harness: scratchpad `pty-spike/src/bin/session_spike.rs` (throwaway). Companions: +pty-ownership, mail-wake, group-room, model-selection; helm #320 for the socket. diff --git a/daemon/test.sh b/daemon/test.sh new file mode 100755 index 0000000..38202a3 --- /dev/null +++ b/daemon/test.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# The daemon gate — the pi/-style carve-out: run when daemon/ changed, needs only the +# Rust toolchain, and the Swift gate never learns about it (bench-roadmap, invariant 9). +# +# Order matters: build the whole workspace BEFORE testing, because the conformance suite +# runs the real benchd binary as a subprocess and locates it beside its own — a test run +# without the build finds nothing and says so. +set -euo pipefail +cd "$(dirname "$0")" + +cargo fmt --check +cargo clippy --workspace --all-targets -- -D warnings +cargo build --workspace +cargo test --workspace diff --git a/docs/future-planning/README.md b/docs/future-planning/README.md index 3e65e9c..2c59dff 100644 --- a/docs/future-planning/README.md +++ b/docs/future-planning/README.md @@ -1,6 +1,7 @@ # future-planning -**Nothing in this directory is built, and nothing in it authorises building.** +**M0 — the daemon skeleton at `daemon/` — is built (PR #340); everything else in this +directory remains unbuilt, and nothing in it authorises building.** These documents describe a proposed successor to helm — a headless daemon owning ptys and state, with the app as a thin face. They are here so the reasoning survives and can be argued diff --git a/docs/future-planning/bench-roadmap.md b/docs/future-planning/bench-roadmap.md index daad8a7..06385a4 100644 --- a/docs/future-planning/bench-roadmap.md +++ b/docs/future-planning/bench-roadmap.md @@ -96,6 +96,8 @@ Each is argued in the audit doc; this is the checklist form. ## M0 — Skeleton and isolation +**Landed 2026-08-18, PR #340** — with the spike verdicts in `daemon/spikes/`. + **Goal:** `daemon/` exists, runs, and is safe to develop against on the machine that hosts the developers. @@ -150,6 +152,14 @@ against the real CLI. **Prove:** two instances (live + suite) run side by side with zero shared state. **Unwire:** nothing. +> **Reordering, operator-ruled 2026-08-18: mail lands before the attention queue.** +> *"Attention can go later because we don't know where we want attention right now."* +> The tap dependency that originally ordered them is softened by measurement — the +> spike verdicts in `daemon/spikes/` proved the pty-paste wake on all three runtimes +> with a plain pty-quiet idle gate, so mail's wake path does not wait for taps. +> Milestone numbers below are left as written; the running order is **M5a (split out +> below) → M2 (mail) → M1 (attention)**. + ## M1 — Taps and the attention queue (purely additive) **Goal:** the first new capability helm never had, proving daemon + socket + CLI + taps @@ -263,6 +273,21 @@ from local disk. **Unwire:** `WorkspacePersistence`'s workbench half, `BenchSnapshotModel` (the snapshot becomes a benchd projection of the event stream, same file shape for readers). +> **Split, operator-ruled 2026-08-18: M5a is pulled forward to land before mail; M5b +> stays here.** The pty spike proved daemon-owned ptys hosting full TUIs, which makes +> the cheap half cheap and the wake story clean: +> +> - **M5a — the daemon-session core, before mail.** benchd owns ptys for **new spawns +> only**: `spawn`/`attach`/`close`/minimal `resume` verbs, a dtach-grade raw byte +> relay for attach (ring-buffer replay, resize; no VT grid, no painter), viewed by +> running `bench attach` inside an ordinary helm pane. Mail's wake then pastes into +> a pty benchd owns — uniform across claude, codex and pi, no per-runtime transports. +> Accepted interim costs, chosen knowingly: attach-hosted agents lose helm's pane +> `agent` records and mark-routing identity until M4/M5b; canvas-push passthrough +> through the relay is assumed and must be spiked before relying on it. +> - **M5b — everything below this note as written**: the painter, VT-grid state, +> migrating existing libghostty panes, and the unwiring list. Unchanged. + ## M5 — Ptys move, per-pane **Goal:** benchd owns the processes; the face becomes a painter; sessions survive the